Where is my Python code run?
When you run a Python Component in an Ascend Flow, that code executes on an Ascend Flow runner. Flow runners can handle computation in two ways: either by pushing computation to your Data Plane (Snowflake, Databricks, BigQuery), or computing locally by pulling data into the Flow runner's memory.
Although Flow runners have limited memory and CPU resources, they provide more flexibility for Python code than Data Plane execution. You can use the full Python ecosystem, including custom libraries and external API integrations.
Typically, heavy computation is pushed down to the Data Plane in order to leverage infrastructure that's built for data processing. The exceptions lie in explicit calls to methods that materialize data in the local Flow runner memory, for example: to_pandas() or to_parquet().
You can always choose to execute computation locally within Flow runners, even when using cloud warehouses. However, without understanding the concepts in this document, you could inadvertently write code that causes higher costs, out-of-memory (OOM) errors, poor performance, or other issues. The key is understanding the tradeoffs between local execution and push-down so that you can make informed decisions based on your specific Data Plane and use case.
Memory management
Memory usage varies dramatically by data format and processing library. A 1 GB compressed Parquet file processed with DuckDB typically uses far less memory than a 1 GB uncompressed CSV file containing complex string data processed with pandas. More importantly, data can expand 10 times or more when loaded into memory; in other words, a 1 GB file can consume 10 GB of RAM.
In general, monitor your memory usage to prevent OOM crashes that will halt your pipeline. Choose efficient formats (Parquet over CSV) and libraries (DuckDB/Polars over pandas) when processing data locally in Flow runners.
In Workspaces, concurrent Flow runs execute on the same pod. Increasing resource limits helps, but only up to a point; if too many jobs run, you'll hit the pod's limit. Meanwhile, Deployments spin up a dedicated pod per each Flow run.
Monitor memory usage
Ascend provides several ways to monitor and track memory usage:
Check current resource allocation: View your Flow runner's resources in the Size and Resources section of your Workspace or Deployment settings. This shows you the limits you've configured and helps you determine if you need to increase resources. As a general rule of thumb, if you're processing a 10 GB Parquet file in an Ascend Flow runner, use the largest available Flow runner to handle potential memory expansion.
Monitor Flow run execution: When Flow runners exceed memory limits, they crash with an exit code -9 and you'll see an error message like:
Flow runner process for extract-load-duckdb/fr-0198e19d-587f-7322-9115-0d892bef053a exited with code -9.
Exit code -9 usually means the kernel's Out-Of-Memory (OOM) killer terminated the process after it exceeded its memory limit.
Consider increasing the runner size or optimizing the workflow's memory usage.
Avoid memory-inefficient patterns: Avoid materializing entire datasets with methods like to_pandas() wherever possible.
Dataframe execution patterns
Different dataframe libraries use varying execution models that affect where your code runs.
Lazy evaluation (Polars, DuckDB, Ibis) defers operations until you explicitly materialize results, while eager evaluation (pandas) executes operations immediately.
Push-down computation compiles your operations to SQL and executes them on your Data Plane rather than in Flow runners.
Choose your dataframe library based on where you want computation to happen:
- For external Data Plane execution: Use Ibis to push computation to your warehouse, Snowpark for Snowflake, or PySpark for distributed processing on Spark clusters.
- For efficient local execution: Use Ibis, Polars, DuckDB, or PyArrow for memory-efficient processing.
- For hybrid execution on DuckLake: Use DuckDB for local compute with decoupled storage and metadata management. See our concept guide to learn more about this architecture.
Data Plane execution details
BigQuery
In Ascend, the BigQuery Data Plane exclusively uses Ibis for Python Components, which means computation remains within BigQuery until you explicitly call materialization methods like to_pandas(). When you call these materialization methods, the data is pulled from BigQuery and processed on the Flow runner.
Snowflake
Snowflake offers both Ibis and Snowpark execution models. Ascend supports Snowpark natively while also providing Ibis as an alternative option.
With Ibis, computation remains within Snowflake until you call materialization methods like to_pandas(), which pull data to the Flow runner.
As for Snowpark, most dataframe operations (filtering, aggregations, joins, transformations) are executed as SQL in Snowflake through lazy evaluation. Operations that require data to leave Snowflake, like to_pandas() or collect() are executed on the Flow runner.
Databricks
Ascend supports two main compute types in Databricks, each with distinct tradeoffs: SQL warehouses and all-purpose clusters.
SQL warehouses are recommended for most use cases because they start up in seconds and can be Photon-enabled for better performance. With SQL warehouses, you can author both SQL Components and Python Components. Python Components must use Ibis, which compiles your Python code to SQL and executes it directly on the warehouse, giving you the flexibility of Python with the speed of SQL execution. When you call materialization methods like to_pandas(), the data is pulled from Databricks and processed on the Flow runner.
To use PySpark, you must use all-purpose clusters. This is because PySpark requires a full Apache Spark runtime environment with distributed computing capabilities, while SQL warehouses are optimized specifically for SQL execution and don't provide the Spark driver and executor infrastructure that PySpark depends on. In contrast, Ibis works with SQL warehouses because it compiles Python operations into SQL queries rather than requiring Spark's distributed processing framework.
All-purpose clusters support both SQL and PySpark but typically come with startup times of several minutes in default configurations. However, you can reduce this latency through various Databricks infrastructure optimizations such as cluster pools, instance pools, or keeping clusters warm. Consider these startup time tradeoffs when designing time-sensitive workflows, though all-purpose clusters can also be Photon-enabled for improved performance.
DuckLake
DuckLake represents a hybrid execution model that delivers the benefits of executing on Ascend Flow runners with data stored in standard storage infrastructure (Postgres and cloud object storage). While platforms like Snowflake and Databricks support Iceberg tables and catalog-based architectures to varying degrees, DuckLake operates differently within Ascend's ecosystem. Unlike other Data Planes where you bring your own compute infrastructure, Ascend operates the compute for DuckLake Data Planes, running DuckDB within Flow runners while maintaining complete separation between storage, metadata, and compute.
This architecture makes DuckLake uniquely decoupled from other data platform dependencies. Where other Data Planes require managed warehouse infrastructure, DuckLake needs only a Postgres database for metadata management and cloud object storage for data. Ascend handles all compute orchestration, eliminating the complexity of managing separate compute resources.
DuckDB processes queries locally in Flow runners. While Flow runners themselves have minimal startup time, there is still latency from connecting to Postgres for metadata and reading from object storage. However, this is typically faster than cold-starting warehouse infrastructure. This gives you infrastructure control and cost optimization benefits—you pay only for actual query execution time. While modern warehouses like Snowflake and Databricks also support scaling to zero, DuckLake's lightweight dependency model can offer advantages in scenarios where simplified infrastructure is preferred over full warehouse platforms.
Key considerations include memory limitations since queries run on Flow runner compute, which constrains large aggregations or joins by available memory. Network latency when reading from object storage may also affect performance compared to warehouse-native storage optimizations.
DuckLake excels when you need local execution flexibility with centralized data management, particularly for workloads that benefit from cost efficiency and simplified infrastructure without requiring massive scale or platform-specific features.
Scenarios
This section outlines the specifics you should consider when planning the best execution patterns for your use case.
Cross-cloud costs
Avoid pulling data across cloud providers and regions. For example, if your data is on GCP in us-west2 but Ascend Flow runners operate on AWS in us-east-1, materializing and aggregating data locally may incur significant ingress/egress costs.
For example, BigQuery egress costs alone range from $0.02/GiB to $0.08/GiB depending on whether BigQuery classifies the cross-cloud transfer as within Northern America or cross-continent. For 1 TB of data, this translates to $20.48 (conservative) to $81.92 (high estimate) in egress fees—before accounting for AWS ingress costs and Flow runner compute time.
Keep computation in the same cloud region to avoid these transfer fees entirely.
Local processing for small datasets
Use local Flow runner execution for small datasets downstream of larger jobs. When your pipeline produces aggregated results, summary statistics, or filtered subsets that are significantly smaller than the original dataset, consider pulling this data to Flow runners for final processing.
You can also execute DuckDB in-memory on Flow runners regardless of your Data Plane choice, providing efficient local processing capabilities. The DuckLake catalog takes this a step further by making DuckDB the primary execution engine while maintaining decoupled storage and metadata management.
Choose your execution model
Understanding where to execute code—in a Data Plane with external compute or locally in Flow runners—is crucial for building efficient, cost-effective workflows. Consider the specific characteristics of your workload and infrastructure.
External Data Plane compute
Ascend Data Planes: Snowflake, BigQuery, Databricks
Best for: Large-scale data processing, complex SQL operations, team collaboration
Key characteristics: Purpose-built engines with query optimization and auto-scaling. Pay for warehouse compute time with variable startup delays. Limited to SQL and supported frameworks, but can handle datasets larger than Flow runner memory.
Local execution with external Data Planes
Ascend Data Planes: Snowflake, BigQuery, Databricks
Best for: Custom algorithms, external API integrations, small datasets, rapid prototyping
Key characteristics: Single-node execution in Flow runners using libraries like DuckDB and Polars. Full Python ecosystem access with immediate startup and easier debugging. Limited by Flow runner memory but no data transfer costs within a given region.
Data Plane with compute on Flow runners
Ascend Data Planes: DuckDB with DuckLake
Best for: Cost-conscious analytics, simplified infrastructure, SQL performance with Python flexibility
Key characteristics: A Data Plane that uses local Flow runner compute with DuckDB optimization. Minimal infrastructure dependencies (Postgres + object storage) with no separate warehouse costs. Combines SQL performance with full Python access while remaining memory-constrained.
Decision framework
Consider these factors when choosing your execution model:
Dataset size and complexity: For datasets that fit comfortably in Flow runner memory (typically 10 or fewer GB of Parquet depending on operations), local execution with efficient libraries can be faster and cheaper than warehouse execution. For larger datasets or complex joins across multiple large tables, warehouse Data Planes typically provide better performance.
Development velocity: If you're prototyping, debugging complex business logic, or integrating with external APIs, local execution offers faster iteration cycles. For production workloads with established SQL patterns, Data Plane execution often provides better long-term maintainability.
Infrastructure preferences: DuckLake minimizes infrastructure dependencies while still providing optimized analytical processing. Data Planes with external compute require more infrastructure but offer enterprise features like governance, security, and team collaboration tools.
Cost sensitivity: Analyze your specific cost structure including compute time, data transfer, and infrastructure overhead. Small to medium workloads often cost less with local execution, while large-scale processing typically benefits from external Data Plane economies of scale.
Next steps
Now that you know how to write code that executes Python in the optimal place, check out the different Data Planes and Transforms:
- 🏢 Data Planes - Configure and manage your compute infrastructure
- 🐍 Python Transforms - Build data transformations using Python and popular libraries
- 📊 SQL Transforms - Write efficient SQL transformations that run directly on your Data Plane
- 🔄 Incremental Transforms - Process only new or changed data for better performance
- ⚡ PySpark Transforms - Leverage Apache Spark for large-scale distributed processing