Flow Run
Defines the run-specific parameters for a Flow, one flow can have multiple Flow runs
FlowRun
Below are the properties for the FlowRun. Each property links to the specific details section further down in this page.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| flow_run | Yes |
Property Details
FlowRunOptions
Options for a Flow Run
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| parameters | object with property values of type None | No | Dictionary of parameters to use for resource. | |
| defaults | array[None] | No | List of default configs with filters that can be applied to a resource config. | |
| description | string | No | Brief description of what the model does. | |
| metadata | No | Meta information of a Flow run. In most cases, it doesn't affect the system behavior but may be helpful to analyze project resources. | ||
| run_tests | True | boolean | No | Boolean flag indicating whether to run tests after processing data. |
| store_test_results | boolean | No | Boolean flag indicating whether to store test results. | |
| components | array[string] | No | List of Component names to run. | |
| component_categories | array[string] | No | List of Component categories to run. | |
| halt_flow_on_error | boolean | No | Boolean flag indicating whether to halt the Flow on error. | |
| disable_optimizers | boolean | No | Boolean flag indicating whether to disable optimizers. | |
| disable_incremental_metadata_collection | boolean | No | Boolean flag indicating whether to disable collection of Incremental Read and Transform Component metadata. | |
| full_refresh | False | boolean | No | Boolean flag indicating whether to perform a full refresh of each Component. ⚠ If true, will drop all internal data and metadata tables/views and re-compute them from scratch. |
| update_materialization_type | False | boolean | No | Boolean flag indicating whether to update Component materialization types (e.g., changing types between 'simple', 'view', 'incremental', and 'smart'). ⚠ If materialization type changes are detected, existing data and metadata tables/views will be dropped and re-computed from scratch. Otherwise, existing data and metadata tables/views will be preserved and type changes will result in an error. |
| backfill_missing_statistics | True | boolean | No | Boolean flag indicating whether to backfill block statistics for existing data blocks that don't have statistics yet. If true (default), statistics will be computed and stored for data blocks that don't have them yet. |
| runner_overrides | RunnerConfig | No | Override runner configuration for this specific flow run. If not specified, inherits from the flow's runner configuration, or the deployment/workspace defaults. | |
| name | string | No | Flow run name. | |
| flow_name | string | Yes | Name of the Flow to run. | |
| event_start_time | string | No | Event start time to be used for time-series processing. | |
| event_end_time | string | No | Event end time to be used for time-series processing. |
FlowRunMetadata
Meta-information of a flow run. In most cases, it doesn't affect the system behavior but may be helpful to analyze project resources.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| source | No | The origin or source information for the resource. | ||
| source_event_uuid | string | No | Event UUID associated with creation of this resource. | |
| backfill_run | string | No | Name of the backfill run that scheduled this Flow run. | |
| created_by | No | User who scheduled this Flow run. |
ConfigFilter
Filter used to target configuration settings to a specific Flow and/or Component.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| kind | string ("Flow", "Component") | Yes | Resource kind to target with this configuration. | |
| name | Any of: string array[string] array[None] | Yes | Name of the resource to target with this configuration. | |
| flow_name | Any of: string array[string] array[None] | No | Name of the Flow to target with this configuration. | |
| spec | Any of: | No | Dictionary of parameters to use for the resource. |
ComponentSpec
Specification for configuration applied to a component at runtime based on the config filter.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| skip | boolean | No | Boolean flag indicating whether to skip processing for the Component or not. | |
| retry_strategy | No | Retry strategy configuration options for the Component if any exceptions are encountered. | ||
| data_maintenance | No | The data maintenance configuration options for the Component. | ||
| data_plane | One of: SnowflakeDataPlane BigQueryDataPlane DuckdbDataPlane DatabricksDataPlane | No | Data Plane-specific configuration options for Components. |
FlowSpec
Specification for configuration applied to a Flow at runtime based on the config filter.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| data_plane | No | The data plane that will be used for the flow at runtime. | ||
| runner | RunnerConfig | No | Runner configuration. | |
| component_concurrency | integer | No | Maximum number of concurrent Components to run within this Flow. |
DataPlane
The external warehouse where data is persisted throughout the Flow runs, and where primary computation on the data itself occurs.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| connection_name | string | No | ||
| metadata_storage_location_prefix | string | No | Prefix to prepend to the names of metadata tables created for this Flow. The prefix may include database/project/etc. and schema/dataset/etc where applicable. If not provided, metadata tables are stored alongside the output data tables per the Data Plane's Connection configuration. |
RegexFilter
A filter used to target resources based on a regex pattern.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| regex | string | Yes | The regex to filter the resources. |
RunnerConfig
Configuration for the flow runner
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| size | Any of: RuntimeSize CustomRuntimeSize | No | Runtime size configuration. Can be: (1) a tier name string (X-Small, Small, Medium, Large, X-Large), or (2) a CustomRuntimeSize object with tier-based or fully custom resources. |
CustomRuntimeSize
Runtime size configuration with flexible resource specification. Supports two modes: 1. Tier-based: Specify a tier with optional resource overrides 2. Fully custom: Specify CPU directly with optional memory/disk Either 'tier' or 'cpu' must be provided (or both).
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| tier | RuntimeSize | No | Base size tier (X-Small, Small, Medium, Large, X-Large). Required unless 'cpu' is specified. | |
| cpu | string | No | CPU allocation in whole cores (e.g., '1', '4', '8'). Required unless 'tier' is specified. | |
| memory | string | No | Memory allocation with unit suffix (e.g., '32Gi', '4G', '512Mi'). For high memory, use a Highmem tier instead. | |
| disk | string | No | Disk allocation with unit suffix (e.g., '100Gi', '1Ti', '500G') |
RuntimeSize
Enumeration of standard runtime size tiers. Each tier corresponds to specific resource allocations (CPU, memory, disk). Highmem variants provide 2x memory for memory-intensive workloads.
No properties defined.
BigQueryDataPlane
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| bigquery | Yes | BigQuery configuration options. |
BigQueryDataPlaneOptions
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| partition_by | Any of: | No | Partition By clause for the table. | |
| cluster_by | array[string] | No | Clustering keys to be added to the table. |
BigQueryRangePartitioning
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| field | string | Yes | Field to partition by. | |
| range | Yes | Range partitioning options. |
BigQueryTimePartitioning
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| field | string | Yes | Field to partition by. | |
| granularity | string ("DAY", "HOUR", "MONTH", "YEAR") | Yes | Granularity of the time partitioning. |
DatabricksDataPlane
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| databricks | cluster_by: null pyspark_job_cluster_id: null table_properties: null | No | Databricks configuration options. |
DatabricksDataPlaneOptions
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| table_properties | object with property values of type string | No | Table properties to include when creating the data table. This setting is equivalent to the CREATE TABLE ... TBLPROPERTIES clause. Please refer to the Databricks documentation at https://docs.databricks.com/aws/en/delta/table-properties for available properties depending on your Data Plane. | |
| pyspark_job_cluster_id | string | No | ID of the compute cluster to use for PySpark jobs. | |
| cluster_by | array[string] | No | Clustering keys to be added to the table. |
DuckdbDataPlane
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| duckdb | ducklake: null | No | DuckDB configuration options. |
DuckDbDataPlaneOptions
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| ducklake | No | DuckLake-specific data plane configuration options including table compaction settings. |
DuckLakeDataPlaneOptions
DuckLake-specific data plane configuration options.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| manual_table_compaction | True | boolean | No | Enable manual table compaction for DuckLake tables. |
| metadata_small_file_compaction | count_threshold: 10 file_size_limit: 100 ratio_threshold: null | SmallFileCompactionSettings | No | Settings for compacting metadata tables. |
| data_small_file_compaction | count_threshold: 50 file_size_limit: 100 ratio_threshold: 0.25 | SmallFileCompactionSettings | No | Settings for compacting data tables. |
| partition_by | array[string] | No | Partition keys to be added to the table. Can be column names or expressions (e.g., ['part_key']). | |
| rewrite_data_files | True | boolean | No | Call DuckLake's rewrite_data_files() maintenance operation to optimize table storage. |
| rewrite_data_files_delete_threshold | number | No | Delete threshold for ducklake_rewrite_data_files operation (0.0-1.0). If set to None, DuckLake's default value (0.95) will be used. |
FabricDataPlane
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| fabric | spark_session_config: null | No | Fabric configuration options. |
FabricDataPlaneOptions
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| spark_session_config | No | Spark session configuration. |
RangeOptions
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| start | integer | Yes | Start of the range partitioning. | |
| end | integer | Yes | End of the range partitioning. | |
| interval | integer | Yes | Interval of the range partitioning. |
SmallFileCompactionSettings
Settings for small file compaction thresholds.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| file_size_limit | 100 | integer | No | Files smaller than this size (in MB) are considered 'small' and eligible for compaction. |
| count_threshold | 10 | integer | No | Run compaction if the number of small files exceeds this threshold. |
| ratio_threshold | number | No | Percentage (0.0-1.0) of small files relative to total files. If set, both absolute count AND ratio must pass for compaction to be triggered. If None, only absolute count check is performed. |
SnowflakeDataPlane
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| snowflake | Yes | Snowflake configuration options. |
SnowflakeDataPlaneOptions
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| cluster_by | array[string] | No | Clustering keys to be added to the table. |
SynapseDataPlane
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| synapse | spark_session_config: null | No | Synapse configuration options. |
SynapseDataPlaneOptions
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| spark_session_config | No | Spark session configuration. |
LivySparkSessionConfig
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| pool | string | No | Pool to use for the Spark session. | |
| driver_memory | string | No | Memory to use for the Spark driver. | |
| driver_cores | integer | No | Number of cores to use for the Spark driver. | |
| executor_memory | string | No | Memory to use for the Spark executor. | |
| executor_cores | integer | No | Number of cores to use for each executor. | |
| num_executors | integer | No | Number of executors to use for the Spark session. | |
| session_key_override | string | No | Key to use for the Spark session. | |
| max_concurrent_sessions | integer | No | Maximum number of concurrent sessions allowed for this configuration. |
User
User information.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| name | string | No | ||
| string | No | |||
| service_account | string | No |
DataMaintenance
Data maintenance configuration options for Components.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| enabled | boolean | No | Boolean flag indicating whether data maintenance is enabled for the Component. |
ResourceLocation
The origin or source information for the resource.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| path | string | Yes | Path within repository files where the resource is defined. | |
| first_line_number | integer | No | First line number within path file where the resource is defined. |
RetryStrategy
Retry strategy configuration for Component operations. This configuration leverages the tenacity library to implement robust retry mechanisms. The configuration options directly map to tenacity's retry parameters. Details on the tenacity library can be found here: https://tenacity.readthedocs.io/en/latest/api.html#retry-main-api Current implementation includes: - stop_after_attempt: Maximum number of retry attempts - stop_after_delay: Give up on retries one attempt before you would exceed the delay. - retry_clauses: Pattern-specific retry rules with their own max_attempts. Will need to supply at least one of the parameters (stop_after_attempt, stop_after_delay, or retry_clauses). Additional retry parameters will be added as needed to support more complex use cases.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| stop_after_attempt | integer | No | Number of retry attempts before giving up. If set to None, it will not stop after any number of attempts. | |
| stop_after_delay | integer | No | Maximum time (in seconds) to spend on retries before giving up. If set to None, it will not stop after any time delay. | |
| retry_clauses | array[None] | No | Pattern-specific retry rules evaluated in order. First matching pattern wins. Non-matching errors use global stop_after_attempt/stop_after_delay. |
RetryClause
A retry rule matching errors by regex pattern.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| pattern | string | Yes | Regex pattern to match against exception message (case-insensitive). | |
| max_attempts | integer | Yes | Maximum retry attempts for errors matching this pattern. |