Runner Configuration
Configuration for the flow runner
RunnerConfig
info
RunnerConfig is defined beneath the following ancestor nodes in the YAML structure:
Below are the properties for the RunnerConfig. Each property links to the specific details section further down in this page.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| size | RuntimeSize | No | Override the size of the flow runner. If not specified, the flow runner inherits the size from the deployment or workspace. |
Property Details
BackfillRun
Defines the parameters for a backfill run.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| backfill_run | Yes | Backfill run options. |
BackfillRunOptions
Options for a backfill run.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| description | string | No | Brief description of what the model does. | |
| metadata | No | Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources. | ||
| name | string | Yes | The name of the model | |
| flow_name | string | Yes | Name of the Flow that is to be backfilled. | |
| start_time | string | Yes | Start time of the time range to be backfilled. | |
| end_time | string | Yes | End time of the time range to be backfilled. | |
| granularity | string ("day", "week", "month") | Yes | Time granularity to use for backfill. Must be one of: 'day', 'week', 'month'. The backfill runner divides the date range into Flow runs of this granularity and launches these Flow runs. | |
| max_concurrent_flow_runs | 1 | integer | No | Maximum number of concurrent Flow runs used for backfill. This is used to limit the number of Flow runners (and hence cluster resources) that are launched simultaneously. |
| backfill_order | string ("forward_chronological", "reverse_chronological") | No | Order to use for backfilling - either forward or reverse chronological order. | |
| flow_run_options | No | Additional options for each Flow run launched during the backfill. | ||
| run_final_sync | boolean | No | Boolean flag indicating whether to run a final sync after concurrent backfill Flow runs. This final sync is a single Flow run that is executed without any time parameters, and is meant to sync the data to the latest state and capture any missing time intervals. |
Flow
A Flow is the primary unit of execution in Ascend and contains a collection of Components assembled into a directed acyclic graph (DAG).
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| flow | Yes |
FlowOptions
Defines the options for a Flow
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| parameters | object with property values of type None | No | Dictionary of parameters to use for resource. | |
| defaults | array[None] | No | List of default configs with filters that can be applied to a resource config. | |
| description | string | No | Brief description of what the model does. | |
| metadata | No | Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources. | ||
| name | string | Yes | The name of the model | |
| data_plane | No | Data plane to use for the flow. | ||
| version | string | No | Flow version. | |
| bootstrap | string | No | Bootstrap command to run within the Docker container. | |
| runner | RunnerConfig | No | Runner configuration. | |
| component_concurrency | integer | No | Maximum number of concurrent Components to run within this Flow. |
FlowRun
Defines the run-specific parameters for a Flow, one flow can have multiple Flow runs
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| flow_run | Yes |
FlowRunBaseOptions
Base options for a Flow Run
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| parameters | object with property values of type None | No | Dictionary of parameters to use for resource. | |
| defaults | array[None] | No | List of default configs with filters that can be applied to a resource config. | |
| description | string | No | Brief description of what the model does. | |
| metadata | No | Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources. | ||
| run_tests | True | boolean | No | Boolean flag indicating whether to run tests after processing data. |
| store_test_results | boolean | No | Boolean flag indicating whether to store test results. | |
| components | array[string] | No | List of Component names to run. | |
| component_categories | array[string] | No | List of Component categories to run. | |
| halt_flow_on_error | boolean | No | Boolean flag indicating whether to halt the Flow on error. | |
| disable_optimizers | boolean | No | Boolean flag indicating whether to disable optimizers. | |
| disable_incremental_metadata_collection | boolean | No | Boolean flag indicating whether to disable collection of Incremental Read and Transform Component metadata. | |
| full_refresh | False | boolean | No | Boolean flag indicating whether to perform a full refresh of each Component. ⚠ If true, will drop all internal data and metadata tables/views and re-compute them from scratch. |
| update_materialization_type | False | boolean | No | Boolean flag indicating whether to update Component materialization types (e.g., changing types between 'simple', 'view', 'incremental', and 'smart'). ⚠ If materialization type changes are detected, existing data and metadata tables/views will be dropped and re-computed from scratch. Otherwise, existing data and metadata tables/views will be preserved and type changes will result in an error. |
| runner_overrides | RunnerConfig | No | Override runner configuration for this specific flow run. If not specified, inherits from the flow's runner configuration, or the deployment/workspace defaults. |
FlowRunOptions
Options for a Flow Run
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| parameters | object with property values of type None | No | Dictionary of parameters to use for resource. | |
| defaults | array[None] | No | List of default configs with filters that can be applied to a resource config. | |
| description | string | No | Brief description of what the model does. | |
| metadata | No | Meta information of a Flow run. In most cases, it doesn't affect the system behavior but may be helpful to analyze project resources. | ||
| run_tests | True | boolean | No | Boolean flag indicating whether to run tests after processing data. |
| store_test_results | boolean | No | Boolean flag indicating whether to store test results. | |
| components | array[string] | No | List of Component names to run. | |
| component_categories | array[string] | No | List of Component categories to run. | |
| halt_flow_on_error | boolean | No | Boolean flag indicating whether to halt the Flow on error. | |
| disable_optimizers | boolean | No | Boolean flag indicating whether to disable optimizers. | |
| disable_incremental_metadata_collection | boolean | No | Boolean flag indicating whether to disable collection of Incremental Read and Transform Component metadata. | |
| full_refresh | False | boolean | No | Boolean flag indicating whether to perform a full refresh of each Component. ⚠ If true, will drop all internal data and metadata tables/views and re-compute them from scratch. |
| update_materialization_type | False | boolean | No | Boolean flag indicating whether to update Component materialization types (e.g., changing types between 'simple', 'view', 'incremental', and 'smart'). ⚠ If materialization type changes are detected, existing data and metadata tables/views will be dropped and re-computed from scratch. Otherwise, existing data and metadata tables/views will be preserved and type changes will result in an error. |
| runner_overrides | RunnerConfig | No | Override runner configuration for this specific flow run. If not specified, inherits from the flow's runner configuration, or the deployment/workspace defaults. | |
| name | string | No | Flow run name. | |
| flow_name | string | Yes | Name of the Flow to run. | |
| event_start_time | string | No | Event start time to be used for time-series processing. | |
| event_end_time | string | No | Event end time to be used for time-series processing. |
RuntimeSize
Enumeration of available runtime sizes for deployments, workspaces, and flow runners. Each size corresponds to specific resource allocations (CPU, memory, disk).
No properties defined.