Flow
A flow is the primary unit of execution in Ascend and contains a collection of components assembled into a directed acyclic graph (DAG).
Examples​
- simple_flow_name_description.yaml
- flow_with_runner_and_version.yaml
- flow_with_specific_data_plane.yaml
flow:
name: "SimpleFlow"
description: "This is a simple flow with only a name and description."
flow:
name: "MyFlow"
description: "A flow with a specific runner and version."
runner: "custom-runner"
version: "1.0.0"
flow:
name: "ExampleFlow"
description: "A flow with a specific data plane."
data_plane:
connection_name: "specific-connection"
Flow​
Below are the properties for the Flow
. Each property links to the specific details section further down in this page.
Property | Default | Type | Required | Description |
---|---|---|---|---|
flow | FlowOptions | Yes |
Property Details​
FlowOptions​
Defines the options for a Flow
Property | Default | Type | Required | Description |
---|---|---|---|---|
parameters | object with property values of type None | No | Dictionary of parameters to use for resource. | |
defaults | array[ConfigFilter] | No | List of default configs with filters that can be applied to a resource config. | |
description | string | No | A brief description of what the model does. | |
metadata | ResourceMetadata | No | Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources. | |
name | string | Yes | The name of the model | |
data_plane | DataPlane | No | Data plane to use for the flow. | |
version | string | No | The version of the flow. | |
bootstrap | string | No | Bootstrap command to run within the Docker container. | |
runner | ascend | string | No | Runner id to use for running the flow. defaults to 'ascend' |
ConfigFilter​
A filter used to target configuration settings to a specific flow and/or component.
Property | Default | Type | Required | Description |
---|---|---|---|---|
kind | string ("Flow", "Component") | Yes | The kind of the resource to apply the config to. | |
name | Any of:   string   array[string]   RegexFilter   array[RegexFilter] | Yes | Name of the resource to apply the config to. | |
flow_name | string | No | Name of the flow to apply the config to. | |
spec | Any of:   FlowSpec   ComponentSpec | No | Dictionary of parameters to use for the resource. |
ComponentSpec​
Specification for configuration applied to a component at runtime based on the config filter.
Property | Default | Type | Required | Description |
---|---|---|---|---|
data_plane |   One of:     SnowflakeDataPlane     BigQueryDataPlane     DuckdbDataPlane     SynapseDataPlane     FabricDataPlane     DatabricksDataPlane | No | Data Plane-specific configuration options for a component. | |
skip | False | boolean | No |
FlowSpec​
Specification for configuration applied to a flow at runtime based on the config filter.
Property | Default | Type | Required | Description |
---|---|---|---|---|
data_plane | DataPlane | No | The data plane that will be used for the flow at runtime. |
DataPlane​
The external warehouse where data is persisted throughout the flow runs, and where primary computation on the data itself occurs.
Property | Default | Type | Required | Description |
---|---|---|---|---|
connection_name | string | No | ||
metadata_storage_location_prefix | string | No | Prefix to prepend to the names of metadata tables created for this flow. The prefix may include database/project/etc and schema/dataset/etc where applicable. If not provided, metadata tables are stored alongside the output data tables per the data plane's connection configuration. |
RegexFilter​
A filter used to target resources based on a regex pattern.
Property | Default | Type | Required | Description |
---|---|---|---|---|
regex | string | Yes | The regex to filter the resources. |
BigQueryDataPlane​
Property | Default | Type | Required | Description |
---|---|---|---|---|
bigquery | BigQueryDataPlaneOptions | Yes | BigQuery configuration options. |
BigQueryDataPlaneOptions​
Property | Default | Type | Required | Description |
---|---|---|---|---|
partition_by | Any of:   BigQueryRangePartitioning   BigQueryTimePartitioning | No | Partition By clause for the table. | |
cluster_by | array[string] | No | Clustering keys to be added to the table. |
BigQueryRangePartitioning​
Property | Default | Type | Required | Description |
---|---|---|---|---|
field | string | Yes | Field to partition by. | |
range | RangeOptions | Yes | Range partitioning options. |
BigQueryTimePartitioning​
Property | Default | Type | Required | Description |
---|---|---|---|---|
field | string | Yes | Field to partition by. | |
granularity | string ("DAY", "HOUR", "MONTH", "YEAR") | Yes | Granularity of the time partitioning. |
DatabricksDataPlane​
Property | Default | Type | Required | Description |
---|---|---|---|---|
databricks | cluster_by: null pyspark_job_cluster_id: null table_properties: null | DatabricksDataPlaneOptions | No | Databricks configuration options. |
DatabricksDataPlaneOptions​
Property | Default | Type | Required | Description |
---|---|---|---|---|
table_properties | object with property values of type string | No | Table properties to include when creating the data table. This setting is equivalent to the CREATE TABLE ... TBLPROPERTIES clause. Please refer to the Databricks documentation at https://docs.databricks.com/aws/en/delta/table-properties for available properties depending on your data plane. | |
pyspark_job_cluster_id | string | No | The ID of the compute cluster to use for PySpark jobs. | |
cluster_by | array[string] | No | Clustering keys to be added to the table. |
DuckdbDataPlane​
Property | Default | Type | Required | Description |
---|---|---|---|---|
duckdb | DuckDbDataPlaneOptions | No | Duckdb configuration options. |
DuckDbDataPlaneOptions​
No properties defined.
FabricDataPlane​
Property | Default | Type | Required | Description |
---|---|---|---|---|
fabric | spark_session_config: null | FabricDataPlaneOptions | No | Fabric configuration options. |
FabricDataPlaneOptions​
Property | Default | Type | Required | Description |
---|---|---|---|---|
spark_session_config | LivySparkSessionConfig | No | Spark session configuration. |
RangeOptions​
Property | Default | Type | Required | Description |
---|---|---|---|---|
start | integer | Yes | Start of the range partitioning. | |
end | integer | Yes | End of the range partitioning. | |
interval | integer | Yes | Interval of the range partitioning. |
SnowflakeDataPlane​
Property | Default | Type | Required | Description |
---|---|---|---|---|
snowflake | SnowflakeDataPlaneOptions | Yes | Snowflake configuration options. |
SnowflakeDataPlaneOptions​
Property | Default | Type | Required | Description |
---|---|---|---|---|
cluster_by | array[string] | No | Clustering keys to be added to the table. |
SynapseDataPlane​
Property | Default | Type | Required | Description |
---|---|---|---|---|
synapse | spark_session_config: null | SynapseDataPlaneOptions | No | Synapse configuration options. |
SynapseDataPlaneOptions​
Property | Default | Type | Required | Description |
---|---|---|---|---|
spark_session_config | LivySparkSessionConfig | No | Spark session configuration. |
LivySparkSessionConfig​
Property | Default | Type | Required | Description |
---|---|---|---|---|
pool | string | No | The pool to use for the Spark session. | |
driver_memory | string | No | The memory to use for the Spark driver. | |
driver_cores | integer | No | The number of cores to use for the Spark driver. | |
executor_memory | string | No | The memory to use for the Spark executor. | |
executor_cores | integer | No | The number of cores to use for each executor. | |
num_executors | integer | No | The number of executors to use for the Spark session. | |
session_key_override | string | No | The key to use for the Spark session. | |
max_concurrent_sessions | integer | No | The maximum number of concurrent sessions of this spec to create. |
ResourceMetadata​
Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.
Property | Default | Type | Required | Description |
---|---|---|---|---|
source | ResourceLocation | No | The origin or source information for the resource. | |
source_event_uuid | string | No | UUID of the event that is associated with creation of this resource. |
ResourceLocation​
The origin or source information for the resource.
Property | Default | Type | Required | Description |
---|---|---|---|---|
path | string | Yes | Path within repository files where the resource is defined. | |
first_line_number | integer | No | First line number within path file where the resource is defined. |