Skip to main content
Version: 3.0.0

Flow

A flow is the primary unit of execution in Ascend and contains a collection of components assembled into a directed acyclic graph (DAG).

Examples​

flow:
name: "SimpleFlow"
description: "This is a simple flow with only a name and description."

Flow​

Below are the properties for the Flow. Each property links to the specific details section further down in this page.

PropertyDefaultTypeRequiredDescription
flowYes

Property Details​

FlowOptions​

Defines the options for a Flow

PropertyDefaultTypeRequiredDescription
parametersobject with property values of type None
NoDictionary of parameters to use for resource.
defaultsarray[None]
NoList of default configs with filters that can be applied to a resource config.
descriptionstring
NoA brief description of what the model does.
metadataNoMeta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.
namestringYesThe name of the model
data_planeNoData plane to use for the flow.
versionstring
NoThe version of the flow.
bootstrapstring
NoBootstrap command to run within the Docker container.
runnerascendstring
NoRunner id to use for running the flow. defaults to 'ascend'

ConfigFilter​

A filter used to target configuration settings to a specific flow and/or component.

PropertyDefaultTypeRequiredDescription
kindstring ("Flow", "Component")YesThe kind of the resource to apply the config to.
nameAny of:
  string
  array[string]
  array[None]
YesName of the resource to apply the config to.
flow_namestring
NoName of the flow to apply the config to.
specAny of:
NoDictionary of parameters to use for the resource.

ComponentSpec​

Specification for configuration applied to a component at runtime based on the config filter.

PropertyDefaultTypeRequiredDescription
skipboolean
NoA boolean flag indicating whether to skip processing for the component or not.
retry_strategyNoThe retry strategy configuration options for the component if any exceptions are encountered.
data_plane  One of:
    SnowflakeDataPlane
    BigQueryDataPlane
    DatabricksDataPlane
NoData Plane-specific configuration options for a component.

FlowSpec​

Specification for configuration applied to a flow at runtime based on the config filter.

PropertyDefaultTypeRequiredDescription
data_planeNoThe data plane that will be used for the flow at runtime.

DataPlane​

The external warehouse where data is persisted throughout the flow runs, and where primary computation on the data itself occurs.

PropertyDefaultTypeRequiredDescription
connection_namestring
No
metadata_storage_location_prefixstring
NoPrefix to prepend to the names of metadata tables created for this flow. The prefix may include database/project/etc and schema/dataset/etc where applicable. If not provided, metadata tables are stored alongside the output data tables per the data plane's connection configuration.

RegexFilter​

A filter used to target resources based on a regex pattern.

PropertyDefaultTypeRequiredDescription
regexstringYesThe regex to filter the resources.

BigQueryDataPlane​

PropertyDefaultTypeRequiredDescription
bigqueryYesBigQuery configuration options.

BigQueryDataPlaneOptions​

PropertyDefaultTypeRequiredDescription
partition_byAny of:
NoPartition By clause for the table.
cluster_byarray[string]
NoClustering keys to be added to the table.

BigQueryRangePartitioning​

PropertyDefaultTypeRequiredDescription
fieldstringYesField to partition by.
rangeYesRange partitioning options.

BigQueryTimePartitioning​

PropertyDefaultTypeRequiredDescription
fieldstringYesField to partition by.
granularitystring ("DAY", "HOUR", "MONTH", "YEAR")YesGranularity of the time partitioning.

DatabricksDataPlane​

PropertyDefaultTypeRequiredDescription
databrickscluster_by: null
pyspark_job_cluster_id: null
table_properties: null
NoDatabricks configuration options.

DatabricksDataPlaneOptions​

PropertyDefaultTypeRequiredDescription
table_propertiesobject with property values of type string
NoTable properties to include when creating the data table. This setting is equivalent to the CREATE TABLE ... TBLPROPERTIES clause. Please refer to the Databricks documentation at https://docs.databricks.com/aws/en/delta/table-properties for available properties depending on your data plane.
pyspark_job_cluster_idstring
NoThe ID of the compute cluster to use for PySpark jobs.
cluster_byarray[string]
NoClustering keys to be added to the table.

DuckdbDataPlane​

PropertyDefaultTypeRequiredDescription
duckdb
NoDuckdb configuration options.

DuckDbDataPlaneOptions​

No properties defined.

FabricDataPlane​

PropertyDefaultTypeRequiredDescription
fabricspark_session_config: null
NoFabric configuration options.

FabricDataPlaneOptions​

PropertyDefaultTypeRequiredDescription
spark_session_configNoSpark session configuration.

RangeOptions​

PropertyDefaultTypeRequiredDescription
startintegerYesStart of the range partitioning.
endintegerYesEnd of the range partitioning.
intervalintegerYesInterval of the range partitioning.

SnowflakeDataPlane​

PropertyDefaultTypeRequiredDescription
snowflakeYesSnowflake configuration options.

SnowflakeDataPlaneOptions​

PropertyDefaultTypeRequiredDescription
cluster_byarray[string]
NoClustering keys to be added to the table.

SynapseDataPlane​

PropertyDefaultTypeRequiredDescription
synapsespark_session_config: null
NoSynapse configuration options.

SynapseDataPlaneOptions​

PropertyDefaultTypeRequiredDescription
spark_session_configNoSpark session configuration.

LivySparkSessionConfig​

PropertyDefaultTypeRequiredDescription
poolstring
NoThe pool to use for the Spark session.
driver_memorystring
NoThe memory to use for the Spark driver.
driver_coresinteger
NoThe number of cores to use for the Spark driver.
executor_memorystring
NoThe memory to use for the Spark executor.
executor_coresinteger
NoThe number of cores to use for each executor.
num_executorsinteger
NoThe number of executors to use for the Spark session.
session_key_overridestring
NoThe key to use for the Spark session.
max_concurrent_sessionsinteger
NoThe maximum number of concurrent sessions of this spec to create.

ResourceMetadata​

Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.

PropertyDefaultTypeRequiredDescription
sourceNoThe origin or source information for the resource.
source_event_uuidstring
NoUUID of the event that is associated with creation of this resource.

ResourceLocation​

The origin or source information for the resource.

PropertyDefaultTypeRequiredDescription
pathstringYesPath within repository files where the resource is defined.
first_line_numberinteger
NoFirst line number within path file where the resource is defined.

RetryStrategy​

Retry strategy configuration for component operations. This configuration leverages the tenacity library to implement robust retry mechanisms. The configuration options directly map to tenacity's retry parameters. Details on the tenacity library can be found here: https://tenacity.readthedocs.io/en/latest/api.html#retry-main-api Current implementation includes: - stop_after_attempt: Maximum number of retry attempts - stop_after_delay: Give up on retries one attempt before you would exceed the delay. Will need to supply at least one of the two parameters. Additional retry parameters will be added as needed to support more complex use cases.

PropertyDefaultTypeRequiredDescription
stop_after_attemptinteger
NoThe number of attempts before giving up, if None is set, will not stop after any attempts.
stop_after_delayinteger
NoGive up on retries one attempt before you would exceed the delay, if None is set, will not stop after any attempts.