Version: 3.0.0

Flow Run

Defines the run-specific parameters for a Flow, one flow can have multiple Flow runs

FlowRun

Below are the properties for the FlowRun. Each property links to the specific details section further down in this page.

Property	Default	Type	Required	Description
flow_run			Yes

Property Details

FlowRunOptions

Options for a Flow Run

Property	Default	Type	Required	Description
parameters		object with property values of type None	No	Dictionary of parameters to use for resource.
defaults		array[None]	No	List of default configs with filters that can be applied to a resource config.
description		string	No	A brief description of what the model does.
metadata			No	Meta information of a flow run. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.
run_tests	True	boolean	No	A boolean flag indicating whether to run tests after processing the data.
store_test_results		boolean	No	A boolean flag indicating whether to store the test results.
components		array[string]	No	List of component names to run.
component_categories		array[string]	No	List of component categories to run.
halt_flow_on_error		boolean	No	A boolean flag indicating whether to halt the flow on error.
disable_optimizers		boolean	No	A boolean flag indicating whether to disable optimizers.
disable_incremental_metadata_collection		boolean	No	A boolean flag indicating whether to disable collection incremental RC/Transform metadata.
full_refresh	False	boolean	No	A boolean flag indicating whether to perform a full refresh of each component. ⚠ If true, will drop all internal data and metadata tables/views and re-compute from scratch.
update_materialization_type	False	boolean	No	A boolean flag indicating whether to update component materialization types (e.g., changing types between 'simple', 'view', 'incremental', and 'smart'). ⚠ If materialization type changes are detected, existing data and metadata tables/views will be dropped and re-computed from scratch. Otherwise, existing data and metadata tables/views will be preserved, and type changes will result in an error.
name		string	No	The name of the FlowRun.
flow_name		string	Yes	The name of the flow that is to be run.
event_start_time		string	No	Event start time to be used for time-series processing.
event_end_time		string	No	Event end time to be used for time-series processing.

FlowRunMetadata

Meta-information of a flow run. In most cases, it doesn't affect the system behavior but may be helpful to analyze project resources.

Property	Type	Required	Description
source		No	The origin or source information for the resource.
source_event_uuid	string	No	UUID of the event that is associated with creation of this resource.
backfill_run	string	No	Name of the backfill run that scheduled this flow run.
created_by		No	Information of the user who scheduled this flow run.

ConfigFilter

A filter used to target configuration settings to a specific flow and/or component.

Property	Type	Required	Description
kind	string ("Flow", "Component")	Yes	The kind of the resource to apply the config to.
name	Any of: string array[string] array[None]	Yes	Name of the resource to apply the config to.
flow_name	string	No	Name of the flow to apply the config to.
spec	Any of:	No	Dictionary of parameters to use for the resource.

ComponentSpec

Specification for configuration applied to a component at runtime based on the config filter.

Property	Type	Required	Description
skip	boolean	No	A boolean flag indicating whether to skip processing for the component or not.
retry_strategy		No	The retry strategy configuration options for the component if any exceptions are encountered.
data_plane	One of: SnowflakeDataPlane BigQueryDataPlane DatabricksDataPlane	No	Data Plane-specific configuration options for a component.

FlowSpec

Specification for configuration applied to a flow at runtime based on the config filter.

Property	Default	Type	Required	Description
data_plane			No	The data plane that will be used for the flow at runtime.

DataPlane

The external warehouse where data is persisted throughout the flow runs, and where primary computation on the data itself occurs.

Property	Default	Type	Required	Description
connection_name		string	No
metadata_storage_location_prefix		string	No	Prefix to prepend to the names of metadata tables created for this flow. The prefix may include database/project/etc and schema/dataset/etc where applicable. If not provided, metadata tables are stored alongside the output data tables per the data plane's connection configuration.

RegexFilter

A filter used to target resources based on a regex pattern.

Property	Default	Type	Required	Description
regex		string	Yes	The regex to filter the resources.

BigQueryDataPlane

Property	Default	Type	Required	Description
bigquery			Yes	BigQuery configuration options.

BigQueryDataPlaneOptions

Property	Default	Type	Required	Description
partition_by		Any of:	No	Partition By clause for the table.
cluster_by		array[string]	No	Clustering keys to be added to the table.

BigQueryRangePartitioning

Property	Default	Type	Required	Description
field		string	Yes	Field to partition by.
range			Yes	Range partitioning options.

BigQueryTimePartitioning

Property	Default	Type	Required	Description
field		string	Yes	Field to partition by.
granularity		string ("DAY", "HOUR", "MONTH", "YEAR")	Yes	Granularity of the time partitioning.

DatabricksDataPlane

Property	Default	Type	Required	Description
databricks	cluster_by: null pyspark_job_cluster_id: null table_properties: null		No	Databricks configuration options.

DatabricksDataPlaneOptions

Property	Type	Required	Description
table_properties	object with property values of type string	No	Table properties to include when creating the data table. This setting is equivalent to the `CREATE TABLE ... TBLPROPERTIES` clause. Please refer to the Databricks documentation at https://docs.databricks.com/aws/en/delta/table-properties for available properties depending on your data plane.
pyspark_job_cluster_id	string	No	The ID of the compute cluster to use for PySpark jobs.
cluster_by	array[string]	No	Clustering keys to be added to the table.

DuckdbDataPlane

Property	Default	Type	Required	Description
duckdb			No	Duckdb configuration options.

DuckDbDataPlaneOptions

No properties defined.

FabricDataPlane

Property	Default	Type	Required	Description
fabric	spark_session_config: null		No	Fabric configuration options.

FabricDataPlaneOptions

Property	Default	Type	Required	Description
spark_session_config			No	Spark session configuration.

RangeOptions

Property	Type	Required	Description
start	integer	Yes	Start of the range partitioning.
end	integer	Yes	End of the range partitioning.
interval	integer	Yes	Interval of the range partitioning.

SnowflakeDataPlane

Property	Default	Type	Required	Description
snowflake			Yes	Snowflake configuration options.

SnowflakeDataPlaneOptions

Property	Default	Type	Required	Description
cluster_by		array[string]	No	Clustering keys to be added to the table.

SynapseDataPlane

Property	Default	Type	Required	Description
synapse	spark_session_config: null		No	Synapse configuration options.

SynapseDataPlaneOptions

Property	Default	Type	Required	Description
spark_session_config			No	Spark session configuration.

LivySparkSessionConfig

Property	Type	Required	Description
pool	string	No	The pool to use for the Spark session.
driver_memory	string	No	The memory to use for the Spark driver.
driver_cores	integer	No	The number of cores to use for the Spark driver.
executor_memory	string	No	The memory to use for the Spark executor.
executor_cores	integer	No	The number of cores to use for each executor.
num_executors	integer	No	The number of executors to use for the Spark session.
session_key_override	string	No	The key to use for the Spark session.
max_concurrent_sessions	integer	No	The maximum number of concurrent sessions of this spec to create.

User

User information.

Property	Type	Required
name	string	No
email	string	No
service_account	string	No

ResourceLocation

The origin or source information for the resource.

Property	Default	Type	Required	Description
path		string	Yes	Path within repository files where the resource is defined.
first_line_number		integer	No	First line number within `path` file where the resource is defined.

RetryStrategy

Retry strategy configuration for component operations. This configuration leverages the tenacity library to implement robust retry mechanisms. The configuration options directly map to tenacity's retry parameters. Details on the tenacity library can be found here: https://tenacity.readthedocs.io/en/latest/api.html#retry-main-api Current implementation includes: - stop_after_attempt: Maximum number of retry attempts - stop_after_delay: Give up on retries one attempt before you would exceed the delay. Will need to supply at least one of the two parameters. Additional retry parameters will be added as needed to support more complex use cases.

Property	Default	Type	Required	Description
stop_after_attempt		integer	No	The number of attempts before giving up, if None is set, will not stop after any attempts.
stop_after_delay		integer	No	Give up on retries one attempt before you would exceed the delay, if None is set, will not stop after any attempts.

FlowRun​

Property Details​

FlowRunOptions​

FlowRunMetadata​

ConfigFilter​

ComponentSpec​

FlowSpec​

DataPlane​

RegexFilter​

BigQueryDataPlane​

BigQueryDataPlaneOptions​

BigQueryRangePartitioning​

BigQueryTimePartitioning​

DatabricksDataPlane​

DatabricksDataPlaneOptions​

DuckdbDataPlane​

DuckDbDataPlaneOptions​

FabricDataPlane​

FabricDataPlaneOptions​

RangeOptions​

SnowflakeDataPlane​

SnowflakeDataPlaneOptions​

SynapseDataPlane​

SynapseDataPlaneOptions​

LivySparkSessionConfig​

User​

ResourceLocation​

RetryStrategy​