Backfill Run

Defines the parameters for a backfill run.

BackfillRun

Below are the properties for the BackfillRun. Each property links to the specific details section further down in this page.

Property	Default	Type	Required	Description
backfill_run		BackfillRunOptions	Yes	Backfill run options.

Property Details

BackfillRunOptions

Options for a backfill run.

Property	Default	Type	Required	Description
name		string	No	The name of the model
description		string	No	A brief description of what the model does.
metadata		ResourceMetadata	No	Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.
flow_name		string	Yes	The name of the flow that is to be backfilled.
start_time		string	Yes	Start time of the time range to be backfilled.
end_time		string	Yes	End time of the time range to be backfilled.
granularity		string ("day", "week", "month")	Yes	The time granularity to use for backfill. Must be one of: 'day', 'week', 'month'. The backfill runner will divide the date range into flow runs of this granularity and launch these flow runs.
max_concurrent_flow_runs		integer	No	The maximum number of concurrent flow runs used for backfill. This is used to limit the number of flow runners (and hence cluster resources) that are launched at once.
backfill_order		string ("forward_chronological", "reverse_chronological")	No	The order to use for backfilling - either forward or reverse chronological order.
flow_run_options		FlowRunBaseOptions	No	Additional options for each flow run launched during the backfill.
run_final_sync		boolean	No	A boolean flag indicating whether to run a final sync after the running concurrent backfill flow runs. This final sync is a single flow run that is executed without any time parameters, and is meant to sync the data to the latest state and capture any missing time intervals.

FlowRunBaseOptions

Base options for a Flow Run

Property	Default	Type	Required	Description
name		string	No	The name of the model
description		string	No	A brief description of what the model does.
metadata		ResourceMetadata	No	Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.
parameters		object	No	Dictionary of parameters to use for resource.
defaults		array[ConfigFilter]	No	List of default configs with filters that can be applied to a resource config.
run_tests	True	boolean	No	A boolean flag indicating whether to run tests after processing the data.
store_test_results		boolean	No	A boolean flag indicating whether to store the test results.
components		array[string]	No	List of component names to run.
component_categories		array[string]	No	List of component categories to run.
halt_flow_on_error		boolean	No	A boolean flag indicating whether to halt the flow on error.
disable_optimizers		boolean	No	A boolean flag indicating whether to disable optimizers.
disable_incremental_metadata_collection		boolean	No	A boolean flag indicating whether to disable collection incremental RC/Transform metadata.

ConfigFilter

A filter used to target configuration settings to a specific flow and/or component.

Property	Default	Type	Required	Description
kind		string ("Flow", "Component")	Yes	The kind of the resource to apply the config to.
name		Any of:   string   array[string]   RegexFilter   array[RegexFilter]	Yes	Name of the resource to apply the config to.
flow_name		string	No	Name of the flow to apply the config to.
spec		Any of:   FlowSpec   ComponentSpec	No	Dictionary of parameters to use for the resource.

ComponentSpec

Specification for configuration applied to a component at runtime based on the config filter.

Property	Default	Type	Required	Description
data_plane		One of:     SnowflakeDataPlane     BigQueryDataPlane     DuckdbDataPlane     SynapseDataPlane	No	Data Plane-specific configuration options for a component.
skip	False	boolean	No

FlowSpec

Specification for configuration applied to a flow at runtime based on the config filter.

Property	Default	Type	Required	Description
data_plane		DataPlane	No	The data plane that will be used for the flow at runtime.

DataPlane

The external warehouse where data is persisted throughout the flow runs, and where primary computation on the data itself occurs.

Property	Default	Type	Required	Description
connection_name		string	No
metadata_storage_location_prefix		string	No	Prefix to prepend to the names of metadata tables created for this flow. The prefix may include database/project/etc and schema/dataset/etc where applicable. If not provided, metadata tables are stored alongside the output data tables per the data plane's connection configuration.

RegexFilter

A filter used to target resources based on a regex pattern.

Property	Default	Type	Required	Description
regex		string	Yes	The regex to filter the resources.

BigQueryDataPlane

Property	Default	Type	Required	Description
bigquery		BigQueryDataPlaneOptions	Yes	BigQuery configuration options.

BigQueryDataPlaneOptions

Property	Default	Type	Required	Description
partition_by		Any of:   BigQueryRangePartitioning   BigQueryTimePartitioning	No	Partition By clause for the table.
cluster_by		array[string]	No	Clustering keys to be added to the table.

BigQueryRangePartitioning

Property	Default	Type	Required	Description
field		string	Yes	Field to partition by.
range		RangeOptions	Yes	Range partitioning options.

BigQueryTimePartitioning

Property	Default	Type	Required	Description
field		string	Yes	Field to partition by.
granularity		string ("DAY", "HOUR", "MONTH", "YEAR")	Yes	Granularity of the time partitioning.

DuckdbDataPlane

Property	Default	Type	Required	Description
duckdb		DuckDbDataPlaneOptions	No	Duckdb configuration options.

DuckDbDataPlaneOptions

No properties defined.

RangeOptions

Property	Default	Type	Required	Description
start		integer	Yes	Start of the range partitioning.
end		integer	Yes	End of the range partitioning.
interval		integer	Yes	Interval of the range partitioning.

SnowflakeDataPlane

Property	Default	Type	Required	Description
snowflake		SnowflakeDataPlaneOptions	Yes	Snowflake configuration options.

SnowflakeDataPlaneOptions

Property	Default	Type	Required	Description
cluster_by		array[string]	No	Clustering keys to be added to the table.

SynapseDataPlane

Property	Default	Type	Required	Description
synapse	spark_session_config: null	SynapseDataPlaneOptions	No	Synapse configuration options.

SynapseDataPlaneOptions

Property	Default	Type	Required	Description
spark_session_config		LivySparkSessionConfig	No	Spark session configuration.

LivySparkSessionConfig

Property	Default	Type	Required	Description
pool		string	No	The pool to use for the Spark session.
driver_memory		string	No	The memory to use for the Spark driver.
driver_cores		integer	No	The number of cores to use for the Spark driver.
executor_memory		string	No	The memory to use for the Spark executor.
executor_cores		integer	No	The number of cores to use for each executor.
num_executors		integer	No	The number of executors to use for the Spark session.
session_key_override		string	No	The key to use for the Spark session.

ResourceMetadata

Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.

Property	Default	Type	Required	Description
source		ResourceLocation	No	The origin or source information for the resource.
source_event_uuid		string	No	UUID of the event that is associated with creation of this resource.

ResourceLocation

The origin or source information for the resource.

Property	Default	Type	Required	Description
path		string	Yes	Path within repository files where the resource is defined.
first_line_number		integer	No	First line number within path file where the resource is defined.