BigQuery Data Plane Configuration

BigQueryDataPlane

info

BigQueryDataPlane is defined beneath the following ancestor nodes in the YAML structure:

• Component
• CustomPythonReadComponent
• Flow
• FlowOptions
• FlowRun
• FlowRunOptions
• Profile
• ProfileOptions
• Project
• ProjectOptions
• ConfigFilter
• ComponentSpec
• ReadComponent
• TransformComponent

Below are the properties for the BigQueryDataPlane. Each property links to the specific details section further down in this page.

Property	Default	Type	Required	Description
bigquery		BigQueryDataPlaneOptions	Yes	BigQuery configuration options.

Property Details

Component

A component is a fundamental building block of a data flow. Types of components that are supported include: read, transform, task, test, and more.

Property	Default	Type	Required	Description
component		One of:   ReadComponent   TransformComponent   TaskComponent   SingularTestComponent   CustomPythonReadComponent   WriteComponent   CompoundComponent   AliasedTableComponent   ExternalTableComponent	Yes	Configuration options for the component.

CustomPythonReadComponent

A component that reads data using user-defined, custom Python code.

Property	Default	Type	Required	Description
data_plane		One of:     SnowflakeDataPlane     BigQueryDataPlane     DuckdbDataPlane     SynapseDataPlane	No	Data Plane-specific configuration options for a component.
name		string	No	The name of the model
description		string	No	A brief description of what the model does.
metadata		ResourceMetadata	No	Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.
flow_name		string	No	The name of the flow that the component belongs to.
skip		boolean	No	A boolean flag indicating whether to skip processing for the component or not.
data_maintenance		DataMaintenance	No	The data maintenance configuration options for the component.
tests		ComponentTestColumn	No	Defines tests to run on the data of this component.
custom_python_read		CustomPythonReadOptions	Yes

Flow

A flow is the primary unit of execution in Ascend and contains a collection of components assembled into a directed acyclic graph (DAG).

Property	Default	Type	Required	Description
flow		FlowOptions	Yes

FlowOptions

Defines the options for a Flow

Property	Default	Type	Required	Description
name		string	No	The name of the model
description		string	No	A brief description of what the model does.
metadata		ResourceMetadata	No	Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.
parameters		object	No	Dictionary of parameters to use for resource.
defaults		array[ConfigFilter]	No	List of default configs with filters that can be applied to a resource config.
data_plane		DataPlane	No	Data plane to use for the flow.
version		string	No	The version of the flow.
bootstrap		string	No	Bootstrap command to run within the Docker container.
runner	ascend	string	No	Runner id to use for running the flow. defaults to 'ascend'

FlowRun

Defines the run-specific parameters for a Flow, one flow can have multiple Flow runs

Property	Default	Type	Required	Description
flow_run		FlowRunOptions	Yes

FlowRunOptions

Options for a Flow Run

Property	Default	Type	Required	Description
name		string	No	The name of the FlowRun.
description		string	No	A brief description of what the model does.
metadata		ResourceMetadata	No	Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.
parameters		object	No	Dictionary of parameters to use for resource.
defaults		array[ConfigFilter]	No	List of default configs with filters that can be applied to a resource config.
flow_name		string	Yes	The name of the flow that is to be run.
run_tests	True	boolean	No	A boolean flag indicating whether to run tests after processing the data.
store_test_results		boolean	No	A boolean flag indicating whether to store the test results.
components		array[string]	No	List of component names to run.
component_categories		array[string]	No	List of component categories to run.
halt_flow_on_error		boolean	No	A boolean flag indicating whether to halt the flow on error.
disable_optimizers		boolean	No	A boolean flag indicating whether to disable optimizers.
disable_incremental_metadata_collection		boolean	No	A boolean flag indicating whether to disable collection incremental RC/Transform metadata.

Profile

A profile is a set of configuration options (and parameters) that define the target where customer code is compiled/run.

Property	Default	Type	Required	Description
profile		ProfileOptions	Yes	Options (and parameters) for the profile.

ProfileOptions

Configuration options (and parameters) for a profile.

Property	Default	Type	Required	Description
pip_packages		array[string]	No	Python PIP packages to install
name		string	No	The name of the model
description		string	No	A brief description of what the model does.
metadata		ResourceMetadata	No	Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.
parameters		object	No	Dictionary of parameters to use for resource.
defaults		array[ConfigFilter]	No	List of default configs with filters that can be applied to a resource config.

Project

A project is a group of related connections, flows/components, profiles, vaults, automations and other code/configuration artifacts. Project files define the mapping of filesystem paths to different kinds of artifacts that the platform can access when running flows for the project.

Property	Default	Type	Required	Description
project		ProjectOptions	Yes

ProjectOptions

Options that can be specified for a project.

Property	Default	Type	Required	Description
pip_packages		array[string]	No	Python PIP packages to install
name		string	No	The name of the model
description		string	No	A brief description of what the model does.
metadata		ResourceMetadata	No	Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.
parameters		object	No	Dictionary of parameters to use for resource.
defaults		array[ConfigFilter]	No	List of default configs with filters that can be applied to a resource config.
version		string	No	The version of the project.
connections	['connections/']	array[string]	No	List of connection definition folders used in the project.
flows	['flows/']	array[string]	No	List of flow definition folders used in the project.
profiles	['profiles/']	array[string]	No	List of profile definition folders used in the project.
sources	['src/']	array[string]	No	List of source definition folders used in the project.
tests	['tests/']	array[string]	No	List of test definition folders used in the project.
vaults	['vaults/']	array[string]	No	List of vault definition folders used in the project.
actions	['actions/']	array[string]	No	List of action definition folders used in the project.
automations	['automations/']	array[string]	No	List of automation definition folders used in the project.
sensors	['sensors/']	array[string]	No	List of sensor definition folders used in the project.
ssh_tunnels	['ssh_tunnels/']	array[string]	No	List of SSH tunnel definition folders used in the project.
applications	['applications/']	array[string]	No	List of Application definition folders used in the project.

ConfigFilter

A filter used to target configuration settings to a specific flow and/or component.

Property	Default	Type	Required	Description
kind		string ("Flow", "Component")	Yes	The kind of the resource to apply the config to.
name		Any of:   string   array[string]   RegexFilter   array[RegexFilter]	Yes	Name of the resource to apply the config to.
flow_name		string	No	Name of the flow to apply the config to.
spec		Any of:   FlowSpec   ComponentSpec	No	Dictionary of parameters to use for the resource.

ComponentSpec

Specification for configuration applied to a component at runtime based on the config filter.

Property	Default	Type	Required	Description
data_plane		One of:     SnowflakeDataPlane     BigQueryDataPlane     DuckdbDataPlane     SynapseDataPlane	No	Data Plane-specific configuration options for a component.
skip	False	boolean	No

ReadComponent

A component that reads data from a data system.

Property	Default	Type	Required	Description
data_plane		One of:     SnowflakeDataPlane     BigQueryDataPlane     DuckdbDataPlane     SynapseDataPlane	No	Data Plane-specific configuration options for a component.
name		string	No	The name of the model
description		string	No	A brief description of what the model does.
metadata		ResourceMetadata	No	Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.
flow_name		string	No	The name of the flow that the component belongs to.
skip		boolean	No	A boolean flag indicating whether to skip processing for the component or not.
data_maintenance		DataMaintenance	No	The data maintenance configuration options for the component.
tests		ComponentTestColumn	No	Defines tests to run on the data of this component.
read		One of:   GenericFileReadComponent   LocalFileReadComponent   S3ReadComponent   GcsReadComponent   AbfsReadComponent   HttpReadComponent   MSSQLReadComponent   MySQLReadComponent   OracleReadComponent   PostgresReadComponent   SnowflakeReadComponent   BigQueryReadComponent	Yes	The read component that reads data from a data system.

TransformComponent

A component that executes SQL or Python code to transform data.

Property	Default	Type	Required	Description
data_plane		One of:     SnowflakeDataPlane     BigQueryDataPlane     DuckdbDataPlane     SynapseDataPlane	No	Data Plane-specific configuration options for a component.
name		string	No	The name of the model
description		string	No	A brief description of what the model does.
metadata		ResourceMetadata	No	Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.
flow_name		string	No	The name of the flow that the component belongs to.
skip		boolean	No	A boolean flag indicating whether to skip processing for the component or not.
data_maintenance		DataMaintenance	No	The data maintenance configuration options for the component.
tests		ComponentTestColumn	No	Defines tests to run on the data of this component.
transform		One of:   SqlTransform   PythonTransform   SnowparkTransform	Yes	The transform component that executes SQL or Python code to transform data.

BigQueryDataPlaneOptions

Property	Default	Type	Required	Description
partition_by		Any of:   BigQueryRangePartitioning   BigQueryTimePartitioning	No	Partition By clause for the table.
cluster_by		array[string]	No	Clustering keys to be added to the table.

BigQueryRangePartitioning

Property	Default	Type	Required	Description
field		string	Yes	Field to partition by.
range		RangeOptions	Yes	Range partitioning options.

BigQueryTimePartitioning

Property	Default	Type	Required	Description
field		string	Yes	Field to partition by.
granularity		string ("DAY", "HOUR", "MONTH", "YEAR")	Yes	Granularity of the time partitioning.

RangeOptions

Property	Default	Type	Required	Description
start		integer	Yes	Start of the range partitioning.
end		integer	Yes	End of the range partitioning.
interval		integer	Yes	Interval of the range partitioning.