BigQuery Connection

Connection to a Google BigQuery data warehouse.

Examples

bigquery_connection_default.yaml

connection:
  bigquery:
    project: my-gcp-project
    dataset: my-dataset
    location: US

bigquery_connection_with_service_account_key.yaml

connection:
  bigquery:
    project: my-gcp-project
    key: ${vaults.my_vault_name.my_service_account_key}

bigquery_advanced_config.yaml

connection:
  bigquery:
    project: my-gcp-project
    max_query_length: 500000  # The maximum query length to allow.
    max_concurrent_queries: 100  # The maximum number of concurrent queries to allow.
    max_combined_sql_statements: 500  # The maximum number of combined SQL statements to allow.
    create_dataset_if_not_exists: true  # A boolean flag indicating whether to create the dataset if it does not exist.
    default_table_expiration_days: 30  # Specifies the default expiration time, in days, for tables in this dataset. You can override this value when you create a table.

BigQueryConnection

info

BigQueryConnection is defined beneath the following ancestor nodes in the YAML structure:

• Connection

Below are the properties for the BigQueryConnection. Each property links to the specific details section further down in this page.

Property	Default	Type	Required	Description
description		string	No	A brief description of what the model does.
metadata		ResourceMetadata	No	Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.
name		string	Yes	The name of the model
bigquery		BigQueryConnectionOptions	Yes	The BigQuery connection options.

Property Details

Connection

A connection to a data source/sink.

Property	Default	Type	Required	Description
connection		One of:   S3Connection   GcsConnection   AbfsConnection   LocalFileConnection   SnowflakeConnection   BigQueryConnection   MSSQLConnection   MySQLConnection   OracleConnection   PostgresConnection   HttpConnection   DuckDBConnection   SynapseConnection   SFTPConnection   FabricConnection   DatabricksConnection	Yes	The connection to the data system.

BigQueryConnectionOptions

BigQuery connection options.

Property	Default	Type	Required	Description
max_query_length		integer	No	The maximum query length to allow.
max_concurrent_queries		integer	No	The maximum number of concurrent queries to allow.
max_combined_sql_statements		integer	No	The maximum number of combined SQL statements to allow.
project		string	Yes	The Google Cloud project to connect to.
dataset		string	No	The BigQuery dataset to connect to.
key		string	No	The GCP service account credentials to use for the BigQuery connection.
location		string	No	The BigQuery location to use.
create_dataset_if_not_exists	False	boolean	No	A boolean flag indicating whether to create the dataset if it does not exist.
default_table_expiration_days		integer	No	Specifies the default expiration time, in days, for tables in this dataset. You can override this value when you create a table.
cast_nano_timestap_to_micro_timestamp	True	boolean	No	Controls automatic conversion of nanosecond timestamps to BigQuery timestamps. When loading parquet files containing nanosecond-precision timestamps, BigQuery stores them as INT64 since it only supports microsecond precision. When enabled, this setting will automatically cast these INT64 values to timestamps, with some precision loss (nanoseconds to microseconds).

ResourceMetadata

Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.

Property	Default	Type	Required	Description
source		ResourceLocation	No	The origin or source information for the resource.
source_event_uuid		string	No	UUID of the event that is associated with creation of this resource.

ResourceLocation

The origin or source information for the resource.

Property	Default	Type	Required	Description
path		string	Yes	Path within repository files where the resource is defined.
first_line_number		integer	No	First line number within path file where the resource is defined.