BigQuery Connection

Connection to a Google BigQuery data warehouse.

Examples

bigquery_connection_default.yaml

connection:
  bigquery:
    project: my-gcp-project
    dataset: my-dataset
    location: US

bigquery_connection_with_service_account_key.yaml

connection:
  bigquery:
    project: my-gcp-project
    key: ${vaults.my_vault_name.my_service_account_key}

bigquery_advanced_config.yaml

connection:
  bigquery:
    project: my-gcp-project
    max_query_length: 500000  # The maximum query length to allow.
    max_concurrent_queries: 100  # The maximum number of concurrent queries to allow.
    max_combined_sql_statements: 500  # The maximum number of combined SQL statements to allow.
    create_dataset_if_not_exists: true  # A boolean flag indicating whether to create the dataset if it does not exist.
    default_table_expiration_days: 30  # Specifies the default expiration time, in days, for tables in this dataset. You can override this value when you create a table.

BigQueryConnection

info

BigQueryConnection is defined beneath the following ancestor nodes in the YAML structure:

• Connection

Below are the properties for the BigQueryConnection. Each property links to the specific details section further down in this page.

Property	Default	Type	Required	Description
description		string	No	Brief description of what the model does.
metadata			No	Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.
name		string	Yes	The name of the model
bigquery			Yes

Property Details

Connection

Data source/sink Connection.

Property	Default	Type	Required	Description
connection		One of:   S3Connection   GcsConnection   AbfsConnection   LocalFileConnection   SnowflakeConnection   BigQueryConnection   MSSQLConnection   MySQLConnection   OracleConnection   PostgresConnection   HttpConnection   DuckDBConnection   SFTPConnection   DatabricksConnection	Yes	Data system Connection.

BigQueryConnectionOptions

BigQuery Connection options.

Property	Default	Type	Required	Description
max_query_length		integer	No	Maximum combined query length permitted during query batching.
max_concurrent_queries		integer	No	Maximum number of concurrent queries permitted.
max_combined_sql_statements		integer	No	Maximum number of combined SQL statements permitted during query batching.
project		string	Yes	Google Cloud project.
dataset		string	No	BigQuery dataset.
key		string	No	GCP service account credentials to use for the BigQuery Connection.
location		string	No	BigQuery location.
create_dataset_if_not_exists	False	boolean	No	Boolean flag indicating whether to create the dataset if it does not exist.
default_table_expiration_days		integer	No	Specifies the default expiration time in days for tables in this dataset. You can override this value when you create a table.
cast_nano_timestamp_to_micro_timestamp	True	boolean	No	Controls automatic conversion of nanosecond timestamps to BigQuery timestamps. When loading parquet files containing nanosecond-precision timestamps, BigQuery stores them as INT64 since it only supports microsecond precision. When enabled, this setting will automatically cast these INT64 values to timestamps, with some precision loss (nanoseconds to microseconds).

ResourceMetadata

Meta information of a resource. In most cases, it doesn't affect the system behavior but may be helpful to analyze Project resources.

Property	Default	Type	Required	Description
source			No	The origin or source information for the resource.
source_event_uuid		string	No	Event UUID associated with creation of this resource.

ResourceLocation

The origin or source information for the resource.

Property	Default	Type	Required	Description
path		string	Yes	Path within repository files where the resource is defined.
first_line_number		integer	No	First line number within path file where the resource is defined.