GCS Connection

Connection to a bucket in Google Cloud Storage.

Examples

simple_gcs_connection.yaml

connection:
  gcs:
    root: "gs://my-bucket/"

gcs_connection_config.yaml

connection:
    gcs:
      root: "gs://my-bucket/"
      key: "${vaults.my_vault_name.service_account_key}"

gcsconnection_setup.yaml

connection:
    gcs:
      root: "gs://my-complex-bucket/"
      key: "${vaults.my_vault_name.service_account_key}"

GcsConnection

info

GcsConnection is defined beneath the following ancestor nodes in the YAML structure:

• Connection

Below are the properties for the GcsConnection. Each property links to the specific details section further down in this page.

Property	Default	Type	Required	Description
description		string	No	A brief description of what the model does.
metadata			No	Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.
name		string	Yes	The name of the model
gcs			Yes	The GCS connection options.

Property Details

Connection

A connection to a data source/sink.

Property	Default	Type	Required	Description
connection		One of:   S3Connection   GcsConnection   AbfsConnection   LocalFileConnection   SnowflakeConnection   BigQueryConnection   MSSQLConnection   MySQLConnection   OracleConnection   PostgresConnection   HttpConnection   DuckDBConnection   SFTPConnection   DatabricksConnection	Yes	The connection to the data system.

GcsConnectionOptions

Google Cloud Storage connection options.

Property	Default	Type	Required	Description
root		string	Yes	the GCS URL for the root prefix.
key		string	No	The GCP service account credentials to use for the GCS connection.

ResourceMetadata

Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.

Property	Default	Type	Required	Description
source			No	The origin or source information for the resource.
source_event_uuid		string	No	UUID of the event that is associated with creation of this resource.

ResourceLocation

The origin or source information for the resource.

Property	Default	Type	Required	Description
path		string	Yes	Path within repository files where the resource is defined.
first_line_number		integer	No	First line number within path file where the resource is defined.