GCS Connection

Connection to a bucket in Google Cloud Storage.

Examples

simple_gcs_connection.yaml

connection:
  gcs:
    root: "gs://my-bucket/"

gcs_connection_config.yaml

connection:
  gcs:
    project: my-gcp-project
    root: "gs://my-bucket/"
    key: "${secret.my_vault_name.service_account_key}"

gcsconnection_setup.yaml

connection:
  gcs:
    project: my-gcp-project
    root: "gs://my-complex-bucket/"
    key: "${secret.my_vault_name.service_account_key}"

GcsConnection

info

GcsConnection is defined beneath the following ancestor nodes in the YAML structure:

• Connection

Below are the properties for the GcsConnection. Each property links to the specific details section further down in this page.

Property	Default	Type	Required	Description
name		string	No	The name of the model
description		string	No	A brief description of what the model does.
metadata		ResourceMetadata	No	Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.
gcs		GcsConnectionOptions	Yes	The GCS connection options.

Property Details

Connection

A connection to a data source/sink.

Property	Default	Type	Required	Description
connection		One of:   S3Connection   GcsConnection   AbfsConnection   LocalFileConnection   SnowflakeConnection   BigQueryConnection   MSSQLConnection   MySQLConnection   OracleConnection   PostgresConnection   HttpConnection   DuckDBConnection   CustomPythonConnection   SynapseConnection	Yes	The connection to the data system.

GcsConnectionOptions

Google Cloud Storage connection options.

Property	Default	Type	Required	Description
project		string	No	The GCP project to connect to.
root		string	Yes	the GCS URL for the root prefix.
key		string	No	The GCP service account credentials to use for the GCS connection.

ResourceMetadata

Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.

Property	Default	Type	Required	Description
source		ResourceLocation	No	The origin or source information for the resource.
source_event_uuid		string	No	UUID of the event that is associated with creation of this resource.

ResourceLocation

The origin or source information for the resource.

Property	Default	Type	Required	Description
path		string	Yes	Path within repository files where the resource is defined.
first_line_number		integer	No	First line number within path file where the resource is defined.