S3 Connection

Connection to a bucket in Amazon S3.

Examples

simple_s3_connection.yaml

connection:
  s3:
    root: "s3://my-bucket/"

s3connection_setup.yaml

connection:
  s3:
    root: "s3://my-bucket/"
    region: "us-west-2"
    aws_access_key_id: "${vaults.my_vault_name.aws_access_key_id}"
    aws_secret_access_key: "${vaults.my_vault_name.aws_secret_access_key}"

S3Connection

info

S3Connection is defined beneath the following ancestor nodes in the YAML structure:

• Connection

Below are the properties for the S3Connection. Each property links to the specific details section further down in this page.

Property	Default	Type	Required	Description
description		string	No	A brief description of what the model does.
metadata			No	Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.
name		string	Yes	The name of the model
s3			Yes	The S3 connection options.

Property Details

Connection

A connection to a data source/sink.

Property	Default	Type	Required	Description
connection		One of:   S3Connection   GcsConnection   AbfsConnection   LocalFileConnection   SnowflakeConnection   BigQueryConnection   MSSQLConnection   MySQLConnection   OracleConnection   PostgresConnection   HttpConnection   DuckDBConnection   SFTPConnection   DatabricksConnection	Yes	The connection to the data system.

S3ConnectionOptions

Amazon S3 connection options.

Property	Default	Type	Required	Description
region		string	No	The AWS region to connect to.
root		string	Yes	the s3 URL for the root prefix.
aws_access_key_id		string	No	The access key ID to use for the S3 connection.
aws_secret_access_key		string	No	The secret access key to use for the S3 connection.

ResourceMetadata

Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.

Property	Default	Type	Required	Description
source			No	The origin or source information for the resource.
source_event_uuid		string	No	UUID of the event that is associated with creation of this resource.

ResourceLocation

The origin or source information for the resource.

Property	Default	Type	Required	Description
path		string	Yes	Path within repository files where the resource is defined.
first_line_number		integer	No	First line number within path file where the resource is defined.