S3 Connection

Amazon S3 bucket Connection.

Examples

simple_s3_connection.yaml

connection:
  s3:
    root: "s3://my-bucket/"

s3connection_setup.yaml

connection:
  s3:
    root: "s3://my-bucket/"
    region: "us-west-2"
    aws_access_key_id: "${vaults.my_vault_name.aws_access_key_id}"
    aws_secret_access_key: "${vaults.my_vault_name.aws_secret_access_key}"

S3Connection

info

S3Connection is defined beneath the following ancestor nodes in the YAML structure:

• Connection

Below are the properties for the S3Connection. Each property links to the specific details section further down in this page.

Property	Default	Type	Required	Description
description		string	No	Brief description of what the model does.
metadata			No	Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.
name		string	Yes	The name of the model
s3			Yes	The S3 Connection options.

Property Details

Connection

Data source/sink Connection.

Property	Default	Type	Required	Description
connection		One of:   S3Connection   GcsConnection   AbfsConnection   LocalFileConnection   SnowflakeConnection   BigQueryConnection   MSSQLConnection   MySQLConnection   OracleConnection   PostgresConnection   HttpConnection   DuckDBConnection   SFTPConnection   DatabricksConnection	Yes	Data system Connection.

S3ConnectionOptions

Amazon S3 Connection options.

Property	Default	Type	Required	Description
region		string	No	AWS region to connect to.
root		string	Yes	S3 URL for the root prefix.
aws_access_key_id		string	No	Access key ID for the S3 Connection.
aws_secret_access_key		string	No	Secret access key for the S3 Connection.
enable_default_credential_chain	False	boolean	No	If True, uses the default credential chain for S3 authentication if no explicit credentials are provided.
role_arn		string	No	Role ARN to assume when reading from S3.

ResourceMetadata

Meta information of a resource. In most cases, it doesn't affect the system behavior but may be helpful to analyze Project resources.

Property	Default	Type	Required	Description
source			No	The origin or source information for the resource.
source_event_uuid		string	No	Event UUID associated with creation of this resource.

ResourceLocation

The origin or source information for the resource.

Property	Default	Type	Required	Description
path		string	Yes	Path within repository files where the resource is defined.
first_line_number		integer	No	First line number within path file where the resource is defined.