S3 Connection
Connection to a bucket in Amazon S3.
Examples​
- simple_s3_connection.yaml
- s3connection_setup.yaml
connection:
s3:
root: "s3://my-bucket/"
connection:
s3:
root: "s3://my-bucket/"
region: "us-west-2"
aws_access_key_id: "${vaults.my_vault_name.aws_access_key_id}"
aws_secret_access_key: "${vaults.my_vault_name.aws_secret_access_key}"
S3Connection​
info
S3Connection
is defined beneath the following ancestor nodes in the YAML structure:
Below are the properties for the S3Connection
. Each property links to the specific details section further down in this page.
Property | Default | Type | Required | Description |
---|---|---|---|---|
description | string | No | A brief description of what the model does. | |
metadata | No | Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources. | ||
name | string | Yes | The name of the model | |
s3 | Yes | The S3 connection options. |
Property Details​
Connection​
A connection to a data source/sink.
Property | Default | Type | Required | Description |
---|---|---|---|---|
connection | One of:   S3Connection   GcsConnection   AbfsConnection   LocalFileConnection   SnowflakeConnection   BigQueryConnection   MSSQLConnection   MySQLConnection   OracleConnection   PostgresConnection   HttpConnection   DuckDBConnection   SFTPConnection   DatabricksConnection | Yes | The connection to the data system. |
S3ConnectionOptions​
Amazon S3 connection options.
Property | Default | Type | Required | Description |
---|---|---|---|---|
region | string | No | The AWS region to connect to. | |
root | string | Yes | the s3 URL for the root prefix. | |
aws_access_key_id | string | No | The access key ID to use for the S3 connection. | |
aws_secret_access_key | string | No | The secret access key to use for the S3 connection. |
ResourceMetadata​
Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.
Property | Default | Type | Required | Description |
---|---|---|---|---|
source | No | The origin or source information for the resource. | ||
source_event_uuid | string | No | UUID of the event that is associated with creation of this resource. |
ResourceLocation​
The origin or source information for the resource.
Property | Default | Type | Required | Description |
---|---|---|---|---|
path | string | Yes | Path within repository files where the resource is defined. | |
first_line_number | integer | No | First line number within path file where the resource is defined. |