Role Based Authentication
AwsRoleBasedAuthentication
info
AwsRoleBasedAuthentication is defined beneath the following ancestor nodes in the YAML structure:
Below are the properties for the AwsRoleBasedAuthentication. Each property links to the specific details section further down in this page.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| aws_access_key_id | string | No | AWS access key ID to use for the DB Connection. If not provided, the default credential chain will be used. | |
| aws_secret_access_key | string | No | AWS secret access key to use for the DB Connection. If not provided, the default credential chain will be used. | |
| region | string | No | AWS region to use for the DB Connection. If not provided, the default credential chain will be used. | |
| role_arn | string | Yes | AWS Role ARN to use for the DB Connection. Required for AWS IAM authentication. |
Property Details
Connection
Data source/sink Connection.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| connection | One of: S3Connection GcsConnection AbfsConnection LocalFileConnection SnowflakeConnection BigQueryConnection MSSQLConnection MySQLConnection OracleConnection PostgresConnection HttpConnection DuckDBConnection SFTPConnection DatabricksConnection | Yes | Data system Connection. |
DuckDBConnection
Connection to a DuckDB database.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| description | string | No | A brief description of what the model does. | |
| metadata | No | Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources. | ||
| name | string | Yes | The name of the model | |
| duckdb | Yes |
DuckDBConnectionOptions
DuckDB Connection options.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| max_query_length | integer | No | Maximum combined query length to allow during query batching. | |
| max_concurrent_queries | integer | No | Maximum number of concurrent queries to allow. | |
| max_combined_sql_statements | integer | No | Maximum number of combined SQL statements to allow during query batching. | |
| path | string | No | The path to the DuckDB database file. Use ':memory:' for in-memory databases. This setting is ignored when DuckLake configuration is present. | |
| memory_limit | integer | No | The memory limit to use for the DuckDB Connection. | |
| ducklake | No | Configuration for using DuckLake with this DuckDB Connection. | ||
| schema | string | No | Schema to use for the DuckDB Connection. If left empty, the Flow name will be used as the schema name at runtime | |
| init_sql | string | No | SQL to run when the DuckLake Connection is initialized. |
DuckLakeOptions
DuckLake configuration options.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| metadata_connection_name | string | No | Name of the Ascend Connection to use for DuckLake metadata storage (Postgres Connections only). | |
| data_connection_name | string | No | Name of the Ascend Connection to use for DuckLake data storage (Local Files, GCS, S3, or ABFS Connections supported). | |
| metadata_connection | Any of: InlinePostgresConnection ASCEND_MANAGED | No | Inline metadata Connection configuration (Postgres Connections only). | |
| data_connection | Any of: One of: InlineLocalFileConnection InlineS3Connection InlineGcsConnection InlineAbfsConnection ASCEND_MANAGED | No | Inline data Connection configuration (Local Files, GCS, S3, or ABFS Connections supported). | |
| catalog | string | No | The name of the DuckLake catalog to use. If not provided, the catalog name will be inferred from the profile name. | |
| metadata_schema | default | string | No | The schema name within the Postgres database to use for DuckLake metadata storage. If left as the default value of 'default', the schema name will be replaced with the catalog name at runtime. |
| data_path | ascendlake/data | string | No | Path within the data Connection root where DuckLake data files will be stored. The catalog name will always be appended to the path provided. |
| local_mode | False | boolean | No | If set to True, sets up the DuckLake Connection with local storage, bypassing the metadata_connection(_name), data_connection(_name), and max_concurrent_queries settings. This is useful for rapid testing and development. |
| ducklake_max_retry_count | 100 | integer | No | The value to set for the 'ducklake_max_retry_count' DuckLake configuration setting. Defaults to 100. |
InlinePostgresConnection
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| postgres | Yes |
MySQLConnection
MySQL database Connection.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| description | string | No | A brief description of what the model does. | |
| metadata | No | Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources. | ||
| name | string | Yes | The name of the model | |
| ssh_tunnel | string | No | ||
| mysql | Yes |
MySQLConnectionOptions
MySQL Connection options.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| user | string | Yes | User to use for the DB Connection. | |
| password | string | No | Password to use for the DB Connection. | |
| aws_iam_auth | AwsRoleBasedAuthentication | No | AWS IAM authentication to use for the DB Connection. | |
| host | string | Yes | MySQL host to connect to. | |
| database | string | Yes | MySQL database to connect to. | |
| port | integer | No | MySQL port to connect to. | |
| ssl | Any of: boolean | No | SSL options to use for the MySQL Connection. |
PostgresConnection
Connection to a PostgreSQL database.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| description | string | No | A brief description of what the model does. | |
| metadata | No | Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources. | ||
| name | string | Yes | The name of the model | |
| ssh_tunnel | string | No | ||
| postgres | Yes |
PostgresConnectionOptions
PostgreSQL Connection options.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| user | string | Yes | User to use for the DB Connection. | |
| password | string | No | Password to use for the DB Connection. | |
| aws_iam_auth | AwsRoleBasedAuthentication | No | AWS IAM authentication to use for the DB Connection. | |
| host | string | Yes | PostgreSQL host to connect to. | |
| database | string | Yes | PostgreSQL database to connect to. | |
| schema | public | string | No | PostgreSQL schema to use. |
| port | integer | No | PostgreSQL port to connect to. |