ABFS Connection
InlineAbfsConnection
info
InlineAbfsConnection is defined beneath the following ancestor nodes in the YAML structure:
Below are the properties for the InlineAbfsConnection. Each property links to the specific details section further down in this page.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| abfs | Yes | ABFS Connection options. |
Property Details
Connection
Data source/sink Connection.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| connection | One of: S3Connection GcsConnection AbfsConnection LocalFileConnection SnowflakeConnection BigQueryConnection MSSQLConnection MySQLConnection OracleConnection PostgresConnection HttpConnection DuckDBConnection SFTPConnection DatabricksConnection | Yes | Data system Connection. |
DuckDBConnection
Connection to a DuckDB database.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| description | string | No | Brief description of what the model does. | |
| metadata | No | Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources. | ||
| name | string | Yes | The name of the model | |
| duckdb | Yes |
DuckDBConnectionOptions
DuckDB Connection options.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| max_query_length | integer | No | Maximum combined query length permitted during query batching. | |
| max_concurrent_queries | integer | No | Maximum number of concurrent queries permitted. | |
| max_combined_sql_statements | integer | No | Maximum number of combined SQL statements permitted during query batching. | |
| path | string | No | Path to the DuckDB database file. Use ':memory:' for in-memory databases. This setting is ignored when DuckLake configuration is present. | |
| memory_limit | integer | No | Memory limit to use for the DuckDB Connection. | |
| ducklake | No | Configuration for using DuckLake with this DuckDB Connection. | ||
| motherduck | No | Configuration for using MotherDuck with this DuckDB Connection. | ||
| schema | string | No | Schema to use for the DuckDB Connection. If left empty, the Flow name will be used as the schema name at runtime | |
| init_sql | string | No | SQL to run when the DuckLake Connection is initialized. |
DuckLakeOptions
DuckLake configuration options.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| metadata_connection_name | string | No | Name of the Ascend Connection for DuckLake metadata storage (Postgres Connections only). | |
| data_connection_name | string | No | Name of the Ascend Connection for DuckLake data storage (Local File, GCS, S3, or ABFS Connections supported). | |
| metadata_connection | One of: InlinePostgresConnection InlineS3Connection InlineGcsConnection InlineAbfsConnection | No | Inline metadata Connection configuration. | |
| data_connection | One of: InlineLocalFileConnection InlineS3Connection InlineGcsConnection InlineAbfsConnection | No | Inline data Connection configuration (Local File, GCS, S3, or ABFS Connections supported). | |
| catalog | string | No | Name of the DuckLake catalog to use. If not provided, the catalog name will be inferred from the profile name. | |
| metadata_schema | default | string | No | Schema name within the Postgres database to use for DuckLake metadata storage. If left as the default value of 'default', the schema name will be replaced with the catalog name at runtime. |
| data_path | ascendlake/data | string | No | Path within the data Connection root where DuckLake data files will be stored. The catalog name will always be appended to the path provided. |
| metadata_path | ascendlake/metadata | string | No | Path within the data Connection root where DuckLake metadata files will be stored. The catalog name will always be appended to the path provided. This is only used when the metadata connection is using Inline Blob Storage. |
| data_inlining_row_limit | 0 | integer | No | The number of rows directly written to inlined data tables within the metadata catalog. If 0, turns off data inlining. |
| local_mode | False | boolean | No | If set to True, sets up the DuckLake Connection with local storage, bypassing the metadata_connection(_name), data_connection(_name), and max_concurrent_queries settings. This is useful for rapid testing and development. |
| ducklake_max_retry_count | 100 | integer | No | The value to set for the 'ducklake_max_retry_count' DuckLake configuration setting. Defaults to 100. |
| use_httpfs | False | boolean | No | Use HTTPFS to access data for available providers. |
| cache | No | (experimental) Local cache options for DuckLake. | ||
| use_nightly | False | boolean | No | If True, installs the DuckLake extension from the nightly (core_nightly) repository instead of the stable version. This allows access to the latest features and bug fixes but may be less stable. |
| force_install | False | boolean | No | If True, uses FORCE INSTALL when installing the DuckLake extension. This ensures the extension is reinstalled even if already present, which can be useful in Workspaces to ensure the latest version is used. |
AbfsConnectionOptions
Azure Blob File System Connection options.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| account | string | No | Azure Blob File System account name to connect to. | |
| root | string | Yes | abfs[s] URL for the root prefix. | |
| shared_key | string | No | Azure Blob File System shared key to use for the ABFS Connection. | |
| service_principal | No | Azure Blob File System service principal in JSON to use for the ABFS connection. The JSON should include a key named 'client_id' for the client ID, a key named 'client_secret' for the client secret, and a key named 'tenant_id' for the tenant ID. | ||
| enable_default_credential | False | boolean | No | If True, enables use of the default credential for the ABFS connection if no other credentials are provided. |
AbfsServicePrincipal
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| client_id | string | Yes | Client ID for the service principal. | |
| client_secret | string | Yes | Client secret for the service principal. | |
| tenant_id | string | Yes | Tenant ID for the service principal. |