Skip to main content

DuckDB Connection

Connection to a DuckDB database.

Examples

connection:
duckdb:
path: /path/to/your/duckdb/file

DuckDBConnection

info

DuckDBConnection is defined beneath the following ancestor nodes in the YAML structure:

Below are the properties for the DuckDBConnection. Each property links to the specific details section further down in this page.

PropertyDefaultTypeRequiredDescription
descriptionstring
NoA brief description of what the model does.
metadataNoMeta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.
namestringYesThe name of the model
duckdbYesThe DuckDB connection options.

Property Details

Connection

A connection to a data source/sink.

PropertyDefaultTypeRequiredDescription
connectionOne of:
  S3Connection
  GcsConnection
  AbfsConnection
  LocalFileConnection
  SnowflakeConnection
  BigQueryConnection
  MSSQLConnection
  MySQLConnection
  OracleConnection
  PostgresConnection
  HttpConnection
  DuckDBConnection
  SFTPConnection
  DatabricksConnection
YesThe connection to the data system.

DuckDBConnectionOptions

DuckDB connection options.

PropertyDefaultTypeRequiredDescription
max_query_lengthinteger
NoThe maximum query length to allow.
max_concurrent_queriesinteger
NoThe maximum number of concurrent queries to allow.
max_combined_sql_statementsinteger
NoThe maximum number of combined SQL statements to allow.
pathstring
NoThe path to the DuckDB database file. Use ':memory:' for in-memory databases. This setting is ignored when ducklake configuration is present.
memory_limitinteger
NoThe memory limit to use for the DuckDB connection.
ducklakeNoConfiguration for using DuckLake with this DuckDB connection.
schemastringNoThe schema to use for the DuckDB connection. If left empty, the flow name will be used as the schema name at runtime
init_sqlstring
NoSQL to run when the DuckLake connection is initialized.

DuckLakeOptions

DuckLake configuration options.

PropertyDefaultTypeRequiredDescription
metadata_connection_namestring
NoThe name of the Ascend Connection to use for DuckLake metadata storage (Postgres connections only).
data_connection_namestring
NoThe name of the Ascend Connection to use for DuckLake data storage (Local Files, GCS, S3, or ABFS connections supported).
metadata_connectionAny of:
  InlinePostgresConnection
  ASCEND_MANAGED
NoInline metadata connection configuration (Postgres connections only).
data_connectionAny of:
  One of:
    InlineLocalFileConnection
    InlineS3Connection
    InlineGcsConnection
    InlineAbfsConnection
  ASCEND_MANAGED
NoInline data connection configuration (Local Files, GCS, S3, or ABFS connections supported).
metadata_schemadefaultstringNoThe schema name within the Postgres database to use for DuckLake metadata storage. If left as the default value of 'default', the schema name will be replaced with the profile name at runtime.
data_pathascendlake/datastringNoPath within the data connection root where DuckLake data files will be stored.

InlineAbfsConnection

PropertyDefaultTypeRequiredDescription
abfsYesThe ABFS connection options.

InlineGcsConnection

PropertyDefaultTypeRequiredDescription
gcsYesThe GCS connection options.

GcsConnectionOptions

Google Cloud Storage connection options.

PropertyDefaultTypeRequiredDescription
rootstringYesthe GCS URL for the root prefix.
keystring
NoThe GCP service account credentials to use for the GCS connection.

InlineLocalFileConnection

PropertyDefaultTypeRequiredDescription
local_fileYesThe local file connection options.

InlinePostgresConnection

PropertyDefaultTypeRequiredDescription
postgresYesThe PostgreSQL connection options.

InlineS3Connection

PropertyDefaultTypeRequiredDescription
s3YesThe S3 connection options.

LocalFileConnectionOptions

Local file connection options.

PropertyDefaultTypeRequiredDescription
rootstringYesThe root directory for the local file connection.

PostgresConnectionOptions

PostgreSQL connection options.

PropertyDefaultTypeRequiredDescription
hoststringYesThe PostgreSQL host to connect to.
userstringYesThe PostgreSQL user to connect as.
passwordstringYesThe PostgreSQL password to use for the connection.
databasestringYesThe PostgreSQL database to connect to.
schemapublicstring
NoThe PostgreSQL schema to use.
portinteger
NoThe PostgreSQL port to connect to.

S3ConnectionOptions

Amazon S3 connection options.

PropertyDefaultTypeRequiredDescription
regionstring
NoThe AWS region to connect to.
rootstringYesthe s3 URL for the root prefix.
aws_access_key_idstring
NoThe access key ID to use for the S3 connection.
aws_secret_access_keystring
NoThe secret access key to use for the S3 connection.
enable_default_credential_chainFalsebooleanNoIf True, enables use of the default credential chain for the S3 connection if no other credentials are provided.

AbfsConnectionOptions

Azure Blob File System connection options.

PropertyDefaultTypeRequiredDescription
accountstring
NoThe Azure Blob File System account name to connect to.
rootstringYesThe abfs[s] URL for the root prefix.
shared_keystring
NoThe Azure Blob File System shared key to use for the ABFS connection.
service_principalNoThe Azure Blob File System service principal in JSON to use for the ABFS connection. The JSON should include a key named 'client_id' for the client ID, a key named 'client_secret' for the client secret, and a key named 'tenant_id' for the tenant ID.
enable_default_credentialFalsebooleanNoIf True, enables use of the default credential for the ABFS connection if no other credentials are provided.

AbfsServicePrincipal

PropertyDefaultTypeRequiredDescription
client_idstringYesThe client ID for the service principal.
client_secretstringYesThe client secret for the service principal.
tenant_idstringYesThe tenant ID for the service principal.

ResourceMetadata

Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.

PropertyDefaultTypeRequiredDescription
sourceNoThe origin or source information for the resource.
source_event_uuidstring
NoUUID of the event that is associated with creation of this resource.

ResourceLocation

The origin or source information for the resource.

PropertyDefaultTypeRequiredDescription
pathstringYesPath within repository files where the resource is defined.
first_line_numberinteger
NoFirst line number within path file where the resource is defined.