ABFS Connection

Connection to a container in Azure Blob File System using ABFS protocol.

Examples

abfs_connection_shared_key.yaml

connection:
  abfs:
    root: "abfs://my-container@myaccount.dfs.core.windows.net/"
    shared_key: "${secret.my_vault_name.shared_key}"

abfs_connection_service_principal.yaml

connection:
  abfs:
    root: "abfs://my-container@myaccount.dfs.core.windows.net/"
    service_principal:
      client_id: "${secret.my_vault_name.client_id}"
      client_secret: "${secret.my_vault_name.client_secret}"
      tenant_id: "${secret.my_vault_name.tenant_id}"

AbfsConnection

info

AbfsConnection is defined beneath the following ancestor nodes in the YAML structure:

• Connection

Below are the properties for the AbfsConnection. Each property links to the specific details section further down in this page.

Property	Default	Type	Required	Description
name		string	No	The name of the model
description		string	No	A brief description of what the model does.
metadata		ResourceMetadata	No	Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.
abfs		AbfsConnectionOptions	Yes	The ABFS connection options.

Property Details

Connection

A connection to a data source/sink.

Property	Default	Type	Required	Description
connection		One of:   S3Connection   GcsConnection   AbfsConnection   LocalFileConnection   SnowflakeConnection   BigQueryConnection   MSSQLConnection   MySQLConnection   OracleConnection   PostgresConnection   HttpConnection   DuckDBConnection   CustomPythonConnection   SynapseConnection	Yes	The connection to the data system.

AbfsConnectionOptions

Azure Blob File System connection options.

Property	Default	Type	Required	Description
account		string	No	The Azure Blob File System account name to connect to.
root		string	Yes	The abfs[s] URL for the root prefix.
shared_key		string	No	The Azure Blob File System shared key to use for the ABFS connection.
service_principal		AbfsServicePrincipal	No	The Azure Blob File System service principal in JSON to use for the ABFS connection. The JSON should include a key named 'client_id' for the client ID, a key named 'client_secret' for the client secret, and a key named 'tenant_id' for the tenant ID.

AbfsServicePrincipal

Property	Default	Type	Required	Description
client_id		string	Yes	The client ID for the service principal.
client_secret		string	Yes	The client secret for the service principal.
tenant_id		string	Yes	The tenant ID for the service principal.

ResourceMetadata

Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.

Property	Default	Type	Required	Description
source		ResourceLocation	No	The origin or source information for the resource.
source_event_uuid		string	No	UUID of the event that is associated with creation of this resource.

ResourceLocation

The origin or source information for the resource.

Property	Default	Type	Required	Description
path		string	Yes	Path within repository files where the resource is defined.
first_line_number		integer	No	First line number within path file where the resource is defined.