Databricks Connection
DatabricksConnection
info
DatabricksConnection is defined beneath the following ancestor nodes in the YAML structure:
Below are the properties for the DatabricksConnection. Each property links to the specific details section further down in this page.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| description | string | No | A brief description of what the model does. | |
| metadata | ResourceMetadata | No | Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources. | |
| name | string | Yes | The name of the model | |
| databricks | DatabricksConnectionOptions | Yes | The Databricks connection options. |
Property Details
Connection
A connection to a data source/sink.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| connection | One of: S3Connection GcsConnection AbfsConnection LocalFileConnection SnowflakeConnection BigQueryConnection MSSQLConnection MySQLConnection OracleConnection PostgresConnection HttpConnection DuckDBConnection SynapseConnection SFTPConnection FabricConnection DatabricksConnection | Yes | The connection to the data system. |
DatabricksConnectionOptions
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| max_query_length | integer | No | The maximum query length to allow. | |
| max_concurrent_queries | integer | No | The maximum number of concurrent queries to allow. | |
| max_combined_sql_statements | integer | No | The maximum number of combined SQL statements to allow. | |
| workspace_url | string | Yes | The workspace instance URL of the Databricks workspace, e.g. 'https://dbc-a1b2345c-d6e7.cloud.databricks.com'. | |
| client_id | string | Yes | The client ID for the Databricks Service Principal. | |
| client_secret | string | Yes | The client secret for the Databricks Service Principal. | |
| http_path | string | Yes | The HTTP path of the Databricks SQL warehouse or compute cluster to use for non-PySpark jobs. | |
| cluster_id | string | Yes | The ID of the compute cluster to use for PySpark jobs. | |
| catalog | string | No | The Unity Catalog to use. | |
| schema | string | No | The Unity Catalog schema to use. |
ResourceMetadata
Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| source | ResourceLocation | No | The origin or source information for the resource. | |
| source_event_uuid | string | No | UUID of the event that is associated with creation of this resource. |
ResourceLocation
The origin or source information for the resource.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| path | string | Yes | Path within repository files where the resource is defined. | |
| first_line_number | integer | No | First line number within path file where the resource is defined. |