Profile
A profile is a set of configuration options (and parameters) that define the target where customer code is compiled/run.
Examples​
- profile_parameters.yaml
- profile_skip_slow_components.yaml
- profile_flow_data_plane_config.yaml
profile:
parameters:
param1: value1
param2: value2
profile:
defaults:
- kind: Component
name:
regex: "^(?!slow).*"
spec:
skip: true
profile:
defaults:
- kind: Flow
name:
regex: "^my-flow-.*$"
spec:
data_plane:
connection_name: "my-connection"
Profile​
Below are the properties for the Profile
. Each property links to the specific details section further down in this page.
Property | Default | Type | Required | Description |
---|---|---|---|---|
profile | ProfileOptions | Yes | Options (and parameters) for the profile. |
Property Details​
ProfileOptions​
Configuration options (and parameters) for a profile.
Property | Default | Type | Required | Description |
---|---|---|---|---|
pip_packages | array[string] | No | Python PIP packages to install | |
parameters | object with property values of type None | No | Dictionary of parameters to use for resource. | |
defaults | array[ConfigFilter] | No | List of default configs with filters that can be applied to a resource config. | |
description | string | No | A brief description of what the model does. | |
metadata | ResourceMetadata | No | Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources. | |
name | string | Yes | The name of the model |
ConfigFilter​
A filter used to target configuration settings to a specific flow and/or component.
Property | Default | Type | Required | Description |
---|---|---|---|---|
kind | string ("Flow", "Component") | Yes | The kind of the resource to apply the config to. | |
name | Any of:   string   array[string]   RegexFilter   array[RegexFilter] | Yes | Name of the resource to apply the config to. | |
flow_name | string | No | Name of the flow to apply the config to. | |
spec | Any of:   FlowSpec   ComponentSpec | No | Dictionary of parameters to use for the resource. |
ComponentSpec​
Specification for configuration applied to a component at runtime based on the config filter.
Property | Default | Type | Required | Description |
---|---|---|---|---|
data_plane |   One of:     SnowflakeDataPlane     BigQueryDataPlane     DuckdbDataPlane     SynapseDataPlane     FabricDataPlane     DatabricksDataPlane | No | Data Plane-specific configuration options for a component. | |
skip | False | boolean | No |
FlowSpec​
Specification for configuration applied to a flow at runtime based on the config filter.
Property | Default | Type | Required | Description |
---|---|---|---|---|
data_plane | DataPlane | No | The data plane that will be used for the flow at runtime. |
DataPlane​
The external warehouse where data is persisted throughout the flow runs, and where primary computation on the data itself occurs.
Property | Default | Type | Required | Description |
---|---|---|---|---|
connection_name | string | No | ||
metadata_storage_location_prefix | string | No | Prefix to prepend to the names of metadata tables created for this flow. The prefix may include database/project/etc and schema/dataset/etc where applicable. If not provided, metadata tables are stored alongside the output data tables per the data plane's connection configuration. |
RegexFilter​
A filter used to target resources based on a regex pattern.
Property | Default | Type | Required | Description |
---|---|---|---|---|
regex | string | Yes | The regex to filter the resources. |
BigQueryDataPlane​
Property | Default | Type | Required | Description |
---|---|---|---|---|
bigquery | BigQueryDataPlaneOptions | Yes | BigQuery configuration options. |
BigQueryDataPlaneOptions​
Property | Default | Type | Required | Description |
---|---|---|---|---|
partition_by | Any of:   BigQueryRangePartitioning   BigQueryTimePartitioning | No | Partition By clause for the table. | |
cluster_by | array[string] | No | Clustering keys to be added to the table. |
BigQueryRangePartitioning​
Property | Default | Type | Required | Description |
---|---|---|---|---|
field | string | Yes | Field to partition by. | |
range | RangeOptions | Yes | Range partitioning options. |
BigQueryTimePartitioning​
Property | Default | Type | Required | Description |
---|---|---|---|---|
field | string | Yes | Field to partition by. | |
granularity | string ("DAY", "HOUR", "MONTH", "YEAR") | Yes | Granularity of the time partitioning. |
DatabricksDataPlane​
Property | Default | Type | Required | Description |
---|---|---|---|---|
databricks | cluster_by: null pyspark_job_cluster_id: null table_properties: null | DatabricksDataPlaneOptions | No | Databricks configuration options. |
DatabricksDataPlaneOptions​
Property | Default | Type | Required | Description |
---|---|---|---|---|
table_properties | object with property values of type string | No | Table properties to include when creating the data table. This setting is equivalent to the CREATE TABLE ... TBLPROPERTIES clause. Please refer to the Databricks documentation at https://docs.databricks.com/aws/en/delta/table-properties for available properties depending on your data plane. | |
pyspark_job_cluster_id | string | No | The ID of the compute cluster to use for PySpark jobs. | |
cluster_by | array[string] | No | Clustering keys to be added to the table. |
DuckdbDataPlane​
Property | Default | Type | Required | Description |
---|---|---|---|---|
duckdb | DuckDbDataPlaneOptions | No | Duckdb configuration options. |
DuckDbDataPlaneOptions​
No properties defined.
FabricDataPlane​
Property | Default | Type | Required | Description |
---|---|---|---|---|
fabric | spark_session_config: null | FabricDataPlaneOptions | No | Fabric configuration options. |
FabricDataPlaneOptions​
Property | Default | Type | Required | Description |
---|---|---|---|---|
spark_session_config | LivySparkSessionConfig | No | Spark session configuration. |
RangeOptions​
Property | Default | Type | Required | Description |
---|---|---|---|---|
start | integer | Yes | Start of the range partitioning. | |
end | integer | Yes | End of the range partitioning. | |
interval | integer | Yes | Interval of the range partitioning. |
SnowflakeDataPlane​
Property | Default | Type | Required | Description |
---|---|---|---|---|
snowflake | SnowflakeDataPlaneOptions | Yes | Snowflake configuration options. |
SnowflakeDataPlaneOptions​
Property | Default | Type | Required | Description |
---|---|---|---|---|
cluster_by | array[string] | No | Clustering keys to be added to the table. |
SynapseDataPlane​
Property | Default | Type | Required | Description |
---|---|---|---|---|
synapse | spark_session_config: null | SynapseDataPlaneOptions | No | Synapse configuration options. |
SynapseDataPlaneOptions​
Property | Default | Type | Required | Description |
---|---|---|---|---|
spark_session_config | LivySparkSessionConfig | No | Spark session configuration. |
LivySparkSessionConfig​
Property | Default | Type | Required | Description |
---|---|---|---|---|
pool | string | No | The pool to use for the Spark session. | |
driver_memory | string | No | The memory to use for the Spark driver. | |
driver_cores | integer | No | The number of cores to use for the Spark driver. | |
executor_memory | string | No | The memory to use for the Spark executor. | |
executor_cores | integer | No | The number of cores to use for each executor. | |
num_executors | integer | No | The number of executors to use for the Spark session. | |
session_key_override | string | No | The key to use for the Spark session. | |
max_concurrent_sessions | integer | No | The maximum number of concurrent sessions of this spec to create. |
ResourceMetadata​
Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.
Property | Default | Type | Required | Description |
---|---|---|---|---|
source | ResourceLocation | No | The origin or source information for the resource. | |
source_event_uuid | string | No | UUID of the event that is associated with creation of this resource. |
ResourceLocation​
The origin or source information for the resource.
Property | Default | Type | Required | Description |
---|---|---|---|---|
path | string | Yes | Path within repository files where the resource is defined. | |
first_line_number | integer | No | First line number within path file where the resource is defined. |