SQL Task Component
TaskSqlComponent
info
TaskSqlComponent is defined beneath the following ancestor nodes in the YAML structure:
Below are the properties for the TaskSqlComponent. Each property links to the specific details section further down in this page.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| dependencies | array[None] | No | List of dependencies that must complete before this component runs. | |
| event_time | string | No | Timestamp column in the component output used to represent event time. | |
| sql | string | No | SQL query to execute for the generic component. |
Property Details
Component
A component is a fundamental building block of a data flow. Types of components that are supported include: read, transform, task, test, and more.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| component | One of: CustomPythonReadComponent ApplicationComponent AliasedTableComponent ExternalTableComponent | Yes | Configuration options for the component. |
TaskComponent
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| skip | boolean | No | A boolean flag indicating whether to skip processing for the component or not. | |
| retry_strategy | No | The retry strategy configuration options for the component if any exceptions are encountered. | ||
| description | string | No | A brief description of what the model does. | |
| metadata | No | Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources. | ||
| name | string | Yes | The name of the model | |
| flow_name | string | No | The name of the flow that the component belongs to. | |
| data_maintenance | No | The data maintenance configuration options for the component. | ||
| tests | No | Defines tests to run on the data of this component. | ||
| task | One of: TaskSqlComponent TaskPythonComponent | Yes |
InputComponent
Specification for input components, including how partitioning behaviors should be handled. This additional metadata is required when a component is used as an input to other components in a flow.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| flow | string | Yes | Name of the parent flow that the input component belongs to. | |
| name | string | Yes | The input component name. | |
| alias | string | No | The alias to use for the input component. | |
| partition_spec | Any of: string ("full_reduction", "map") | No | The type of partitioning to apply to the component's input data before processing the component's logic. Input partitioning is applied before the component's logic is executed. | |
| where | string | No | An optional filter condition to apply to the input component's data. | |
| partition_binding | Any of: string | No | An optional partition binding specification to apply to the component on a per-output-partition basis against other inputs' partitions. |
PartitionBinding
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| logical_operator | logical_operator | string ("AND", "OR") | No | The logical operator to use to combine the partition binding predicates provided |
| predicates | predicates | array[string] | No | The list of partition binding predicates to apply to the input component's data |
RepartitionSpec
Specification for repartitioning operations on input component's data
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| repartition | No | Options for repartitioning the input component's data. |
RepartitionOptions
Options for repartitioning the input component's data.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| partition_by | string | Yes | The column to partition by. | |
| granularity | string | Yes | The granularity to use for the partitioning. |