Application Component
Specification for an application component. An application component is generated by invoking a reusable Application, producing sub-components.
ApplicationComponent
ApplicationComponent is defined beneath the following ancestor nodes in the YAML structure:
Below are the properties for the ApplicationComponent. Each property links to the specific details section further down in this page.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| skip | boolean | No | A boolean flag indicating whether to skip processing for the component or not. | |
| retry_strategy | No | The retry strategy configuration options for the component if any exceptions are encountered. | ||
| description | string | No | A brief description of what the model does. | |
| metadata | No | Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources. | ||
| name | string | Yes | The name of the model | |
| flow_name | string | No | The name of the flow that the component belongs to. | |
| application | Yes | Configuration options for Application Component. |
Property Details
Component
A component is a fundamental building block of a data flow. Types of components that are supported include: read, transform, task, test, and more.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| component | One of: CustomPythonReadComponent ApplicationComponent AliasedTableComponent ExternalTableComponent | Yes | Configuration options for the component. |
ApplicationComponentOptions
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| dependencies | array[None] | No | List of dependencies that must complete before this component runs. | |
| application_id | string | Yes | Reference to the Application resource that generates sub-components. | |
| config | object with property values of type None | Yes | Configuration passed to the Application to generate sub-components. | |
| components | array[string] | No | Names of the generated sub-components. |
ResourceMetadata
Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| source | No | The origin or source information for the resource. | ||
| source_event_uuid | string | No | UUID of the event that is associated with creation of this resource. |
ResourceLocation
The origin or source information for the resource.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| path | string | Yes | Path within repository files where the resource is defined. | |
| first_line_number | integer | No | First line number within path file where the resource is defined. |
RetryStrategy
Retry strategy configuration for component operations. This configuration leverages the tenacity library to implement robust retry mechanisms. The configuration options directly map to tenacity's retry parameters. Details on the tenacity library can be found here: https://tenacity.readthedocs.io/en/latest/api.html#retry-main-api Current implementation includes: - stop_after_attempt: Maximum number of retry attempts - stop_after_delay: Give up on retries one attempt before you would exceed the delay. Will need to supply at least one of the two parameters. Additional retry parameters will be added as needed to support more complex use cases.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| stop_after_attempt | integer | No | The number of attempts before giving up, if None is set, will not stop after any attempts. | |
| stop_after_delay | integer | No | Give up on retries one attempt before you would exceed the delay, if None is set, will not stop after any attempts. |
InputComponent
Specification for input components, including how partitioning behaviors should be handled. This additional metadata is required when a component is used as an input to other components in a flow.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| flow | string | Yes | Name of the parent flow that the input component belongs to. | |
| name | string | Yes | The input component name. | |
| alias | string | No | The alias to use for the input component. | |
| partition_spec | Any of: string ("full_reduction", "map") | No | The type of partitioning to apply to the component's input data before processing the component's logic. Input partitioning is applied before the component's logic is executed. | |
| where | string | No | An optional filter condition to apply to the input component's data. | |
| partition_binding | Any of: string | No | An optional partition binding specification to apply to the component on a per-output-partition basis against other inputs' partitions. |
PartitionBinding
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| logical_operator | logical_operator | string ("AND", "OR") | No | The logical operator to use to combine the partition binding predicates provided |
| predicates | predicates | array[string] | No | The list of partition binding predicates to apply to the input component's data |
RepartitionSpec
Specification for repartitioning operations on input component's data
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| repartition | No | Options for repartitioning the input component's data. |
RepartitionOptions
Options for repartitioning the input component's data.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| partition_by | string | Yes | The column to partition by. | |
| granularity | string | Yes | The granularity to use for the partitioning. |