Aliased Table
A component that makes data in a pre-existing table available in the Ascend Flow.
Examples
- aliased_table_reference.yaml
# This YAML configuration is for referencing another table in Ascend
component:
alias:
# Specify the location of the existing table to alias
location: "my_database.my_schema.my_table"
AliasedTableComponent
AliasedTableComponent
is defined beneath the following ancestor nodes in the YAML structure:
Below are the properties for the AliasedTableComponent
. Each property links to the specific details section further down in this page.
Property | Default | Type | Required | Description |
---|---|---|---|---|
skip | boolean | No | A boolean flag indicating whether to skip processing for the component or not. | |
retry_strategy | No | The retry strategy configuration options for the component if any exceptions are encountered. | ||
description | string | No | A brief description of what the model does. | |
metadata | No | Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources. | ||
name | string | Yes | The name of the model | |
flow_name | string | No | The name of the flow that the component belongs to. | |
data_maintenance | No | The data maintenance configuration options for the component. | ||
tests | No | Defines tests to run on the data of this component. | ||
alias | Yes | Configuration options for the aliased table component. |
Property Details
Component
A component is a fundamental building block of a data flow. Types of components that are supported include: read, transform, task, test, and more.
Property | Default | Type | Required | Description |
---|---|---|---|---|
component | One of: CustomPythonReadComponent ApplicationComponent AliasedTableComponent ExternalTableComponent | Yes | Configuration options for the component. |
AliasedTableOptions
Configuration options for the AliasedTable component.
Property | Default | Type | Required | Description |
---|---|---|---|---|
dependencies | array[None] | No | List of dependencies that must complete before this component runs. | |
event_time | string | No | Timestamp column in the component output used to represent event time. | |
location | string | No | Location of the table to alias. |
ComponentTestOptions
Options for component tests, including data quality tests and schema checks.
ColumnTestPython
Test to validate data using a Python function for a single column.
Property | Default | Type | Required | Description |
---|---|---|---|---|
severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
name | string | Yes | ||
python | Yes | Configuration options for the Python column test. |
ColumnTestPythonOptions
Property | Default | Type | Required | Description |
---|---|---|---|---|
entrypoint | string | Yes | The entrypoint for the python transform function. | |
source | string | Yes | The source file for the python transform function. | |
params | object with property values of type None | No | Parameters for the Python test function. | |
is_asset_test | boolean | No |
ColumnTestSql
Test to validate data using an SQL query for a single column.
Property | Default | Type | Required | Description |
---|---|---|---|---|
severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
name | string | Yes | ||
sql | string | No | SQL query that tests data for conditions. |
CombinationUniqueTest
Test to check if a value is unique.
Property | Default | Type | Required | Description |
---|---|---|---|---|
severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
combination_unique | Yes | Test to check if a value is unique. |
CombinationUniqueTestOptions
Configuration options for the unique test.
Property | Default | Type | Required | Description |
---|---|---|---|---|
columns | array[string] | Yes | The combination of columns to check for uniqueness. |
ComponentSchemaTest
Test to validate that component columns match expected types.
Property | Default | Type | Required | Description |
---|---|---|---|---|
severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
match | exact | string ("exact", "ignore_missing") | No | The type of schema matching to perform. 'exact' requires all columns to be present, 'ignore_missing' allows for missing columns. |
columns | object with property values of type string | No | A mapping of column names to their expected types. |
CountDistinctEqualTest
Test to check if the number of distinct values is equal to a certain number.
Property | Default | Type | Required | Description |
---|---|---|---|---|
severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
count_distinct_equal | Yes |
CountDistinctEqualTestOptions
Configuration options for the count_distinct_equal test.
Property | Default | Type | Required | Description |
---|---|---|---|---|
count | integer | Yes | The number of distinct values to expect. | |
group_by_columns | array[string] | No | The columns to group by. |
CountEqualTest
Test to check if the number of rows is equal to a certain number.
Property | Default | Type | Required | Description |
---|---|---|---|---|
severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
count_equal | Yes | Configuration options for the the count_equal test. |
CountEqualTestOptions
Configuration options for the count_equal test.
Property | Default | Type | Required | Description |
---|---|---|---|---|
count | integer | Yes | The number of rows to expect. |
CountGreaterThanOrEqualTest
Test to check if the number of rows is greater than or equal to a certain number.
Property | Default | Type | Required | Description |
---|---|---|---|---|
severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
count_greater_than_or_equal | Yes |
CountGreaterThanOrEqualTestOptions
Configuration options for the count_greater_than_or_equal test.
Property | Default | Type | Required | Description |
---|---|---|---|---|
count | integer | Yes | The value to compare against. | |
group_by_columns | array[string] | No | The columns to group by. |
CountGreaterThanTest
Test to check if the number of rows is greater than a certain number.
Property | Default | Type | Required | Description |
---|---|---|---|---|
severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
count_greater_than | Yes |
CountGreaterThanTestOptions
Configuration options for the count_greater_than test.
Property | Default | Type | Required | Description |
---|---|---|---|---|
count | integer | Yes | The value to compare against. | |
group_by_columns | array[string] | No | The columns to group by. |
CountLessThanOrEqualTest
Test to check if the number of rows is greater than or equal to a certain number.
Property | Default | Type | Required | Description |
---|---|---|---|---|
severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
count_less_than_or_equal | Yes |
CountLessThanOrEqualTestOptions
Configuration options for the count_less_than_or_equal test.
Property | Default | Type | Required | Description |
---|---|---|---|---|
count | integer | Yes | The value to compare against. | |
group_by_columns | array[string] | No | The columns to group by. |
CountLessThanTest
Test to check if the number of rows is less than a certain number.
Property | Default | Type | Required | Description |
---|---|---|---|---|
severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
count_less_than | Yes |
CountLessThanTestOptions
Configuration options for the count_less_than test.
Property | Default | Type | Required | Description |
---|---|---|---|---|
count | integer | Yes | The value to compare against. | |
group_by_columns | array[string] | No | The columns to group by. |
DataMaintenance
Data maintenance configuration options for the component.
Property | Default | Type | Required | Description |
---|---|---|---|---|
enabled | boolean | No | A boolean flag indicating whether data maintenance is enabled for the component. |
DateInRangeTest
Test to check if a date is within a certain range.
Property | Default | Type | Required | Description |
---|---|---|---|---|
severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
date_in_range | Yes |
DateInRangeTestOptions
Configuration options for the date_in_range test.
Property | Default | Type | Required | Description |
---|---|---|---|---|
min | string | Yes | The minimum value to expect. | |
max | string | Yes | The maximum value to expect. |
GreaterThanOrEqualTest
Test to check if a value is greater than or equal to a certain number.
Property | Default | Type | Required | Description |
---|---|---|---|---|
severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
greater_than_or_equal | Yes |
GreaterThanOrEqualTestOptions
Configuration options for the greater_than_or_equal test.
Property | Default | Type | Required | Description |
---|---|---|---|---|
value | Any of: integer number string | Yes | The value to compare against. |
GreaterThanTest
Test to check if a value is greater than a certain number.
Property | Default | Type | Required | Description |
---|---|---|---|---|
severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
greater_than | Yes |
GreaterThanTestOptions
Configuration options for the greater_than test.
Property | Default | Type | Required | Description |
---|---|---|---|---|
value | Any of: integer number string | Yes | The value to compare against. |
InRangeTest
Test to check if a value is within a certain range.
Property | Default | Type | Required | Description |
---|---|---|---|---|
severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
in_range | Yes |
InRangeTestOptions
Configuration options for the in_range test.
Property | Default | Type | Required | Description |
---|---|---|---|---|
min | Any of: integer number string | Yes | The minimum value to expect. | |
max | Any of: integer number string | Yes | The maximum value to expect. |
InSetTest
Test to check if a value is in a set of values.
Property | Default | Type | Required | Description |
---|---|---|---|---|
severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
in_set | Yes |
InSetTestOptions
Configuration options for the in_set test.
Property | Default | Type | Required | Description |
---|---|---|---|---|
values | array[Any of: (integer, number, string)] | Yes | The set of values to expect. |
LessThanOrEqualTest
Test to check if a value is less than or equal to a certain number.
Property | Default | Type | Required | Description |
---|---|---|---|---|
severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
less_than_or_equal | Yes |
LessThanOrEqualTestOptions
Configuration options for the less_than_or_equal test.
Property | Default | Type | Required | Description |
---|---|---|---|---|
value | Any of: integer number string | Yes | The value to compare against. |
LessThanTest
Test to check if a value is less than a certain number.
Property | Default | Type | Required | Description |
---|---|---|---|---|
severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
less_than | Yes |
LessThanTestOptions
Configuration options for the less_than test.
Property | Default | Type | Required | Description |
---|---|---|---|---|
value | Any of: integer number string | Yes | The value to compare against. |
MeanInRangeTest
Test to check if a value is within a certain mean range.
Property | Default | Type | Required | Description |
---|---|---|---|---|
severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
mean_in_range | Yes |
MeanInRangeTestOptions
Configuration options for the mean_in_range test.
Property | Default | Type | Required | Description |
---|---|---|---|---|
min | Any of: integer number string | Yes | The minimum value to expect. | |
max | Any of: integer number string | Yes | The maximum value to expect. |
NotEmptyTest
Test to check if a value is not empty.
Property | Default | Type | Required | Description |
---|---|---|---|---|
severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
not_empty | No | Test to check if a value is not empty. |
NotNullTest
Test to check if a value is not null.
Property | Default | Type | Required | Description |
---|---|---|---|---|
severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
not_null | No | Test to check if a value is not null. |
StddevInRangeTest
Test to check if a value is within a certain standard deviation range.
Property | Default | Type | Required | Description |
---|---|---|---|---|
severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
stddev_in_range | Yes |
StddevInRangeTestOptions
Configuration options for the stddev_in_range test.
Property | Default | Type | Required | Description |
---|---|---|---|---|
min | Any of: integer number string | Yes | The minimum value to expect. | |
max | Any of: integer number string | Yes | The maximum value to expect. |
SubstringMatchTest
Test to check if a value contains a substring.
Property | Default | Type | Required | Description |
---|---|---|---|---|
severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
substring_match | Yes |
SubstringMatchTestOptions
Configuration options for the substring_match test.
Property | Default | Type | Required | Description |
---|---|---|---|---|
substring | string | Yes | The substring to search for. |
UniqueTest
Test to check if a value is unique.
Property | Default | Type | Required | Description |
---|---|---|---|---|
severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
unique | No | Test to check if a value is unique. |
NoTestOptions
Configuration options for tests that have no test body definition (not_null, unique, etc.).
No properties defined.
ResourceMetadata
Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.
Property | Default | Type | Required | Description |
---|---|---|---|---|
source | No | The origin or source information for the resource. | ||
source_event_uuid | string | No | UUID of the event that is associated with creation of this resource. |
ResourceLocation
The origin or source information for the resource.
Property | Default | Type | Required | Description |
---|---|---|---|---|
path | string | Yes | Path within repository files where the resource is defined. | |
first_line_number | integer | No | First line number within path file where the resource is defined. |
RetryStrategy
Retry strategy configuration for component operations. This configuration leverages the tenacity library to implement robust retry mechanisms. The configuration options directly map to tenacity's retry parameters. Details on the tenacity library can be found here: https://tenacity.readthedocs.io/en/latest/api.html#retry-main-api Current implementation includes: - stop_after_attempt: Maximum number of retry attempts - stop_after_delay: Give up on retries one attempt before you would exceed the delay. Will need to supply at least one of the two parameters. Additional retry parameters will be added as needed to support more complex use cases.
Property | Default | Type | Required | Description |
---|---|---|---|---|
stop_after_attempt | integer | No | The number of attempts before giving up, if None is set, will not stop after any attempts. | |
stop_after_delay | integer | No | Give up on retries one attempt before you would exceed the delay, if None is set, will not stop after any attempts. |
InputComponent
Specification for input components, including how partitioning behaviors should be handled. This additional metadata is required when a component is used as an input to other components in a flow.
Property | Default | Type | Required | Description |
---|---|---|---|---|
flow | string | Yes | Name of the parent flow that the input component belongs to. | |
name | string | Yes | The input component name. | |
alias | string | No | The alias to use for the input component. | |
partition_spec | Any of: string ("full_reduction", "map") | No | The type of partitioning to apply to the component's input data before processing the component's logic. Input partitioning is applied before the component's logic is executed. | |
where | string | No | An optional filter condition to apply to the input component's data. | |
partition_binding | Any of: string | No | An optional partition binding specification to apply to the component on a per-output-partition basis against other inputs' partitions. |
PartitionBinding
Property | Default | Type | Required | Description |
---|---|---|---|---|
logical_operator | logical_operator | string ("AND", "OR") | No | The logical operator to use to combine the partition binding predicates provided |
predicates | predicates | array[string] | No | The list of partition binding predicates to apply to the input component's data |
RepartitionSpec
Specification for repartitioning operations on input component's data
Property | Default | Type | Required | Description |
---|---|---|---|---|
repartition | No | Options for repartitioning the input component's data. |
RepartitionOptions
Options for repartitioning the input component's data.
Property | Default | Type | Required | Description |
---|---|---|---|---|
partition_by | string | Yes | The column to partition by. | |
granularity | string | Yes | The granularity to use for the partitioning. |