Custom Python Read Component
Component that reads data using user-defined custom Python code.
CustomPythonReadComponent
info
CustomPythonReadComponent is defined beneath the following ancestor nodes in the YAML structure:
Below are the properties for the CustomPythonReadComponent. Each property links to the specific details section further down in this page.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| data_plane | One of: SnowflakeDataPlane BigQueryDataPlane DuckdbDataPlane DatabricksDataPlane | No | Data Plane-specific configuration options for Components. | |
| skip | boolean | No | Boolean flag indicating whether to skip processing for the Component or not. | |
| retry_strategy | No | Retry strategy configuration options for the Component if any exceptions are encountered. | ||
| data_maintenance | No | The data maintenance configuration options for the Component. | ||
| description | string | No | Brief description of what the model does. | |
| metadata | No | Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources. | ||
| name | string | Yes | The name of the model | |
| flow_name | string | No | Name of the Flow that the Component belongs to. | |
| tests | No | Defines tests to run on this Component's data. | ||
| custom_python_read | Yes |
Property Details
Component
A Component is a fundamental building block of a data Flow. Supported Component types include: Read, Transform, Task, Test, and more.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| component | One of: CustomPythonReadComponent ApplicationComponent AliasedTableComponent ExternalTableComponent DbtNodeComponent | Yes | Component configuration options. |
CustomPythonReadOptions
Configuration options for the Custom Python Read Component.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| dependencies | array[None] | No | List of dependencies that must complete before this Component runs. | |
| event_time | string | No | Timestamp column in the Component output used to represent Event time. | |
| strategy | full | Any of: full IncrementalStrategy PartitionedStrategy | No | Ingest strategy. |
| python | Any of: | Yes | Python code to execute for ingesting data. |
PartitionedListRead
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| list | Yes | Python function that lists partitions in the source. | ||
| read | Yes | Python function that reads a partition from the source. |
PythonBase
Base class for Python-based Components and resources.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| entrypoint | string | Yes | Entry point for the Python Transform function. | |
| source | string | Yes | Source file for the Python Transform function. |
BigQueryDataPlane
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| bigquery | Yes | BigQuery configuration options. |
BigQueryDataPlaneOptions
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| partition_by | Any of: | No | Partition By clause for the table. | |
| cluster_by | array[string] | No | Clustering keys to be added to the table. |
BigQueryRangePartitioning
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| field | string | Yes | Field to partition by. | |
| range | Yes | Range partitioning options. |
BigQueryTimePartitioning
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| field | string | Yes | Field to partition by. | |
| granularity | string ("DAY", "HOUR", "MONTH", "YEAR") | Yes | Granularity of the time partitioning. |
ComponentTestOptions
Options for component tests, including data quality tests and schema checks.
ColumnTestPython
Test to validate data using a Python function for a single column.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
| name | string | Yes | ||
| python | Yes | Configuration options for the Python column test. |
ColumnTestPythonOptions
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| entrypoint | string | Yes | Entry point for the Python Transform function. | |
| source | string | Yes | Source file for the Python Transform function. | |
| params | object with property values of type None | No | Parameters for the Python test function. | |
| is_asset_test | boolean | No |
ColumnTestSql
Test to validate data using an SQL query for a single column.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
| name | string | Yes | ||
| sql | string | No | SQL query that tests data for conditions. |
CombinationUniqueTest
Test to check if a value is unique.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
| combination_unique | Yes | Test to check if a value is unique. |
CombinationUniqueTestOptions
Configuration options for the unique test.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| columns | array[string] | Yes | Combination of columns to check for uniqueness. |
ComponentSchemaTest
Test to validate that component columns match expected types.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
| match | exact | string ("exact", "ignore_missing") | No | The type of schema matching to perform. 'exact' requires all columns to be present, 'ignore_missing' allows for missing columns. |
| columns | object with property values of type string | No | Mapping of column names to their expected types. |
CountDistinctEqualTest
Test to check if the number of distinct values is equal to a certain number.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
| count_distinct_equal | Yes |
CountDistinctEqualTestOptions
Configuration options for the count_distinct_equal test.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| count | integer | Yes | Number of distinct values to expect. | |
| group_by_columns | array[string] | No | Columns to group by. |
CountEqualTest
Test to check if the number of rows is equal to a certain number.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
| count_equal | Yes | Configuration options for the the count_equal test. |
CountEqualTestOptions
Configuration options for the count_equal test.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| count | integer | Yes | Number of rows to expect. |
CountGreaterThanOrEqualTest
Test to check if the number of rows is greater than or equal to a certain number.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
| count_greater_than_or_equal | Yes |
CountGreaterThanOrEqualTestOptions
Configuration options for the count_greater_than_or_equal test.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| count | integer | Yes | Value to compare against. | |
| group_by_columns | array[string] | No | Columns to group by. |
CountGreaterThanTest
Test to check if the number of rows is greater than a certain number.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
| count_greater_than | Yes |
CountGreaterThanTestOptions
Configuration options for the count_greater_than test.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| count | integer | Yes | Value to compare against. | |
| group_by_columns | array[string] | No | Columns to group by. |
CountLessThanOrEqualTest
Test to check if the number of rows is greater than or equal to a certain number.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
| count_less_than_or_equal | Yes |
CountLessThanOrEqualTestOptions
Configuration options for the count_less_than_or_equal test.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| count | integer | Yes | Value to compare against. | |
| group_by_columns | array[string] | No | Columns to group by. |
CountLessThanTest
Test to check if the number of rows is less than a certain number.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
| count_less_than | Yes |
CountLessThanTestOptions
Configuration options for the count_less_than test.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| count | integer | Yes | Value to compare against. | |
| group_by_columns | array[string] | No | Columns to group by. |
DatabricksDataPlane
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| databricks | cluster_by: null pyspark_job_cluster_id: null table_properties: null | No | Databricks configuration options. |
DatabricksDataPlaneOptions
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| table_properties | object with property values of type string | No | Table properties to include when creating the data table. This setting is equivalent to the CREATE TABLE ... TBLPROPERTIES clause. Please refer to the Databricks documentation at https://docs.databricks.com/aws/en/delta/table-properties for available properties depending on your Data Plane. | |
| pyspark_job_cluster_id | string | No | ID of the compute cluster to use for PySpark jobs. | |
| cluster_by | array[string] | No | Clustering keys to be added to the table. |
DateInRangeTest
Test to check if a date is within a certain range.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
| date_in_range | Yes |
DateInRangeTestOptions
Configuration options for the date_in_range test.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| min | string | Yes | Minimum value to expect. | |
| max | string | Yes | Maximum value to expect. |
DuckdbDataPlane
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| duckdb | ducklake: null | No | DuckDB configuration options. |
DuckDbDataPlaneOptions
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| ducklake | No | DuckLake-specific data plane configuration options including table compaction settings. |
DuckLakeDataPlaneOptions
DuckLake-specific data plane configuration options.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| manual_table_compaction | True | boolean | No | Enable manual table compaction for DuckLake tables. |
| metadata_small_file_compaction | count_threshold: 10 file_size_limit: 100 ratio_threshold: null | SmallFileCompactionSettings | No | Settings for compacting metadata tables. |
| data_small_file_compaction | count_threshold: 50 file_size_limit: 100 ratio_threshold: 0.25 | SmallFileCompactionSettings | No | Settings for compacting data tables. |
| partition_by | array[string] | No | Partition keys to be added to the table. Can be column names or expressions (e.g., ['part_key']). | |
| rewrite_data_files | True | boolean | No | Call DuckLake's rewrite_data_files() maintenance operation to optimize table storage. |
| rewrite_data_files_delete_threshold | number | No | Delete threshold for ducklake_rewrite_data_files operation (0.0-1.0). If set to None, DuckLake's default value (0.95) will be used. |
FabricDataPlane
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| fabric | spark_session_config: null | No | Fabric configuration options. |
FabricDataPlaneOptions
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| spark_session_config | No | Spark session configuration. |
GreaterThanOrEqualTest
Test to check if a value is greater than or equal to a certain number.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
| greater_than_or_equal | Yes |
GreaterThanOrEqualTestOptions
Configuration options for the greater_than_or_equal test.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| value | Any of: integer number string | Yes | Value to compare against. |
GreaterThanTest
Test to check if a value is greater than a certain number.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
| greater_than | Yes |
GreaterThanTestOptions
Configuration options for the greater_than test.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| value | Any of: integer number string | Yes | Value to compare against. |
InRangeTest
Test to check if a value is within a certain range.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
| in_range | Yes |
InRangeTestOptions
Configuration options for the in_range test.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| min | Any of: integer number string | Yes | Minimum value to expect. | |
| max | Any of: integer number string | Yes | Maximum value to expect. |
InSetTest
Test to check if a value is in a set of values.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
| in_set | Yes |
InSetTestOptions
Configuration options for the in_set test.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| values | array[Any of: (integer, number, string)] | Yes | Set of values to expect. |
LessThanOrEqualTest
Test to check if a value is less than or equal to a certain number.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
| less_than_or_equal | Yes |
LessThanOrEqualTestOptions
Configuration options for the less_than_or_equal test.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| value | Any of: integer number string | Yes | Value to compare against. |
LessThanTest
Test to check if a value is less than a certain number.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
| less_than | Yes |
LessThanTestOptions
Configuration options for the less_than test.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| value | Any of: integer number string | Yes | Value to compare against. |
MeanInRangeTest
Test to check if a value is within a certain mean range.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
| mean_in_range | Yes |
MeanInRangeTestOptions
Configuration options for the mean_in_range test.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| min | Any of: integer number string | Yes | Minimum value to expect. | |
| max | Any of: integer number string | Yes | Maximum value to expect. |
NotEmptyTest
Test to check if a value is not empty.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
| not_empty | No | Test to check if a value is not empty. |
NotNullTest
Test to check if a value is not null.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
| not_null | No | Test to check if a value is not null. |
RangeOptions
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| start | integer | Yes | Start of the range partitioning. | |
| end | integer | Yes | End of the range partitioning. | |
| interval | integer | Yes | Interval of the range partitioning. |
SmallFileCompactionSettings
Settings for small file compaction thresholds.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| file_size_limit | 100 | integer | No | Files smaller than this size (in MB) are considered 'small' and eligible for compaction. |
| count_threshold | 10 | integer | No | Run compaction if the number of small files exceeds this threshold. |
| ratio_threshold | number | No | Percentage (0.0-1.0) of small files relative to total files. If set, both absolute count AND ratio must pass for compaction to be triggered. If None, only absolute count check is performed. |
SnowflakeDataPlane
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| snowflake | Yes | Snowflake configuration options. |
SnowflakeDataPlaneOptions
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| cluster_by | array[string] | No | Clustering keys to be added to the table. |
IncrementalStrategy
Incremental Processing Strategy.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| incremental | Any of: append MergeStrategy | Yes | Incremental processing strategy. | |
| on_schema_change | string ("ignore", "fail", "append_new_columns", "sync_all_columns", "smart") | No | Policy to apply when schema changes are detected. Defaults to 'fail' if not provided. |
PartitionedStrategy
Partitioned Ingest Strategy. The user is expected to provide 2 functions, a list function that lists partitions in the source, and a read function that reads a partition from the source.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| partitioned | No | Options for partitioning data. | ||
| on_schema_change | string ("ignore", "fail", "append_new_columns", "sync_all_columns", "smart") | No | Policy to apply when schema changes are detected. Defaults to 'fail' if not provided. |
PartitionedOptions
Options related to partition optimization - in particular, the policy that determines which partitions to ingest.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| enable_substitution_by_partition_name | boolean | Yes | Enable substitution by partition name. | |
| output_type | table | string ("table", "view") | No | Output type for partitioned data. Must be either 'table' or 'view'. This strategy only applies to Transforms. |
SCDType2Strategy
The SCD Type 2 strategy allows users to track changes to records over time, by tracking the start and end times for each version of a record. A brief overview of the strategy can be found at https://en.wikipedia.org/wiki/Slowly_changing_dimension#Type_2:_add_new_row.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| scd_type_2 | No | Options for SCD Type 2 strategy. |
StddevInRangeTest
Test to check if a value is within a certain standard deviation range.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
| stddev_in_range | Yes |
StddevInRangeTestOptions
Configuration options for the stddev_in_range test.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| min | Any of: integer number string | Yes | Minimum value to expect. | |
| max | Any of: integer number string | Yes | Maximum value to expect. |
SubstringMatchTest
Test to check if a value contains a substring.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
| substring_match | Yes |
SubstringMatchTestOptions
Configuration options for the substring_match test.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| substring | string | Yes | Substring to search for. |
SynapseDataPlane
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| synapse | spark_session_config: null | No | Synapse configuration options. |
SynapseDataPlaneOptions
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| spark_session_config | No | Spark session configuration. |
LivySparkSessionConfig
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| pool | string | No | Pool to use for the Spark session. | |
| driver_memory | string | No | Memory to use for the Spark driver. | |
| driver_cores | integer | No | Number of cores to use for the Spark driver. | |
| executor_memory | string | No | Memory to use for the Spark executor. | |
| executor_cores | integer | No | Number of cores to use for each executor. | |
| num_executors | integer | No | Number of executors to use for the Spark session. | |
| session_key_override | string | No | Key to use for the Spark session. | |
| max_concurrent_sessions | integer | No | Maximum number of concurrent sessions allowed for this configuration. |
UniqueTest
Test to check if a value is unique.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
| unique | No | Test to check if a value is unique. |
NoTestOptions
Configuration options for tests that have no test body definition (not_null, unique, etc.).
No properties defined.
DataMaintenance
Data maintenance configuration options for Components.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| enabled | boolean | No | Boolean flag indicating whether data maintenance is enabled for the Component. |
ResourceMetadata
Meta information of a resource. In most cases, it doesn't affect the system behavior but may be helpful to analyze Project resources.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| source | No | The origin or source information for the resource. | ||
| source_event_uuid | string | No | Event UUID associated with creation of this resource. |
ResourceLocation
The origin or source information for the resource.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| path | string | Yes | Path within repository files where the resource is defined. | |
| first_line_number | integer | No | First line number within path file where the resource is defined. |
RetryStrategy
Retry strategy configuration for Component operations. This configuration leverages the tenacity library to implement robust retry mechanisms. The configuration options directly map to tenacity's retry parameters. Details on the tenacity library can be found here: https://tenacity.readthedocs.io/en/latest/api.html#retry-main-api Current implementation includes: - stop_after_attempt: Maximum number of retry attempts - stop_after_delay: Give up on retries one attempt before you would exceed the delay. - retry_clauses: Pattern-specific retry rules with their own max_attempts. Will need to supply at least one of the parameters (stop_after_attempt, stop_after_delay, or retry_clauses). Additional retry parameters will be added as needed to support more complex use cases.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| stop_after_attempt | integer | No | Number of retry attempts before giving up. If set to None, it will not stop after any number of attempts. | |
| stop_after_delay | integer | No | Maximum time (in seconds) to spend on retries before giving up. If set to None, it will not stop after any time delay. | |
| retry_clauses | array[None] | No | Pattern-specific retry rules evaluated in order. First matching pattern wins. Non-matching errors use global stop_after_attempt/stop_after_delay. |
RetryClause
A retry rule matching errors by regex pattern.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| pattern | string | Yes | Regex pattern to match against exception message (case-insensitive). | |
| max_attempts | integer | Yes | Maximum retry attempts for errors matching this pattern. |
InputComponent
Specification for input Components defining how partitioning behaviors should be handled. This metadata is required when a Component serves as an input to other Components within a Flow. The reshape parameter controls how input data is partitioned and processed. It accepts either full for full reduction operations or map for partition-wise operations.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| flow | string | Yes | Name of the parent Flow that the input Component belongs to. | |
| name | string | Yes | Name of the input Component. | |
| alias | string | No | Alias to use for the input Component. | |
| partition_spec | Any of: string ("full_reduction", "map") | No | Internal specification for how Component input data should be partitioned before processing. This field is populated based on the user-facing reshape parameter in ref() calls, which accepts full (for full reduction operations) or map (for partition-wise operations). Input partitioning is applied before the Component's logic is executed. | |
| where | string | No | Optional filter condition to apply to the input Component's data. | |
| partition_binding | Any of: string | No | Optional partition binding specification to apply to the Component on a per-output-partition basis against other inputs' partitions. |
MergeStrategy
Strategy that involves merging new data with existing data by updating existing records that match the unique key.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| merge | No | Options for merge strategy. |
KeyOptions
Column options needed for merge and SCD Type 2 strategies, such as unique key and deletion column name.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| unique_key | string | Yes | Column or comma-separated set of columns used as a unique identifier for records, aiding in the merge process. | |
| deletion_column | string | No | Column name used in the upstream source for soft-deleting records. Used when replicating data from a source that supports soft-deletion. If provided, the merge strategy will be able to detect deletions and mark them as deleted in the destination. If not provided, the merge strategy will not be able to detect deletions. | |
| merge_update_columns | Any of: string array[string] | No | List of columns to include when updating values in merge. These columns are mutually exclusive with respect to the columns in merge_exclude_columns. | |
| merge_exclude_columns | Any of: string array[string] | No | List of columns to exclude when updating values in merge. These columns are mutually exclusive with respect to the columns in merge_update_columns. | |
| incremental_predicates | Any of: string array[string] | No | List of conditions to filter incremental data. |
PartitionBinding
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| logical_operator | logical_operator | string ("AND", "OR") | No | TLogical operator to use to combine the partition binding predicates provided |
| predicates | predicates | array[string] | No | List of partition binding predicates to apply to the input Component's data |
RepartitionSpec
Specification for repartitioning operations on input Component's data
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| repartition | No | Options for repartitioning the input Component's data. |
RepartitionOptions
Options for repartitioning the input Component's data.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| partition_by | string | Yes | Column to partition by. | |
| granularity | string | Yes | Granularity to use for the partitioning. |