External Table

Component that constructs and updates an External Table. Currently supported for Snowflake only.

Examples

external_table_example.yaml

component:
  external_table:
    location: "@my_namespace.my_ext_stage/path"
    file_format: "my_file_format"
    pattern: ".*[.]csv"
    auto_refresh: true
    partitions:
      - name: partition_column
        data_type: STRING
        expression: EXTRACT(YEAR FROM my_date_column)
    aws_sns_topic: "arn:aws:sns:us-west-2:123456789012:my_sns_topic"
    integration: "my_integration"

ExternalTableComponent

info

ExternalTableComponent is defined beneath the following ancestor nodes in the YAML structure:

• Component

Below are the properties for the ExternalTableComponent. Each property links to the specific details section further down in this page.

Property	Default	Type	Required	Description
data_plane		One of:     SnowflakeDataPlane     BigQueryDataPlane     DuckdbDataPlane     DatabricksDataPlane	No	Data Plane-specific configuration options for Components.
skip		boolean	No	Boolean flag indicating whether to skip processing for the Component or not.
retry_strategy			No	Retry strategy configuration options for the Component if any exceptions are encountered.
data_maintenance			No	The data maintenance configuration options for the Component.
description		string	No	Brief description of what the model does.
metadata			No	Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.
name		string	Yes	The name of the model
flow_name		string	No	Name of the Flow that the Component belongs to.
tests			No	Defines tests to run on this Component's data.
external_table		Any of:	Yes	Configuration options for the External Table Component.

Property Details

Component

A Component is a fundamental building block of a data Flow. Supported Component types include: Read, Transform, Task, Test, and more.

Property	Default	Type	Required	Description
component		One of:   CustomPythonReadComponent   ApplicationComponent   AliasedTableComponent   ExternalTableComponent	Yes	Component configuration options.

BigQueryExternalTableOptions

Configuration options for the an External Table Component in BigQuery. Currently not implemented, just stubbed for future reference.

Property	Default	Type	Required	Description
dependencies		array[None]	No	List of dependencies that must complete before this Component runs.
event_time		string	No	Timestamp column in the Component output used to represent Event time.
location		string	No

SnowflakeExternalTableOptions

Configuration options for the an External Table Component in Snowflake.

Property	Default	Type	Required	Description
dependencies		array[None]	No	List of dependencies that must complete before this Component runs.
event_time		string	No	Timestamp column in the Component output used to represent Event time.
location		string	No	Snowflake Stage containing the data files, in the format @[namespace.]ext_stage_name[/path].
file_format		string	No	The file format configuration. See the Snowflake documentation at https://docs.snowflake.com/en/sql-reference/sql/create-external-table#required-parameters for more information.
pattern		string	No	RegEx pattern to match files in the stage.
auto_refresh		boolean	No	Determines whether Snowflake should auto-refresh the table.
partitions		array[None]	No	List of virtual columns to compute and partition the table by.
columns		array[None]	No	List of virtual columns to compute.
aws_sns_topic		string	No	Amazon Resource Name (ARN) for the SNS topic for your S3 bucket (required for auto-refreshing tables from S3 using SNS).
integration		string	No	Name of the notification integration (required for auto-refreshing tables from GCS or Azure Blob Store).

SnowflakeVirtualColumnSpec

Configuration options for the Custom Python Read Component.

Property	Default	Type	Required	Description
name		string	Yes	Name of the virtual column.
data_type		string	Yes	Data type of the virtual column.
description		string	No	Description of the virtual column.
expression		string	No	SQL expression that computes the value for the virtual column.

BigQueryDataPlane

Property	Default	Type	Required	Description
bigquery			Yes	BigQuery configuration options.

BigQueryDataPlaneOptions

Property	Default	Type	Required	Description
partition_by		Any of:	No	Partition By clause for the table.
cluster_by		array[string]	No	Clustering keys to be added to the table.

BigQueryRangePartitioning

Property	Default	Type	Required	Description
field		string	Yes	Field to partition by.
range			Yes	Range partitioning options.

BigQueryTimePartitioning

Property	Default	Type	Required	Description
field		string	Yes	Field to partition by.
granularity		string ("DAY", "HOUR", "MONTH", "YEAR")	Yes	Granularity of the time partitioning.

ComponentTestOptions

Options for component tests, including data quality tests and schema checks.

Property	Default	Type	Required	Description
columns		object with property values of type array[One of: (Any of: (not_null, NotNullTest), Any of: (not_empty, NotEmptyTest), Any of: (unique, UniqueTest), CombinationUniqueTest, InRangeTest, DateInRangeTest, InSetTest, SubstringMatchTest, CountDistinctEqualTest, CountGreaterThanOrEqualTest, CountGreaterThanTest, CountLessThanOrEqualTest, CountLessThanTest, CountEqualTest, GreaterThanTest, LessThanTest, GreaterThanOrEqualTest, LessThanOrEqualTest, MeanInRangeTest, StddevInRangeTest, ColumnTestSql, ColumnTestPython)]	No	List of column-level data quality tests for a Component.
component		array[One of: (Any of: (not_null, NotNullTest), Any of: (not_empty, NotEmptyTest), Any of: (unique, UniqueTest), CombinationUniqueTest, InRangeTest, DateInRangeTest, InSetTest, SubstringMatchTest, CountDistinctEqualTest, CountGreaterThanOrEqualTest, CountGreaterThanTest, CountLessThanOrEqualTest, CountLessThanTest, CountEqualTest, GreaterThanTest, LessThanTest, GreaterThanOrEqualTest, LessThanOrEqualTest, MeanInRangeTest, StddevInRangeTest, ColumnTestSql, ColumnTestPython)]	No	List of Component-level tests.
schema			No	List of schema checks for a Component.

ColumnTestPython

Test to validate data using a Python function for a single column.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
name		string	Yes
python			Yes	Configuration options for the Python column test.

ColumnTestPythonOptions

Property	Default	Type	Required	Description
entrypoint		string	Yes	Entry point for the Python Transform function.
source		string	Yes	Source file for the Python Transform function.
params		object with property values of type None	No	Parameters for the Python test function.
is_asset_test		boolean	No

ColumnTestSql

Test to validate data using an SQL query for a single column.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
name		string	Yes
sql		string	No	SQL query that tests data for conditions.

CombinationUniqueTest

Test to check if a value is unique.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
combination_unique			Yes	Test to check if a value is unique.

CombinationUniqueTestOptions

Configuration options for the unique test.

Property	Default	Type	Required	Description
columns		array[string]	Yes	Combination of columns to check for uniqueness.

ComponentSchemaTest

Test to validate that component columns match expected types.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
match	exact	string ("exact", "ignore_missing")	No	The type of schema matching to perform. 'exact' requires all columns to be present, 'ignore_missing' allows for missing columns.
columns		object with property values of type string	No	Mapping of column names to their expected types.

CountDistinctEqualTest

Test to check if the number of distinct values is equal to a certain number.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
count_distinct_equal			Yes

CountDistinctEqualTestOptions

Configuration options for the count_distinct_equal test.

Property	Default	Type	Required	Description
count		integer	Yes	Number of distinct values to expect.
group_by_columns		array[string]	No	Columns to group by.

CountEqualTest

Test to check if the number of rows is equal to a certain number.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
count_equal			Yes	Configuration options for the the count_equal test.

CountEqualTestOptions

Configuration options for the count_equal test.

Property	Default	Type	Required	Description
count		integer	Yes	Number of rows to expect.

CountGreaterThanOrEqualTest

Test to check if the number of rows is greater than or equal to a certain number.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
count_greater_than_or_equal			Yes

CountGreaterThanOrEqualTestOptions

Configuration options for the count_greater_than_or_equal test.

Property	Default	Type	Required	Description
count		integer	Yes	Value to compare against.
group_by_columns		array[string]	No	Columns to group by.

CountGreaterThanTest

Test to check if the number of rows is greater than a certain number.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
count_greater_than			Yes

CountGreaterThanTestOptions

Configuration options for the count_greater_than test.

Property	Default	Type	Required	Description
count		integer	Yes	Value to compare against.
group_by_columns		array[string]	No	Columns to group by.

CountLessThanOrEqualTest

Test to check if the number of rows is greater than or equal to a certain number.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
count_less_than_or_equal			Yes

CountLessThanOrEqualTestOptions

Configuration options for the count_less_than_or_equal test.

Property	Default	Type	Required	Description
count		integer	Yes	Value to compare against.
group_by_columns		array[string]	No	Columns to group by.

CountLessThanTest

Test to check if the number of rows is less than a certain number.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
count_less_than			Yes

CountLessThanTestOptions

Configuration options for the count_less_than test.

Property	Default	Type	Required	Description
count		integer	Yes	Value to compare against.
group_by_columns		array[string]	No	Columns to group by.

DatabricksDataPlane

Property	Default	Type	Required	Description
databricks	cluster_by: null pyspark_job_cluster_id: null table_properties: null		No	Databricks configuration options.

DatabricksDataPlaneOptions

Property	Default	Type	Required	Description
table_properties		object with property values of type string	No	Table properties to include when creating the data table. This setting is equivalent to the CREATE TABLE ... TBLPROPERTIES clause. Please refer to the Databricks documentation at https://docs.databricks.com/aws/en/delta/table-properties for available properties depending on your Data Plane.
pyspark_job_cluster_id		string	No	ID of the compute cluster to use for PySpark jobs.
cluster_by		array[string]	No	Clustering keys to be added to the table.

DateInRangeTest

Test to check if a date is within a certain range.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
date_in_range			Yes

DateInRangeTestOptions

Configuration options for the date_in_range test.

Property	Default	Type	Required	Description
min		string	Yes	Minimum value to expect.
max		string	Yes	Maximum value to expect.

DuckdbDataPlane

Property	Default	Type	Required	Description
duckdb	ducklake_data_table_compaction:   small_file_count_threshold: 50   small_file_ratio_threshold: 0.25   small_file_record_count_limit: 100000 ducklake_metadata_table_compaction:   small_file_count_threshold: 10   small_file_ratio_threshold: null   small_file_record_count_limit: 10		No	DuckDB configuration options.

DuckDbDataPlaneOptions

Property	Default	Type	Required	Description
ducklake_metadata_table_compaction	small_file_count_threshold: 10 small_file_ratio_threshold: null small_file_record_count_limit: 10	DuckLakeTableCompactionSettings	No	Settings for compacting metadata tables. If present, metadata table compaction is enabled.
ducklake_data_table_compaction	small_file_count_threshold: 50 small_file_ratio_threshold: 0.25 small_file_record_count_limit: 100000	DuckLakeTableCompactionSettings	No	Settings for compacting data tables. If present, data table compaction is enabled.

DuckLakeTableCompactionSettings

Settings for DuckLake table compaction.

Property	Default	Type	Required	Description
small_file_record_count_limit	10	integer	No	Files with fewer records than this limit are considered 'small'.
small_file_count_threshold	10	integer	No	Run manual table compaction if the number of files with fewer than small_file_record_count_limit records exceeds this threshold.
small_file_ratio_threshold		number	No	Percentage (0.0-1.0) of small files relative to total files. If set, both absolute count AND ratio must pass for compaction to be triggered. If None, only absolute count check is performed.

FabricDataPlane

Property	Default	Type	Required	Description
fabric	spark_session_config: null		No	Fabric configuration options.

FabricDataPlaneOptions

Property	Default	Type	Required	Description
spark_session_config			No	Spark session configuration.

GreaterThanOrEqualTest

Test to check if a value is greater than or equal to a certain number.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
greater_than_or_equal			Yes

GreaterThanOrEqualTestOptions

Configuration options for the greater_than_or_equal test.

Property	Default	Type	Required	Description
value		Any of:   integer   number   string	Yes	Value to compare against.

GreaterThanTest

Test to check if a value is greater than a certain number.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
greater_than			Yes

GreaterThanTestOptions

Configuration options for the greater_than test.

Property	Default	Type	Required	Description
value		Any of:   integer   number   string	Yes	Value to compare against.

InRangeTest

Test to check if a value is within a certain range.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
in_range			Yes

InRangeTestOptions

Configuration options for the in_range test.

Property	Default	Type	Required	Description
min		Any of:   integer   number   string	Yes	Minimum value to expect.
max		Any of:   integer   number   string	Yes	Maximum value to expect.

InSetTest

Test to check if a value is in a set of values.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
in_set			Yes

InSetTestOptions

Configuration options for the in_set test.

Property	Default	Type	Required	Description
values		array[Any of: (integer, number, string)]	Yes	Set of values to expect.

LessThanOrEqualTest

Test to check if a value is less than or equal to a certain number.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
less_than_or_equal			Yes

LessThanOrEqualTestOptions

Configuration options for the less_than_or_equal test.

Property	Default	Type	Required	Description
value		Any of:   integer   number   string	Yes	Value to compare against.

LessThanTest

Test to check if a value is less than a certain number.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
less_than			Yes

LessThanTestOptions

Configuration options for the less_than test.

Property	Default	Type	Required	Description
value		Any of:   integer   number   string	Yes	Value to compare against.

MeanInRangeTest

Test to check if a value is within a certain mean range.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
mean_in_range			Yes

MeanInRangeTestOptions

Configuration options for the mean_in_range test.

Property	Default	Type	Required	Description
min		Any of:   integer   number   string	Yes	Minimum value to expect.
max		Any of:   integer   number   string	Yes	Maximum value to expect.

NotEmptyTest

Test to check if a value is not empty.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
not_empty			No	Test to check if a value is not empty.

NotNullTest

Test to check if a value is not null.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
not_null			No	Test to check if a value is not null.

RangeOptions

Property	Default	Type	Required	Description
start		integer	Yes	Start of the range partitioning.
end		integer	Yes	End of the range partitioning.
interval		integer	Yes	Interval of the range partitioning.

SnowflakeDataPlane

Property	Default	Type	Required	Description
snowflake			Yes	Snowflake configuration options.

SnowflakeDataPlaneOptions

Property	Default	Type	Required	Description
cluster_by		array[string]	No	Clustering keys to be added to the table.

StddevInRangeTest

Test to check if a value is within a certain standard deviation range.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
stddev_in_range			Yes

StddevInRangeTestOptions

Configuration options for the stddev_in_range test.

Property	Default	Type	Required	Description
min		Any of:   integer   number   string	Yes	Minimum value to expect.
max		Any of:   integer   number   string	Yes	Maximum value to expect.

SubstringMatchTest

Test to check if a value contains a substring.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
substring_match			Yes

SubstringMatchTestOptions

Configuration options for the substring_match test.

Property	Default	Type	Required	Description
substring		string	Yes	Substring to search for.

SynapseDataPlane

Property	Default	Type	Required	Description
synapse	spark_session_config: null		No	Synapse configuration options.

SynapseDataPlaneOptions

Property	Default	Type	Required	Description
spark_session_config			No	Spark session configuration.

LivySparkSessionConfig

Property	Default	Type	Required	Description
pool		string	No	Pool to use for the Spark session.
driver_memory		string	No	Memory to use for the Spark driver.
driver_cores		integer	No	Number of cores to use for the Spark driver.
executor_memory		string	No	Memory to use for the Spark executor.
executor_cores		integer	No	Number of cores to use for each executor.
num_executors		integer	No	Number of executors to use for the Spark session.
session_key_override		string	No	Key to use for the Spark session.
max_concurrent_sessions		integer	No	Maximum number of concurrent sessions allowed for this configuration.

UniqueTest

Test to check if a value is unique.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
unique			No	Test to check if a value is unique.

NoTestOptions

Configuration options for tests that have no test body definition (not_null, unique, etc.).

No properties defined.

DataMaintenance

Data maintenance configuration options for Components.

Property	Default	Type	Required	Description
enabled		boolean	No	Boolean flag indicating whether data maintenance is enabled for the Component.
manual_table_compaction		boolean	No	Boolean flag indicating whether manual table compaction is enabled for the Component. This is currently only relevant for DuckLake Data Planes.
manual_table_compaction_record_count_threshold	10	integer	No	Consider files with fewer than this number of records in determining whether to perform manual table compaction. This is currently only relevant for DuckLake Data Planes.
manual_table_compaction_file_count_threshold	10	integer	No	Run manual table compaction if the number of files with fewer than manual_table_compaction_record_count_threshold records exceeds this threshold. This is currently only relevant for DuckLake Data Planes.

ResourceMetadata

Meta information of a resource. In most cases, it doesn't affect the system behavior but may be helpful to analyze Project resources.

Property	Default	Type	Required	Description
source			No	The origin or source information for the resource.
source_event_uuid		string	No	Event UUID associated with creation of this resource.

ResourceLocation

The origin or source information for the resource.

Property	Default	Type	Required	Description
path		string	Yes	Path within repository files where the resource is defined.
first_line_number		integer	No	First line number within path file where the resource is defined.

RetryStrategy

Retry strategy configuration for Component operations. This configuration leverages the tenacity library to implement robust retry mechanisms. The configuration options directly map to tenacity's retry parameters. Details on the tenacity library can be found here: https://tenacity.readthedocs.io/en/latest/api.html#retry-main-api Current implementation includes: - stop_after_attempt: Maximum number of retry attempts - stop_after_delay: Give up on retries one attempt before you would exceed the delay. Will need to supply at least one of the two parameters. Additional retry parameters will be added as needed to support more complex use cases.

Property	Default	Type	Required	Description
stop_after_attempt		integer	No	Number of retry attempts before giving up. If set to None, it will not stop after any number of attempts.
stop_after_delay		integer	No	Maximum time (in seconds) to spend on retries before giving up. If set to None, it will not stop after any time delay.

InputComponent

Specification for input Components defining how partitioning behaviors should be handled. This metadata is required when a Component serves as an input to other Components within a Flow. The reshape parameter controls how input data is partitioned and processed. It accepts either full for full reduction operations or map for partition-wise operations.

Property	Default	Type	Required	Description
flow		string	Yes	Name of the parent Flow that the input Component belongs to.
name		string	Yes	Name of the input Component.
alias		string	No	Alias to use for the input Component.
partition_spec		Any of:   string ("full_reduction", "map")	No	Internal specification for how Component input data should be partitioned before processing. This field is populated based on the user-facing reshape parameter in ref() calls, which accepts full (for full reduction operations) or map (for partition-wise operations). Input partitioning is applied before the Component's logic is executed.
where		string	No	Optional filter condition to apply to the input Component's data.
partition_binding		Any of:   string	No	Optional partition binding specification to apply to the Component on a per-output-partition basis against other inputs' partitions.

PartitionBinding

Property	Default	Type	Required	Description
logical_operator	logical_operator	string ("AND", "OR")	No	TLogical operator to use to combine the partition binding predicates provided
predicates	predicates	array[string]	No	List of partition binding predicates to apply to the input Component's data

RepartitionSpec

Specification for repartitioning operations on input Component's data

Property	Default	Type	Required	Description
repartition			No	Options for repartitioning the input Component's data.

RepartitionOptions

Options for repartitioning the input Component's data.

Property	Default	Type	Required	Description
partition_by		string	Yes	Column to partition by.
granularity		string	Yes	Granularity to use for the partitioning.