Version: 3.0.0

Custom Python Read

A component that reads data using user-defined, custom Python code.

CustomPythonReadComponent

info

CustomPythonReadComponent is defined beneath the following ancestor nodes in the YAML structure:

Component

Below are the properties for the CustomPythonReadComponent. Each property links to the specific details section further down in this page.

Property	Type	Required	Description
data_plane	One of: SnowflakeDataPlane BigQueryDataPlane DuckdbDataPlane SynapseDataPlane	No	Data Plane-specific configuration options for a component.
name	string	No	The name of the model
description	string	No	A brief description of what the model does.
metadata	ResourceMetadata	No	Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.
flow_name	string	No	The name of the flow that the component belongs to.
skip	boolean	No	A boolean flag indicating whether to skip processing for the component or not.
data_maintenance	DataMaintenance	No	The data maintenance configuration options for the component.
skip_for_time_series_runs	boolean	No	A boolean flag indicating whether to skip processing for this component in time-series runs.
tests	ComponentTestColumn	No	Defines tests to run on the data of this component.
custom_python_read	CustomPythonReadOptions	Yes

Property Details

Component

A component is a fundamental building block of a data flow. Types of components that are supported include: read, transform, task, test, and more.

Property	Default	Type	Required	Description
component		One of: ReadComponent TransformComponent TaskComponent SingularTestComponent CustomPythonReadComponent WriteComponent CompoundComponent AliasedTableComponent ExternalTableComponent	Yes	Configuration options for the component.

CustomPythonReadOptions

Configuration options for the custom Python read component.

Property	Default	Type	Required	Description
event_time		string	No	Timestamp column in the component output used to represent event time.
entrypoint		string	Yes	The entrypoint for the python transform function.
source		string	Yes	The source file for the python transform function.
ingest_mode	full	string ("incremental", "full")	No	The ingest mode for the custom Python read connector ('incremental' or 'full'). Defaults to 'full'.
materialization		PartitionMaterialization	No	Configuration options for how data is materialized after it is read.

BigQueryDataPlane

Property	Default	Type	Required	Description
bigquery		BigQueryDataPlaneOptions	Yes	BigQuery configuration options.

BigQueryDataPlaneOptions

Property	Default	Type	Required	Description
partition_by		Any of: BigQueryRangePartitioning BigQueryTimePartitioning	No	Partition By clause for the table.
cluster_by		array[string]	No	Clustering keys to be added to the table.

BigQueryRangePartitioning

Property	Default	Type	Required	Description
field		string	Yes	Field to partition by.
range		RangeOptions	Yes	Range partitioning options.

BigQueryTimePartitioning

Property	Default	Type	Required	Description
field		string	Yes	Field to partition by.
granularity		string ("DAY", "HOUR", "MONTH", "YEAR")	Yes	Granularity of the time partitioning.

ComponentTestColumn

Property	Type	Required	Description
columns	object with property values of type array[One of: (Any of: (string ("not_null"), NotNullTest), Any of: (string ("not_empty"), NotEmptyTest), Any of: (string ("unique"), UniqueTest), CombinationUniqueTest, InRangeTest, DateInRangeTest, InSetTest, SubstringMatchTest, CountDistinctEqualTest, CountGreaterThanOrEqualTest, CountGreaterThanTest, CountLessThanOrEqualTest, CountLessThanTest, CountEqualTest, GreaterThanTest, LessThanTest, GreaterThanOrEqualTest, LessThanOrEqualTest, MeanInRangeTest, StddevInRangeTest, ColumnTestSql, ColumnTestPython)]	No	List of tests to run on columns the data after processing for validation purposes. Used in the context of a component.
component	array[One of: (Any of: (string ("not_null"), NotNullTest), Any of: (string ("not_empty"), NotEmptyTest), Any of: (string ("unique"), UniqueTest), CombinationUniqueTest, InRangeTest, DateInRangeTest, InSetTest, SubstringMatchTest, CountDistinctEqualTest, CountGreaterThanOrEqualTest, CountGreaterThanTest, CountLessThanOrEqualTest, CountLessThanTest, CountEqualTest, GreaterThanTest, LessThanTest, GreaterThanOrEqualTest, LessThanOrEqualTest, MeanInRangeTest, StddevInRangeTest, ColumnTestSql, ColumnTestPython)]	No	List of component level tests.
schema	ComponentSchemaTest	No	List of the component's schema level tests.

ColumnTestPython

Test to validate data using a Python function for a single column.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
name		string	Yes
python		ColumnTestPythonOptions	Yes	Configuration options for the Python column test.

ColumnTestPythonOptions

Property	Type	Required	Description
entrypoint	string	Yes	The entrypoint for the python transform function.
source	string	Yes	The source file for the python transform function.
params	object	No	Parameters for the Python test function.
is_asset_test	boolean	No

ColumnTestSql

Test to validate data using an SQL query for a single column.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
name		string	Yes
sql		string	No	SQL query that tests data for conditions.

CombinationUniqueTest

Test to check if a value is unique.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
combination_unique		CombinationUniqueTestOptions	Yes	Test to check if a value is unique.

CombinationUniqueTestOptions

Configuration options for the unique test.

Property	Default	Type	Required	Description
columns		array[string]	Yes	The combination of columns to check for uniqueness.

ComponentSchemaTest

Test to validate that component columns match expected types.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
match	exact	string ("exact", "ignore_missing")	No	The type of schema matching to perform. 'exact' requires all columns to be present, 'ignore_missing' allows for missing columns.
columns		object with property values of type string	No	A mapping of column names to their expected types.

CountDistinctEqualTest

Test to check if the number of distinct values is equal to a certain number.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
count_distinct_equal		CountDistinctEqualTestOptions	Yes	Configuration options for the count_distinct_equal test.

CountDistinctEqualTestOptions

Configuration options for the count_distinct_equal test.

Property	Default	Type	Required	Description
count		integer	Yes	The number of distinct values to expect.
group_by_columns		array[string]	No	The columns to group by.

CountEqualTest

Test to check if the number of rows is equal to a certain number.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
count_equal		CountEqualTestOptions	Yes	Configuration options for the the count_equal test.

CountEqualTestOptions

Configuration options for the count_equal test.

Property	Default	Type	Required	Description
count		integer	Yes	The number of rows to expect.

CountGreaterThanOrEqualTest

Test to check if the number of rows is greater than or equal to a certain number.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
count_greater_than_or_equal		CountGreaterThanOrEqualTestOptions	Yes	Configuration options for the count_greater_than_or_equal test.

CountGreaterThanOrEqualTestOptions

Configuration options for the count_greater_than_or_equal test.

Property	Default	Type	Required	Description
count		integer	Yes	The value to compare against.
group_by_columns		array[string]	No	The columns to group by.

CountGreaterThanTest

Test to check if the number of rows is greater than a certain number.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
count_greater_than		CountGreaterThanTestOptions	Yes	Configuration options for the count_greater_than test.

CountGreaterThanTestOptions

Configuration options for the count_greater_than test.

Property	Default	Type	Required	Description
count		integer	Yes	The value to compare against.
group_by_columns		array[string]	No	The columns to group by.

CountLessThanOrEqualTest

Test to check if the number of rows is greater than or equal to a certain number.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
count_less_than_or_equal		CountLessThanOrEqualTestOptions	Yes	Configuration options for the count_less_than_or_equal test.

CountLessThanOrEqualTestOptions

Configuration options for the count_less_than_or_equal test.

Property	Default	Type	Required	Description
count		integer	Yes	The value to compare against.
group_by_columns		array[string]	No	The columns to group by.

CountLessThanTest

Test to check if the number of rows is less than a certain number.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
count_less_than		CountLessThanTestOptions	Yes	Configuration options for the count_less_than test.

CountLessThanTestOptions

Configuration options for the count_less_than test.

Property	Default	Type	Required	Description
count		integer	Yes	The value to compare against.
group_by_columns		array[string]	No	The columns to group by.

DataMaintenance

Data maintenance configuration options for the component.

Property	Default	Type	Required	Description
enabled		boolean	No	A boolean flag indicating whether data maintenance is enabled for the component.

DateInRangeTest

Test to check if a date is within a certain range.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
date_in_range		DateInRangeTestOptions	Yes	Configuration options for the date_in_range test.

DateInRangeTestOptions

Configuration options for the date_in_range test.

Property	Default	Type	Required	Description
min		string	Yes	The minimum value to expect.
max		string	Yes	The maximum value to expect.

DuckdbDataPlane

Property	Default	Type	Required	Description
duckdb		DuckDbDataPlaneOptions	No	Duckdb configuration options.

DuckDbDataPlaneOptions

No properties defined.

GreaterThanOrEqualTest

Test to check if a value is greater than or equal to a certain number.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
greater_than_or_equal		GreaterThanOrEqualTestOptions	Yes	Configuration options for the greater_than_or_equal test.

GreaterThanOrEqualTestOptions

Configuration options for the greater_than_or_equal test.

Property	Default	Type	Required	Description
value		Any of: integer number string	Yes	The value to compare against.

GreaterThanTest

Test to check if a value is greater than a certain number.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
greater_than		GreaterThanTestOptions	Yes	Configuration options for the greater_than test.

GreaterThanTestOptions

Configuration options for the greater_than test.

Property	Default	Type	Required	Description
value		Any of: integer number string	Yes	The value to compare against.

InRangeTest

Test to check if a value is within a certain range.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
in_range		InRangeTestOptions	Yes	Configuration options for the in_range test.

InRangeTestOptions

Configuration options for the in_range test.

Property	Default	Type	Required	Description
min		Any of: integer number string	Yes	The minimum value to expect.
max		Any of: integer number string	Yes	The maximum value to expect.

InSetTest

Test to check if a value is in a set of values.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
in_set		InSetTestOptions	Yes	Configuration options for the in_set test.

InSetTestOptions

Configuration options for the in_set test.

Property	Default	Type	Required	Description
values		array[Any of: (integer, number, string)]	Yes	The set of values to expect.

LessThanOrEqualTest

Test to check if a value is less than or equal to a certain number.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
less_than_or_equal		LessThanOrEqualTestOptions	Yes	Configuration options for the less_than_or_equal test.

LessThanOrEqualTestOptions

Configuration options for the less_than_or_equal test.

Property	Default	Type	Required	Description
value		Any of: integer number string	Yes	The value to compare against.

LessThanTest

Test to check if a value is less than a certain number.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
less_than		LessThanTestOptions	Yes	Configuration options for the less_than test.

LessThanTestOptions

Configuration options for the less_than test.

Property	Default	Type	Required	Description
value		Any of: integer number string	Yes	The value to compare against.

MeanInRangeTest

Test to check if a value is within a certain mean range.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
mean_in_range		MeanInRangeTestOptions	Yes	Configuration options for the mean_in_range test.

MeanInRangeTestOptions

Configuration options for the mean_in_range test.

Property	Default	Type	Required	Description
min		Any of: integer number string	Yes	The minimum value to expect.
max		Any of: integer number string	Yes	The maximum value to expect.

NotEmptyTest

Test to check if a value is not empty.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
not_empty		NoTestOptions	No	Test to check if a value is not empty.

NotNullTest

Test to check if a value is not null.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
not_null		NoTestOptions	No	Test to check if a value is not null.

RangeOptions

Property	Type	Required	Description
start	integer	Yes	Start of the range partitioning.
end	integer	Yes	End of the range partitioning.
interval	integer	Yes	Interval of the range partitioning.

SnowflakeDataPlane

Property	Default	Type	Required	Description
snowflake		SnowflakeDataPlaneOptions	Yes	Snowflake configuration options.

SnowflakeDataPlaneOptions

Property	Default	Type	Required	Description
cluster_by		array[string]	No	Clustering keys to be added to the table.

PartitionMaterialization

Container for options for how data is materialized and stored for partitioned case.

Property	Default	Type	Required	Description
partitioned		PartitionedOptions	No	Field for options for partitioning data.

PartitionedOptions

Options for partitioning data.

Property	Default	Type	Required	Description
enable_substitution_by_partition_name		boolean	Yes	Enable substitution by partition name.
on_schema_change		string ("ignore", "fail", "append_new_columns", "sync_all_columns")	No	Policy to apply when schema changes are detected.
output_type	table	string ("table", "view")	No	Output type for partitioned data. Must be either 'table' or 'view'.

StddevInRangeTest

Test to check if a value is within a certain standard deviation range.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
stddev_in_range		StddevInRangeTestOptions	Yes	Configuration options for the stddev_in_range test.

StddevInRangeTestOptions

Configuration options for the stddev_in_range test.

Property	Default	Type	Required	Description
min		Any of: integer number string	Yes	The minimum value to expect.
max		Any of: integer number string	Yes	The maximum value to expect.

SubstringMatchTest

Test to check if a value contains a substring.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
substring_match		SubstringMatchTestOptions	Yes	Configuration options for the substring_match test.

SubstringMatchTestOptions

Configuration options for the substring_match test.

Property	Default	Type	Required	Description
substring		string	Yes	The substring to search for.

SynapseDataPlane

Property	Default	Type	Required	Description
synapse	spark_session_config: null	SynapseDataPlaneOptions	No	Synapse configuration options.

SynapseDataPlaneOptions

Property	Default	Type	Required	Description
spark_session_config		LivySparkSessionConfig	No	Spark session configuration.

LivySparkSessionConfig

Property	Type	Required	Description
pool	string	No	The pool to use for the Spark session.
driver_memory	string	No	The memory to use for the Spark driver.
driver_cores	integer	No	The number of cores to use for the Spark driver.
executor_memory	string	No	The memory to use for the Spark executor.
executor_cores	integer	No	The number of cores to use for each executor.
num_executors	integer	No	The number of executors to use for the Spark session.
session_key_override	string	No	The key to use for the Spark session.

UniqueTest

Test to check if a value is unique.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
unique		NoTestOptions	No	Test to check if a value is unique.

NoTestOptions

Configuration options for tests that have no test body definition (not_null, unique, etc.).

No properties defined.

ResourceMetadata

Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.

Property	Default	Type	Required	Description
source		ResourceLocation	No	The origin or source information for the resource.
source_event_uuid		string	No	UUID of the event that is associated with creation of this resource.

ResourceLocation

The origin or source information for the resource.

Property	Default	Type	Required	Description
path		string	Yes	Path within repository files where the resource is defined.
first_line_number		integer	No	First line number within `path` file where the resource is defined.

CustomPythonReadComponent​

Property Details​

Component​

CustomPythonReadOptions​

BigQueryDataPlane​

BigQueryDataPlaneOptions​

BigQueryRangePartitioning​

BigQueryTimePartitioning​

ComponentTestColumn​

ColumnTestPython​

ColumnTestPythonOptions​

ColumnTestSql​

CombinationUniqueTest​

CombinationUniqueTestOptions​

ComponentSchemaTest​

CountDistinctEqualTest​

CountDistinctEqualTestOptions​

CountEqualTest​

CountEqualTestOptions​

CountGreaterThanOrEqualTest​

CountGreaterThanOrEqualTestOptions​

CountGreaterThanTest​

CountGreaterThanTestOptions​

CountLessThanOrEqualTest​

CountLessThanOrEqualTestOptions​

CountLessThanTest​

CountLessThanTestOptions​

DataMaintenance​

DateInRangeTest​

DateInRangeTestOptions​

DuckdbDataPlane​

DuckDbDataPlaneOptions​

GreaterThanOrEqualTest​

GreaterThanOrEqualTestOptions​

GreaterThanTest​

GreaterThanTestOptions​

InRangeTest​

InRangeTestOptions​

InSetTest​

InSetTestOptions​

LessThanOrEqualTest​

LessThanOrEqualTestOptions​

LessThanTest​

LessThanTestOptions​

MeanInRangeTest​

MeanInRangeTestOptions​

NotEmptyTest​

NotNullTest​

RangeOptions​

SnowflakeDataPlane​

SnowflakeDataPlaneOptions​

PartitionMaterialization​

PartitionedOptions​

StddevInRangeTest​

StddevInRangeTestOptions​

SubstringMatchTest​

SubstringMatchTestOptions​

SynapseDataPlane​

SynapseDataPlaneOptions​

LivySparkSessionConfig​

UniqueTest​

NoTestOptions​

ResourceMetadata​

ResourceLocation​