Schema Test

Test to validate that component columns match expected types.

ComponentSchemaTest

info

ComponentSchemaTest is defined beneath the following ancestor nodes in the YAML structure:

• Component
• AliasedTableComponent
• CustomPythonReadComponent
• ExternalTableComponent
• ReadComponent
• TaskComponent
• TransformComponent
• ComponentTestOptions

Below are the properties for the ComponentSchemaTest. Each property links to the specific details section further down in this page.

Property	Default	Type	Required	Description
severity	error	string ("error", "warn")	No	The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing.
match	exact	string ("exact", "ignore_missing")	No	The type of schema matching to perform. 'exact' requires all columns to be present, 'ignore_missing' allows for missing columns.
columns		object with property values of type string	No	A mapping of column names to their expected types.

Property Details

Component

A Component is a fundamental building block of a data Flow. Supported Component types include: Read, Transform, Task, Test, and more.

Property	Default	Type	Required	Description
component		One of:   CustomPythonReadComponent   ApplicationComponent   AliasedTableComponent   ExternalTableComponent	Yes	Component configuration options.

AliasedTableComponent

Component that makes data in a pre-existing table available in the Ascend Flow.

Property	Default	Type	Required	Description
skip		boolean	No	Boolean flag indicating whether to skip processing for the Component or not.
retry_strategy			No	Retry strategy configuration options for the Component if any exceptions are encountered.
description		string	No	A brief description of what the model does.
metadata			No	Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.
name		string	Yes	The name of the model
flow_name		string	No	Name of the Flow that the Component belongs to.
data_maintenance			No	The data maintenance configuration options for the Component.
tests			No	Defines tests to run on this Component's data.
alias			Yes	Configuration options for the aliased table component.

CustomPythonReadComponent

Component that reads data using user-defined custom Python code.

Property	Default	Type	Required	Description
data_plane		One of:     SnowflakeDataPlane     BigQueryDataPlane     DuckdbDataPlane     DatabricksDataPlane	No	Data Plane-specific configuration options for a component.
skip		boolean	No	Boolean flag indicating whether to skip processing for the Component or not.
retry_strategy			No	Retry strategy configuration options for the Component if any exceptions are encountered.
description		string	No	A brief description of what the model does.
metadata			No	Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.
name		string	Yes	The name of the model
flow_name		string	No	Name of the Flow that the Component belongs to.
data_maintenance			No	The data maintenance configuration options for the Component.
tests			No	Defines tests to run on this Component's data.
custom_python_read			Yes

ExternalTableComponent

Component that constructs and updates an External Table. Currently supported for Snowflake only.

Property	Default	Type	Required	Description
skip		boolean	No	Boolean flag indicating whether to skip processing for the Component or not.
retry_strategy			No	Retry strategy configuration options for the Component if any exceptions are encountered.
description		string	No	A brief description of what the model does.
metadata			No	Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.
name		string	Yes	The name of the model
flow_name		string	No	Name of the Flow that the Component belongs to.
data_maintenance			No	The data maintenance configuration options for the Component.
tests			No	Defines tests to run on this Component's data.
external_table		Any of:	Yes	Configuration options for the External Table component.

ReadComponent

Component that reads data from a system.

Property	Default	Type	Required	Description
data_plane		One of:     SnowflakeDataPlane     BigQueryDataPlane     DuckdbDataPlane     DatabricksDataPlane	No	Data Plane-specific configuration options for a component.
skip		boolean	No	Boolean flag indicating whether to skip processing for the Component or not.
retry_strategy			No	Retry strategy configuration options for the Component if any exceptions are encountered.
description		string	No	A brief description of what the model does.
metadata			No	Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.
name		string	Yes	The name of the model
flow_name		string	No	Name of the Flow that the Component belongs to.
data_maintenance			No	The data maintenance configuration options for the Component.
tests			No	Defines tests to run on this Component's data.
read		One of:   GenericFileReadComponent   LocalFileReadComponent   SFTPReadComponent   S3ReadComponent   GcsReadComponent   AbfsReadComponent   HttpReadComponent   MSSQLReadComponent   MySQLReadComponent   OracleReadComponent   PostgresReadComponent   SnowflakeReadComponent   BigQueryReadComponent   DatabricksReadComponent	Yes	Read component that reads data from a system.

TaskComponent

Property	Default	Type	Required	Description
skip		boolean	No	Boolean flag indicating whether to skip processing for the Component or not.
retry_strategy			No	Retry strategy configuration options for the Component if any exceptions are encountered.
description		string	No	A brief description of what the model does.
metadata			No	Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.
name		string	Yes	The name of the model
flow_name		string	No	Name of the Flow that the Component belongs to.
data_maintenance			No	The data maintenance configuration options for the Component.
tests			No	Defines tests to run on this Component's data.
task		One of:   TaskSqlComponent   TaskPythonComponent	Yes

TransformComponent

Component that executes SQL or Python code to transform data.

Property	Default	Type	Required	Description
data_plane		One of:     SnowflakeDataPlane     BigQueryDataPlane     DuckdbDataPlane     DatabricksDataPlane	No	Data Plane-specific configuration options for a component.
skip		boolean	No	Boolean flag indicating whether to skip processing for the Component or not.
retry_strategy			No	Retry strategy configuration options for the Component if any exceptions are encountered.
description		string	No	A brief description of what the model does.
metadata			No	Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.
name		string	Yes	The name of the model
flow_name		string	No	Name of the Flow that the Component belongs to.
data_maintenance			No	The data maintenance configuration options for the Component.
tests			No	Defines tests to run on this Component's data.
transform		One of:   SqlTransform   PythonTransform   SnowparkTransform   PySparkTransform	Yes	The Transform Component that executes SQL or Python code to transform data.

ComponentTestOptions

Options for component tests, including data quality tests and schema checks.

Property	Default	Type	Required	Description
columns		object with property values of type array[One of: (Any of: (not_null, NotNullTest), Any of: (not_empty, NotEmptyTest), Any of: (unique, UniqueTest), CombinationUniqueTest, InRangeTest, DateInRangeTest, InSetTest, SubstringMatchTest, CountDistinctEqualTest, CountGreaterThanOrEqualTest, CountGreaterThanTest, CountLessThanOrEqualTest, CountLessThanTest, CountEqualTest, GreaterThanTest, LessThanTest, GreaterThanOrEqualTest, LessThanOrEqualTest, MeanInRangeTest, StddevInRangeTest, ColumnTestSql, ColumnTestPython)]	No	List of column-level data quality tests for a component.
component		array[One of: (Any of: (not_null, NotNullTest), Any of: (not_empty, NotEmptyTest), Any of: (unique, UniqueTest), CombinationUniqueTest, InRangeTest, DateInRangeTest, InSetTest, SubstringMatchTest, CountDistinctEqualTest, CountGreaterThanOrEqualTest, CountGreaterThanTest, CountLessThanOrEqualTest, CountLessThanTest, CountEqualTest, GreaterThanTest, LessThanTest, GreaterThanOrEqualTest, LessThanOrEqualTest, MeanInRangeTest, StddevInRangeTest, ColumnTestSql, ColumnTestPython)]	No	List of component-level tests.
schema			No	List of schema checks for a component.