Python Test
Test to validate data using a Python function for a single column.
ColumnTestPython
info
ColumnTestPython is defined beneath the following ancestor nodes in the YAML structure:
Below are the properties for the ColumnTestPython. Each property links to the specific details section further down in this page.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| severity | error | string ("error", "warn") | No | The severity level for issues raised by the test. Default is 'error'. Use 'error' for critical issues that should interrupt flow processing. Use 'warn' for warnings/minor issues that should not interrupt flow processing. |
| name | string | Yes | ||
| python | ColumnTestPythonOptions | Yes | Configuration options for the Python column test. |
Property Details
Component
A component is a fundamental building block of a data flow. Types of components that are supported include: read, transform, task, test, and more.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| component | One of: ReadComponent TransformComponent TaskComponent SingularTestComponent CustomPythonReadComponent WriteComponent CompoundComponent AliasedTableComponent ExternalTableComponent | Yes | Configuration options for the component. |
AliasedTableComponent
A component that reads makes the data in a pre-existing table available in the Ascend Flow.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| name | string | No | The name of the model | |
| description | string | No | A brief description of what the model does. | |
| metadata | ResourceMetadata | No | Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources. | |
| flow_name | string | No | The name of the flow that the component belongs to. | |
| skip | boolean | No | A boolean flag indicating whether to skip processing for the component or not. | |
| data_maintenance | DataMaintenance | No | The data maintenance configuration options for the component. | |
| tests | ComponentTestColumn | No | Defines tests to run on the data of this component. | |
| alias | AliasedTableOptions | Yes | Configuration options for the aliased table component. |
CustomPythonReadComponent
A component that reads data using user-defined, custom Python code.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| data_plane | One of: SnowflakeDataPlane BigQueryDataPlane DuckdbDataPlane SynapseDataPlane | No | Data Plane-specific configuration options for a component. | |
| name | string | No | The name of the model | |
| description | string | No | A brief description of what the model does. | |
| metadata | ResourceMetadata | No | Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources. | |
| flow_name | string | No | The name of the flow that the component belongs to. | |
| skip | boolean | No | A boolean flag indicating whether to skip processing for the component or not. | |
| data_maintenance | DataMaintenance | No | The data maintenance configuration options for the component. | |
| tests | ComponentTestColumn | No | Defines tests to run on the data of this component. | |
| custom_python_read | CustomPythonReadOptions | Yes |
ExternalTableComponent
A component that constructs and updates an External Table. Currently supported for Snowflake only.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| name | string | No | The name of the model | |
| description | string | No | A brief description of what the model does. | |
| metadata | ResourceMetadata | No | Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources. | |
| flow_name | string | No | The name of the flow that the component belongs to. | |
| skip | boolean | No | A boolean flag indicating whether to skip processing for the component or not. | |
| data_maintenance | DataMaintenance | No | The data maintenance configuration options for the component. | |
| tests | ComponentTestColumn | No | Defines tests to run on the data of this component. | |
| external_table | Any of: SnowflakeExternalTableOptions BigQueryExternalTableOptions | Yes | Configuration options for the External Table component. |
ReadComponent
A component that reads data from a data system.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| data_plane | One of: SnowflakeDataPlane BigQueryDataPlane DuckdbDataPlane SynapseDataPlane | No | Data Plane-specific configuration options for a component. | |
| name | string | No | The name of the model | |
| description | string | No | A brief description of what the model does. | |
| metadata | ResourceMetadata | No | Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources. | |
| flow_name | string | No | The name of the flow that the component belongs to. | |
| skip | boolean | No | A boolean flag indicating whether to skip processing for the component or not. | |
| data_maintenance | DataMaintenance | No | The data maintenance configuration options for the component. | |
| tests | ComponentTestColumn | No | Defines tests to run on the data of this component. | |
| read | One of: GenericFileReadComponent LocalFileReadComponent S3ReadComponent GcsReadComponent AbfsReadComponent HttpReadComponent MSSQLReadComponent MySQLReadComponent OracleReadComponent PostgresReadComponent SnowflakeReadComponent BigQueryReadComponent | Yes | The read component that reads data from a data system. |
TaskComponent
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| name | string | No | The name of the model | |
| description | string | No | A brief description of what the model does. | |
| metadata | ResourceMetadata | No | Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources. | |
| flow_name | string | No | The name of the flow that the component belongs to. | |
| skip | boolean | No | A boolean flag indicating whether to skip processing for the component or not. | |
| data_maintenance | DataMaintenance | No | The data maintenance configuration options for the component. | |
| tests | ComponentTestColumn | No | Defines tests to run on the data of this component. | |
| task | One of: TaskSqlComponent TaskPythonComponent | Yes |
TransformComponent
A component that executes SQL or Python code to transform data.
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| data_plane | One of: SnowflakeDataPlane BigQueryDataPlane DuckdbDataPlane SynapseDataPlane | No | Data Plane-specific configuration options for a component. | |
| name | string | No | The name of the model | |
| description | string | No | A brief description of what the model does. | |
| metadata | ResourceMetadata | No | Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources. | |
| flow_name | string | No | The name of the flow that the component belongs to. | |
| skip | boolean | No | A boolean flag indicating whether to skip processing for the component or not. | |
| data_maintenance | DataMaintenance | No | The data maintenance configuration options for the component. | |
| tests | ComponentTestColumn | No | Defines tests to run on the data of this component. | |
| transform | One of: SqlTransform PythonTransform SnowparkTransform | Yes | The transform component that executes SQL or Python code to transform data. |
ComponentTestColumn
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| columns | object | No | List of tests to run on columns the data after processing for validation purposes. Used in the context of a component. | |
| component | array[One of: (CombinationUniqueTest, InRangeTest, DateInRangeTest, InSetTest, SubstringMatchTest, CountDistinctEqualTest, CountGreaterThanOrEqualTest, CountGreaterThanTest, CountLessThanOrEqualTest, CountLessThanTest, CountEqualTest, GreaterThanTest, LessThanTest, GreaterThanOrEqualTest, LessThanOrEqualTest, MeanInRangeTest, StddevInRangeTest, ColumnTestSql, ColumnTestPython)] | No | List of component level tests. | |
| schema | ComponentSchemaTest | No | List of the component's schema level tests. |
ColumnTestPythonOptions
| Property | Default | Type | Required | Description |
|---|---|---|---|---|
| entrypoint | string | Yes | The entrypoint for the python transform function. | |
| source | string | Yes | The source file for the python transform function. | |
| params | object | No | Parameters for the Python test function. | |
| is_asset_test | boolean | No |