Test YAML

In this guide, you'll learn how to add data quality and validation tests to your YAML Components to ensure data integrity in your pipelines.

Prerequisites

• Ascend Flow

info

For a comprehensive overview of test types and when to use them, see our Tests concept guide.

Test behavior

Tests accept a severity parameter that can be set to error or warn.

error is the default severity, meaning that failed tests cause the entire Component to fail. To log warnings instead of failing, set severity: warn:

columns:
  id:
    - not_null:
        severity: warn

Test format

YAML Components support both column-level and Component-level native tests through a dedicated tests section. These tests validate data quality automatically during pipeline execution.

Add native tests to YAML Components by specifying a tests section with component and columns subsections:

component:
  # ... component configuration ...
  tests:
    component:
      - test_name:
          parameter: value
    columns:
      column_name:
        - test_name
        - test_name_with_params:
            parameter: value

Component-level tests

Component-level tests validate the entire output of your Component. Common use cases include verifying record counts and schema validation.

Verify that your Component produces exactly the expected number of records:

component:
  read:
    connection: my_connection
    # ... read configuration ...
  tests:
    component:
      - count_equal:
          count: 1000

Column-level tests

Column-level tests validate individual columns in your dataset. Specify tests under the column name in the columns section.

Basic validation tests

Check for null values and empty strings:

component:
  read:
    connection: my_connection
    # ... read configuration ...
  tests:
    columns:
      user_id:
        - not_null
        - not_empty
      email:
        - not_null
        - unique

Value range tests

Validate that numeric columns fall within expected ranges:

component:
  read:
    connection: my_connection
    # ... read configuration ...
  tests:
    columns:
      age:
        - greater_than:
            value: 0
        - less_than:
            value: 120

Complete example

Here's a comprehensive example of a GCS Read Component with both Component-level and column-level tests:

read_bigquery_with_tests.yaml

component:
  read:
    connection: read_gcs_lake
    gcs:
      path: ottos-expeditions/lakev0/generated/events/sales_store.parquet/year=
      include:
        - glob: "*/month=*/day=*/*.parquet"
  tests:
    component:
      - count_equal:
          count: 8
    columns:
      _ascend_partition_uuid:
        - not_null
        - not_empty
        - count_distinct_equal:
            count: 2
      id:
        - unique
        - not_null
      price:
        - greater_than:
            value: 0
        - less_than:
            value: 1000000
      timestamp:
        - not_null
        - greater_than:
            value: "'2023-08-25 00:00:00'"
        - less_than:
            value: "'2023-08-27 00:00:00'"
      ascender_id:
        - not_null
        - not_empty

This example demonstrates:

• Component test: Verifies exactly 8 records are returned
• Partition validation: Ensures partition UUIDs are present and exactly 2 distinct values exist
• Data integrity: Validates transaction IDs are unique and not null
• Business rules: Ensures transaction amounts are positive and reasonable
• Time bounds: Confirms timestamps fall within the expected date range
• Contact information: Validates customer emails are present and not empty

Next steps

• Learn about SQL tests
• Explore Python tests
• Review the Tests concept guide