Tests

Ascend offers two approaches to validation: tests applied within Components (like Read Components and Transforms) and standalone Test Components dedicated solely to validation. Both ensure your data meets quality and integrity criteria, from simple column checks to complex cross-dataset validations.

Key features

1. Versatility: Apply tests at both column and Component levels to enforce data quality checks tailored to your specific needs. Create reusable tests that work across multiple Components.
2. Customization: Build custom tests in SQL and Python to define validation logic beyond the native options.
3. Inline validation: Define tests directly within data processing Components for immediate, context-aware validation.
4. Automation: Run tests automatically during data processing for continuous quality assurance without manual intervention.

Test types

info

See the Test Reference for more information on the syntax and capabilities of all test types.

Test Type	Description
Column-level test	Pre-defined tests that validate individual columns in your data. These tests check properties like non-null values, uniqueness, and value ranges. Add them directly to Component decorators or as Jinja blocks in SQL.
Component-level test	Pre-defined tests that validate the entire output of a Component rather than specific columns. These tests check table counts, schema validation, and other Component-level properties. Add them directly to Component decorators or as Jinja blocks in SQL.
Test Component	Dedicated Components whose sole purpose is testing. These Components contain custom logic in Python or SQL and perform complex validations that aren't possible with native tests.
Singular test	A kind of Test Component decorated with @singular_test that run custom validation logic and return a TestResult object. Use these for specialized test cases that require custom processing.

Test Components (the last two rows in the table) are standalone entities dedicated specifically to validation tasks, while column-level and Component-level tests are added to existing data processing Components.

Best practices for tests

• Use clear naming: Name Test Components with a test_ prefix. For example, a Component that validates customer IDs should be named test_customer_ids.
• Test comprehensively: Implement various tests to cover all critical aspects of data quality and integrity. Combine native tests for common validations with custom tests for specific requirements.
• Review and adjust regularly: Update tests as data schemas and business requirements evolve to maintain their relevance and effectiveness.
• Consider performance: Optimize custom tests to minimize their impact on data processing times while maintaining thorough validation.

Conclusion

Tests are essential to Ascend's data engineering platform, ensuring that your data pipelines produce reliable, high-quality data. By using both native and custom tests, you can:

• Enforce strict data quality standards
• Automate validation processes
• Reduce the risk of data errors

Effectively implementing tests is crucial for maintaining data integrity and accuracy within your Ascend workflows.

Next steps

Apply what you've learned by following our data tests tutorial, which walks through implementing each test type with practical examples.