Schema change strategies

Ascend's schema change strategies allow your pipelines to adapt automatically, reducing maintenance overhead and ensuring continuous data flow.

Why is schema change handling important?

Data schemas evolve over time as business requirements change. Without proper schema change handling, your data pipelines can break when:

• New columns are added to source tables
• Column data types change
• Columns are renamed or removed
• Table structures are reorganized

Full refresh

Run Components and Flows in full-refresh mode to resolve schema mismatch errors and completely refresh the Data Plane schema for that Component. Full-refresh rebuilds the entire dataset from scratch, ensuring the schema is up-to-date.

For targeted schema updates, you can perform a full-refresh on individual Components or a selected subset of Components by using the Component context menu.

Schema change strategies

Ascend Components provide several options for handling schema changes through the on_schema_change parameter in Read and Transform Components.

sync_all_columns

The sync_all_columns strategy synchronizes the target table's columns to exactly match the source schema. When using this strategy:

• Columns present in the source but missing in the target are added
• Columns in the target but not in the source are removed
• Data types of existing columns will be updated to match the source
• Original column order is preserved for existing columns, with new columns added at the end

This strategy ensures the target always fully reflects the current structure of the source, potentially at the cost of dropping columns that are no longer needed.

append_new_columns

The append_new_columns strategy adds new columns from the source to the target table without modifying existing columns. When using this strategy:

• New columns detected in the source data are added to the target
• Existing columns in the target remain unchanged, even if they no longer exist in the source
• Data type changes to existing columns are not propagated

This approach preserves all current data and structure, simply extending the schema to accommodate new fields as they appear over time.

ignore

The ignore strategy maintains the existing target schema without any modifications:

• No columns are added, removed, or modified regardless of source schema changes
• The system will attempt to write data using the existing schema
• If incompatible data is encountered (e.g., missing required columns), an error will occur
• This is useful when you want to strictly control schema through other means

fail

The fail strategy provides the strictest approach to schema management:

• Any schema mismatch between source and target immediately raises an error
• No automatic schema modifications are permitted
• Forces manual intervention for any schema changes

Summary

Strategy	When to Use	Description
sync_all_columns	Ensure target tables always reflect current source structure	Synchronizes target schema to exactly match source, adding new columns and removing obsolete ones
append_new_columns	Preserve historical columns while adding new ones	Adds new columns from source but keeps all existing target columns unchanged
ignore	Maintain existing schema without modifications	No schema changes are made; uses existing target schema regardless of source changes
fail	Need strict control over schema changes	Raises an error on any schema mismatch, forcing manual intervention
full-refresh	Making significant schema changes to ensure data consistency	Rebuilds entire dataset from scratch with updated schema

Next steps

Follow our guide to implement schema change strategies in your Ascend pipelines and maintain robust, resilient Flows.