Partitioned Write Strategy with Schema Change

Container for specifying the partitioned write strategy that supports different behaviors when schema changes.

PartitionedWriteStrategyWithSchemaChange

info

PartitionedWriteStrategyWithSchemaChange is defined beneath the following ancestor nodes in the YAML structure:

Component
WriteComponent
BigQueryWriteComponent
MySQLWriteComponent
OracleWriteComponent
PostgresWriteComponent
SnowflakeWriteComponent

Below are the properties for the PartitionedWriteStrategyWithSchemaChange. Each property links to the specific details section further down in this page.

Property	Default	Type	Required	Description
partitioned			Yes	Options to use when writing partitioned data to a Write Component.
on_schema_change		string ("ignore", "fail", "append_new_columns", "sync_all_columns")	No	Policy to apply when schema changes are detected. Defaults to 'fail' if not provided.

Property Details

Component

A Component is a fundamental building block of a data Flow. Supported Component types include: Read, Transform, Task, Test, and more.

Property	Default	Type	Required	Description
component		One of: CustomPythonReadComponent ApplicationComponent AliasedTableComponent ExternalTableComponent FivetranComponent	Yes	Component configuration options.

WriteComponent

Property	Type	Required	Description
skip	boolean	No	Boolean flag indicating whether to skip processing for the Component or not.
retry_strategy		No	Retry strategy configuration options for the Component if any exceptions are encountered.
description	string	No	Brief description of what the model does.
metadata		No	Meta information of a resource. In most cases it doesn't affect the system behavior but may be helpful to analyze project resources.
name	string	Yes	The name of the model
flow_name	string	No	Name of the Flow that the Component belongs to.
write	One of: BigQueryWriteComponent SnowflakeWriteComponent S3WriteComponent SFTPWriteComponent GcsWriteComponent AbfsWriteComponent MySQLWriteComponent OracleWriteComponent PostgresWriteComponent	Yes

BigQueryWriteComponent

Component that writes data to a BigQuery table.

Property	Default	Type	Required	Description
dependencies		array[None]	No	List of dependencies that must complete before this Component runs.
connection		string	Yes	Name of the Connection to use for writing data.
input			Yes	Input component name.
normalize		boolean	No	Boolean flag indicating if the output column names should be normalized to a standard naming convention when writing.
preserve_case		boolean	No	Boolean flag indicating if the case of the column names should be preserved when writing.
uppercase		boolean	No	Boolean flag indicating if the column names should be transformed to uppercase when writing.
strategy	full: mode: drop_and_recreate	Any of: snapshot FullWriteStrategy IncrementalWriteStrategyWithSchemaChange PartitionedWriteStrategyWithSchemaChange	No	Resource for write strategy.
pre_sql		Any of: string array[string]	No	SQL statements to execute before the main write operation. Can be a single SQL statement string or multiple statements as a list of strings.
post_sql		Any of: string array[string]	No	SQL statements to execute after the main write operation. Can be a single SQL statement string or multiple statements as a list of strings.
bigquery			Yes

MySQLWriteComponent

Component that writes data to a MySQL table

Property	Default	Type	Required	Description
dependencies		array[None]	No	List of dependencies that must complete before this Component runs.
connection		string	Yes	Name of the Connection to use for writing data.
input			Yes	Input component name.
normalize		boolean	No	Boolean flag indicating if the output column names should be normalized to a standard naming convention when writing.
preserve_case		boolean	No	Boolean flag indicating if the case of the column names should be preserved when writing.
uppercase		boolean	No	Boolean flag indicating if the column names should be transformed to uppercase when writing.
strategy	full: mode: drop_and_recreate	Any of: snapshot FullWriteStrategy IncrementalWriteStrategyWithSchemaChange PartitionedWriteStrategyWithSchemaChange	No	Resource for write strategy.
pre_sql		Any of: string array[string]	No	SQL statements to execute before the main write operation. Can be a single SQL statement string or multiple statements as a list of strings.
post_sql		Any of: string array[string]	No	SQL statements to execute after the main write operation. Can be a single SQL statement string or multiple statements as a list of strings.
mysql			Yes

OracleWriteComponent

Component that writes data to an Oracle table

Property	Default	Type	Required	Description
dependencies		array[None]	No	List of dependencies that must complete before this Component runs.
connection		string	Yes	Name of the Connection to use for writing data.
input			Yes	Input component name.
normalize		boolean	No	Boolean flag indicating if the output column names should be normalized to a standard naming convention when writing.
preserve_case		boolean	No	Boolean flag indicating if the case of the column names should be preserved when writing.
uppercase		boolean	No	Boolean flag indicating if the column names should be transformed to uppercase when writing.
strategy	full: mode: drop_and_recreate	Any of: snapshot FullWriteStrategy IncrementalWriteStrategyWithSchemaChange PartitionedWriteStrategyWithSchemaChange	No	Resource for write strategy.
pre_sql		Any of: string array[string]	No	SQL statements to execute before the main write operation. Can be a single SQL statement string or multiple statements as a list of strings.
post_sql		Any of: string array[string]	No	SQL statements to execute after the main write operation. Can be a single SQL statement string or multiple statements as a list of strings.
oracle			Yes

PostgresWriteComponent

Component that writes data to a Postgres table

Property	Default	Type	Required	Description
dependencies		array[None]	No	List of dependencies that must complete before this Component runs.
connection		string	Yes	Name of the Connection to use for writing data.
input			Yes	Input component name.
normalize		boolean	No	Boolean flag indicating if the output column names should be normalized to a standard naming convention when writing.
preserve_case		boolean	No	Boolean flag indicating if the case of the column names should be preserved when writing.
uppercase		boolean	No	Boolean flag indicating if the column names should be transformed to uppercase when writing.
strategy	full: mode: drop_and_recreate	Any of: snapshot FullWriteStrategy IncrementalWriteStrategyWithSchemaChange PartitionedWriteStrategyWithSchemaChange	No	Resource for write strategy.
pre_sql		Any of: string array[string]	No	SQL statements to execute before the main write operation. Can be a single SQL statement string or multiple statements as a list of strings.
post_sql		Any of: string array[string]	No	SQL statements to execute after the main write operation. Can be a single SQL statement string or multiple statements as a list of strings.
postgres			Yes

SnowflakeWriteComponent

Component that writes data to a Snowflake table.

Property	Default	Type	Required	Description
dependencies		array[None]	No	List of dependencies that must complete before this Component runs.
connection		string	Yes	Name of the Connection to use for writing data.
input			Yes	Input component name.
normalize		boolean	No	Boolean flag indicating if the output column names should be normalized to a standard naming convention when writing.
preserve_case		boolean	No	Boolean flag indicating if the case of the column names should be preserved when writing.
uppercase		boolean	No	Boolean flag indicating if the column names should be transformed to uppercase when writing.
strategy	full: mode: drop_and_recreate	Any of: snapshot FullWriteStrategy IncrementalWriteStrategyWithSchemaChange PartitionedWriteStrategyWithSchemaChange	No	Resource for write strategy.
pre_sql		Any of: string array[string]	No	SQL statements to execute before the main write operation. Can be a single SQL statement string or multiple statements as a list of strings.
post_sql		Any of: string array[string]	No	SQL statements to execute after the main write operation. Can be a single SQL statement string or multiple statements as a list of strings.
snowflake			Yes

PartitionedWriteStrategyOptions

Resource options for incremental writes, including mode selection and criteria for detecting deletions and unique records.

Property	Default	Type	Required	Description
mode			Yes	Specifies the mode to use when writing data in partitions: 'append' to append new or modified partitions, 'insert_overwrite' to insert new partitions and replace/overwrite modified partitions, and 'sync' to encompass both 'insert_overwrite' functionality and to delete partitions when deleted at the source.
partition_col		string	No	Column name used for partitioning. Uses the internal Ascend partition identifier by default.

PartitionedWriteModeEnum

No properties defined.