Skip to main content

Write to S3

This guide shows you how to create an AWS S3 Write Component.

Prerequisites​

  • S3 Connection with write permissions
  • An Ascend Flow with a Component that contains data

Create a new Write Component​

From your workspace Super Graph view, follow these steps to create your Write Component:

  1. Double-click the Flow where you want to create your Component
  2. Right-click on any Component
  3. Hover over Create Downstream -> Write, and select your target Connection menu
  4. Complete the form with these details:
    • Select your Flow
    • Enter a descriptive Component Name like write_mysql form

Configure your S3 Write Component

Follow these steps to configure your S3 Write Component:

  1. Set up your Connection
    • Enter your S3 Connection name in the connection field
  2. Define your data source
    • Set input to the Component that contains your source data
  3. Configure the write destination
    • Set up the s3 write connector options
    • Specify your target table name, schema, and other required properties
  4. Choose a write strategy

    Select the strategy that best fits your use case:

    StrategyDescriptionBest for
    full (default)Replaces the entire target table during each Flow RunReference tables, complete data refreshes
    partitionedUpdates only the partitions that have changedTime-series data, regional datasets, date-partitioned tables
    snapshotCreates flexible output as a single file or multiple chunksData exports, analytical datasets, flexible output formats

For detailed guidance on when to use each strategy, see the write strategies guide.

Examples​

Choose the write strategy that best fits your use case:

note

Full write is the default strategy used when no strategy is explicitly specified.

This example shows an S3 Write Component using a full write strategy that outputs data in chunks for optimal performance with large datasets.

full_s3.yaml
component:
write:
connection: write_s3
input:
name: my_component
flow: my_flow
strategy:
full:
mode: drop_and_recreate
s3:
path: /some_other_dir/my_data.parquet
formatter: parquet

Output: Multiple files like part_001.parquet, part_002.parquet, etc. in the specified directory.

🎉 Congratulations! You successfully created an S3 Write Component in Ascend.