Write to AWS S3

This guide shows you how to create an AWS S3 Write Component.

Prerequisites

• S3 Connection with write permissions
• An Ascend Flow with a Component that contains data

Create a new Write Component

Begin from your workspace Super Graph view. Follow these steps to create your Write Component:

Using the Component Form

1. Double-click the Flow where you want to create your component
2. Right-click on any Component
3. Hover over Create Downstream -> Write, and select your target Connection
4. Complete the form with these details:
   • Select your Flow
   • Enter a descriptive Component Name like write_mysql

Using the Files Panel

1. Open the Files panel in the top left corner
2. Navigate to and select your desired Flow
3. Right-click on the components directory and choose New file
4. Name your file with a descriptive name like write_mysql.yaml and press enter

Configure your S3 Write Component

Follow these steps to set up your S3 Write Component:

1. Configure your Connection
   • Enter your S3 Connection name in the connection field
2. Specify a data source
   • Set input to the Component that contains your source data
3. Define the write target
   • Configure the s3 write connector options
   • Specify your target table name, schema, and other required properties
4. Select a write strategy

   Choose one of these strategies based on your use case:

   Strategy	Description	Best For
   full	Replaces entire target during each Flow Run	Reference tables, complete data refreshes
   partitioned	Updates only modified partitions	Time-series data, regional datasets, date-partitioned data

5. Set strategy-specific options
   • For full: use drop_and_recreate mode
   • For partitioned: use append, insert_overwrite, or sync mode

See the Write Components reference guide for all available options.

Examples

Choose the write strategy that best fits your use case:

Full write strategy

This example shows an S3 Write Component that uses a full write strategy. This strategy replaces all data in the target location during each Flow Run, ensuring a complete refresh.

write_s3_full.yaml

component:
  write:
    connection: write_s3
    input:
      name: my_component
      flow: my_flow
    strategy:
      full:
        mode: drop_and_recreate
    s3:
      path: /some_other_dir
      formatter: parquet

Partitioned write strategy

This example shows an S3 Write Component that uses a partitioned write strategy. This strategy only updates modified partitions while preserving existing partitions, which is more efficient for large datasets.

write_s3_partitioned.yaml

component:
  write:
    connection: write_s3
    input:
      name: my_component
      flow: my_flow
    strategy:
      partitioned:
        mode: append
    s3:
      path: /some_parquet_dir
      formatter: parquet

File Naming Convention

By default, Ascend will generate file names for each partition (or for the whole dataset if not partitioned) using a deterministic template. The default template is typically:

component_name/partition_values/part-unique_identifier.formatter_extension

• component_name: The name of your write component.
• partition_values: If your input is partitioned, this will be a directory structure representing the partition keys and values (e.g., date=2024-06-01/country=US/).
• part-unique_identifier: A unique identifier for the file, often tied to the manifest (see below).
• formatter_extension: The file format extension, e.g., parquet.

Customizing with partition_template: If you specify a partition_template in your config, you can control the subdirectory and file naming pattern.

🎉 Congratulations! You successfully created an S3 Write Component in Ascend.