Create an S3 Read Component

This guide walks you through creating a Read Component that ingests data from S3.

Prerequisites

• Ascend Flow
• S3 Connection

Create a new Component

Begin from your workspace Super Graph view. Follow these steps to create your component:

Using the Component Form

1. Double-click the Flow where you want to create your component
2. Right-click anywhere in the Flow Graph
3. Hover over Create Component, then over Read in the expanded menu, and click From Scratch
4. Complete the form with these details:
   • Select your Flow
   • Enter a descriptive Component Name like read_sales
   • Select YAML as your file type

Using the Files Panel

1. Open the files panel in the top left corner
2. Navigate to and select your desired Flow
3. Right-click on the components directory and choose New file
4. Name your file with a descriptive name like read_sales.yaml and press enter

Create your S3 Read Component

Structure your S3 Read Component following this pattern:

1. Reference your S3 connection: Specify which S3 connection to read from
2. Add the s3 key: Configure the specific S3 bucket settings
   • path: Specify the path within the bucket to read from
   • include: Define file filtering patterns to include specific files
3. Add parser settings: Specify how to parse the files (auto-detection or specific format)
4. Add load strategy options: Configure how data is loaded from S3

Example

read_s3.yaml

component:
  read:
    connection: read_s3
    s3:
      path: listing/binary/
      include:
        - suffix: csv
        - created_at:
            after: 2024-01-01
      parser: auto
      load_strategy:
        max_size: 1GB

This example shows how to:

• Read from a specific directory (listing/binary/) within your S3 bucket
• Include only CSV files created after January 1, 2024
• Use automatic format detection for parsing
• Set a maximum load size of 1GB to optimize performance

For a complete list of configuration options and advanced settings, see this reference guide.

🎉 Congratulations! You successfully created an S3 Read Component in Ascend.