Write to Azure Blob Storage

This guide shows you how to create an Azure Blob Storage Write Component.

Prerequisites

• Azure Blob Storage Connection with write permissions
• An Ascend Flow with a Component that contains data

Create a new Write Component

Begin from your workspace Super Graph view. Follow these steps to create your Write Component:

Using the Component Form

1. Double-click the Flow where you want to create your component
2. Right-click on any Component
3. Hover over Create Downstream -> Write, and select your target Connection
4. Complete the form with these details:
   • Select your Flow
   • Enter a descriptive Component Name like write_mysql

Using the Files Panel

1. Open the Files panel in the top left corner
2. Navigate to and select your desired Flow
3. Right-click on the components directory and choose New file
4. Name your file with a descriptive name like write_mysql.yaml and press enter

Configure your Azure Blob Storage Write Component

Follow these steps to set up your Azure Blob Storage Write Component:

1. Configure your Connection
   • Enter your Azure Blob Storage Connection name in the connection field
2. Specify a data source
   • Set input to the Component that contains your source data
3. Define the write target
   • Configure the azure blob storage write connector options
   • Specify your target table name, schema, and other required properties
4. Select a write strategy

   Choose one of these strategies based on your use case:

   Strategy	Description	Best For
   full	Replaces entire target during each Flow Run	Reference tables, complete data refreshes
   partitioned	Updates only modified partitions	Time-series data, regional datasets, date-partitioned data

5. Set strategy-specific options
   • For full: use drop_and_recreate mode
   • For partitioned: use append, insert_overwrite, or sync mode

See the Write Components reference guide for all available options.

Example

write_abfss.yaml

component:
  write:
    connection: write_abfss
    input:
      name: my_component
      flow: my_flow
    abfs:
      path: /some-other-dir
      formatter: parquet

This configuration:

• References an Azure Blob Storage connection named write_abfss
• Uses my_component from my_flow as the input data
• Applies a parquet format
• Writes to Azure Blob Storage in /some-other-dir

File Naming Convention

By default, Ascend will generate file names for each partition (or for the whole dataset if not partitioned) using a deterministic template. The default template is typically:

{component_name}/{partition_values}/part-{unique_identifier}.{formatter_extension}

• component_name: The name of your write component.
• partition_values: If your input is partitioned, this will be a directory structure representing the partition keys and values (e.g., date=2024-06-01/country=US/).
• part-unique_identifier: A unique identifier for the file, often tied to the manifest (see below).
• formatter_extension: The file format extension, e.g., parquet.

Customizing with partition_template: If you specify a partition_template in your config, you can control the subdirectory and file naming pattern.

🎉 Congratulations! You successfully created an Azure Blob Storage Write Component in Ascend.