Automate and orchestrate your Flows

In this tutorial, you'll learn how to use Ascend's Automation framework to trigger Flows in response to real-time system Events and Sensors. You'll create responsive, intelligent pipelines that run precisely when they're needed.

You can use the built-in Ascend-native Automations or build with the extensible automation framework to integrate your tools, define custom behaviors, and build reusable automations that fit your workflows.

What you'll learn

• How to create Automations using both the UI and file-based approaches
• How to configure different types of triggers for your Automations
• How to implement custom Python Sensors and Actions
• How to structure common Automation patterns

Prerequisites

• At least one Ascend Flow for scheduling, or multiple Flows for trigger-based automation
• Python knowledge (if building custom Sensors or Actions)

How Automations work

Before we start building, let's understand the key components of Automations:

• Sensors: Define when to trigger an Action (time-based or custom logic)
• Events: Listen for system events like Flow or Component run status changes
• Actions: Define what happens when a Sensor or Event is triggered (running Flows or custom logic)

Ascend offers two approaches to building Automations:

1. Ascend-native Automations: Built-in capabilities that require no coding and cover common use cases:

   • Time-based scheduling with cron expressions
   • Flow and Component run status monitoring
   • Triggering Flow runs based on events

2. Custom Automations: Custom Python-based logic for specialized needs:

   • Custom Sensors to detect complex conditions or integrate with external systems
   • Custom Actions to perform advanced operations or interact with third-party tools
   • Full flexibility to implement business-specific automation requirements

Combining these elements lets you create powerful workflows tailored to your use case.

Create your Automation

You can create Automations using either the configuration UI or the Files panel. Both approaches are covered below.

Configuration UI

Using the configuration UI

1. Scroll down in the Build panel until you reach the Automations section, and click the + to create a new Automation

2. The configuration UI will open in a dedicated tab

3. Add a descriptive name to your Automation

tip

Name your Automation before adding Sensors, Events, and Actions.

4. Apply any Sensors, Events, and Actions relevant to your use case. Click Save after each addition

If built-in Ascend Automations meet your needs, you're done with the creation step! The sections below explain how to extend your Automation with custom Events and Python code.

Add a custom Event filter

1. Click + Add next to Events
2. Select Custom for the Filter Preset
3. In the Custom field, enter your SQL filter expression

tip

The filter uses SQL syntax to query JSON event data. For example, to filter events where the Flow name is "metrics", use:

json_extract_string(event, '$.data.flow') = 'metrics'

This extracts the flow name from the event JSON and compares it to the specified value.

4. Under Event Types, enter a descriptive name for your custom Event and press Enter. Delete any native Event Types by clicking Remove next to their names
5. After saving your custom Event, navigate to the Code tab in the top right to confirm that your sql_filter is correctly specified in the Automation YAML

Add a custom Python Sensor

1. Click + Add next to Sensors
2. Select Python Function for the Sensor Type
3. Enter a descriptive function name and write your Python code directly in the editor

note

The name you give your Sensor or Action becomes the name of the Python function you're creating. Choose clear, descriptive names that reflect their purpose.

Add a custom Python Action

1. Click + Add next to Actions
2. Select Python Function for the Action Type
3. Enter a descriptive function name and write your Python code directly in the editor

Files Panel

Using the Files panel

1. Navigate to the Files panel
2. Right-click on the Automations folder and select New file
3. Give your Automation a descriptive name like downstreams-bigquery-test.yaml

If built-in Ascend Automations meet your needs, you're done with the creation step! The sections below explain how to extend your Automation with custom Python code.

Add custom Python code

To extend your Automation with custom Python:

1. Create a folder structure src > automations > [your_automation_name] (e.g., test_automation)
2. Within that folder, create separate files for your actions and sensors code
3. Reference these files in your Automation configuration (covered in the next section)
4. Your final file structure should look like this:

Configure your Automation

When structuring your Automation, include these key elements:

• Set enabled to true to run your Automation in Deployments, or false to disable it
• Define your triggers under the triggers field:
  • For Events: specify sql_filters and types
  • For Sensors: specify type, name, and a cron config with a schedule
• For Actions: define the type, name, config, and the target flow

Example Automation patterns

Below are common Automation patterns you can implement in your workflows.

Run a Flow on a schedule

This example shows a timer Sensor triggered hourly via a cron job. When triggered, the Action automatically runs the extract-load-bigquery Flow.

Configuration UI

YAML

extract_load_bigquery.yaml

automation:
  enabled: true
  name: extract-load-bigquery
  triggers:
    sensors:
      - type: timer
        name: cron-timer
        config:
          schedule:
            cron: '0 * * * *'
  actions:
    - type: run_flow
      name: run-extract-load-bigquery
      config:
        flow: extract-load-bigquery

Trigger downstream Flows automatically

This example demonstrates a SQL filter Event triggered when an upstream Flow transform-bigquery runs. When triggered, the Action automatically runs a downstream Flow metrics-bigquery.

Configuration UI

YAML

downstreams_bigquery.yaml

automation:
  enabled: true
  name: downstreams-bigquery
  triggers:
    events:
      - sql_filter: json_extract_string(event, '$.data.flow') = 'transform-bigquery'
        types:
          - FlowRunSuccess
  actions:
    - type: run_flow
      name: run-metrics-bigquery
      config:
        flow: metrics-bigquery

Implement fully custom Automations

This example shows a fully custom Automation that combines:

• A custom Python Sensor script
• A custom Python Action script
• A custom Event that filters for specific Flow runs using the SQL expression:

json_extract_string(event, '$.data.flow') = 'metrics'

Configuration UI

YAML

test_automation.yaml

automation:
  name: test_automation
  enabled: false
  triggers:
    sensors:
      - name: custom_sensor
        type: function
        config:
          python:
            entrypoint: automations.test_automation.sensors.custom_sensor.custom_sensor
    events:
      - sql_filter: json_extract_string(event, '$.data.flow') = 'metrics'
        types:
          - custom_event
  actions:
    - type: function
      name: custom_action
      config:
        python:
          entrypoint: automations.test_automation.actions.custom_action.custom_action

Next steps

• Explore detailed documentation in our reference guides for Automations, Sensors, Events, and Actions
• Deepen your understanding with our Automation concept guide