Create your first Flow

Overview

Your Ascend Instance comes pre-configured with everything you need to start building data pipelines: a Repository, the Otto's Expeditions Project, a Workspace, Connections, and more.

This quickstart guide focuses on the essentials of delightful Ascend development. You'll learn how to:

• Create a Flow (data pipeline)
• Build a Read Component with data quality tests
• Create a Transform Component to process your data
• Implement a Task for custom operations
• Execute your Flow in both a Workspace for development and a Deployment for production

Project

Ascend provides multiple Projects out of the box. The simplest approach is using the Default Project, which lets you explore Ascend immediately without any setup.

For a more comprehensive experience, you can optionally switch to the Otto's Expeditions Project through your Workspace settings to see advanced features.

This quickstart works with either the Default or Otto's Expeditions Project.

Alternatively, customize the Project configuration for your existing data infrastructure and specific requirements.

Create a Flow

A Flow is the foundation of your data pipeline. Let's create one to organize your Components:

Form

1. Right-click anywhere in the Super Graph (the overview of all Flows in your Project) and select Create Flow
2. Fill out the form with the Flow name developer-quickstart. Other fields are optional
3. Click Save to create your Flow
4. Click the collapse icon in the top right to return to the Super Graph view

Files panel

1. Right-click the flows folder and select New folder
2. Name your folder developer-quickstart
3. Right-click the developer-quickstart folder and select New file
4. Name your file developer-quickstart.yaml
5. Copy the following configuration into your new file:
   developer-quickstart.yaml

   flow: {}
   note

   A components folder will be automatically created within the developer-quickstart folder to hold your Flow Components later. This is expected!

6. Click the collapse icon in the top right to return to the Super Graph view

note

You'll collapse tabs the same way after each Component creation step in order to return to the Flow Graph view.

Create a Read Component

Throughout this quickstart, we'll be recreating Components from Otto's Expeditions, where Otto the data pipeline goat creates unforgettable mountain adventures. Let's start by building a Read Component that pulls route closure data from an existing Local File Connection.

note

Typically, you would create a Connection before building Components, but we'll be leveraging an existing Local File Connection.

UI

1. Navigate to your developer-quickstart Flow using the build panel or by double-clicking its node in the Super Graph

2. Right-click anywhere in the Flow Graph background (the graph showing your current Flow's Components)

3. Hover over Create Component, then Read, and choose From Scratch

   

4. Name your Component read_route_closures and ensure YAML is selected as the file type

5. Copy the following configuration:

Files panel

1. In the Files panel, right-click the components folder inside your developer-quickstart Flow and select New file 1. Name your file read_route_closures.yaml 1. Copy the following configuration into your new file:

read_route_closures.yaml

component:
  read:
    connection: read_local_files
    local_file:
      parser: auto
      path: route_closures.csv

Return to the Flow Graph view.

Create a Transform

Next, let's create a Transform that uses our Read Component as input and cleans the column names in our data:

1. In the Files panel, right-click the components folder inside your developer-quickstart Flow and select New file
2. Name your file route_closures.py
3. Copy the following code into your new file:

route_closures.py

import ibis
import ottos_expeditions.lib.transform as T
from ascend.application.context import ComponentExecutionContext
from ascend.resources import ref, transform


@transform(inputs=[ref("read_route_closures")])
def route_closures(read_route_closures: ibis.Table, context: ComponentExecutionContext) -> ibis.Table:
    route_closures = T.clean(read_route_closures)
    return route_closures

Return to the Flow Graph view.

tip

The custom cleaning function used here is shared code from the ottos_expeditions/lib folder.

Create a Task

Finally, let's create a Task that demonstrates Ascend's logging capabilities by logging each route that gets updated:

UI

1. Right-click anywhere in the Flow Graph background (the graph showing your current Flow's Components)

2. Hover over Create Component, then choose Task

   

3. Name your Component task_update_route_closures_calendar and ensure Python is selected as the file type

4. Copy the following code:

Files panel

1. In the Files panel, right-click the components folder inside your developer-quickstart Flow and select New file
2. Name your file task_update_route_closures_calendar.py
3. Copy the following code into your new file:

task_update_route_closures_calendar.py

import ibis
from ascend.application.context import ComponentExecutionContext
from ascend.common.events import log
from ascend.resources import ref, task


@task(
    dependencies=[
        ref("route_closures"),
    ]
)
def task_update_route_closures_calendar(
    route_closures: ibis.Table,
    context: ComponentExecutionContext,
) -> None:
    for route in route_closures["route_id"].to_pyarrow().to_pylist():
        log(f"Updating route {route}")

Return to the Flow Graph view.

Run your Flow in a Workspace

workspace vs. deployment

Workspaces are ideal for development, with editable code and Automations disabled. Deployments are ideal for production with read-only code and Automations enabled. Refer to our concept docs to learn more.

Now it's time to see your pipeline in action! Your completed Flow should display all three Components connected in sequence:

Click Run Flow in the build info panel in the top left, and select Run in the modal that appears.

Watch as Ascend processes your data through each Component - from reading the route closure data, to cleaning it with your Transform, to logging updates with your Task.

Run your Flow from the Super Graph

You can also run Flows from the Super Graph in both Workspaces and Deployments. Right-click the developer-quickstart Flow and choose Entire Flow.

Run your Flow in a Deployment

Now, let's promote your code to the Production Deployment and run it in an isolated environment.

note

In a typical DataOps lifecycle, you would promote through Development and Staging before pushing to Production. However, since trial Instances have only one Production Deployment, we'll use a simplified promotion process.

1. From your Workspace, navigate to the Source Control panel in the left sidebar and click Open Git Log & actions.

   

2. In the Git log window, select Merge to Deployment and choose production. Click Merge in the confirmation dialog to promote your code to the Production Deployment

3. From the Ascend homepage, locate the Production Deployment and click on it to open the Deployments Dashboard

4. In the Deployments Dashboard, select the developer-quickstart Flow from the left sidebar

5. Click the Run Flow button in the top right corner to run your pipeline:

6. In the run configuration modal, click Run to start the execution

7. Monitor the execution progress using the Flow runs matrix, which displays the status of both the overall Flow and its individual Components:

🎉 Congratulations! You've successfully built and deployed your first Ascend Flow, complete with a Read Component, Transform, and Task!

Next steps

Ready to dive deeper? Here's what to explore next:

• 💭 Explore core Ascend concepts to build a deeper understanding of the platform
• 📊 Master DataOps best practices for robust production workflows
• 🧪 Add data quality tests to validate and monitor your data pipelines
• ⚙️ Learn more about Deployments to promote your pipelines from development to production