Skip to main content

Quickstart for the Developer plan

preview feature

We're still working on this one! Expect changes and note that standard SLAs don't apply, so don't rely on this for production workloads yet. Refer to our release stages to learn more.

Introduction​

Your Ascend Instance comes pre-configured with everything you need to start building data pipelines: a Repository, Otto's Expeditions Project, a Workspace, Connections, and more.

This quickstart guide focuses on the essentials of delightful Ascend development. You'll learn how to:

Create a Flow​

A Flow is the foundation of your data pipeline. Let's create one to organize your Components:

  1. Right-click anywhere in the Super Graph (the overview of all Flows in your Project) and select Create Flow

    folder

  2. Fill out the form with these details:

    • Flow name: developer-quickstart
    • Version: 1.0.0

    Other fields are optional.

    folder

  3. Scroll down and click Save to create your Flow

Create a Read Component​

Throughout this quickstart, we'll be recreating Components from Otto's Expeditions, where Otto the data pipeline goat creates unforgettable mountain adventures. Let's start by building a Read Component that pulls route closure data from an existing Local File Connection.

note

Typically, you'd create a Connection before building Components, but we'll be leveraging an existing Local File Connection.

  1. Navigate to your developer-quickstart Flow using the Build panel or by double-clicking its node in the Super Graph

  2. Right-click anywhere in the Flow Graph background (the graph showing your current Flow's Components)

  3. Hover over Create Component, then Read, and choose From Scratch

    folder

  4. Name your Component read_route_closures and ensure YAML is selected as the file type

  5. Copy the following configuration:

read_route_closures.yaml
component:
  read:
    connection: read_local_files
    local_file:
      parser: auto
      path: route_closures.csv

Create a Transform​

Next, let's create a Transform that uses our Read Component as input and cleans the column names in our data:

  1. In the Files panel, right-click the components folder inside your developer-quickstart Flow and select New File
  2. Name your file route_closures.py
  3. Copy the following code into your new file:
route_closures.py
import ibis
import ottos_expeditions.lib.transform as T
from ascend.application.context import ComponentExecutionContext
from ascend.resources import ref, transform


@transform(inputs=[ref("read_route_closures")])
def route_closures(read_route_closures: ibis.Table, context: ComponentExecutionContext) -> ibis.Table:
    route_closures = T.clean(read_route_closures)
    return route_closures
tip

The custom cleaning function used here is shared code from the ottos_expeditions/lib folder.

Create a Task​

Finally, let's create a Task that demonstrates Ascend's logging capabilities by logging each route that gets updated:

  1. Right-click anywhere in the Flow Graph background (the graph showing your current Flow's Components)

  2. Hover over Create Component, then choose Task

    folder

  3. Name your Component task_update_route_closures_calendar and ensure Python is selected as the file type

  4. Copy the following code:

task_update_route_closures_calendar.py
import ibis
from ascend.application.context import ComponentExecutionContext
from ascend.common.events import log
from ascend.resources import ref, task


@task(
    dependencies=[
        ref("route_closures"),
    ]
)
def task_update_route_closures_calendar(
    route_closures: ibis.Table,
    context: ComponentExecutionContext,
) -> None:
    for route in route_closures["route_id"].to_pyarrow().to_pylist():
        log(f"Updating route {route}")

Run your Flow in a Workspace​

Now it's time to see your pipeline in action! Your completed Flow should display all three Components connected in sequence:

flow-graph

Click Run Flow in the build info panel in the top left, and select Run in the modal that appears.

folder

Watch as Ascend processes your data through each Component - from reading the route closure data, to cleaning it with your Transform, to logging updates with your Task.

workspace vs. deployment

Workspaces are ideal for development, with editable code and Automations disabled. Deployments are ideal for production with read-only code and Automations enabled. Refer to our concept docs to learn more.

Run your Flow in a Deployment​

Now, let's promote your code to the Development Deployment and run it in an isolated environment.

  1. From your Workspace, navigate to the Source Control panel in the left sidebar and click Open Git Log & Actions.

    source-control

  2. In the Git log window, select Merge to Deployment and choose development. Click Merge in the confirmation dialog to promote your code to the Development Deployment. Merge

  3. From the Ascend homepage, locate the Development Deployment and click on it to open the Deployments Dashboard.

    note

    The screenshots below show the Demo Development Deployment, but you should use the regular Development Deployment.

    Home

  4. In the Deployments Dashboard, select the developer-quickstart Flow from the left sidebar. Dash

  5. Click the Run Flow button in the top right corner to run your pipeline: Run

  6. In the run configuration modal, click Run to start the execution.

  7. Monitor the execution progress through the Flow runs matrix, which displays the status of both the overall Flow and its individual Components: Matrix

🎉 Congratulations! You've successfully built and deployed your first Ascend Flow, complete with a Read Component, Transform, and Task!

Next steps​

Ready to dive deeper? Here's what to explore next:

  • 💭 Explore core Ascend concepts to build a deeper understanding of the platform
  • 📊 Master DataOps best practices for robust production workflows
  • 🧪 Add data quality tests to validate and monitor your data pipelines
  • ⚙️ Learn more about Deployments to promote your pipelines from development to production
Last updated on