Lab 2: Orchestration, Automation, & Observability

Schedule your pipeline to run automatically and set up triggers for your agents to respond to system events.

In this lab, you'll move from manual pipeline runs to a fully automated DataOps workflow. You'll schedule daily runs, create triggers for your agents to respond to system events, and set up weekly summary reports.

Prerequisites

• Completed Lab 1: Ingestion & Transformation

Step 1: Schedule daily pipeline runs

Your carbon intensity analysis is most useful when it's up to date. Let's schedule the pipeline to run every day at midnight UTC so we have up to date data to schedule our operations for the each day.

Open Otto (Ctrl+I) and paste:

Please create an automation that will run this flow daily at midnight UTC

Otto will create an Automation resource that:

• Triggers your Flow on a daily schedule
• Runs at midnight UTC to have fresh data each morning

Step 2: Create a failure alert with AI root cause analysis

When pipelines fail, typically you want to know immediately — and you want to understand why so that you can resolve the error to reduce business impact.

Up to 50% of data engineering time is spent on this work of maintaining pipelines and resolving errors. However, with AI, we can automate this process to save time and reduce business impact.

Let's create an agentic automation that uses the Context, tools, and triggers framework to perform critical responses and root cause analysis when failures occur.

Paste this prompt:

Please create an automation that triggers Otto to run when this flow fails. I would like you to perform a complete root cause analysis based on my git history and runtime performance history. Please include proposed fixes. Then send this summary to me in a well formatted email. This automation should require no human intervention so our instructions should include detailed expectations and clearly state that you cannot ask for help from the user while generating the RCA, Proposed Response, & Email.

This creates an intelligent automation that:

1. Triggers when the Flow fails
2. Uses the Context of my Flow run history and git history to identify the root cause
3. And leverages Tools to send and email with the summary of its findings

To test this automation, we can ask Otto to perform the task based on a recent failure:

Please find the latest flow run error and perform the tasks in this automation. Please follow the instructions exactly, and send the results to me in a well formatted email. (If there are no recent errors, send me an example email with a fabricated error.)

tip

This is agentic DataOps in action — Otto autonomously investigates and reports on failures without human intervention! You can further extend these agents to perform additional actions by giving agents access to tools via MCP servers. While we won't be doing that for this lab, you can follow this guide to build your own MCP connections to external services.

Step 3: Set up weekly summary reports

The last automation we will create for this lab will trigger Otto to send a weekly summary of all pipeline activity. You can use this automation to stay informed about your pipeline's health and performance trends.

Paste this prompt:

Please create an automation that sends me a weekly summary of all pipeline activity including successes, failures, and performance trends. Schedule this to send on Friday morning at 9:00 AM UTC.

Otto will create an automation that:

• Runs every Friday morning
• Summarizes all pipeline runs from the past week
• Reports on successes, failures, and performance trends
• Delivers the summary via email

To test this automation, we can ask Otto to perform the task based on our Workspace activity (Note: when we deploy this automations to our Production Deployment in the next step, this automation will trigger Otto to explore the history of that Deployment environment, rather than our Workspace activity).

Please follow the exact instructions in this automation and generate a weekly summary email based on my Workspace activity and send it to me in an email.

Now that we have created and tested our automations, we can deploy them to a Production environment where they will run automatically.

[Optional] Step 4: Create a Deployment and merge your changes

Why Deployments matter

In Ascend, Workspaces and Deployments serve different purposes:

	Workspace	Deployment
Purpose	Development & testing	Production workloads
Git branch	Feature/dev branches	Main branch
Data warehouse	Dev environment	Production environment
Automations	Disabled	Enabled

Workspaces are where you build and test your pipelines. They're isolated sandboxes where you can experiment without affecting production data. This ensures that your production data is always clean and up to date, and not impacted by your development work or testing.

Deployments are where your pipelines run in production. They typically operate on your main Git branch and write to your production data warehouse. Automations only run in Deployments — this is a safety feature that prevents scheduled jobs from accidentally running in development environments and wasting resources.

Ultimately, this enables a complete CI/CD pipeline for your data pipelines, which is a critical part of DataOps and absolutely essential for leveraging AI safely at scale.

Creating a Deployment

1. Open your Settings menu by clicking your profile icon in the top right corner of your screen then click the Settings button.
2. Then click the Projects tab and select the **Default** Project.
3. Click the Add your first Deployment button.
4. Give it a name (e.g., "Production"), select a branch (e.g. main), and choose a Profile (deployment_template).
5. Click Save to create the Deployment.

Merge your automation to Production

1. Return to your Workspace and open the Source Control panel in the Toolbar.
2. Review your changes including the components you created in the previous lab as well as the new automation.
3. Click Merge to Deployment
4. Select your Production Deployment and click Merge to merge your changes .

Once deployed, your Automations will begin running on defined schedules!

tip

To navigate to your Production Deployment you can click into it from the Home screen or use cmd+k (Mac) or ctrl+k (Windows) to search for Production.

Explore your Production Deployment

Once deployed, you can explore your Production Deployment to see your automations running.

1. Click into your Production Deployment from the Home screen or use cmd+k (Mac) or ctrl+k (Windows) to search for Production.
2. Here you can see all the flows and components that are part of your Production Deployment, including the pipeline we built in the previous lab.
3. Click the name of your pipeline to access the run history.
4. From this page, click the Run Flow button to run your pipeline in production.

This is writing data to a different location than when you run the pipeline in your Workspace. This is because the Workspace and the Deployment are using different Profiles that are isolated from each other. This ensures that your production data is always clean and up to date, and not impacted by your development work or testing.

In your Deployment, you have access to all the metadata associated with your Production Deployment, including the run history of your pipeline and code changes. All this metadata is stored and used by the agents as context (and Triggers) to perform their tasks.

Checkpoint

By the end of this lab, you should have:

• Created a daily schedule automation
• Created a failure alert with Otto-powered RCA
• Created a weekly summary automation
• Deployed your automations to Production

Need help?

Ask a bootcamp instructor or reach out in the Ascend Community Slack.

Next steps

Continue to Lab 3: Data Visualization to create visualizations that show optimal operation times!