Skip to main content
Version: 3.0.0

Tasks

Task Components allow you to integrate complex application and business logic directly into your data pipelines. Tasks are components within Ascend that run specific code, such as Python scripts or SQL queries. They handle complex operations, apply business logic, or interact with data in ways that predefined components do not cover.

Types of Tasks​

info

See the Task Reference for more information on the syntax and capabilities of all task types.

  • Python Tasks: Execute Python code within the flow to perform data manipulation, apply conditional logic, make API calls, and more.
  • SQL Tasks: Execute SQL queries to interact directly with data stores, perform complex joins, aggregations, and other SQL operations.

How Tasks Work​

  • Integration with Flows: Tasks are embedded directly into Ascend flows and run in sequence with other components like data reads, writes, and transformations, ensuring they execute in the context of the workflow.
  • Parameterization and Inputs: Tasks can accept parameters, allowing them to adjust their behavior based on input data or external conditions, which is crucial for dynamic workflows.

Use Cases​

  • Custom Data Processing: Use Python Tasks for data cleaning, advanced calculations, or real-time data enrichment via external APIs.
  • Conditional Execution: Introduce decision-making logic with Tasks that evaluate conditions, such as rerouting data based on quality checks or triggering alerts.
  • External System Integration: Connect and interact with external systems, send data to third-party services, execute remote commands, or integrate with other applications.

Examples​

SQL​

The following example shows a SQL Task that will be executed on the Flow's Data Plane.

components/my_task.sql.jinja
{{
  config(
    type="generic",
    dependencies=[ref("my_table")]
  )
}}
CALL my_stored_procedure(
  'param1',
  'param2',
  123
);

Python​

The following example shows a Python Task that will be executed within the Flow's runtime environment.

components/my_task.py
@component(tests=[])
def my_python_task(context):
  print("Hello, world!")

Benefits​

  • Enhanced Flexibility: Implement any business logic directly within Ascend, extending its data orchestration capabilities.
  • Business Logic Integration: Embed business rules within flows to align the data pipeline with organizational needs, reducing the need for separate processing steps.
  • Streamlined Workflows: Handle complex logic directly within Ascend, reducing the need for external processing and manual intervention, which improves efficiency.
Last updated on