Python Task

Task Components in Ascend let you execute arbitrary Python code as a Component. Unlike Transforms, Python Tasks aren't directly tied to the Data Plane, giving you more flexibility.

Structure your Task Component

The @task decorator defines dependencies and execution context for your Python Task. This decorator specifies input data and any Components that your Task depends on.

Example

Here's a Python Task Component example:

task.py

import ibis
from ascend.application.context import ComponentExecutionContext
from ascend.common.events import log
from ascend.resources import ref, task


@task(dependencies=[ref("top_guides")])
def task_top_guide_metrics(
    top_guides: ibis.Table,
    context: ComponentExecutionContext,
) -> None:
    # Compute metrics using Ibis expressions
    metrics_expr = top_guides.aggregate(
        average_customer_rating=top_guides["customer_rating"].mean(),
        total_expeditions_led=top_guides["total_expeditions_led"].sum(),
    )
    # Materialize the results as a pyarrow Table
    metrics_arrow = metrics_expr.to_pyarrow()
    average_customer_rating = metrics_arrow.column("average_customer_rating")[0].as_py()
    total_expeditions_led = metrics_arrow.column("total_expeditions_led")[0].as_py()

    log(f"Average Customer Rating: {average_customer_rating}")
    log(f"Total Expeditions Led: {total_expeditions_led}")

This example computes metrics using Ibis expressions and logs the results:

• Task decorator: The @task decorator establishes dependencies. In this case, it depends on the top_guides Component which provides the input data.

• Aggregation: The code computes metrics with Ibis expressions, calculating average customer ratings and total expeditions led by aggregating data from the top_guides table.

• Materialization: Results are converted to a PyArrow table with individual metrics extracted for logging.

• Logging: The computed metrics are logged to provide insights into average customer ratings and total expeditions.

Next steps

• Learn more about Tasks in our concept guide
• Explore Task options in our reference guide