Configure dynamic pipelines with Parameters

Parameters in Ascend provide a powerful mechanism for dynamically configuring your data flows and components. Rather than hardcoding values, Parameters allow you to create flexible, reusable, and maintainable data pipelines that can adapt to different environments and scenarios.

What you'll learn

In this tutorial, you'll learn:

• How Parameters work within the Ascend platform
• The hierarchy and precedence of Parameter settings
• How to define and use Parameters in various component types
• Best practices for securely managing Parameters
• Common Parameter use cases and examples

Hierarchy of Parameters

Parameters can be set at multiple levels in Ascend, with more specific settings overriding more general ones:

1. Flow Run Parameters (highest priority)

   • Applied to a specific execution of a flow
   • Ideal for one-time configuration changes

2. Flow Parameters

   • Defined within a flow definition
   • Apply to all runs of that flow unless overridden

3. Profile Parameters

   • Defined in profiles to be used across multiple flows
   • Great for environment-specific configurations

4. Project Parameters (lowest priority)

   • Applied globally across your project
   • Used for organization-wide settings

This hierarchy allows you to set default values at higher levels while providing the flexibility to override them for specific scenarios when needed.

How Parameters work

Parameters in Ascend are stored in Vaults and can be referenced using the ${...} notation. This approach provides both security and convenience:

• Secure storage: Sensitive information is kept in centralized, access-controlled Vaults
• Dynamic references: Use the ${parameters.<param_name>} syntax to reference Parameters
• Real-time resolution: Parameter values are resolved at runtime, allowing for flexibility

Examples

Let's explore how to use Parameters in different component types:

YAML Read Component (e.g., S3, ABFS, Database)

You can define Parameters at the flow or component level and reference them in your YAML using ${parameters.<param_name>}:

component:
  read:
    connection: read_gcs_lake
    gcs:
      path: ${parameters.gcs_lake_path}
      include:
        - glob: "*/month=*/day=*/*.parquet"

This allows you to easily change the input path without modifying the component definition.

YAML Write Component

Parameters can be used for output paths, table names, or other configuration details:

component:
  write:
    connection: write_s3
    input:
      name: my_component
      flow: my_flow
    s3:
      path: {$parameters.s3_path}
      formatter: csv
      manifest:
        name: _ascend_manifest.json

YAML Alias or External Table Component

Parameters can define locations or other properties for alias components:

component:
  alias:
    location: "${parameters.database}.${parameters.schema}.MY_TABLE_NAME"

Python Read Component

In Python components, Parameters are accessed via context.Parameters:

read_params.py

import pandas as pd
from ascend.application.context import ComponentExecutionContext
from ascend.resources import read


@read()
def read_guides(context: ComponentExecutionContext) -> pd.DataFrame:
    df = pd.read_csv(context.parameters["guides_path"])
    return df

Python Transform Component

Parameters are accessed the same way in Transform components:

transform_params.py

from ascend.resources import application_subcomponent_name, ref, test, transform


@transform(
    name=application_subcomponent_name("python_transform"),
    inputs=[
        ref("ft_all_cabs_enriched"),
    ],
    tests=[test("count_greater_than_or_equal", count=0)],
)
def python_transform(ft_all_cabs_enriched, context):
    trip_distance_threshold = context.parameters["trip_distance_threshold"]
    app_context = context.application_component_context
    vendor_id = app_context.config["parameters"]["vendor_id"]

    tb = (
        ft_all_cabs_enriched.filter((ft_all_cabs_enriched.trip_distance > trip_distance_threshold) & (ft_all_cabs_enriched.vendor_id == vendor_id))
        .mutate(tip_percentage=((ft_all_cabs_enriched.tip_amount * 100) / ft_all_cabs_enriched.fare_amount))
        .select("dropoff_zone", "pickup_zone", "tip_percentage", "pickup_datetime")
    )
    return tb

Python Task Component

Task components also access Parameters via context.Parameters:

task_params.py

import ibis
from ascend.application.context import ComponentExecutionContext
from ascend.common.events import log
from ascend.resources import ref, task


@task(dependencies=[ref("top_guides")])
def task_top_guide_metrics(
    top_guides: ibis.Table,
    context: ComponentExecutionContext,
) -> None:
    # Access parameters with defaults
    min_rating = context.parameters.get("min_rating", 0)
    min_expeditions = context.parameters.get("min_expeditions", 0)

    # Filter using parameters
    filtered = top_guides.filter((top_guides["customer_rating"] >= min_rating) & (top_guides["total_expeditions_led"] >= min_expeditions))

    # Compute metrics using Ibis expressions
    metrics_expr = filtered.aggregate(
        average_customer_rating=filtered["customer_rating"].mean(),
        total_expeditions_led=filtered["total_expeditions_led"].sum(),
    )
    # Materialize the results as a pyarrow Table
    metrics_arrow = metrics_expr.to_pyarrow()
    average_customer_rating = metrics_arrow.column("average_customer_rating")[0].as_py()
    total_expeditions_led = metrics_arrow.column("total_expeditions_led")[0].as_py()

    log(f"Average Customer Rating: {average_customer_rating}")
    log(f"Total Expeditions Led: {total_expeditions_led}")

Use cases

Parameters are particularly valuable for:

1. Environment configuration

   • Different database connections for dev, test, and production
   • Varying resource allocations based on environment needs

2. Path management

   • Dynamic input/output locations
   • Date-based or version-based path construction

3. Runtime behavior control

   • Feature flags for enabling/disabling functionality
   • Processing thresholds or limits

4. Secure credential management

   • Database credentials
   • API keys and tokens

Next steps

Now that you understand how Parameters work in Ascend, you can:

1. Define Parameters in Profiles to set default configurations
2. Use Parameters in Flows to create adaptive data pipelines

For more detailed information, refer to the Parameters concept guide.