Create a Custom Application

Applications are powerful, reusable sub-flows that bundle multiple components into a single visual unit within your Flow Graph. They allow you to package complex logic into modular, reusable building blocks that simplify your data pipelines.

Custom Application Setup

1. Create your Python definition - Write a Python file that implements the Ascend Application interface. This lets you programmatically generate components based on configuration parameters.

2. Organize your code - Place your Python file in a subfolder within your project's src directory. Each Application needs a unique name within the Project.

   Example directory structure:

   my_project/
   ├── src/
   │   └── filter_app.py

3. Create a filter Application - Here's a basic example that filters data based on a threshold:

filter_app.py

import yaml
from pydantic import BaseModel
from ascend.application.application import Application, ApplicationBuildContext, application
from ascend.models.component.component import Component

class FilterConfig(BaseModel):
    input_dataset: str
    filter_column: str
    threshold: int = 100

@application(name="filter_app")
class FilterApplication(Application[FilterConfig]):
    config_model = FilterConfig

    def components(self, config: FilterConfig, context: ApplicationBuildContext):
        component_yaml = f"""
name: {context.compound_component_name}_filtered
transform:
  strategy:
    partitioned:
      enable_substitution_by_partition_name: false
      output_type: table
  inputs:
  - flow: {context.flow_build_context.flow_name}
    name: {config.input_dataset}
    partition_spec: full_reduction
  sql: |-
    SELECT *
    FROM {{{{ ref(\"{config.input_dataset}\") }}}}
    WHERE {config.filter_column} > {config.threshold}
"""
        # Wrap the loaded YAML in a dict with a 'component' key
        component_dict = {"component": yaml.safe_load(component_yaml)}
        return [Component(**component_dict)]

4. Configure the Application - Create a YAML file that references your Custom Application code and provides specific parameters:

filter_app.yaml

component:
  application:
    application_id: filter_app  # Matches your Python file name
    config:
      input_dataset: YOUR_INPUT_DATASET  # Replace with your actual dataset name
      filter_column: YOUR_COLUMN_NAME    # Replace with your column to filter on
      threshold: 100                     # Replace with your desired threshold

When you apply this configuration, your Python code will dynamically generate the Transform component that filters your data based on the threshold you specified.

Key points to remember

• Replace placeholders (like YOUR_INPUT_DATASET, YOUR_COLUMN_NAME) with your actual values
• The application_id in your YAML must match your Python implementation
• Custom Applications give you full flexibility to create multiple components with complex logic

🎉 Congratulations – You've learned how to create a simple Custom Application that you can adapt to your specific needs!