Parameters

Parameters in Ascend provide a powerful mechanism for dynamically configuring your data flows and components. They allow for greater flexibility by enabling settings to be adjusted at runtime rather than being hardcoded. Parameters can have arbitrary structures and are resolved in real-time, meaning that changes to parameters immediately affect how resources operate.

Benefits of Using Parameters

• Flexibility: Easily adapt configurations to different environments and execution contexts.
• Reusability: Avoid code duplication by parameterizing common configuration values.
• Maintainability: Simplify configuration management by centralizing parameter definitions.
• Version Control: Track changes to parameter values over time.

Use Cases

Parameters are flexible and adaptable. Each organization, instance, and project will utilize different parameters based on their unique needs. Common use cases include:

• Retrieving Credentials: Securely store secrets in Vaults and reference them as parameters to avoid hardcoding sensitive information.
• Environment-Specific Adjustments: Control settings like database connections or API endpoints that vary between development, staging, and production.
• Dynamic Data Paths: Define file paths or data locations that change between runs or datasets.
• Scalable Operations: Adjust batch sizes, processing limits, or resource allocations based on workload demands.
• Flow Control: Control data flow behaviors, such as enabling or disabling flows, setting data ingestion time ranges, or specifying processing conditions.
• Data Filtering: Filter data based on criteria like date ranges or specific columns.
• Resource Allocation: Define settings like data plane type (e.g., Snowflake, Databricks) or compute resources for processing tasks.
• Conditional Logic: Use parameters in conditional logic to manage component or task execution within a flow.
• Customization and Reusability: Make flows and components more dynamic and adaptable by using parameters for customization.

How Parameters Work

Parameters are typically defined in Profiles within a Project, and can have arbitrary structures. For example:

profile:
  parameters:
    my_param_1: my_value_1
    my_param_2: my_value_2

Parameters are most commonly used to configure Connections and Components. They are referenced with ${...} syntax.

connection:
  s3:
    root: ${param.my_param_1}

Parameters are variables you define in Ascend to reuse values across different Connections, Components, Flows, and more. They provide a consistent way to manage variable values across your project.

Secrets

Secrets are a type of parameter that are stored in a Vault and also referenced with ${...} syntax. Different from Parameters, however, Secrets are referenced with the secret keyword.

warning

You should always store secrets in a Vault and reference them as a Secret. Do not hardcode secrets in your code or profiles.

connection:
  s3:
    root: ${param.my_param_1}
    key: ${secret.my_vault.my_secret_1}

Parameter Inheritance and Precedence

Ascend resolves parameter values at runtime based on a well-defined precedence order. The most specific parameter definition takes precedence. The order is as follows:

1. Flow Run Parameters (highest priority)
2. Flow Parameters
3. Profile Parameters
4. Project Parameters (lowest priority)

This ensures that the most relevant parameter value is always used, allowing for flexible overrides and customizations.

Best Practices

• Consistent Naming: Use consistent naming conventions to keep parameter usage clear and manageable across components and workflows.
• Documentation and Comments: Include comments in parameter definitions to explain their purpose and impact. This practice aids in understanding how parameters influence workflows.
• Testing Parameter Values: Thoroughly test parameters in controlled environments to ensure expected behavior and avoid unintended changes.

Conclusion

Parameters are a powerful tool in Ascend that enhance flexibility, reusability, and maintainability in your data workflows. By understanding how parameters work and following best practices, you can create more dynamic and adaptable systems that are easier to manage and scale.