Parameters

Parameters in Ascend provide a powerful mechanism for dynamically configuring your data flows and components. They allow for greater flexibility by enabling settings to be adjusted at runtime rather than being hardcoded. This concept guide explores what Parameters are, why they're valuable, and the theory behind their operation.

Benefits

• Flexibility: Easily adapt configurations to different environments and execution contexts.
• Reusability: Avoid code duplication by parameterizing common configuration values.
• Maintainability: Simplify configuration management by centralizing Parameter definitions.
• Version control: Track changes to Parameter values over time.
• Security: Safely manage sensitive information through vault integrations.

Types of Parameters

Parameters in Ascend can be categorized based on their purpose and scope:

Configuration Parameters

Used to define operational aspects of your data pipelines:

• Database connection details
• File paths and storage locations
• Processing options and flags

Runtime Parameters

Dynamically set at execution time to control behavior:

• Date ranges for data processing
• Filtering criteria for datasets
• Feature flags to enable/disable functionality

Secure Parameters

Protected values that require special handling:

• API keys and tokens
• Database credentials
• Encryption keys and certificates

Use cases

Parameters are flexible and adaptable. Each organization, instance, and project will utilize different Parameters based on their unique needs. Common use cases include:

• Environment-specific adjustments: Control settings like database connections or API endpoints that vary between development, staging, and production.
• Dynamic data paths: Define file paths or data locations that change between runs or datasets.
• Scalable operations: Adjust batch sizes, processing limits, or resource allocations based on workload demands.
• Flow control: Control data flow behaviors, such as enabling or disabling flows, setting data ingestion time ranges, or specifying processing conditions.
• Data filtering: Filter data based on criteria like date ranges or specific columns.
• Resource allocation: Define settings like data plane type (e.g., Snowflake, Databricks) or compute resources for processing tasks.
• Conditional logic: Use Parameters in conditional logic to manage component or task execution within a flow.
• Customization and reusability: Make flows and components more dynamic and adaptable by using Parameters for customization.

Parameter theory and architecture

Definition and structure

Parameters are typically defined in Profiles within a Project and can have arbitrary structures:

profile:
  parameters:
    my_param_1: my_value_1
    my_param_2: my_value_2
    nested:
      param_a: value_a
      param_b: value_b

Reference syntax

Parameters are referenced using the ${...} notation, which tells Ascend to resolve the value at runtime:

connection:
  s3:
    root: ${parameters.my_param_1}

Secrets vs. Parameters

Secrets are secured credentials that are managed separately from Parameters and processed through different security systems. While both Secrets and Parameters use a similar reference syntax, they serve distinct purposes.

The example below illustrates the difference in syntax between referencing Parameters and Secrets:

connection:
  s3:
    root: ${parameters.my_param_1}
    key: ${vaults.my_vault.my_Secret_1}

Parameters are used for configurable values, while Secrets should be used for sensitive information like credentials, tokens, and keys.

Parameters can be defined in Project configurations and Profile definitions, making them available throughout your Project. In contrast, Secrets must be stored in secure Vaults and can only be referenced (not directly defined) in other resources such as Connections, Components, and Flows.

warning

Always store Secrets in a Vault and reference them as a Secret. Never hardcode Secrets in your Profiles.

Parameter inheritance and precedence

Ascend resolves Parameter values at runtime based on a well-defined precedence order. The most specific Parameter definition takes precedence:

1. Flow Run Parameters (highest priority)
2. Flow Parameters
3. Profile Parameters
4. Project Parameters (lowest priority)

This hierarchy ensures that the most relevant Parameter value is always used, allowing for flexible overrides and customizations.

Resolution Algorithm

When a Parameter is referenced, Ascend follows these steps:

1. Check if the Parameter is defined at the Flow Run level
2. If not found, check the Flow level
3. If not found, check the Profile level
4. If not found, check the Project level
5. If not found anywhere, return a resolution error or default value (if specified)

Best practices

Naming and organization

• Consistent naming: Use consistent naming conventions to keep Parameter usage clear and manageable across components and workflows.
• Logical grouping: Organize Parameters in logical groups based on their purpose or the components they configure.
• Descriptive names: Choose clear, descriptive names that indicate each Parameter's purpose.

Documentation and maintenance

• Documentation: Include comments in Parameter definitions to explain their purpose and impact.
• Version control: Maintain Parameter definitions under version control alongside your code.
• Auditing: Regularly review Parameter usage and remove unused Parameters.

Security considerations

• Sensitive data: Always use Vaults for storing sensitive information.
• Least privilege: Limit access to Parameter definitions based on the principle of least privilege.
• Validation: Implement validation for Parameter values to prevent misuse or injection attacks.

Testing

• Testing Parameter values: Thoroughly test Parameters in controlled environments to ensure expected behavior and avoid unintended changes.
• Default values: Provide sensible default values for optional Parameters.
• Error handling: Implement robust error handling for missing or invalid Parameters.

Conclusion

Parameters are a powerful tool in Ascend that enhance flexibility, reusability, and maintainability in your data workflows. By understanding the concept of Parameters and following best practices, you can create more dynamic and adaptable systems that are easier to manage and scale.

For hands-on implementation examples and practical guidance, refer to the Parameters tutorial.