Flows

Flows are the backbone of data processing in Ascend, and generally follow the ELT (Extract, Load, Transform) pattern of data processing. You create a Data Flow with a combination of Components to ingest, transform, and write data to an external location. The Data Flow you design is a blueprint that is sent to the Ascend DataAware Automation Engine which intelligently orchestrates data movement and processing. These steps are executed on specific Data Planes like Snowflake, Databricks, or BigQuery.

Key Features

1. Structured Data Processing: Flows provide a clear approach to data engineering, defining how data is ingested, transformed, and output.
2. Component Integration: By integrating Read Components, Transforms, and Write Components, Flows enable modular and efficient data pipeline construction.
3. Scalability and Flexibility: Flows are built for scalability, handling various data volumes, velocities, and types.
4. Monitoring and Management: Flows integrate with Ascend's monitoring tools, allowing you to track performance, manage resources, and troubleshoot pipeline issues.

Flow Structure

info

For complete details on the options available when customizing your Flow, see the Flow Reference.

A Flow in Ascend is defined by its YAML configuration, and includes flow specific parameters, and flow specific bootstrap scripts. All resources for a Flow are contained within the Flow's directory, including the components/ directory, which contains the Components for the Flow.

Best Practices

• Modular Design: Design Flows in modular blocks to enable easier updates, testing, and reuse of components.
• Efficiency Optimization: Use features like incremental processing and partitioning to ensure efficient Flow execution at scale.
• Comprehensive Testing: Integrate thorough testing to ensure data quality and integrity.
• Monitoring and Maintenance: Regularly monitor performance and resource utilization, adjusting as needed for optimal operation.

Conclusion

Flows are central to data pipeline orchestration in Ascend. By understanding their structure, capabilities, and best practices, you can effectively streamline your data engineering processes—from ingestion to the delivery of insights.