Flow Runs

A Flow Run is a single execution of a Flow. It represents the actual processing of data according to the blueprint defined in your Flow. Each Flow Run has a unique identifier and generates logs, metrics, and lineage information, providing valuable insights into the execution process. Flow Runs operate within a specific Environment and utilize the resources defined in the associated Data Plane.

Key Features

• Environment Context: Flow Runs execute within a specific Environment (Development, Staging, Production), ensuring isolation and preventing interference between different stages of the development lifecycle.
• Runtime: Flow Runs are executed within a specific runtime environment, which can be either a Workspace or a Deployment.
• Logging and Metrics: Flow Runs generate detailed logs and metrics, capturing execution progress, timestamps, and any errors encountered. This information is crucial for debugging and performance analysis.
• Data Plane Utilization: Flow Runs leverage the resources and configurations of the associated Data Plane, determining where data is stored and processed.
• Parameters: Flow Runs can be configured with parameters, which are used to customize the execution of the Flow.
• Unique Identifier: Each Flow Run is assigned a unique ID, allowing you to track and monitor individual executions.
• Lineage Tracking: Flow Runs automatically track data lineage, providing a clear picture of how data is transformed and moved throughout the pipeline.

Initiating Flow Runs

Flow Runs can be initiated in several ways:

• Manually: Trigger a Flow Run directly through the Ascend UI or API. This is useful for testing and ad-hoc data processing.
• Scheduled: Configure automated Flow Runs on a recurring schedule (e.g., hourly, daily, weekly). This is essential for regular data updates and reporting.
• Triggered by Events: Start Flow Runs based on external events, such as messages from other systems. This enables real-time data processing and responsiveness.
• Automations: Leverage Ascend's automation capabilities to orchestrate complex workflows and trigger Flow Runs based on specific conditions or events.

Monitoring and Managing Flow Runs

Ascend provides tools to monitor the status and progress of Flow Runs. You can view logs, metrics, and lineage information to understand the execution process and identify any issues. Furthermore, you can manage Flow Runs by stopping them prematurely or retrying failed executions.

Conclusion

Flow Runs are the operational heart of Ascend, bringing your data flows to life. By understanding their features, initiation methods, and management tools, you can effectively execute and monitor your data pipelines, ensuring efficient and reliable data processing. Leveraging best practices for Flow Runs will contribute to the overall success and scalability of your data engineering efforts within Ascend.