Build

Building in Ascend encompasses a series of interconnected concepts designed to streamline and enhance the data engineering process. This document serves as an overview, elucidating the fundamental components involved in building within the Ascend platform. Each concept plays a unique role in the data pipeline, from sourcing data to transforming and ultimately writing the data to a target destination. For detailed information on each concept, refer to their specific explainer documents linked below.

Flows

Flows are the backbone of Ascend data pipelines, defining the sequence and logic of data transformations and transfers. They orchestrate the movement of data across various components, ensuring data is processed efficiently and accurately from source to destination.

Data Planes

Data Planes in Ascend provide the infrastructure layer for executing flows. They are responsible for managing the compute resources required to process data, enabling scalable and secure data operations within the platform.

Read Components

Read Components are the entry points for data into the Ascend platform. They allow Ascend to connect with various data sources, such as databases, file storage systems, and cloud services, to ingest data into the platform for processing.

Transforms

Transforms in Ascend represent the data processing logic. They can be defined using SQL or Python to perform transformations on the ingested data. Transforms are crucial for cleaning, aggregating, enriching, and otherwise preparing data for analysis or further processing.

Tasks

Tasks give you the freedom to write nearly any SQL or Python code you would like, to run in sequence in the context of a flow. You can write multiline SQL statements that do nearly anything, or copy/paste nearly any Python code and run it in your flow with almost no modifications.

Write Components

Write Components enable the export of processed data to external systems, databases, or storage solutions. They are the exit points for data leaving the Ascend platform, ensuring that the transformed data is available for use in downstream applications or for storage.

Tests

Tests in Ascend provide mechanisms for validating data quality and integrity throughout the data pipeline. They can be applied at various stages of the pipeline to ensure that data meets the specified criteria and standards, reducing errors and improving data reliability.

Conclusion

Building in Ascend leverages a comprehensive set of components designed to facilitate efficient data engineering practices. By understanding the roles and relationships of these components, users can create robust, scalable, and reliable data pipelines that support a wide range of data processing and analytics applications.