Foundational Concepts
Ascend's core primitives, known as resources, serve as the essential building blocks for constructing your daily data flows.
Project
A Project organizes all related artifacts for a specific data engineering initiative. It acts as a container for connections, flows, components, code, configurations, and more, providing a structured environment for development, testing, and deployment.
Connection
A Connection link Ascend to external data sources, enabling data ingestion, transformation, and delivery. They include configuration details and credentials for interacting with systems like warehouses, databases, cloud storage, APIs, and more.
Flow
A Flow is a single unit of execution — a data processing flow that defines the sequence of operations required to read, transform, and/or write data, or even execute arbitrary code as a task. Flows are composed of Components, which represent individual processing steps.
A flow is represented as a directed acyclic graph (DAG) that defines the dependencies between components, and the order in which they should be executed. This is a fancy way of saying, some steps may run in parallel, some steps may run in sequence, and some steps may run only after other steps have completed. Typical data/control flow in data pipelines never runs in a loop/cycle, and this is why a DAG representation is used.
Component
A Component is a reusable building block that encapsulates a specific data processing operation, such as reading data from a source, transforming data, or writing data to a destination. Components can be used within Flows to define the processing logic and dependencies of a data pipeline.
Many of the component types in Ascend support "Smart" logic, which allows you to define the component logic in a declarative manner, without having to write as much code. These Smart Components track additional metadata about the data they are processing, which can be used to drive more sophisticated and dynamic behavior in your flows.
There are several types and subtypes of components in Ascend. The basic component types include:
- Read: Read data from a source
- Transform: Transform data
- Write: Write data to a destination
- Task: Execute arbitrary code
- Test: Validate data integrity
In addition, there are multiple advanced component types that you can use to extend the functionality of your flows and increase reusability. These include:
- Application: a reusable sub-flow consisting of one or more components
- Compound Component: Combine multiple components into a single component
Data Plane
A Data Plane is the underlying infrastructure where data storage, processing, and computation occur. It's the execution environment for the component logic of your Flows. Ascend supports various Data Planes (Snowflake, Databricks, BigQuery, etc.), allowing you to choose the best fit for your needs.
It is strongly recommended that you set up a separate Data Plane for your Development, Staging, and Production environments, and keep these isolated from the Instance Store.
For more information on how Ascend manages infrastructure resources and best practices on secure data storage and processing, see System Architecture.