Data Planes

Data Planes are the foundation for executing Flows within Ascend. Each Flow connects to a specific Data Plane that determines where your data is stored and processed. This architecture optimizes data processing by considering data locality, processing capacity, and storage requirements.

What is a Data Plane?

A Data Plane serves as the storage and processing infrastructure for all data handled by your Ascend workflows. Every Component execution result becomes an object stored in your Data Plane. Without a Data Plane, there would be no location to store, organize, and process your data and metadata.

Key benefits

• Data locality: Processes data close to where it's stored, minimizing data movement and reducing latency
• Customizable infrastructure: Configure Data Planes to match specific workflow requirements with custom compute resources and storage configurations
• Security and compliance: Leverages the security features and compliance frameworks of the underlying platforms
• Centralized integration: Combine multiple Data Planes in a Super Graph for complex workflows

How Data Planes work

Each Flow is configured to run on a specific Data Plane and connects through a Connection. When a Flow executes, Ascend uses the Connection to access the Data Plane and performs different actions based on the component type:

• Read Components: Extract data from external systems via the Component's Connection and write it to the Data Plane
• Transform Components: Process data within the Data Plane by generating and executing appropriate instructions (SQL or otherwise)
• Write Components: Retrieve data from the Data Plane and output it to external systems through the Component's Connection
• Tests: Execute directly on the Data Plane
• Task Components: Run code on the Data Plane when supported, or execute within your Ascend Workspace or Deployment

Configuring Data Planes

You can customize your Data Plane configuration using Parameters at multiple levels:

• Project level
• Profile level
• Flow level

Architecture considerations

When designing data workflows in Ascend, consider these Data Plane factors to maximize performance and efficiency:

• Platform selection: Choose a Data Plane that aligns with your existing data infrastructure, processing requirements, and preferred cloud ecosystem
• Performance tuning: Optimize for speed and cost-efficiency by configuring processing power and storage patterns
• Security implementation: Configure Data Planes to comply with organizational security policies by utilizing platform-specific security features

Conclusion

Data Planes provide the essential infrastructure that powers Ascend's data processing capabilities. Their scalability, flexibility, and integration with leading cloud platforms enable you to build efficient, secure, and compliant data workflows. Understanding Data Planes helps you effectively leverage Ascend for sophisticated data engineering tasks.