Data Planes

Data Planes are essential for executing Flows within Ascend. Each Flow is linked to a specific Data Plane, which determines where data is stored and processed. This linkage ensures data processing occurs in an optimized environment by considering data locality, processing power, and storage capabilities.

Benefits

• Data Locality: Processes data near its storage location, reducing movement and latency.
• Customization: Configure Data Planes to meet workflow-specific needs, including custom compute resources and storage setups.
• Security and Compliance: Adheres to the security and compliance standards of the underlying platforms.

How Data Planes Work in Ascend

Each Flow is configured to run on a specific Data Plane, which must be connected via a Connection. When the Flow is run, Ascend will use the Connection to access the Data Plane, and depending on the component type, perform the appropriate actions:

• Read Components will read data from the external system via the Connection of the Component, and write that data to the Data Plane via the Connection to the Data Plane.
• Transform Components will transform data in the Data Plane by generating the appropriate instructions (SQL or otherwise) and executing those instructions on Data Plane.
• Write Components will read data from the Data Plane via the Connection to the Data Plane, and write that data to the external system via the Connection of the Component.
• Tests will execute on the Data Plane.
• Task Components will execute the code from the Task on the Data Plane (if supported). Otherwise, it will execute the code in your Ascend Workspace or Deployment.

Configuring Data Planes

Data Plane configuration can be customized with Parameters, which can be specified at the Project, Profile, and Flow levels.

Architecture Considerations

When designing Data Flows in Ascend, consider the capabilities and configuration of Data Planes to achieve optimal performance and efficiency.

• Platform Selection: Choose a Data Plane that aligns with your data storage, processing needs, and preferred cloud ecosystem.
• Performance Optimization: Use Data Plane configurability to optimize speed and cost efficiency, considering factors like processing power and data storage patterns.
• Data Security: Configure Data Planes to comply with your organization's data security policies, leveraging the security features of the selected platform.

Conclusion

Data Planes are a critical component of Ascend's platform, providing the necessary infrastructure for large-scale data storage and processing. Their scalability, flexibility, and integration with leading cloud services allow you to build efficient, secure, and compliant data workflows. Understanding the role and capabilities of Data Planes helps you effectively leverage Ascend's platform for advanced data engineering tasks.