Advanced topics
Ascend's advanced capabilities enable sophisticated data engineering workflows through intelligent automation, flexible development approaches, and powerful runtime contexts. These features help teams build more efficient, maintainable, and scalable data pipelines.
Intelligence Core​
Ascend's Intelligence Core is the foundational architecture that combines metadata, automation, and AI in a unified system. This three-layer architecture provides:
- Unified metadata collection - Comprehensive tracking of data, code, and user actions
- DataAwareâ„¢ automation engine - Smart orchestration based on dependencies and changes
- Integrated AI agents - Otto and other AI assistants that understand your data context
The Intelligence Core enables Ascend to deliver up to 7x faster development and 83% cost reduction through intelligent optimization.
Flex code​
Flex code democratizes data pipeline development by supporting both code-based and UI-based workflows in the same platform. This approach allows:
- Data engineers to write complex Python and SQL transformations directly in code
- Data analysts to configure components through intuitive forms and UI
- Mixed teams to collaborate seamlessly regardless of technical background
- Flexible development with real-time switching between code and form views
Flex Code ensures that everyone on your team can contribute effectively to data workflows.
Context objects​
Context objects are the foundation of Ascend's runtime environment, providing components with access to resources, configuration, and services. Key context types include:
- ApplicationContext - Flow-level operations and cross-component coordination
- ComponentExecutionContext - Component-specific execution environment
- IncrementalComponentExecutionContext - Specialized context for efficient incremental processing
Context objects enable dependency injection, resource management, and secure access to data and connections throughout your pipeline execution.
Additional concepts​
Smart Components & Partitioning​
Smart Components leverage Ascend's intelligence to automatically optimize data processing through micro-partitioning and fingerprinting. They track metadata about data and code to determine the most efficient processing strategies without requiring manual optimization.
Materialization​
Materialization strategies in Ascend determine how your data is physically stored and accessed. Options include Views, Tables, and Smart Tables, each optimized for different use cases and performance requirements.
Parameterization & Profiles​
Parameterization allows you to create flexible, reusable Flows using variables. Profiles define runtime parameters and configurations, enabling the same Flow to work across different environments like development, staging, and production.
Automation​
Automation streamlines your data lifecycle by triggering Flow runs based on events, schedules, or custom conditions, reducing manual intervention and ensuring timely data processing.