Skip to main content
Version: 3.0.0

📢 What's new in Gen3

☁️ Gen3 is a ground-up rebuild of the Ascend platform, designed to give you more control, greater scalability, and deeper visibility across your data workflows. It's everything you already love about Ascend – now faster, more flexible, and more extensible.

  • Ascend's new Intelligence Core combines metadata, automation, and AI in a layered architecture, empowering all teams to build pipelines faster and significantly compress processing times.

  • Git-native workflows bring version control, collaboration, and CI/CD alignment to all teams through our flex-code architecture— empowering both low-code users and developers to contribute.

  • Observability features expose detailed pipeline metadata so teams have deeper visibility into their system to diagnose problems quickly, reduce manual investigation, and optimize system behavior.

  • Modular architecture empowers data and analytics teams to manage increasingly large and complex pipelines with improved performance and maintainability.

  • Standardized plugins and extension points enable data platform teams to customize and automate workflows more easily.

➡️ Ready to explore? Join the Gen3 public preview to get early access.

Ascend Gen3 Demo

🚀 Features​

Explore the latest enhancements across our platform, from improved system architectures to optimized project management. This section highlights major new functionalities designed to boost performance and flexibility.

Systems & architecture​

Explore the foundational improvements in our system's architecture, designed to enhance collaboration, resource management, and cloud efficiency.

  • Version control-first design – Collaborate and track changes with Git-native workflows
  • Project-based organization – Organize and manage resources with intuitive, project-centric workflows
  • Optimized cloud footprint – Reduce infrastructure usage with centralized UI and a lightweight, scalable backend
  • Event-driven core – Trigger custom workflows using system-generated events
  • Native Git integration – Automate CI/CD pipelines with built-in support for your Git provider CI/CD

Project & resource management​

Effortlessly create, share, configure, and deploy projects with streamlined processes, allowing you to spend less time on administration and more on engineering innovation.

  • Project structure – Organize and manage your data projects with improved structure and clarity
  • Environments – Configure and maintain development, staging, and production environments with software development best practices
  • Parameterized everything – Reuse and adapt pipelines with flexible, comprehensive parameterization
  • Deployments – Roll out pipelines consistently across environments with simplified deployment workflows Deployments

Industry-leading security​

Protect your data and resources with enterprise-grade security features, ensuring comprehensive access control and secrets management across your organization.

  • Enterprise-grade authentication – Secure instances, projects, and pipelines with OpenID Connect (OIDC)
  • Centralized vault system – Manage secrets, credentials, and sensitive configurations securely across your entire platform

Builder experience​

Discover how our builder experience enhancements simplify component creation and improve user interaction with a modern interface.

  • Simplified component spec - Write components with less boilerplate and more intuitive syntax
  • Components:
    • Partition strategies - Flexible data partitioning for optimal performance
    • Data & job deduplication - Intelligent handling of duplicate data and operations
    • Incremental components - Process only new or changed data efficiently
    • Views - Create and manage virtual tables efficiently
    • Generic tasks - Support for versatile task types, including SQL and Python scripts for complex operations
  • Data applications - Build complex data transformations from simpler reusable building blocks and templates
  • Testing support - Test components easily with built-in sample datasets
  • Modern interface - Navigate an intuitive UI designed for improved productivity
  • Dark mode - Switch between light and dark themes with enhanced visual comfort and accessibility
  • Navigation - Access projects, components, and resources through streamlined menus

Navigation and building experience

Data integration​

Our data integration improvements ensure seamless connectivity and performance across major platforms, enhancing your data processing capabilities.

Performance improvements across all data planes**:​

  • 70% reduction in infrastructure costs*
  • 4x faster ingestion speed*
  • 2x faster runtime execution*
  • 10x increase in concurrent processing*

Data planes: Enhanced connectivity and performance across major platforms.​

Snowflake - Full platform integration including:

  • SQL support
  • Snowpark for advanced data processing
  • Complete access to Snowflake Cortex capabilities BigQuery - Comprehensive SQL support including:
  • BigQuery SQL integration
  • Built-in support for BigQuery AI features

Databricks - Complete lakehouse integration featuring:

  • SQL and PySpark support
  • Full access to AI/ML models in data pipelines
  • Support for both SQL warehouses and clusters
  • Unified compute management

* Performance metrics based on comparative analysis between Gen2 and Gen3 platforms

Data quality​

Enhance your data quality management with automated checks and customizable validation rules, ensuring data integrity across your projects.

  • Automated quality gates - Validate data within components, including read and write components
  • Reusable rule library - Create and share standardized data quality rules across your organization
  • Python-based validation - Write custom data quality checks using familiar Python syntax

Flow management​

Optimize your data flows with advanced planning and execution capabilities, supporting high-frequency and concurrent processing.

  • Gen3 flow planner & optimizer – Improve pipeline performance with intelligence planning and execution
  • Flow runs – Manage and monitor individual pipeline executions with enhanced controls
  • Concurrent & high-frequency flow runs – Execute flows in parallel and at higher frequencies
  • Semantic partition protection – Preserve computed results across code changes to avoid unnecessary reprocessing
  • Optional Smart backfills – Backfill data flexibly with advanced control over reprocessing. Smart Backfills

Automation​

Leverage our automation features to create dynamic workflows triggered by real-time events, enhancing operational efficiency.

  • Event-driven extensibility - Automate workflows dynamically based on real-time platform events and triggers
  • Customizable event triggers - Create custom automation triggers including sensors and events Automation

Observability​

Gain comprehensive insights into your data operations with real-time and historical observability, ensuring full transparency and control.

  • Unified metadata stream & repository - Centralize and track metadata across all pipelines
  • Real-time & historical monitoring - Access metadata on pipeline runs and performance history, including:
    • Live monitoring of active pipeline runs
    • Full execution history with smart commit summaries
    • Performance analytics and trend analysis
    • Complete troubleshooting visibility Observability

AI-powered assistant (Otto 🐐)​

Experience the power of AI with Otto, our assistant that helps you create, optimize, and document your data pipelines effortlessly.

  • Component creation & editing - Generate new pipeline components or modify existing ones with natural language
  • Smart updates & recommendations - Receive intelligent suggestions for pipeline optimization, performance improvements, and descriptive commit messages
  • Automated documentation - Automatically generate and maintain comprehensive documentation for pipelines and components Otto AI

🌟 Improvements​

This section highlights enhancements in component functionality, connector improvements, and overall system optimization to boost performance and usability.

Component improvements​

Data types

  • Timestamp TZ/NTZ - Enhanced timezone handling and support
  • Variant - Flexible data type for semi-structured data
  • JSON - Native JSON data type support

Connector improvements​

Read connectors

  • Automatic schema field detection - Intelligent schema inference for all connectors
  • Customizable schema options - Flexible schema configuration options

Files

  • Advanced filtering - Filter files by last modified, created at, and custom combinations
  • Archive support - Native support for zip & tar archives

Warehouses

  • Enhanced data ingestion - Moving beyond single-table limitations, you can now use SQL queries to filter columns, rows, and join datasets at the source
  • Multi-table support - Easily ingest multiple tables in a single component without complex query writing

Databases

  • Replication strategies - Advanced options for data replication
  • Materialization strategies - Flexible approaches to data materialization

🗣️ Community updates​

Discover ready-to-use examples, comprehensive documentation, and resources created by and for the Ascend community to accelerate your development journey.

Sample projects​

🐐 Otto's Expeditions - Ready-to-use examples for:

Documentation​

New content structure

💬 Terminology changes​

We've updated some terms to better reflect their functionality:

  • Dataflow ➡️ Flow
  • Partitioned component ➡️ Smart table

🔮 Coming soon​

Stay tuned for upcoming innovations, including enhanced AI tools and comprehensive documentation improvements that will streamline your workflow.

✨ Expanded AI capabilities​

  • Coding copilot - Intelligent code suggestions and completions
  • Agentic data engineering - Automated pipeline creation and optimization
  • AI-assisted migration - Use Otto to migrate from legacy data tooling like dbt or Airflow with customizable AI agents

🏢 Enterprise​

  • Organization management - Hierarchical team structures with flexible resource sharing and access controls
  • Enterprise identity & access - Fine-grained access and permission controls
  • External vault integration - Connect to your organization's existing secret management system
Last updated on