📢 What's new

reminder

If you want help trying our latest features, ask Otto, your intelligent data engineering agent!

🗓️ Week of 2025-10-13

🚀 Features

• Discover Ascend's capabilities with our new product overview and welcome video, designed to help you get started with confidence.
• Flow configurations now support Flow runner size overrides, letting you specify custom resource allocation per Flow instead of inheriting from the Deployment or Workspace default settings.
• Flow and Component organization gets a major upgrade with nested subdirectories, complete with collapsible directory views and summary statistics.

🌟 Improvements

• PostgreSQL authentication gets a security boost by switching to pgpass files with automatic IAM token refresh, keeping your credentials fresh and your database connections stable.
• Smart schema is now supported on the Databricks Data Plane, bringing intelligent schema management to more of your data pipelines.
• DuckDB has been upgraded to version 1.4.1, bringing the latest performance enhancements and features to your data processing.

🛠️ Bug fixes

• Before, concurrent DuckDB operations could cause segmentation faults due to missing thread locks with errors like Flow runner process for my_flow/fr-0199d9de-81cb-79f3-96ca-40b9035368aa exited with code -11. Fatal Python error: Segmentation fault. Now, proper locking protects DDL operations, query execution, and batch insertions, keeping your database operations rock solid.
• Previously, Smart Schema components with NULL partitioning versions could fail during concurrent backfills with Invalid partitioning version: None errors. Now, NULL values are gracefully handled as pre-existing Components, preventing workflow interruptions.
• Before, DuckLake Connections on S3 would experience failures after 1 hour due to expired AWS credentials. Now, DuckLake automatically refreshes AWS credentials for S3 Connections every hour through a new credential management system that uses AWS profiles with background thread refresh, ensuring long-running operations continue without interruption.
• Before, the UI had multiple memory leaks from Monaco editors, polling handlers, and stacked window event listeners. Now, proper cleanup ensures your browser stays responsive during long editing sessions.
• Before, prolonged use of the editor tabs in Workspaces could cause memory leaks. We've improved the stability & performance to ensure your browser stays responsive during long editing sessions.
• Before, paused Workspaces would time out trying to load Git status, causing unnecessary 503 errors. Now, paused Workspaces show a friendly message instead of attempting the doomed status check.

🗓️ Week of 2025-10-06

🚀 Features

• 🤖🐐 New Otto capabilities:
  • Otto can now send automated email alerts with optional AI-powered explanations, helping you understand what went wrong and what action to take when your data pipelines hit a snag.
  • Otto's conversation limits expand from 25 to 50 maximum turns per request, reducing scenarios where complex queries run out of interaction capacity before generating complete answers.

🌟 Improvements

• DuckDB now defaults to max_combined_sql_statements=1 when using DuckLake, delivering better performance and more efficient resource utilization.
  • This change prevents memory issues and out-of-memory (OOM) errors that were occurring when combining multiple SQL queries simultaneously, while also addressing CPU usage inefficiencies where processing was limited to 1-2 cores regardless of available resources.
• Build performance soars with up to 67% faster builds for large projects through global Jinja2 template caching, enhanced file I/O optimization, and smarter threading that eliminates unnecessary overhead.
• Otto's Bedrock integration now supports prompt caching for Anthropic models, dramatically reducing costs and latency by reusing common system prompts and user context across requests.
• Flow runner resource allocation becomes more flexible with configurable size overrides, letting you fine-tune CPU, memory, and disk allocation per runner while maintaining consistent resource assignment.

🛠️ Bug fixes

• Before, MCP tool call responses sometimes failed due to serialization issues with complex data types like URLs. Now, proper JSON serialization handles all data types correctly, preventing runtime conversion errors.
• Fixed a segmentation fault in DuckLake by implementing a safer check for the existence of the partitioning version column.

🗓️ Week of 2025-09-29

🚀 Features

• 🤖🐐 New Otto capabilities:
  • Claude Sonnet 4.5 model is now the default LLM powering Otto.
• A delightful guided tour will welcome you to Ascend on your very first visit, showing you all the amazing features waiting to be discovered.

🌟 Improvements

• File upload memory management gets smarter with single-threaded conversion to parquet, creating natural back-pressure to prevent out-of-memory (OOM) crashes.
• DuckDB DDL operations are now properly locked to prevent race conditions when multiple operations run simultaneously with task-threads > 1.
• The Ascend documentation site got a major glow up! ✨ We've sprinkled in some glassmorphic magic, radial gradients, and a whole bunch of other delightful design treats.

🛠️ Bug fixes

• Before, BigQuery Read Components with column casting would throw syntax errors like Syntax error: Expected "<" but got ")" due to missing fully qualified names. Now, these operations use proper table references and work smoothly.
• Before, BigQuery Flows with concurrent components writing to the same table would fail with Transaction is aborted due to concurrent update errors due to BigQuery's recent stricter enforcement of concurrent DML operations. Now, these operations include automatic retry logic with exponential backoff to handle concurrent update errors gracefully and ensure data processing continues without interruption.
• Before, COALESCE expressions in partitioned Data Plane operations could cause SQL type inference issues due to missing explicit type casting for NULL values, potentially leading to errors with strict type-checking database engines like DuckDB 1.4. Now, all COALESCE expressions use proper type casting with explicit CAST(NULL AS {type}) syntax, ensuring consistent and reliable query execution across different database engines.
• Before, adding columns to existing tables on Databricks would fail with `[PARSE_SYNTAX_ERROR] Syntax error at or near 'EXISTS' errors. Now, column addition works correctly across all Data Planes with proper backfill handling.

🗓️ Week of 2025-09-22

🚀 Features

• 🤖🐐 New Otto capabilities:
  • Otto can now use LLMs across providers with Instance-level model management settings, allowing you to enable or disable specific models and set default models across providers. Configure your settings under AI & Models.
  • Otto works with Jinja macros, parameters, Automations, Tasks, and SSH Tunnels with updated rules that focus on agent-specific guidance and lower token usage.
• Smart schema evolution arrives as the new default schema change strategy for Read Components with object storage and Local File Connections on all Data Planes! Your data partitions now store their own innate schemas and intelligently reconcile type differences without copying data every time the output schema changes.
  • This is not a breaking change. Existing Components with these Connection types continue to use the full strategy (previous default). For new Components, smart strategy will be the default.
• Workspace auto-snooze keeps your infrastructure costs in check by automatically pausing inactive Workspaces after a configurable timeout (5-120 minutes), with smart detection of ongoing Flow runs to avoid interrupting active work.
  • Auto-snooze applies to all new Workspaces with a default timeout of 10 minutes.

🌟 Improvements

• Focus on a Component in the build panel to jump to its place in the Flow Graph, or on a Flow to jump to its place in the Super Graph.
• Data Plane Connections now appear in your Flow Connections list, making it easier to access the Connections powering your current Flow. Additionally, you can view the number of Connections being used in a given Flow within the build info panel.
• Lists throughout your Ascend Instance now sort alphabetically for a more organized experience including Deployment, Project, Git branch, Profile, and Environment selectors.

🛠️ Bug fixes

• PostgreSQL Read Components now preserve array and JSON column types instead of converting them to VARCHAR strings. Plus, a new arrays_as_json parameter lets you handle non-mappable array data with ease.
• Before, incremental merger resets would fail with malformed SQL syntax errors like this:

  Parser Error: syntax error at or near '(' DROP TABLE IF EXISTS DuckDbTable(table='postgres__tracking_merge_snapshot_metadata'...)
  • Now, qualified table names generate proper DROP statements across Data Planes, keeping your data transformations humming.

🗓️ Week of 2025-09-15

⚠️ Breaking changes

• GCS and ABFS Connections now use more stable metadata fields (md5Hash, crc32c for GCS; content_md5, etag for ABFS) for fingerprinting to prevent unnecessary re-ingests when storage tiers change. This change requires re-ingesting existing data to adopt the new fingerprinting method.

🌟 Improvements

• Deployment run pages now link directly to individual Flows instead of the Super Graph when exploring builds and errors, making troubleshooting more focused and efficient.
• Flex Code Flow creation forms are streamlined to include only essential fields (name, description, Data Plane Connection, parameters, and defaults), making Flow creation faster and more efficient.
• BigQuery Data Plane replaced custom UDF logic with native IF functions for better performance and cleaner datasets.
• DuckLake on S3 now uses httpfs as the default Connection method, providing improved performance and stability.
• Local file caching support for DuckLake when working with S3, GCS, and Azure Blob Storage, which can significantly improve data access speeds.

🛠️ Bug fixes

• Before, DuckDB's SAFE_STRING macro creation could fail in DuckLake environments, hitting expression depth limits with Max expression depth limit of 1000 exceeded errors. Now, a robust Python UDF fallback ensures consistent data fingerprinting even when the SQL macro throws a tantrum.
• Fixed a critical issue where long-running DuckLake jobs would lose connection to the metadata database, improving job reliability.
• Resolved a bug that prevented DuckDB-based ingestion from working with multiple PostgreSQL Components simultaneously.

🗓️ Week of 2025-09-08

🚀 Features

• Drag files directly from your computer into the file tree to upload them instantly - no need to connect to object storage to work with your own data.

🌟 Improvements

• The Instance status indicator has moved to the right side of the header next to your user menu for a cleaner, more intuitive layout.

🛠️ Bug fixes

• Fixed ascend view connection sample command failing with pyarrow.lib.ArrowInvalid error when PostgreSQL tables contain range type columns (date ranges, time ranges, etc.). The command now converts range types to readable string representations for successful data sampling.
• Incremental reads now properly honor start values during backfill phases, ensuring your data pipelines only process the records they should and preventing duplicate or incorrect data processing.

🗓️ Week of 2025-09-01

🚀 Features

• Database Connections just learned some new authentication moves! MySQL and PostgreSQL databases hosted by AWS can now authenticate using AWS IAM roles.

🛠️ Bug fixes

• Before, Databricks SQL connector compatibility broke due to a renamed API with the following error:

  `AttributeError: module 'databricks.sql.utils' has no attribute 'ResultSetQueueFactory'. Did you mean: 'ThriftResultSetQueueFactory'?``.
  Now, the system gracefully handles both API versions with fallback logic, keeping your Databricks pipelines humming along.
• Before, partition_template functionality wasn't working in blob Write Components, causing data to land in folders with auto-generated UUID names instead of your provided template. Now, datetime templates are properly parsed and interpolated, so your data ends up exactly where you expect it.

🗓️ Week of 2025-08-25

🚀 Features

• 🤖🐐 New Otto capabilities: Otto can now be used as an Action in Automations to intelligently respond to Flow events, analyze complex data patterns, and execute contextual actions across multiple platforms. This breakthrough capability empowers users to create sophisticated Event-driven workflows where Otto actively monitors pipeline health, dynamically adapts responses based on Flow run history, and delivers intelligent Slack notifications like the example below:

  automations/otto-notifications.yaml

  automation:
    name: otto-notifications
    enabled: true
    triggers:
      events:
        - sql_filter: json_extract_string(event, '$.data.flow') = 'transform-demo'
          types:
            - FlowRunError
            - FlowRunSuccess
  
    actions:
      - type: run_otto
        name: run-otto
        config:
          prompt: >-
            # Instructions
  
            You are being invoked at the end of a flow run. Your job is to assess
            changes in flow run behavior, and send a Slack message to keep other users updated.
            You cannot ask for help or clarification; you must do this end-to-end on your own based
            on your own judgement. Your last action should always be to post a message to Slack.
  
            ## Flow run analysis rules
  
            Analyze flow success / failure by fetching the maximum number of flow runs.
  
            1) If the flow run was a success and the previous flow run was also a
            success, send a
                funny message congratulating the team on another smooth run.
  
            2) If the flow run was a success and the previous flow run was not,
            send an exciting
                and congratulatory messages celebrating the flow is now fixed, and congratulating
                whichever user did the fixing (check git history).
  
            3) If the flow run was a failure and the previous flow run was also a
            failure, send a sad
                message imploring somebody to fix it.
  
            4) If the flow run was a failure and the previous flow run was not,
            send a big alert message.
                You should try to identify who broke the flow, and even @ mention them if you can. You
                should also try to analyze the changes and suggest fixes if possible.
  
            Additionally, you should make heavy use of git history and file reading calls to help track
             down details that may be relevant.
  
            ## Slack message and notification
  
            At the end of your analysis, you must post a message to Slack in the appropriate channel.
  
            Your slack post message should:
            - include a link to the flow run:
                https://[runtime.link_url]/flows/[flow_name]/[run_name]
            - be formatted for Slack rendering, which is different than markdown
            - when @ mentioning users, look up their profile so you can reference their actual id

🌟 Improvements

• Application Components can now access Flow parameters during build time, enabling conditional Component generation based on Profile and Flow parameter values.
  • Applications can now make build-time decisions using Flow parameters that were previously only available at runtime, supporting advanced use cases where Component generation must be dynamically controlled based on Flow configuration.
• Enhanced Automation configuration UI prevents layout issues on small screens and added delete buttons to Sensor, Event, and Action cards for better user experience.

🗓️ Week of 2025-08-18

🚀 Features

• Users can now create Ascend-managed Repositories directly within the platform, streamlining the development workflow by eliminating the need for external Git repository setup and streamlining immediate Project development.

⚠️ Breaking changes

• Fixed Oracle connection configuration validation to prevent runtime failures. Added model-level validation to ensure mutually exclusive Oracle Connection parameters (dsn, database, service_name) are handled correctly, providing clear error messages when conflicting parameters are provided instead of failing at runtime.
  • This breaking change requires users with existing Oracle connections using both database and service_name fields to choose one Connection method.

🛠️ Bug fixes

• Fixed landing table duplicate data issue in incremental persist by implementing de-duplication logic to handle potential duplicate records during incremental reads, ensuring pipeline self-healing and preventing UPDATE/MERGE must match at most one source row for each target row errors.

🗓️ Week of 2025-08-11

🚀 Features

• Ascend now supports DuckDB via DuckLake as a Data Plane! Follow our guide to get started.

🌟 Improvements

Database incremental reads now process up to 50% faster by implementing concurrent download and upload operations using threading, reducing processing time for large datasets while also preventing potential disk space errors during data operations.

🛠️ Bug fixes

• Before, users encountered Failed to list resources errors when browsing Connection data that contained variable placeholders (like ${variable_name}) in file paths. Now, users can seamlessly browse and explore Connection data in the explorer interface.
• Before, incremental merge queries failed with syntax errors when column names contained special characters or reserved keywords (like "INTERVAL" in BigQuery). Now, column names are properly quoted in merge queries to prevent runtime errors.
• Before, users experienced job failures and silent schema inconsistencies when running Snowflake DML operations concurrently or when data tables had metadata drift. Now, jobs are protected from Snowflake concurrency limits and schema inconsistencies are caught with clear error messages at the data storage and management layer for partitioned components.

🗓️ Week of 2025-08-04

🌟 Improvements

• Building on last week's settings enhancements, the interface now features a unified design with consistent card styling for improved usability and visual cohesion.

🛠️ Bug fixes

• Fixed PostgreSQL Read Component incremental strategy memory issue - Resolved a bug where PostgreSQL Read Components on any Data Plane using incremental strategies (merge and SCD Type 2) were attempting to load entire tables instead of empty tables during initial runs, which could cause out-of-memory errors. The fix ensures that predicates used in min/max queries are properly applied to data fetching queries, preventing unnecessary full table loads.

🗓️ Week of 2025-07-28

🚀 Features

• Your Flows can now write to PostgreSQL with style! The new PostgreSQL Write Component establishes support for multiple write strategies — snapshot, full, partitioned, and incremental.

🌟 Improvements

• Settings got a makeover! You can now toggle between light and dark mode directly from General settings.

🛠️ Bug fixes

• Fixed an issue where Incremental Read Components with the full-refresh flag enabled would return empty results despite showing success status. Previously, the full-refresh operation only reset the output table but not the metadata table, causing the Component to incorrectly assume all data was already processed. Now, full-refresh properly resets both output and metadata tables, ensuring you get the complete dataset as expected.

🗓️ Week of 2025-07-21

🚀 Features

• 🤖🐐 New Otto capabilities:
  • Otto now provides smart code completion suggestions when you’re developing in the UI, powered by context from rules and files.
  • Otto can now be configured with an Ascend-native Automation to send activity summary emails! This feature keeps you in the loop about all the behind-the-scenes action on your Flow runs without manual monitoring. See our how-to guide for detailed instructions.
• The Files panel just got smarter! You can now drag and drop existing files directly into place, making file management faster and more intuitive.
  • Hold down the option/alt key while dragging files to create copies on the fly.
• The Files panel right-click menu now includes a convenient Duplicate option, letting you clone files with just two clicks.

⚠️ Breaking changes

• The default Write Component strategy changed from Smart (partitioned) to Simple to align with other areas of the product. Although Smart Components are more scalable, they're also more difficult to get started with and can be slower for small, simple datasets.
  • If you were relying on the previous Smart Component default behavior, you can restore it by explicitly setting strategy: partitioned in your Write Component configuration. See our write strategies documentation for more details.

🛠️ Bug fixes

• Previously, the SSH public key input box was appearing for all secret fields, not just SSH-specific ones, causing confusion in forms. Now, the SSH public key input knows its place and appears only for SSH secret fields where it belongs!
• Before, date indicators in the Flow runs timeline view were jumpy and unstable due to CSS issues. Now, we've ensured a smooth and consistent timeline experience with stable positioning.
• Fixed BigQuery Transform queries that failed when referencing newly created columns in the same query (e.g., SELECT '1' as foo, foo as bar). These queries now work by automatically moving column references to a separate CTE.

🗓️ Week of 2025-07-07

🚀 Features

• 🤖🐐 New Otto capabilities:

  • Otto now supports custom agents, letting you create unique personalities and behaviors tailored to your data workflows. Define your agents in markdown, customize their traits, and even select their preferred language for a personalized touch.

    • Tool inclusion for agents supports powerful wildcards: use [category].* for all tools of a specific type, or * for everything, streamlining agent setup. For example:
    otto/agents/custom_agent.md

    ---
    otto:
      agent:
        name: Standup Reporter
        mcp_servers:
          - slack
        model: gpt-4o
        model_settings:
          temperature: 0.1
        tools:
          - "*"
    ---
    
    # Standup Reporter
    
    You are a professional activity reporter specialized in summarizing Ascend platform work.
    
    Your job is to create comprehensive weekly summaries of my Ascend activities and share them with my manager via Slack.
    ...

  • Otto introduces custom rules, enabling you to add special instructions for chat interactions. Your Otto, your rules!

  • Otto now connects to external MCP servers (used in Slack Otto Automations) via simple configuration files, expanding your agents' capabilities.

  • Otto's configuration is more flexible with a new otto.yaml file in a dedicated otto directory, allowing you to configure agents and MCP server access in one centralized place for easy management.

🌟 Improvements

• Write Components for blob storage (ABFS, GCS, S3) are now more adaptable: you can configure the chunk size using the part_file_rows field for change blobs to optimize performance based on your data needs and avoid out-of-memory errors. This example shows a partitioned Write Component with a chunk size of 1000 rows:

write_custom_chunk.yaml

component:
  write:
    connection: write_s3
    input:
      name: my_component
      flow: my_flow
    s3:
      path: /some_other_dir/
      formatter: json
    part_file_rows: 1000

🛠️ Bug fixes

• Previously, database Read Components (PostgreSQL, MySQL, Microsoft SQL Server, Oracle) converted SQL null values to the literal string 'None' when loading query results into PyArrow arrays, which could cause issues in downstream data processing (e.g., treating 'None' as a string instead of a true null). Now, null values are properly preserved in your data.
  • This change affects all pipelines using database Read Components across all Data Planes and applies to any data type that can be null in your source tables.
  • If you previously had to add cleanup steps to handle 'None' string values resulting from this issue, you can now safely remove those from your pipeline.

🗓️ Week of 2025-06-23

🚀 Features

• 🤖🐐 New Otto capabilities: Otto now dazzles with enhanced tool call rendering – see exactly which tools Otto calls, their inputs, and the outputs Otto receives as he works through your request.

🌟 Improvements

• Write Components now deliver significantly improved performance for large uploads through our new chunked blob storage approach:
  • Paths ending with a filename (e.g., .parquet) → single file output (unchanged)
  • Paths ending with a trailing slash (/) → new chunked output with standardized part_<chunkid> format
  • Partition writes now always produce chunked output files
• You can now run Flows from Deployment history pages as long as you're on the latest Project build

🛠️ Bug fixes

• Before, action failures in an Automation would interrupt the entire workflow. Now, individual action failures are isolated, so if one fails, subsequent actions keep humming along, ensuring Automation resilience.

🗓️ Week of 2025-06-16

🚀 Features

• Private Python package repositories are now supported for AWS CodeArtifact and GCP Artifact Registry via pyproject.toml files, enabling installation of internal and proprietary dependencies. See our how-to guide for setup instructions.

🌟 Improvements

• An error is displayed in the Instance header to alert you if your Instance backend is unhealthy:
• Error messages for out-of-memory process terminations such as "Flow runner not found" are now much more enlightening, turning confusion into clarity when things get tight. See an example error message below:

Flow runner process for my_flow/fr-01977f3c-845b-72a1-b225-4327835f8434 exited with code -9.
Exit code -9 usually means the kernel's Out-Of-Memory (OOM) killer terminated the process after it exceeded its memory limit.
Consider increasing the runner size or optimizing the workflow's memory usage.

🛠️ Bug fixes

• Build history calendar days outside the current month are now selectable, so you can revisit any date without resistance.

🗓️ Week of 2025-06-09

🛠️ Bug fixes

• Switching Projects in your Workspace settings now clears out old run history, so you won't see ghost runs from your previous Project haunting your current Workspace .
• The status bar on the Explore tab was showing outdated information and wouldn't refresh properly - now it updates in real-time so you always see current pipeline status.
• Workspace settings URLs were broken and wouldn't take you to the right page - now clicking these links gets you exactly where you need to go.
• Fixed a bug where the size of a Workspace or Deployment was not being respected - now your size settings stick and your Deployments run with the resources you specified.
• "Flow runner not found" errors were vague and hard to debug - now error messages include detailed pod status information so you can quickly identify and fix issues such as OOM, timeout, unexpected states, API/connection errors, and more.
• Flow runners were staying active even when idle, wasting your compute resources - now they automatically shut down when not in use, keeping your costs down and resources available.
• Git operations could hang indefinitely when repositories became unresponsive - now they timeout automatically so you're never left hanging when a repo goes offline.
• Application parameters weren't compatible with Incremental logic in SQL transforms, breaking dynamic data processing - now they work together seamlessly so your Incremental pipelines can use parameterized logic.
• Partition filter and partition value analyzers would interfere with each other during concurrent backfills, causing data processing conflicts - now they run independently so multiple backfills can process different partitions simultaneously without issues.
• Text parser would crash when given string file paths in Arrow and pandas reads, failing your data ingestion - now it handles string paths gracefully so your file loading works reliably regardless of format (path vs. file object).

🗓️ Week of 2025-06-02

🚀 Features

• 🤖🐐 Otto can now pull information about SQL linting via SQLFluff, ensuring your queries are as polished as a shiny new penny.
• Command-K search now lets you zip straight to settings pages, making navigation as swift as a cheetah on roller skates.

🌟 Improvements

• The UI now boasts improved rendering of errors and warnings, making troubleshooting as smooth as a freshly paved road.
• Polars DataFrames are now supported as a valid input/output format in Python transforms, adding more flexibility to your data wrangling toolkit.
• Parquet processing now normalizes column names to avoid any pesky case conflicts.
• Automation sensors now come with timezone support, making scheduling as precise as a Swiss watch.
• Automation failed status is now shown when there is no associated Flow run, keeping you informed at all times.

🛠️ Bug fixes

• Column case sensitivity for Python Read Components on Snowflake is now handled consistently, ensuring your data stays reliable.
• Profile name, Project UUID, and path are now properly persisted into build info, keeping your records accurate and complete.
• Transformations with upstream Incremental Read Components now resolve to the correct table name, keeping your data pipelines on track.
• Unicode and emoji are now handled gracefully, ensuring your text displays beautifully.
• Component/Run state no longer carries over from current builds into historical ones, keeping your build history clean and accurate.
• Timestamps across the UI now include timezone information, ensuring your time references are always precise.
• Record columns can be resized in the Explore view, making data management a breeze.

🗓️ Week of 2025-05-26

🚀 Features

• Your code editor just got smarter! INI, TOML, and SQLFluff files now shine with beautiful syntax highlighting.
• AWS Managed Vault joins the party as your Instance or Environment Vault, giving you more flexibility in secret management.
• 🤖🐐 New Otto capabilities:
  • Otto's validation powers have expanded! Otto can now validate YAML files and test your Connections to catch any issues.
  • Otto's gotten smarter about which tools to bring to the party! He'll automatically have the perfect toolkit ready for whatever environment you're working in (Workspace vs. Deployment)

🌟 Improvements

• Python interfaces in Simple Applications are now streamlined, making it a breeze to migrate your Python Components from regular Flows.
• Connection test errors got a helpful sidekick: a shiny new copy button that makes sharing error details effortless.

🛠️ Bug fixes

• Multi-line comments in SQL and SQL Jinja files now parse correctly and consistently.
• The great .yaml vs .yml debate is over! Both file extensions now work seamlessly without compatibility hiccups.
• Error stack traces were getting a bit verbose and impacting Flow runs, so we optimized them to be more concise and actionable.
• PostgreSQL Connections now handle SSL verification reliably and gracefully manage empty port configurations.
• Clickable error links in Component cards within Deployments are back in action, ready to guide you straight to the problem.
• Arrow keys now work perfectly when renaming files, making the experience feel smooth and polished.

🗓️ Week of 2025-05-19

🚀 Features

• 🤖🐐 Otto's skill set just got a major upgrade! Otto can now:
  • Perform comprehensive Project file health checks, including YAML linting and intelligent fixes
  • Test your Connections and help resolve any issues that arise
  • List and explore all your Connections to help you create Read Components with ease
  • Run entire Flows or individual Components, monitor completion, and help troubleshoot any errors
• Component dependencies broke free from their limitations! You can now add any set of Components as non-data graph dependencies to any other Component, giving you ultimate flexibility.
• Your file browser and tab bar now sport colorful Git status indicators, so you'll always know what's been changed, added, or needs attention.
• Retry logic is now configurable for all Component types, because sometimes persistence pays off.
• Deployment Automations can now take a well-deserved break with the new pause-all functionality.

🌟 Improvements

• The Runs table got a makeover with smarter column widths that actually remember your preferences.
• Automation forms now Flow more smoothly, creating a streamlined experience that just feels right.
• Component build errors became much more helpful, pinpointing exactly which Component is causing trouble while catching a wider range of exceptions.

🛠️ Bug fixes

• Databricks Connections now gracefully reconnect when encountering "Database Invalid session" errors, maintaining stable connectivity.
• File refresh now works as expected, properly updating both cached and open files.
• The repo save button now correctly reflects the current state, staying enabled only when appropriate.
• Branch listing operations are now more reliable and consistent.
• Zooming in on individual nodes in the expanded Application Component graph tab works like a charm again.
• Databricks Connections now handle catalog references more intelligently, preventing unexpected behavior.
• The UI now maintains proper state consistency during rapid save, build, and run operations.
• MySQL Connections with SSL=True now work correctly without throwing exceptions.
• Table references now use their full, proper names in merge operations.
• Build failures caused by out-of-memory conditions are now properly detected and handled gracefully.
• Empty files no longer cause Project builds to fail unexpectedly.

Agentic Data Engineering

Ascend is the industry's first Agentic Data Engineering platform, empowering teams to build and manage data pipelines faster, safely, and at scale. With Ascend's platform, engineers benefit from the assistance of context-aware AI agents that deeply understand their data pipelines.

Meet Otto, the intelligent data engineering agent designed to eliminate repetitive tasks, accelerate innovation, and enable faster development cycles.

Integration with Ascend platform

Otto works seamlessly with other aspects of the Ascend platform:

• Chat with your stack: Engage in natural language conversations with Otto about your entire data infrastructure. Ask questions about data lineage, Component configurations, or pipeline performance, and receive contextual answers that incorporate knowledge of your specific environment.

• In-line code suggestions: Receive intelligent recommendations as you write SQL, Python, or YAML. Otto analyzes your code patterns, data structures, and pipeline context to suggest optimized transformations, efficient joins, and best practices for your specific data plane.

• Background agents: Leverage autonomous agents that continuously monitor your data pipelines, detect anomalies, and proactively suggest optimizations. These agents work silently in the background, identifying performance bottlenecks, data quality issues, and optimization opportunities without manual intervention.

• Custom agents (coming soon): Create specialized AI assistants tailored to your organization's unique needs. Configure agents with specific business logic, data domain expertise, and compliance requirements to automate complex tasks across your data engineering workflows.

By understanding the relationships between these elements, Otto provides contextual assistance that considers your entire data engineering environment.

With Otto, you can:

• Understand data lineage across your entire pipeline with column-level tracing
• Transforms between frameworks with automatic code migration
• Implement robust data quality tests with intelligent recommendations

Discover these capabilities and many more!

➡️ Ready to agentify your data engineering experience? Schedule a demo to see Ascend in action.

Ascend Gen3

☁️ Gen3 is a ground-up rebuild of the Ascend platform, designed to give you more control, greater scalability, and deeper visibility across your data workflows. It's everything you already love about Ascend – now faster, more flexible, and more extensible.

• Ascend's new Intelligence Core combines metadata, automation, and AI in a layered architecture, empowering all teams to build pipelines faster and significantly compress processing times.

• Git-native workflows bring version control, collaboration, and CI/CD alignment to all teams through our Flex Code architecture— empowering both low-code users and developers to contribute.

• Observability features expose detailed pipeline metadata so teams have deeper visibility into their system to diagnose problems quickly, reduce manual investigation, and optimize system behavior.

• Modular architecture empowers data and analytics teams to manage increasingly large and complex pipelines with improved performance and maintainability.

• Standardized plugins and extension points enable data platform teams to customize and automate workflows more easily.

➡️ Ready to explore? Join the Gen3 public preview to get early access.

🚀 Features

Explore the latest enhancements across our platform, from improved system architectures to optimized Project management. This section highlights major new functionalities designed to boost performance and flexibility.

Systems & architecture

Explore the foundational improvements in our system's architecture, designed to enhance collaboration, resource management, and cloud efficiency.

• Version control-first design – Collaborate and track changes with Git-native workflows
• Project-based organization – Organize and manage resources with intuitive, Project-centric workflows
• Optimized cloud footprint – Reduce infrastructure usage with centralized UI and a lightweight, scalable backend
• Event-driven core – Trigger custom workflows using system-generated events
• Native Git integration – Automate CI/CD pipelines with built-in support for your Git provider

Project & resource management

Effortlessly create, share, configure, and deploy Projects with streamlined processes, allowing you to spend less time on administration and more on engineering innovation.

• Project structure – Organize and manage your data Projects with improved structure and clarity
• Environments – Configure and maintain development, staging, and production environments with software development best practices
• Parameterized everything – Reuse and adapt pipelines with flexible, comprehensive parameterization
• Deployments – Roll out pipelines consistently across environments with simplified deployment workflows

Industry-leading security

Protect your data and resources with enterprise-grade security features, ensuring comprehensive access control and secrets management across your organization.

• Enterprise-grade authentication – Secure Instances, Projects, and pipelines with OpenID Connect (OIDC)
• Centralized vault system – Manage secrets, credentials, and sensitive configurations securely across your entire platform

Builder experience

Discover how our builder experience enhancements simplify Component creation and improve user interaction with a modern interface.

• Simplified Component spec - Write Components with less boilerplate and more intuitive syntax
• Components:
  • Partition strategies - Flexible data partitioning for optimal performance
  • Data & job deduplication - Intelligent handling of duplicate data and operations
  • Incremental Components - Process only new or changed data efficiently
  • Views - Create and manage virtual tables efficiently
  • Generic tasks - Support for versatile task types, including SQL and Python scripts for complex operations
• Data Applications - Build complex data transformations from simpler reusable building blocks and templates
• Testing support - Test Components easily with built-in sample datasets
• Modern interface - Navigate an intuitive UI designed for improved productivity
• Dark mode - Switch between light and dark themes with enhanced visual comfort and accessibility
• Navigation - Access Projects, Components, and resources through streamlined menus

Data integration

Our data integration improvements ensure seamless connectivity and performance across major platforms, enhancing your data processing capabilities.

Performance improvements across all data planes**:

• 70% reduction in infrastructure costs*
• 4x faster ingestion speed*
• 2x faster runtime execution*
• 10x increase in concurrent processing*

Data Planes: Enhanced connectivity and performance across major platforms.

Snowflake - Full platform integration including:

• SQL support
• Snowpark for advanced data processing
• Complete access to Snowflake Cortex capabilities BigQuery - Comprehensive SQL support including:
• BigQuery SQL integration
• Built-in support for BigQuery AI features

Databricks - Complete lakehouse integration featuring:

• SQL and PySpark support
• Full access to AI/ML models in data pipelines
• Support for both SQL warehouses and clusters
• Unified compute management

* Performance metrics based on comparative analysis between Gen2 and Gen3 platforms

Data quality

Enhance your data quality management with automated checks and customizable validation rules, ensuring data integrity across your Projects.

• Automated quality gates - Validate data within Components, including Read and Write Components
• Reusable rule library - Create and share standardized data quality rules across your organization
• Python-based validation - Write custom data quality checks using familiar Python syntax

Flow management

Optimize your data Flows with advanced planning and execution capabilities, supporting high-frequency and concurrent processing.

• Gen3 Flow planner & optimizer – Improve pipeline performance with intelligence planning and execution
• Flow runs – Manage and monitor individual pipeline executions with enhanced controls
• Concurrent & high-frequency Flow runs – Execute Flows in parallel and at higher frequencies
• Semantic partition protection – Preserve computed results across code changes to avoid unnecessary reprocessing
• Optional Smart backfills – Backfill data flexibly with advanced control over reprocessing.

Automation

Leverage our Automation features to create dynamic workflows triggered by real-time events, enhancing operational efficiency.

• Event-driven extensibility - Automate workflows dynamically based on real-time platform events and triggers
• Customizable event triggers - Create custom Automation triggers including sensors and events

Observability

Gain comprehensive insights into your data operations with real-time and historical observability, ensuring full transparency and control.

• Unified metadata stream & repository - Centralize and track metadata across all pipelines
• Real-time & historical monitoring - Access metadata on pipeline runs and performance history, including:
  • Live monitoring of active pipeline runs
  • Full execution history with smart commit summaries
  • Performance analytics and trend analysis
  • Complete troubleshooting visibility

AI-powered assistant (Otto 🐐)

Experience the power of AI with Otto, our assistant that helps you create, optimize, and document your data pipelines effortlessly.

• Component creation & editing - Generate new pipeline Components or modify existing ones with natural language
• Smart updates & recommendations - Receive intelligent suggestions for pipeline optimization, performance improvements, and descriptive commit messages
• Automated documentation - Automatically generate and maintain comprehensive documentation for pipelines and Components

🌟 Improvements

This section highlights enhancements in Component functionality, connector improvements, and overall system optimization to boost performance and usability.

Component improvements

Data types

• Timestamp TZ/NTZ - Enhanced timezone handling and support
• Variant - Flexible data type for semi-structured data
• JSON - Native JSON data type support

Connector improvements

Read connectors

• Automatic schema field detection - Intelligent schema inference for all connectors
• Customizable schema options - Flexible schema configuration options

Files

• Advanced filtering - Filter files by last modified, created at, and custom combinations
• Archive support - Native support for zip & tar archives

Warehouses

• Enhanced data ingestion - Moving beyond single-table limitations, you can now use SQL queries to filter columns, rows, and join datasets at the source
• Multi-table support - Easily ingest multiple tables in a single Component without complex query writing

Databases

• Replication strategies - Advanced options for data replication
• Materialization strategies - Flexible approaches to data materialization

🗣️ Community updates

Discover ready-to-use examples, comprehensive documentation, and resources created by and for the Ascend community to accelerate your development journey.

Sample Projects

🐐 Otto's Expeditions - Ready-to-use examples for:

• BigQuery data plane
• Snowflake data plane
• Databricks data plane

Documentation

New content structure

• Quickstarts - Get up and running quickly
• Concepts - Learn core Ascend concepts
• Reference - Comprehensive API and Component documentation
• How-to guides - Set up Projects across various Git platforms

💬 Terminology changes

We've updated some terms to better reflect their functionality:

• Dataflow ➡️ Flow
• Partitioned Component ➡️ Smart table

🔮 Coming soon

Stay tuned for upcoming innovations, including enhanced AI tools and comprehensive documentation improvements that will streamline your workflow.

✨ Expanded AI capabilities

• Coding copilot - Intelligent code suggestions and completions
• Agentic Data Engineering - Automated pipeline creation and optimization
• AI-assisted migration - Use Otto to migrate from legacy data tooling like dbt or Airflow with customizable AI agents

🏢 Enterprise

• Organization management - Hierarchical team structures with flexible resource sharing and access controls
• Enterprise identity & access - Fine-grained access and permission controls
• External vault integration - Connect to your organization's existing secret management system