🛠️ How-to guides

Ascend's how-to guides.

Table of Contents (ToC)

• Agentic Data Engineering
  • Otto
    • Custom agents
    • Configure MCP servers
    • Otto configuration
    • Otto Automations
    • Custom rules
    • Automate pipeline activity summaries
    • Otto prompts
• Infrastructure & security
  • Repository
    • Ascend-managed Repository
    • Manual Repository
    • GitHub
    • Bitbucket
    • GitLab
    • Project
    • Environment identity information
  • AI providers
    • Ascend-managed Bedrock
    • Azure OpenAI
    • OpenAI API key
  • Vaults & secrets
    • Configure Environment Vaults
      • AWS Secrets Manager
      • Azure Key Vault
      • Google Cloud Secret Manager
    • Configure Instance Vaults
      • AWS Secrets Manager
      • Azure Key Vault
      • Google Cloud Secret Manager
    • Provision Vaults
      • AWS Secrets Manager
      • Google Cloud Secret Manager
      • Provision Azure Key Vault
  • Connections
    • Databases
      • Microsoft SQL Server
      • MySQL
      • Oracle
      • PostgreSQL
    • Local File
    • Object stores
      • ABFS
      • GCS
      • S3
    • Sample Connection
• Ingest, extract, load, & manage data
  • Build Read Components
    • Build Python Read Components
      • Simple Python Read
      • Incremental Python Read
      • Smart Python Read
    • Read from Database Sources
      • Incremental Microsoft SQL Server Read
      • Incremental MySQL Read
      • Incremental Oracle Read
      • Incremental PostgreSQL Read
    • Read from Object Store Sources
      • Read from ABFS
      • Read from GCS
      • Read from S3
      • Read from local files
      • Expose original filename and timestamp metadata in Read Components
    • Connections
    • Data Plane tables
  • Data Planes
    • BigQuery
    • Databricks
    • DuckDB Data Plane with DuckLake catalog
      • Ascend-managed DuckLake
      • Self-hosted DuckLake
    • MotherDuck
    • Snowflake
  • Build Write Components
    • Write to Data Plane Destinations
      • Write to BigQuery
      • Write to Snowflake
    • Write to database destinations
      • Write to MySQL
      • Write to Oracle
      • Write to PostgreSQL
    • Write to object store destinations
      • Write to ABFS
      • Write to GCS
      • Write to S3
• Building data pipelines
  • Setup Ascend
    • Profile
    • Workspace
    • Configure dynamic pipelines with parameters
    • Manage custom package dependencies
    • Flow runner size
    • Ignore files with .ascendignore
    • Nested subdirectories
    • Secrets
  • Build Transforms
    • Build PySpark Transforms
      • Simple PySpark Transform
      • Incremental PySpark Transform
      • Smart PySpark Transform
    • Build Python Transforms
      • Simple Python Transform
      • Incremental Python Transform
      • Smart Python Transform
    • Build Snowpark Transforms
      • Simple Snowpark Transform
      • Incremental Snowpark Transform
      • Smart Snowpark Transform
    • Build SQL Transforms
      • Simple SQL Transform
      • Incremental SQL Transform
      • Smart SQL Transform
      • Macros
    • Custom BigQuery partitioning and clustering
  • Test Components
    • Test Python
    • Test SQL
    • Test YAML
  • Build Task Components
    • Python Task
    • SQL Task
  • Application
    • Build a templatized data pipeline
    • Build a Custom Application
• GitOps, deploying, and monitoring data pipelines
  • DataOps best practices
    • Best practices for BigQuery
    • Best practices for Databricks
    • Best practices for Snowflake
    • Power user tips and shortcuts
    • Auto-snooze Workspaces
  • Deploy
• Automation & DataOps
  • Automate your Flows
• Additional platform capabilities
  • Troubleshoot
    • Debug Python in Ascend
    • Debug SQL in Ascend
  • Advanced
    • Cross-Flow references
    • Retry Components on failure
    • Execute Components in order with inputs
    • Concurrency configuration
    • Schema change