Building data pipelines

Learn how to build and structure your data pipelines using Ascend's Components, Transforms, Tests, Tasks, and Applications.

Table of Contents (ToC)

• Setup Ascend
  • Profile
  • Workspace
  • Configure dynamic pipelines with parameters
  • Manage custom package dependencies
  • Flow runner size
  • Ignore files with .ascendignore
  • Nested subdirectories
  • Secrets
• Build Transforms
  • Build PySpark Transforms
    • Simple PySpark Transform
    • Incremental PySpark Transform
    • Smart PySpark Transform
  • Build Python Transforms
    • Simple Python Transform
    • Incremental Python Transform
    • Smart Python Transform
  • Build Snowpark Transforms
    • Simple Snowpark Transform
    • Incremental Snowpark Transform
    • Smart Snowpark Transform
  • Build SQL Transforms
    • Simple SQL Transform
    • Incremental SQL Transform
    • Smart SQL Transform
    • Macros
  • Custom BigQuery partitioning and clustering
• Test Components
  • Test Python
  • Test SQL
  • Test YAML
• Build Task Components
  • Python Task
  • SQL Task
• Application
  • Build a templatized data pipeline
  • Build a Custom Application