Ingest, extract, load, & manage data

Learn how to ingest data from various sources and write it to destinations using Read and Write Components.

Table of Contents (ToC)

• Build Read Components
  • Build Python Read Components
    • Simple Python Read
    • Incremental Python Read
    • Smart Python Read
  • Read from Database Sources
    • Incremental Microsoft SQL Server Read
    • Incremental MySQL Read
    • Incremental Oracle Read
    • Incremental PostgreSQL Read
  • Read from Object Store Sources
    • Read from ABFS
    • Read from GCS
    • Read from S3
    • Read from local files
    • Expose original filename and timestamp metadata in Read Components
  • Connections
  • Data Plane tables
• Data Planes
  • BigQuery
  • Databricks
  • DuckDB Data Plane with DuckLake catalog
    • Ascend-managed DuckLake
    • Self-hosted DuckLake
  • MotherDuck
  • Snowflake
• Build Write Components
  • Write to Data Plane Destinations
    • Write to BigQuery
    • Write to Snowflake
  • Write to database destinations
    • Write to MySQL
    • Write to Oracle
    • Write to PostgreSQL
  • Write to object store destinations
    • Write to ABFS
    • Write to GCS
    • Write to S3