Read from File system Sources
File system Read Components connect to cloud storage services and file systems to ingest data into your pipelines. They support various file formats and offer features like partitioned reading, metadata extraction, and flexible file filtering to efficiently bring data from storage into your Data Plane.
Before you begin
- Ensure you have an Ascend Project and Workspace
- Set up the appropriate file system Connection for your storage provider
File system Read guides
📦 Amazon S3
Read data from S3 buckets with support for various file formats and partitioned data
☁️ Google Cloud Storage
Ingest data from GCS buckets with flexible file filtering and format detection
🔷 Azure Blob Storage
Connect to Azure Blob containers and read data with automatic parsing
📁 Local Files
Read data from local file systems for development and testing
📊 File Metadata
Extract and work with file metadata for advanced data processing
Next steps
After setting up file system reads:
- 🔄 Transform your data with SQL or Python
- 📤 Write data to destinations
- 🧪 Add data tests to validate data quality
- 🔄 Set up Automations to process files automatically