Scaling Agentic Data Systems
Your pilot worked. The pipelines your ingestion agent monitors have run clean for three weeks, the Slack oncall channel has been quiet since the quality agent went live, and the team has stopped manually checking execution logs. Leadership sees the numbers and asks the obvious question: "How fast can we roll this out across the org?" Agentic data engineering stops being a skunkworks experiment and becomes an operating model the whole organization expects to rely on.
This is where most teams discover that the architecture that worked for 20 pipelines doesn't scale linearly to 200. Context windows that were manageable become expensive. Agents that were well-scoped start interfering with each other. The team that built the first five agents gets pulled into every new deployment because governance doesn't exist yet. The problem isn't that the technology stops working — it's that the operational patterns that made the pilot successful were never designed to scale.
By the end of this module, you will be able to:
- Select a context management pattern (include/summarize/exclude) appropriate for pipeline scale
- Design a specialist agent roster matched to a real pipeline portfolio
- Define a minimal agent config schema with required ownership fields
- Choose between namespace isolation and application-layer filtering for multi-tenant deployments
- Design a living documentation system for agent configs
Three dimensions of scale
Scaling an agentic system means something different from scaling a traditional pipeline. You're not just adding resources — you're managing a growing population of autonomous actors that share infrastructure, coordinate on data, and collectively determine your platform's behavior.
| Dimension | What changes | What you need |
|---|---|---|
| Schema and pipeline complexity | More tables and schemas mean more metadata to reason about; lineage graphs grow; execution logs from longer-running pipelines expand the context the agent needs to diagnose failures | Chunking strategies — breaking large schemas, logs, or documentation into retrieval-sized units so agents load only what is relevant to their task; retrieval-augmented context (pulling relevant information from a search index rather than loading all context upfront); per-pipeline token budgets |
| Pipeline count | Manual per-pipeline agent setup becomes unsustainable; generalist approaches accumulate unnecessary context; specialization decisions can no longer be deferred | Template-based creation; shared context libraries; agent templates; specialization architecture that addresses trade-offs as pipeline count grows |
| Team size | Multi-team governance; permission boundaries; agent sprawl risk increases | Version-controlled agent configs, per-team cost attribution, clear ownership model |
Chunking strategies break large schemas, logs, or pipeline documentation into retrieval-sized units — small enough for an agent to load only the context relevant to its current task.
None of these dimensions scale independently. A team that solves schema complexity without solving governance will accumulate agents without visibility. A team that solves governance without specialization will maintain a growing pool of underperforming generalist agents. All three need to evolve together.
Context window management at scale
At pilot scale, context windows are a design consideration. At production scale across 200 pipelines, they're a cost driver and a reliability risk — and without the right signals, you will not know when context or cost drift is happening until something breaks, which is why fleet (a set of agent instances deployed at scale)-level visibility belongs in the same conversation as context design; see Observability for agentic pipelines for patterns that scale with the fleet. The decisions you make about what to include, summarize, and exclude from agent context (applying the context engineering principles from ADE 201 at fleet scale) become one of the most consequential architecture choices you'll face.
The instinct to add more context is almost always wrong. The discipline is knowing what to include, what to summarize, and what to leave out — and automating that decision.
Three context patterns that work at scale:
| Pattern | What belongs here | Rule |
|---|---|---|
| Include | Last execution logs, current schema state, this pipeline's lineage subgraph | Current, local, directly relevant to this run |
| Summarize | Historical failure patterns — "failed 3× in 30 days, always upstream schema" | One-sentence summary beats 1,000 lines of logs, at a fraction of the cost |
| Exclude | Cross-pipeline metadata, global schema catalogs, docs for uninvolved systems | Irrelevant context costs tokens and buries the relevant signal |
Diagram: The ENTRY node is where the diagnosis task begins; color is only emphasis — the label carries the meaning. Include, summarize, and exclude are the three context buckets feeding that task.
At scale, this pattern needs to be automated. Defining which context belongs in each category — per agent type, per pipeline state — as a reusable template replaces manual context assembly and keeps decisions consistent across a growing fleet.
Sub-agents as a context management strategy
Include/summarize/exclude handles most context pressure — but some tasks are genuinely too complex to fit a well-scoped context window. When that happens, the right move is decomposition: an orchestrating agent breaks the work into subtasks and delegates each to a sub-agent with a fresh, focused context window containing only what's relevant to that piece of the work.
The sub-agent completes its subtask and returns a summary — not its full context — to the orchestrator. This keeps every agent's context window targeted while allowing the overall system to handle complex, multi-step work. It's also one of the few ways to parallelize agent work: independent subtasks can run as concurrent sub-agents rather than sequentially in a single growing context.
Example: An orchestrator running a data quality audit might delegate one sub-agent per schema layer (sources, staging, marts). Each sub-agent works in a fresh, narrow context, inspects only its layer, and returns a one-paragraph finding to the orchestrator, which synthesizes the full audit without loading every table into one window.
The tradeoff is coordination overhead, so sub-agents are worth the complexity when a task's context requirements genuinely exceed what scoped include/summarize/exclude can manage.
Agent specialization
The instinct when scaling is to build more general agents — agents that can handle any pipeline, any task type, any data pattern. Evidence from multi-agent systems research — with emerging parallels in LLM-based agent work — suggests the opposite approach works better: specialist agents tend to outperform generalists on bounded, well-defined tasks — not because they know more, but because their limited scope keeps context targeted and tool selection unambiguous.
Research on specialist vs. generalist agents in cooperative multi-agent systems studies this pattern in multi-agent reinforcement learning (MARL) — environments such as StarCraft-style settings and Overcooked-AI, not LLM data pipelines; Additional Reading summarizes the paper. The AgentOrchestra framework (preprint, not peer-reviewed) reports competitive GAIA benchmark results in the authors' evaluation, using hierarchical coordination with specialist routing rather than a single generalist path. We cite both as analogy for pipeline work: ingestion, quality, and transformation each benefit from deep domain focus rather than breadth.
Hire narrow before you hire wide. If you cannot name the bounded task, you do not yet have a specialist — you have a generalist wearing a costume.
The practical implication: instead of a single "data engineering agent" that handles everything from ingestion debugging to transformation optimization, build a roster of specialists with deep context in their domains.
| Specialist type | Domain | What it knows deeply |
|---|---|---|
| Ingestion agent | Source connections, API parsing, schema negotiation | Source system quirks, retry patterns, historical ingestion failures |
| Quality agent | Data quality rules, anomaly detection, baseline distributions | What "normal" looks like for each dataset, quality rule history |
| Transformation agent | SQL patterns, transformation logic, optimization | Lineage for transformation layer, query performance history |
| Operations agent | Scheduling, orchestration, pipeline health | Pipeline dependency graph, historical execution patterns |
| Schema agent | Schema evolution, contract management, impact assessment | Schema history, downstream consumer registry |
Specialists require more upfront design — you need to define their domains, their context packages, their tool scopes — but research and emerging practice suggest specialists often produce more accurate outputs on bounded tasks — though results depend on task design and agent configuration — because their context windows are more targeted and their focus stays within bounded problem spaces.
Preventing agent sprawl
The governance problem that scales fastest is agent sprawl: the proliferation of agents without clear ownership, oversight, or visibility into what each one does and what it costs.
The solution is the same discipline that works for agent configurations: version control. When every agent's config — owner, pipelines served, cost budget, guardrail settings — lives in a git repo the whole team can see, you get sprawl prevention as a side effect of normal collaborative development. A PR to add a new agent is visible to the team. An ownerless config file fails review. There's no separate system to maintain; the repo is the source of truth.
Governance challenges tend to grow faster than the agent fleet. Establish version-controlled agent configs when you have 5 agents — retrofitting visibility to a deployed fleet is significantly harder than starting with it.
Each agent config should capture enough to answer: who owns this, what does it serve, what does it cost, and how does it behave?
# agents/orders_daily_monitor/config.yaml
agent_id: orders_daily_monitor_v2
owner: data-platform-team
team: engineering
created: 2026-01-15
last_updated: 2026-01-15
pipelines_served:
- orders_daily
- orders_hourly_summary
monthly_cost_usd: 847 # example value — actual costs vary by pipeline complexity
token_budget_per_run: 50000
review_required: true
on_call_escalation: "#data-platform-oncall"
status: production
Ownerless agents are a security and cost risk. A config file in version control — visible to the whole team, reviewable before merge, traceable over time — is the simplest form of governance that actually scales.
Multi-tenancy and isolation
When multiple teams share an agentic platform, isolation becomes a security and governance requirement — one of the same concerns you operationalize when you move from pilot to production workloads; the Production readiness module ties those practices to rollout and operational gates. The question is: how do you ensure that Team A's agents can't access Team B's data, consume Team B's token budget, or affect Team B's pipelines?
The more structurally enforced multi-tenant isolation pattern is namespace-per-tenant — each team or customer gets their own isolated agent context, tool permissions, and cost tracking. Application-layer filtering (a single shared agent that checks tenant IDs at runtime) is faster to build but creates a path for cross-tenant data leakage when business logic changes.
Ascend's workspace, environment, and instance hierarchy provides layered isolation boundaries. For multi-tenant agentic workloads, use separate environments or instances per tenant — this gives you the structural separation the generic namespace pattern describes, without relying on application-layer filtering alone.
Per-team cost attribution follows from isolation. When teams operate in separate namespaces, their token consumption, API costs, and infrastructure usage can be tracked and attributed independently. This isn't just accounting — it creates accountability. Teams that can see their agent costs are more deliberate about token budgets, context scope, and agent proliferation.
Estimated time: 15–20 minutes
Good agent documentation doesn't just describe what an agent does today — it stays accurate as agents evolve, gets maintained alongside the configs it describes, and gives any teammate immediate visibility into what's running and why.
Open Otto in your Ascend workspace (sparkles icon, top bar) and paste the prompt below. If you're not on Ascend, use your preferred assistant.
If you don't have AI assistant access, use the prompts as a written self-assessment. Document your answers in a shared doc or use a colleague's review instead.
I want to create living documentation for the agents in my data platform — documentation that gives any teammate immediate visibility into what agents exist, what each one does, and what it costs, and that stays accurate as we add and refine agents over time.
Help me design this. Specifically:
1. What fields should every agent's documentation include to be useful to a teammate who didn't build it?
2. Where should this documentation live so it stays in sync with the agent configs as they change?
3. How do we keep it from going stale — what's the lightweight process to update it when an agent is modified?
4. What's the right format: a structured file per agent, a shared doc, something else?
What to notice: A strong response will connect documentation location to version control — the most durable answer is that documentation lives next to the config files it describes, not in a separate wiki that drifts. Watch whether the AI surfaces the staleness problem unprompted: documentation that isn't updated on the same PR as the config change is documentation that's wrong by definition. If the suggestions are too abstract, push back and ask for an example of what one agent's documentation file would actually look like.
- Specialization at scale: Research on multi-agent specialization directionally favors specialists for bounded, parallelizable tasks; patterns observed in early deployments align with that bias. The principle is directionally supported; empirical data specifically on LLM pipeline agents is still emerging. Build a roster of domain-specific agents (ingestion, quality, transformation, operations, schema) rather than one generalist that handles everything.
- Version control is your governance layer. Keep every agent config in the same repo your team already uses — owner, pipelines served, cost budget, guardrail settings. Visibility and accountability come for free, and a PR to add a new agent keeps the whole team in the loop.
- Namespace-per-tenant is the more structurally enforced option compared to application-layer filtering alone. Structural isolation reduces the class of cross-team interference bugs that application-layer filtering is supposed to prevent but often does not catch under evolving business logic.
You now have 200 pipelines, 5 specialist agent types, and a governance model. The next challenge is getting them to work together — coordination patterns, compound reliability math, and the failure modes that only emerge when agents need to hand off to each other.
With your agent fleet patterns established, the next challenge is getting them to work together — coordination patterns, compound reliability math, and the failure modes that only emerge when agents need to hand off to each other.
Next: Multi-Agent Orchestration →
Additional Reading
- Specialist vs. generalist agents in cooperative multi-agent systems — MARL research on specialist outperformance when tasks have high parallelizability. The paper's experiments use environments such as StarCraft-style settings, particle simulations, and Overcooked-AI — not LLM data pipelines; this module cites it as an analogy for pipeline specialization.
- AgentOrchestra: multi-agent orchestration framework (preprint, not peer-reviewed) — Multi-agent framework reporting competitive GAIA benchmark results in the authors' evaluation; architecture combines hierarchical coordination with specialist routing (see the paper for full claims and methods).
- Context Engineering for ADE — The ADE 201 module on context engineering principles — the foundation for the include/summarize/exclude patterns applied at fleet scale here.
- Multi-Agent Orchestration — The next module: how to coordinate specialist agents once you've built them.