Proprietary system — no public code. This was built inside Methanex Corporation for a regulated compliance workflow, so there is no repository to link. The absence of a code link here is deliberate, not an oversight. Everything below is described at the level of architecture and method, without disclosing internal data or controls.
The problem
Before a Responsible Care audit, the compliance team spent roughly four weeks assembling evidence — pulling procedures, incident records, training logs, and prior findings from systems that don't talk to each other, then cross-referencing them against the audit protocol by hand. That preparation window was the bottleneck: it was slow, it pulled senior people off other work, and any gap discovered late left no time to remediate before the auditor arrived.
Constraints
The constraints, not the model, defined this project.
- Regulatory. Responsible Care audits carry real consequences. A compliance auditor has to be willing to put their name on the output, which means the system's answers have to be traceable back to source documents — no unattributed claims, no confident hallucinations presented as fact.
- Data. Evidence lived across siloed enterprise systems with different access rules. The system needed federated read access — reaching into each source under existing permissions — rather than a copied-out data lake that would create a new governance problem.
- Stakeholder. The users are auditors and compliance staff, not engineers. Trust had to be earned in their language: every answer needed to show its work and let them interrogate it, or they would (correctly) fall back to doing it by hand.
- Non-determinism. The hard part of the whole project: an LLM-driven agent gives different answers on different runs. A workflow that a regulator signs off on cannot behave that way without controls around it.
Approach
First attempt — a single capable agent with tools. The obvious starting point was one orchestrating agent with access to every data source and a large tool set. It worked in demos and fell apart in practice: with 200+ tools and the full protocol in one context, the agent lost the plot on long tasks, mixed up which source a fact came from, and produced work that couldn't be audited step by step. A single context window is the wrong place to hold an entire regulated workflow.
Second attempt — decomposition into specialized sub-agents. I broke the workflow into 22 specialized sub-agents, each owning a narrow slice of the audit (a protocol section, a data domain, a document type) with only the tools and retrieval scope that slice requires. A lead orchestrator runs a Plan-and-Execute pattern: it decomposes the audit request into an explicit plan, dispatches steps to the relevant sub-agents, and composes their outputs — rather than improvising the whole thing token by token.
The evidence layer — where the real engineering was. An agent is only as trustworthy as the ground truth it retrieves, so most of the work sat below the agents:
- Ingestion / crawling. I crawled the siloed source systems — procedures, incident records, training logs, prior findings — and pulled them into a unified corpus instead of leaving 22 agents to reach into 22 systems live.
- Databricks pipelines. Normalization, chunking, and transformation ran in Databricks, turning inconsistent documents into a clean, queryable evidence store with the lineage a regulated setting needs.
- Azure AI Search. Retrieval runs through hybrid (keyword + vector) search so each sub-agent pulls grounded, permission-scoped evidence rather than free-associating — retrieval quality, not prompt cleverness, is what makes the answers defensible.
- Python orchestration. The orchestrator, sub-agents, and eval harness are Python, deployed inside the enterprise's existing Azure identity, data-governance, and security boundaries — no net-new attack surface to get approved.
Making it trustworthy. The reliability work sat around the agents, not inside the prompts:
- Observability on every run — the plan, which sub-agent was invoked, which evidence it retrieved, and what it returned — so any answer could be traced end to end.
- Attribution by construction — outputs cite the underlying evidence, so a compliance auditor reviews sources, not just prose.
- Conversational follow-up — auditors can push back ("show me the incident records behind this") and drill into any step, keeping a human in the loop on judgment calls.
- Bounded scope per agent — narrow tool and retrieval surfaces sharply cut the ways any one agent could go wrong.
Evaluation
For an agentic system in a regulated setting, "is the model accurate?" is the wrong question. The real question is "will the compliance team stake their sign-off on it?" I evaluated on three levels.
Outcome. Pre-audit preparation went from ~4 weeks to ~15 minutes for the covered scope — not a benchmark number but the metric the business actually cares about. More important than the speed: the system was validated in use by the audit team it was built for, not just demoed. Adoption by the intended users, on real audit prep, is the strongest signal an enterprise system gets.
Reliability under non-determinism. Because agent behavior varies run to run, I evaluated it the way you'd evaluate a flaky distributed system, not a static model. I ran repeated executions on the same audit inputs and checked for consistency — does the plan stay stable, do sub-agents pull the same evidence, does the composed answer hold — and I traced divergences through the observability layer to the specific sub-agent and tool call responsible. Decomposition helped here: a narrow agent with a bounded task has far less room to drift than one agent holding the whole protocol.
Traceability. The acceptance test that mattered most was human: could an auditor follow any conclusion back to its source documents and agree with it? That's what turned a non-deterministic system into something a compliance professional would put their name on.
What the metrics don't capture. "4 weeks → 15 min" is honest but incomplete. It covers the scoped workflow, not every corner case an audit can surface, and it front-loads work into building and maintaining 22 agents plus their eval harness. The system compresses preparation; it does not replace the auditor's judgment, and it shouldn't. And a time-savings number says nothing about the failure modes that matter most — a confidently wrong answer is worse than a slow one, which is exactly why attribution and traceability, not latency, were the real deliverables.
What I'd do differently
- Build the evaluation harness before the agents. I treated systematic consistency-testing as something I bolted on once behavior looked good. For a non-deterministic system in a regulated domain, the eval harness is the product — I'd write it first and let it drive the decomposition.
- Fewer, broader agents where I could. 22 specialized agents gave clean bounded scopes but real orchestration and maintenance overhead. Some sub-agents were split more finely than the reliability payoff justified; I'd consolidate the ones whose failure modes never actually diverged.
- Formalize a regression suite of "known-good" audits. I validated consistency by re-running inputs manually. A curated set of audit scenarios with expected evidence trails, run on every prompt or tool change, would catch drift before an auditor ever sees it — the difference between trusting the system once and trusting it every release.