Case Study — Agentic AI & Orchestration
ANA
An audit-ready control-testing automation platform for regulated finance. Given a set of controls and the documents required to test them, ANA runs the multi-step control-testing procedure through a hybrid multi-agent system and produces audit-ready output.
- Role
- Lead AI Engineer
- Team
- AI Product Team of 8
- Status
- In Production
- Domain
- Enterprise Risk (BFSI)
Project Overview
ANA is an audit-ready control-testing automation solution. It takes a set of controls, along with the various documents required to test them, runs the multi-step control-testing procedure, and produces audit-ready output at the end.
Because it operates in BFSI, ANA has to be more than an automation: every result has to be instantly explainable and traceable, decisions have to stay with humans, and the system has to handle documents and data far larger than a model can read directly — all while staying responsive across long conversations.
Why it's difficult
The Engineering Problem
Automating control testing surfaced several hard problems at once. The documents involved can carry millions of entries — data populations far too large to pass into an LLM directly. Sessions are long, so the conversation contextbecomes hard to manage as it grows. The multi-step, multi-agent flow is inherently slow. And because this is BFSI, the whole system must be instantly explainable and traceable, with humans owning the decisions rather than the AI.
The engineering task was to build a control-testing system that handles data and context at that scale, stays fast enough to use, keeps humans in control, and can be audited step by step.
My Role
What I owned on this project:
- Architected ANA's multi-agent system — a hybrid supervisor-plus-hierarchical topology that decomposes control testing into 7 sequential steps and 17+ agents, with actor–critic validation inside each step.
- Solved the large-data and large-context problems: the Python executor for data populations and context comparison for long conversations.
- Owned latency (via a gradient cache) and the human-in-the-loop checkpoints that keep decisions with people.
- Built the evaluation and monitoring approach on Langfuse, including custom deterministic metrics and LLM evaluators, and coordinated a product team of eight engineers.
Organized by engineering problem
Technical Contributions
Large data-population handling
Problem
The documents can contain millions of entries — a data population far too large to pass into an LLM directly.
Approach
Rather than sending the data to the model, a Python executor handles the data population and passes the model information about the data, so the LLM can reason over it effectively.
Outcome
The system reasons over massive data populations without overwhelming the model's context.
Large context & long conversations
Problem
Over long sessions, the conversation context grows and becomes very hard to manage.
Approach
Used context comparison to retrieve the relevant data from previous chats, rather than carrying the entire history forward.
Outcome
Coherent behavior across long conversations without unbounded context growth.
Latency
Problem
A multi-step flow across many agents is inherently slow, and latency was a major problem.
Approach
Introduced a gradient cache to cut the latency of the flow.
Outcome
A control-testing flow responsive enough for real use.
In-step AI validation (actor–critic)
Problem
Each of the control-testing steps can produce a wrong result, and an error early in the flow propagates to the audit-ready output.
Approach
Structured each of the 7 steps as actor–critic sequential agents: an actor performs the step, and a critic validates its output before the flow moves on.
Outcome
Errors are caught inside the step that produced them, rather than surfacing at the end.
Human-in-the-loop decisions
Problem
In BFSI, the AI should not be the one making the decision — a human should.
Approach
Placed human-in-the-loop checkpoints at every point where a decision is made, so the AI proposes and the human decides.
Outcome
Automation that accelerates the work while keeping decision accountability with people.
Explainability & traceability
Problem
Operating in BFSI, ANA must be instantly explainable and traceable.
Approach
Instrumented the system with agent monitoring through Langfuse, so each agent's behavior can be inspected and traced.
Outcome
A control-testing pipeline whose every step can be explained and audited.
Reasoning, not just the result
Engineering Decisions
Why decompose control testing into 7 steps and 17+ agents?
The control-testing procedure has 7 distinct steps. Decomposing it into 7 steps and 17+ agents lets each step be built, monitored, and reasoned about on its own, which is what makes the whole flow explainable and traceable.
Why a hybrid supervisor + hierarchical architecture?
A supervisor coordinates the 7-step flow, while the agents reporting to it are workflow agents that run sequentially. Combining a supervisor with this hierarchical, sequential reporting gives a hybrid topology — coordinated at the top, ordered underneath.
Why actor–critic agents inside each step?
Validation belongs where the work happens. Making each step an actor that produces a result and a critic that checks it means AI validation is built into every step, so a wrong result is caught in place instead of contaminating the final audit output.
Why a Python executor for large data instead of feeding the LLM directly?
Data populations of millions of entries cannot fit in an LLM's context. A deterministic Python executor handles the data and hands the model information about it, so the LLM reasons over the data without ever being flooded by it.
Why human-in-the-loop at every decision checkpoint?
In a regulated setting the AI should not own the decision. Checkpoints keep a human in the loop wherever a decision is made, which is a requirement, not an option.
Why build a dedicated evaluation system?
Agent output in BFSI has to be measured, not assumed. A purpose-built evaluation system — custom deterministic metrics plus LLM evaluators, grounded in the data and its traceability — is what makes quality defensible.
High-level, public view
Architecture
Controls + Required Documents
Inputs for a control test
Supervisor Agent
Coordinates the 7-step control-testing flow
7 Sequential Steps · Actor → Critic
Each step: an actor produces, a critic validates (17+ agents total)
Python Executor
Handles million-entry data populations
Human-in-the-Loop Checkpoints
Human owns each decision
Audit-Ready Output
Explainable & traceable result
Challenges & Trade-offs
Challenge
Massive data vs. the LLM's limits.
Trade-off
Offloading million-entry data populations to a deterministic Python executor made the data tractable, but added a component the model depends on rather than reasoning end-to-end itself.
Challenge
Depth of the agent flow vs. latency.
Trade-off
A 17+-agent flow is inherently slow; a gradient cache recovered responsiveness, at the cost of the caching machinery it introduced.
Challenge
Automation vs. human control.
Trade-off
Human-in-the-loop checkpoints mean ANA does not decide autonomously — trading some throughput for the decision accountability BFSI demands.
How it was assessed
Evaluation
Evaluating the agents was a problem in its own right, so I built a dedicated evaluation system on Langfuse:
- Grounded evaluation in the underlying data and its traceability.
- Defined a set of custom deterministic evaluation metrics for the outputs.
- Added LLM evaluators to assess ANA's output alongside the deterministic metrics.
- Ran monitoring and evaluation over the pipeline through Langfuse.
Impact
- Automates the multi-step control-testing procedure end to end, producing audit-ready output.
- Keeps output instantly explainable and traceable, as BFSI requires.
- Keeps humans in control of decisions through in-the-loop checkpoints.
- One of the AI products delivered for the BFSI sector under my technical lead.
Technologies
Agent Architecture
Data & Context
Observability & Evaluation
Control
Lessons Learned
- Large data populations belong in a deterministic executor, not the LLM context — the model should reason over a description of the data, not millions of raw entries.
- In BFSI, explainability and traceability are hard requirements, met through agent monitoring rather than added after the fact.
- Decisions should stay with humans; checkpoints keep accountability where it belongs.
- Evaluating agents needs its own system — deterministic metrics plus LLM evaluators, grounded in the data and its traceability.
- Validation works best in place: actor–critic agents inside each step catch errors where they happen, before they reach the audit output.
- A hybrid supervisor-plus-hierarchical topology lets a complex, 7-step flow be decomposed into sequential, monitorable agents.