Case Study — Document Intelligence
CovenAce
An AI-powered covenant management platform that autonomously extracts and validates financial covenants from legal credit agreements — turning slow, manual compliance review into a scalable, self-correcting pipeline.
- Role
- Lead AI Engineer
- Duration
- Feb 2025 – Mar 2026
- Status
- In Production
- Recognition
- Financial Express Award
Project Overview
CovenAce automates the extraction and ongoing monitoring of financial covenants — the conditions embedded in credit agreements that borrowers must satisfy. It is used by financial institutions to replace line-by-line manual review of legal documents with an automated, auditable extraction pipeline.
The difficulty is that covenants are expressed in dense, highly variable legal prose. The same obligation can be phrased a dozen ways across a dozen agreements, definitions cross-reference each other, and a missed or misread clause has real compliance consequences. The system has to be both accurate and consistent across that variability.
Why it's difficult
The Engineering Problem
Extracting covenants is an information-extraction problem under adversarial variability: unstructured, inconsistent legal text; entities that only make sense relative to definitions elsewhere in the document; and a low tolerance for both false negatives (missed obligations) and hallucinated content (fabricated obligations). A naive prompt-and-parse approach produces plausible output that silently varies between runs and between document formats.
The real engineering problem was building an extraction system that is accurate on messy input, consistent across document types, verifiable against the source, and fast enough to monitor a live portfolio — rather than a one-shot model call that is none of those things at scale.
My Role
What I owned on this project:
- Architected the end-to-end multi-agent extraction platform — from document ingestion through structured, validated covenant output.
- Designed the self-correcting validation system and the parallel agent architecture that made high-volume processing feasible.
- Built the prompt-optimization framework with aspect-based feedback that systematically improved extraction quality over time.
- Worked directly with business stakeholders to understand the nuance of legal extraction and translate it into system requirements and evaluation criteria.
Organized by engineering problem
Technical Contributions
Accurate extraction under variability
Problem
Covenant language varies enormously across credit agreements, and misclassifying an entity has compliance consequences.
Approach
Built a multi-agent extraction pipeline with a self-correcting validation stage that re-checks entities rather than trusting a single pass.
Outcome
94.47% accuracy and 0.93 recall in entity classification, at 98% consistency across document types.
Throughput for live monitoring
Problem
Sequential extraction over long legal documents was far too slow to monitor a portfolio in near real time.
Approach
Re-architected the inference pipeline around parallel agents, batch processing, and caching of repeated work.
Outcome
Reduced extraction latency by 97% — from roughly 5 hours to about 10 minutes — enabling real-time compliance monitoring.
Systematic quality improvement
Problem
Static prompts plateaued: they handled common phrasings but failed on the long tail of legal edge cases.
Approach
Designed a prompt-optimization framework driven by aspect-based feedback, targeting specific failure modes rather than rewriting prompts by intuition.
Outcome
Improved extraction F1-score by 12% across diverse covenant types.
Reasoning, not just the result
Engineering Decisions
Why a multi-agent pipeline instead of one extraction call?
Splitting extraction, validation, and correction into distinct stages lets each be measured and improved independently, and creates a point where a wrong extraction can be caught before it becomes an output — something a single call cannot do.
Why a self-correcting validation stage?
In legal extraction the expensive error is the confident-but-wrong one. A dedicated validation pass that re-checks entities against the source trades extra compute for a direct attack on that failure mode.
Why structured outputs?
Downstream compliance monitoring needs machine-actionable data, not prose. Constraining the model to structured output makes results verifiable against the document and usable by the rest of the system.
Why optimize prompts with feedback instead of hand-tuning?
Hand-tuning prompts does not scale across covenant types and regresses silently. Aspect-based feedback turns prompt improvement into a measurable, repeatable loop targeted at the specific aspects that were failing.
High-level, public view
Architecture
Credit Agreement
Unstructured legal document
Ingestion & Parsing
Document structuring & segmentation
Parallel Extraction Agents
Covenant & entity extraction
Self-Correcting Validation
Re-checks entities against source
Aspect-Based Feedback Loop
Targets specific failure modes
Structured Covenant Output
Machine-actionable, verifiable
Compliance Monitoring
Near real-time portfolio view
Challenges & Trade-offs
Challenge
Recall vs. hallucination.
Trade-off
Pushing recall to catch every obligation raises the risk of fabricated ones; the validation stage exists to hold both in tension, at the cost of extra passes over the document.
Challenge
Latency vs. thoroughness.
Trade-off
Parallelism and caching cut latency by 97%, but batching and concurrency added pipeline complexity and made failure handling and cache invalidation harder to reason about.
Challenge
Generalization vs. per-type tuning.
Trade-off
Optimizing prompts per covenant type raised F1, but risks overfitting to seen formats — so improvements were validated for consistency across document types, not just on the types being tuned.
How it was measured
Evaluation
Extraction quality was measured with standard, defensible metrics on a corpus of real credit agreements:
- Accuracy and recall on entity classification across diverse credit agreements.
- Consistency across document types — the metric that matters most for auditability at scale.
- F1 tracked per covenant type to detect where prompt optimization helped and where it regressed.
- Latency treated as a first-class quality metric, since near real-time monitoring was a product requirement.
Impact
- Replaced hours of manual line-by-line covenant review with near real-time automated extraction.
- Enabled continuous compliance monitoring across a portfolio rather than point-in-time review.
- Publicly recognized as a Financial Express award-winning product.
Technologies
Agent Frameworks
Models & Methods
Infrastructure
Languages
Lessons Learned
- In complex domains like LegalTech, static prompting is inadequate — self-correction and feedback loops are what handle the long tail of edge cases.
- Consistency across document types is a harder and more valuable target than peak accuracy on any single type.
- Latency is a product feature: cutting it 97% is what turned point-in-time extraction into continuous monitoring.
- A measurable feedback loop beats prompt intuition — improvements you cannot measure will regress silently.