Custom Solution For Credit Document Analysis

The covenants in a bond indenture set out what a company is allowed to do, and they leave plenty of room to devalue your claim. It's difficult to extract exactly how much a borrower can borrow, how much collateral it can move, or how much cash it can pay out to its owners. Those permissions sit in dozens of separate exceptions: some fixed in dollars, some growing as earnings grow, some sharing a limit with another provision elsewhere in the document. Building a pipeline that solves this requires a domain specific solution with expert written evals to judge efficacy. We gave 89 covenant sections to four LLMs and asked each one for every basket. Our solution found 92%, and the frontier solution found 78%.

For the first evaluation we used 89 covenant sections taken from 56 companies. By section type our questions are split between permitted liens, permitted indebtedness, and restricted payments: what a company can pledge, what it can borrow, and what it can pay out to its owners. The split is chosen to mirror real life requirements.

The models worked from the section text alone, one-shot, with no tools. Four systems ran the full set: our pipeline, Claude Sonnet 5, Claude Opus 5, and Meta Muse Spark 1.2.*

The second evaluation is deliberately narrow. Our domain experts wrote a set of questions of the kind that takes a specialist to answer and a specialist to grade: whether a J.Crew-style transfer is available, or how much room a specific carve-out actually provides. Each one requires finding the right provisions, often across more than one document, and working out how they interact.

All three systems answered the same questions. Tara worked from our structured data, while Claude Sonnet 5 and Claude Opus 5 worked from the credit documents themselves.

* Muse Spark is included because it is the highest-scoring model in Harvey's LAB legal benchmark.

Our domain experts wrote a reference answer for each covenant section. We compare what a system returns against the reference one detail at a time, so we report field recall and field precision rather than marking each basket right or wrong.

We graded the answers by hand against a list of facts each one had to contain. The scores show how many of those facts an answer got, how many of the ones that decide the conclusion it got, and how often it took a known trap, a reading the document invites but does not support. On traps, lower is better.

Constellation led on every measure, and led by most on completeness. Across the 89 sections it recovered more of each one than any model did.

The same held on the questions. Tara runs on the structured data the pipeline produces rather than on the document itself. It got more of the facts right, and more of the required facts that carry the conclusion, including on questions whose answer sits in more than one document.

Finding the right language is not the same as ruling on it. Our domain experts built common mistakes into the questions as 'traps', to see how often a system would take one.

Take the restricted payments covenant of CHS/Community Health Systems' 10.875% senior secured notes due 2032. Section 3.3, together with the Permitted Investments definition it leans on, sets out 53 separate permissions, each one a way the company is allowed to move value out to shareholders and affiliates. Twelve of them carry a number: a dollar cap, a percentage, or a leverage test.

Constellation found all twelve. The $110 million a year for buying back employee equity, the $65 million basket for loans to management, the general basket at the greater of $375 million and 3% of total assets, the 3.5x leverage test.

The frontier models failed differently. Each listed the clauses and then left the money off. Claude Sonnet 5 returned five of the twelve with no capacity at all, the $65 million management basket among them. Claude Opus 5 left five blank and attached terms to the $110 million basket that the clause does not contain. Muse Spark dropped or blanked eight, including the investment basket at the greater of $750 million and 5% of total assets. A table with the right clause numbers and empty capacity columns reads as diligent. It is the opposite. The reader is left to find the numbers themselves, without knowing which ones are missing.

Work like this needs a bespoke solution, not a frontier model. It needs domain experts who build the evaluations that can test whether the solution actually works. We built both for credit documents.