Evaluating Jev on legal document classification tasks
We ran TypeSafe's Jev model against one of our custom eval suites (~14.8k tasks). It came back 40x faster and 32x cheaper, a fraction of a point worse on accuracy, and slightly conservative on assigning confidence to its answers.
What is Jev?
Jev is the first in a new class of models (System One Models1) trained with RLCD (Reinforcement Learning for Calibrated Decisions); instead of generating open-ended text, Jev outputs fast, typed, probabilistic options or scores.
1 System One models are a class of AI models built to make fast, structured decisions that software can use directly. A System One model evaluates a state and returns typed answers and probabilities.
To invoke Jev, you send it a state (the context) and one or more questions about that state. In the simplified example below, Jev classifies a debt instrument’s seniority from three choices.
Input
{
"model": "jev-latest",
"state": "The Notes will be senior unsecured obligations and will rank equally with all existing and future senior unsecured indebtedness.",
"questions": {
"debt_seniority": {
"type": "choice",
"instructions": "Classify the seniority and security of the Notes.",
"criteria": {
"senior_secured": "Senior obligations backed by collateral",
"senior_unsecured": "Senior obligations without collateral",
"subordinated": "Obligations ranking below senior debt"
}
}
}
}Output
{
"debt_seniority": {
"type": "choice",
"choice": "senior_unsecured",
"confidence": 0.99,
"probabilities": {
"senior_secured": 0.005,
"senior_unsecured": 0.99,
"subordinated": 0.005
}
}
}Jev correctly selects senior unsecured with 99% confidence and returns probabilities for each alternative, producing a structured result that software can use directly.
The model’s creators aim to align the model’s reported confidence with its observed correctness: a well-calibrated model should be roughly correct 80% of the time across all of the answers it assigns 80% confidence.
The Task
Credit agreements and indentures are typically organised into articles (e.g. Article IV) and Sections (e.g. Section 4.10) that form unique addresses for citing legal text. However, we find that some documents stray from convention, and use their own nomenclature for headings or contain typos and drafting errors.
The first step of Constellation’s production credit agreement pipeline sorts every heading in the document into articles and sections. To do that, each candidate is classified into one of three labels: Article marks a top-level division, Section marks a numbered clause inside it, and Skip covers anything that is not a heading, such as a cross-reference, a table row, a running header, or ordinary contract text. Every candidate gets exactly one label.
Credit documents run hundreds of pages, and each one can define its headings differently. The three images below are excerpts from a publicly-filed Indenture. ARTICLE VII sits on its own line and opens a clear division. Section 7.16 is numbered beneath it and opens a Section. Compare this to the third image, which prints Section 7.16 again inside a definition, where it is only a reference.
Example of a typical credit document
Edge cases
Here are just a few examples of edge cases, where classification can get tricky. Raw documents can deviate from standard patterns, and may even contain unintentional errors. The classifier must be robust to catch both false positives and false negatives.
Examples that break the typical pattern
Evaluation Methodology
The Constellation Lab builds hand-curated eval sets to ensure the highest-quality production AI systems, a task that requires a unique mix of domain and technical expertise. For this task, we curated an eval set of 14,776 candidates for Article and Section headers across 111 credit agreements and indentures.
We use Claude Haiku 4.5 in our production system, which currently scores 100% on our eval suite of 14.8k candidates. In addition to Jev, we tested a number of models: Qwen 3.7 Flash, Qwen 3.7 Plus, Qwen 3.6 Flash and Qwen 3.6 Plus.
Results
Cost versus performance
Accuracy: within half a point of production. Jev labelled 99.5% of the 14,776 candidates correctly. Our production Haiku setup scored 100%, and both Qwen Plus models reached 99.8%, which puts Jev 0.5 points behind production and 0.3 points behind the Plus models. The Flash models fell further back: Qwen 3.6 Flash scored 99.0%, and Qwen 3.7 Flash scored 97.3%.
Overall accuracy
Overall accuracyGetting the whole document right. Candidate-level accuracy tells only part of the story. Our pipeline uses these labels to build a tree of articles and sections, so a single wrong label can throw off a document’s structure. Here the gap widens. Jev produced exactly the right article and section counts for 92.8% of the 111 documents, against 100% for production Haiku, 98.2% for Qwen 3.6 Plus and 97.3% for Qwen 3.7 Plus. The Flash models trailed at 84.7% and 85.6%.
Documents with exact article and section counts
Share of documentsThis is where Jev’s confidence scores become important, as we explain below.
Confidence. A confidence score is only useful if it means what it says. On our data, Jev erred on the side of caution: in every confidence band, Jev was right more often than it claimed to be. Its average confidence was 97.9%, against an actual accuracy of 99.5%, giving an expected calibration error of just 0.016.
Stated confidence against observed accuracy
Most of Jev’s answers came with very high confidence: 81.7% of candidates scored 99% or above, and 94.4% scored 95% or above. More importantly, its mistakes announced themselves. All 70 errors had a confidence below 85%, and none scored higher than 84%.
Distribution of stated confidence
Share of candidatesSpeed: 40x faster. Jev classified a typical document in 0.45 seconds, and 95% of documents finished within 0.78 seconds. Production Haiku took 18.18 seconds per document at the median. The Qwen models ranged from 17 to 50 seconds, with Qwen 3.6 Plus the slowest at 50.3 seconds. For a pipeline that processes documents hundreds of pages long, that’s the difference between waiting and not noticing.
Median seconds to classify one document
Seconds per documentCost: 32x cheaper. The full evaluation cost $0.10 with Jev and $3.14 with production Haiku. Only Qwen 3.7 Flash came in lower, at $0.09, and it was also the least accurate model we tested.
Interestingly, Jev’s savings don’t come from using fewer tokens. Because each question repeats the line of the candidate it asks about, Jev processed around 21,279 input tokens per document, compared with 12,974 for Haiku. The savings come from pricing: Jev charges $0.042 per million input tokens and nothing for output.
Total classification cost
US dollarsWhat this means in production
Being cautious is exactly what you want from a model that decides when to call in a human. Its main cost is a little extra review time on answers that turn out to be correct.
To test this, we set a confidence threshold of 0.80: answers at or above it are accepted automatically, and anything below goes to a human for review. At that threshold, Jev accepted 97.4% of candidates, and 99.98% of those accepted answers were correct. Only 385 candidates went to review, and just three errors slipped through.
At the document level, 76.6% of documents needed no review at all and were entirely correct. Another 21.6% had some candidates flagged for review, and that review caught every error. Most of these needed 12 or fewer candidates checked, though one needed 272. Only two documents (1.8%) had an error pass the threshold unreviewed.
Documents at a 0.80 confidence gate
Share of documentsLimitations and open questions
Jev returns a label and a confidence score, and nothing more. Our production setup also extracts each heading’s number and title, and explains its reasoning. With Jev, our own code has to pull the number and title from the candidate line, including unusual formats such as 2A or 8.02A. The missing explanations matter most when something goes wrong. Haiku writes a short reason for every answer, and we rely on those reasons to trace and correct errors. When Jev gets something wrong, we can see that it did, but not why, which makes it harder to improve our prompts and data.
We also know relatively little about how Jev works. TypeSafe hasn’t published the model’s architecture, its training data, or the details of its RLCD training. TypeSafe says the confidence score reflects the spread of the model’s probabilities, but not exactly how it’s calculated. Our results show how Jev performs on this task and these documents, and we can’t yet say how well they carry over to other tasks.
What it means for our pipeline
On its own, Jev doesn’t replace our production setup. It got 99.5% of candidates right, and for a task where one wrong label can break a document’s structure, we need 100%.
Paired with Haiku, though, Jev lets us keep 100% accuracy while cutting both cost and wait time. We’ve been testing a two-step pipeline:
Step 1. Jev classifies every candidate in a document.
Step 2. If any candidate scores below 0.90 confidence, Haiku reclassifies the whole document, and its labels replace Jev’s.
Since no Jev error on our documents scored above 0.84, a 0.90 threshold gives us a comfortable margin. Every document containing an error goes to Haiku, and the documents Jev keeps are entirely correct.
In practice, 44 of the 111 documents (39.6%) were passed to Haiku. The combined pipeline matched production’s 100% accuracy at half the cost, $1.56 against $3.14. The median time per document fell from 18.18 seconds to 0.60 seconds, and the average fell from 19.9 seconds to 9.5 seconds. Across the volume of documents we process, those savings add up quickly.
Why we test this way
The bigger lesson is about how we choose models. No vendor benchmark could have told us that every Jev error on our documents falls below 0.85 confidence, or that a simple two-step pipeline would halve our costs without losing a single label. Only testing on our own documents, against our own evals, could show that.
Every model we consider goes through the same process, and it only reaches production if it makes our system faster, cheaper or better without giving ground on any of our evaluation metrics. For the analysts, investors and lawyers who rely on our outputs, that means faster answers at lower cost, held to the same standard of accuracy.