Tooling case study
How we used agents to evaluate Jev—and changed our minds about its best use
Published · Research through
Executive summary
We used agents to test Jev on research tasks. The strongest finding was a narrow, practical one: it could help prioritise claims for closer inspection. Its scores could not establish that a claim was correct.
Agents compared approaches and checked results against sources, narrowing the recommendation as evidence accumulated. These remain internal, model-reviewed findings—not an independent human benchmark. Research through 24 September 2026; Jev 1.13.0.
A number can appear in a source and still support the wrong claim. In our transcript checks, revenue had become operating profit, a 60–70% range had become 60%, and a quarterly investment figure had been paired with a cumulative period.
Jev helped bring suspicious claims to the front of a review queue. In the follow-up audit, a blinded model reviewer judged all 30 selected low-scoring claims outside the initial pilot documents defective.
That was the clearest positive result, but it came from recurring numerical errors in one company. It supports further testing of review prioritisation; it does not establish general detection accuracy.
We scored 3,670 extracted claims with the questions unchanged. A fresh model reviewer examined the 30 lowest-scoring claims outside the initial pilot documents, without seeing their scores or knowing the selection group. It judged all 30 defective.
Those claims came from 11 transcripts of one company. They shared recurring numerical extraction patterns. Thirty findings were not 30 independent tests across businesses and error mechanisms, and the result does not establish overall detection accuracy or recall.
Scoring the 3,670 claims took approximately 6.6 minutes. The input-token estimate was about US$0.30 at the price recorded for the experiment. TypeSafe’s pricing documentation describes the token-pricing basis.
That estimate covers Jev scoring only. It excludes review and correction work, is not an invoice, and is not a measured saving against another complete process.
After observing error types in the transcript pilot, we designed a short set of semantic checks and scored 333 claims. Three source-confirmed errors appeared first, fifth and sixteenth in the resulting review order.
The questions were designed after seeing the error types. This was useful exploratory evidence, not a test on examples independent of question design. It motivated a follow-up using the same questions on additional documents.
An earnings-call transcript described EBITDA falling by 20%. An extracted claim turned it into growth of 20%. Jev gave the claim a support score of 0.86. A later model reviewer, reading the source without seeing that score, caught the error.
Checking only suspicious claims would have missed this failure. We found it because we also sampled apparent successes.
Annual reports reinforced the limit: semantic checks and rule-based checks caught different problems, and short excerpts sometimes lacked essential table context. The practical implication is to retain source review and complementary checks, including samples of high-scoring claims. A high score is insufficient for automatic approval.
For claims scoring at least 0.8, the blinded reviewer judged 18 of the 20 sampled claims correct, one defective and one uncertain. The sign reversal was the defective claim.
One error in this sample does not measure the share of all errors that Jev missed. Most score ranges had not received equivalent review, and the results did not establish calibrated probabilities or a safe acceptance threshold.
The same checks were applied to 435 narrative claims from two annual reports of the same company. They surfaced three instances in which a statutory tax rate had been interpreted as an effective tax rate. One describes a rate set by tax rules; the other concerns tax expense relative to profit before tax. Finding the percentage in the source did not make the interpretation correct.
Jev did not reliably prioritise six known defects already identified by rule-based checks, involving units and the distinction between a component and a total. It also raised false alarms when table excerpts lacked the headers needed to interpret their columns. A claim could be correct on the source page while looking doubtful in a short passage.
Neither method covered every problem. Source context and the scope of the question mattered; a score alone could not settle an ambiguity that required reading the page.
An intermediate result is useful only if it helps the completed task. Better-ranked suggestions still needed investigation. More extracted material came with substantial work by a stronger model.
The tests did not establish the broader efficiency gains we were looking for. The missing comparisons matter before adopting the approach or claiming savings. The evidence below separates what improved from what remains uncertain.
Consider a hypothetical research note about electricity demand from data centres. It might inform questions about power supply, technology investment or AI economics. Deciding where it is useful requires understanding its main claim and the question being investigated.
Jev ranked candidates better than keyword search. But in 20 paired cases, agents with Jev suggestions did not show a clear overall efficiency gain: they still had to read and judge the results. We retained ordinary agent search as the default for the tested task.
Each of 20 cases received two fresh agent runs: one with Jev suggestions and one without. Separate model reviewers assessed the resulting decisions without knowing which method produced each answer.
| Measurement | Without Jev | With Jev |
|---|---|---|
| Decisions judged defensible | 20/20 | 20/20 |
| Decisions judged appropriately specific | 18/20 | 19/20 |
| Mean composite time | 46.21 seconds | 45.67 seconds |
| Median composite time | 41.52 seconds | 41.96 seconds |
The small difference in means favoured assistance; the medians slightly favoured the ordinary agent. Jev reduced searches, but did not remove the need to read and validate candidates. These results did not establish a meaningful speed advantage.
The sample was small and retrospective, with one run per method for each case. The reviewers belonged to the same model family as the task agents; their judgments were not independent human ground truth.
Composite timing combined separately measured components. Startup, caching and concurrent execution limited precision. The result concerns this assistance design and sample, not every possible use of Jev. Better search ranking remained a valid finding, even though the case for adopting the tested agent assistance remained unproven.
Across three transcripts from two companies, we compared existing extracted material, passages selected by Jev, and a combination that sent material from Jev to a stronger model for further work. The combination covered more of a separately generated reference set.
But the stronger model had done substantial work, and we had not tested it alone under matched conditions. More coverage was an observed result; attributing it to Jev would have gone beyond the evidence.
Separate full-source model reads generated the comparison reference set. Under the model-based assessment, the combined approach covered 391 of 448 items; the existing extracted material covered 179. These are coverage judgments, not verified numeric-extraction accuracy.
The combination sent about two-thirds of the input units for stronger-model review. It had not demonstrated that Jev removed much of that work. Output formats also differed, so a single percentage could not capture every quality difference.
We needed a fresh experiment in which the stronger model performed the task without Jev, using comparable material and conditions. That comparison had not been run.
The improvement could have come from the stronger model, the additional attention, or the way the task was presented. The reference set was model-generated rather than human gold. The useful next test is to isolate Jev’s contribution and measure the quality of the final claims alongside the total work required.
Fresh agent sessions carried out paired tasks. Separate model reviewers assessed alternatives without method labels. Source checks examined whether numbers, measures and context belonged together.
Those comparisons narrowed the recommendation from several possible research uses to one promising candidate: deciding what deserves closer inspection. They did not justify automatic acceptance, rejection or repair. A separate model reviewer remained a model reviewer, not human ground truth.
Jev, from TypeSafe AI, evaluates supplied material through constrained questions: choosing among options, returning a score or assessing a yes/no proposition. TypeSafe’s API documentation describes the question types. A valid output format does not establish correct interpretation.
Our experiments used Jev 1.13.0 and covered research classification, earnings-call extraction and checks on extracted claims. This article summarises work completed by 24 September 2026. It is not a universal product ranking. Experimental results are our own; vendor links support the product description and pricing basis. Private experimental materials are not included.
A matched stronger-model-only extraction comparison is needed to isolate Jev’s contribution. Broader document coverage and independent human-adjudicated examples would help test whether the review ranking generalises. More of the score distribution must be reviewed before making calibration or missed-error claims.
The present evidence supports checking suspicious claims against sources, retaining rule-based checks and continuing to sample high-scoring claims. The revenue mistaken for profit shows why a review queue matters. The decline mistaken for growth shows why further review remains necessary.