Benchmarking alternatives to Jev

We ran these benchmarks for Farero, our app for running Codex, Claude Code and OpenCode in parallel.

For its attention cards, we tested ScaleDown extraction and summarization, changes to the Jev request format, GPT-6 Luna, Claude Haiku 5.5 and the Jev 1.14 upgrade. The workload combines conversation classification, suggested replies and a brief showing completed work, pending questions, who acts next and restrictions.

Result: retain the existing Jev 1.13 pipeline for this workload. ScaleDown tuning improved known examples but failed most fresh-case fidelity checks. Cheaper Jev requests changed useful selections. Luna and Haiku improved some briefs but introduced critical errors, while Jev 1.14 introduced repeated omissions.

The experiments ran from 15–20 October 2026 and used different datasets and acceptance criteria. Results below are compared within each experiment. The later model screen allowed up to 3× the baseline's complete inference cost; price alone did not qualify a replacement.

ScaleDown

We started with extract-1 for source spans and summarize-1 for readable briefs. The initial six-case comparison was exploratory: extraction and summarization did not perform Jev's classification and reply-selection work, so their lower cost was not evidence of a cheaper replacement pipeline.

Whole-passage extraction retained more qualifications than extracting short labels, but remained inconsistent across repeats. Exact source validation rejected invalid spans. It could not detect a restriction that was never selected. Quote-based summaries had the same limitation: copying source text prevented altered wording in accepted quotes, but still omitted conditions and selected outdated information.

Prompt tuning

We tried prompts for action ownership, latest-message emphasis, explicit restrictions and exact quotations. A later bounded trial followed ScaleDown's optimization workflow. Its selected edit added a standing-restrictions check. A second edit asking for broader conversation-state reconciliation introduced regressions and was rejected.

The final comparison used 20 known synthetic cases and eight fresh held-out synthetic cases. The prior latest prompt (v1) and selected restriction-focused prompt (v2) each ran twice on identical sources with summarize-1 and a 1,000-token output cap. All 112 responses completed without truncation. A fidelity pass required every material source requirement to survive without a changed or invented claim; restrictions and question/decision/ownership accuracy had equal priority.

The 16 fresh outputs are two repeats of eight sources. The tuned prompt preserved prohibitions, consent ownership and automatic continuation more often on known examples. It still failed 11 of 16 fresh outputs. In one case it restored an upload hold after the user had explicitly authorized the upload. In another it described an export limit and then said no restrictions remained. Summaries were about 93% longer, with roughly 15% more input tokens.

These counts supersede the interrupted six-case pilot. The final source audits corrected automated reviewer mistakes and were not blinded to prompt identities. The rubrics were assistant-authored, so the result establishes failure against our stated contract, not a general ScaleDown accuracy score.

Verification and compression

We also tested changing the pipeline around ScaleDown, rather than just its prompt.

The selected-evidence experiment exposed a useful distinction. Some omissions happened before ScaleDown received the text; those are passage-selection failures. Others were introduced during summarization: “Keep the proposal unpublished” became “The proposal is unpublished,” replacing an instruction with a status claim. A statement that a gesture was untested became an unanswered question. Showing the original quotes alongside those summaries did not make the generated claims correct.

ScaleDown cost estimates in these earlier runs used an explicitly assumed $0.05 per million input tokens. They include reused stages and fallback work, but are not verified invoices. The quality findings do not depend on that rate.

Shared-context billing

Our decision request contains 17 independent reply checks and one classification. Jev bills shared context once in a batched request. ScaleDown's Decisions documentation described one internal call per question with summed usage. That distinction made a direct swap less attractive, but it did not establish a 17× or 18× total bill: only the shared state repeats, while the questions have their own token cost.

We tested an alternative arrangement through ScaleDown Classify, putting synthetic reference facts in a shared rubric. It answered all 17 toy questions correctly, including a control with every fact inverted. It reported 9,210 input tokens against 8,425 for facts repeated inside each question—9.3% more. The rubrics differed between request shapes, so this did not isolate the billing mechanism. It demonstrated no saving and did not validate our actual routing checks.

Cheaper Jev requests

Keeping the model fixed also produced tradeoffs. Moving repeated instructions into shared state and shortening source identifiers reduced input size, but changed which replies and passages crossed the selection threshold.

The later validation used 16 checkpoints: 14 synthetic and two correlated checkpoints from one real chat. Shortened identifiers preserved every source field and relationship reversibly.

The judgment-only change had earlier saved 7.1% on a separate 12-case set, so the saving repeated. The later results showed it was not behaviorally equivalent. Both original and shorter judgments also missed one category in the 16-case round. We kept the existing requests; these percentages are measured costs of experimental configurations, not savings shipped to users.

GPT-6 Luna

The structured-brief evaluation used 52 conversation checkpoints: 28 development or previously known cases, 12 fresh synthetic cases and 12 real conversation replays. Each ran twice, giving 104 outputs per configuration.

• Jev baseline: Jev 1.13 makes category and reply decisions, then selects original source passages for the brief.
• Jev + Luna: the same Jev decisions, with Luna generating a structured brief.
• Luna only: Luna generates both decisions and the brief.

Luna reasoning was disabled. All 312 outputs were structurally valid. Semantic review checked the displayed cards against their source conversations. GPT-OSS 20B on Groq was removed before the final run and is not part of these results.

“Fully actionable” requires a valid, useful output with no material classification, fidelity or reply error. Weakening an absolute restriction counted as a critical error. The 24 fresh checkpoints include the 12 real replays; those rows overlap.

Luna’s aggregate gains came mainly from known cases. On the real replays, both candidates reduced the number of checkpoints where both repeats were fully actionable from seven to four. The three critical outputs came from one synthetic case: an unconditional ban on credit-card trials became conditional on approval.

The cost figures reflect substantial caching: more than 99.89% of Luna’s second-repeat input came from cache. They are observed experiment costs, not estimates for new conversations. Luna only also had a 49.76-second p95 on the fresh synthetic subset, which its overall p95 obscures.

Jev 1.14 and Haiku 5.5

The follow-up used 16 checkpoints, again with two repeats: four real replays and twelve synthetic cases. Eight cases came from the earlier evaluation; eight were newly authored around known failure modes. Sources, expected results and the brief prompt were fixed before inference.

Jev 1.13 and 1.14 received identical paired requests apart from the model field. The Haiku configuration reused saved Jev 1.14 decisions and replaced passage selection with brief generation. Haiku ran through OpenRouter’s Anthropic route, with thinking disabled and a 1,600-token output cap.

Jev’s versions have the same published token price. Haiku’s total pipeline cost was 2.13× baseline overall and 2.09× on real replays, within the predeclared 3× limit. Two Haiku outputs hit the token cap; their costs include the failed generation and Jev fallback. No Haiku input-cache discount contributed to these results.

The quality failures were specific:

For the restriction error, the synthetic source said:

do not start any trial that needs a credit card under any circumstance.

Haiku produced:

Do not purchase, subscribe to, or start any trial of LogNest or TraceHive, especially any trial requiring a credit card, without explicit separate purchase approval from the user.

The generated text passed structural and source-reference validation, but changed the scope of permission. The earlier Luna evaluation reproduced the same failure pattern. These were errors in the briefs; no purchase or trial was executed.

Haiku also improved several cases: it restored the question and restriction omitted by Jev 1.14, preserved the local completion, and recovered a missing export-completion detail. Those gains did not satisfy the zero-critical-error requirement.

Candidates screened but not benchmarked

The broader capability and pricing review also considered other brief writers and direct decision-model replacements. It was a documentation and cost screen, not evidence of their quality on our conversations.

These three untested writers fit the later 3× budget screen with Jev retained for decisions. The projections use the Luna run's saved mean input/output sizes, ordinary synchronous prices and no caching. Native tokenization, reasoning, output length and fallback frequency remain unmeasured.

These are the 20 October cost projections, not measured inference costs. Haiku also passed that screen and was then tested above. The same review put standard-price Gemini and Sonnet configurations over budget; DeepSeek had limited headroom, while MiniMax, GLM and Liquid's small generative model needed reasoning-token measurements before qualification.

For direct decisions, we reviewed Liquid d1-ultra, Mercury Decide, Microsoft-Decision-1, Cloudflare Clef-Flash, Luna Decisions and Fastino GLiDE. None received a Farero quality evaluation here. Liquid explicitly bills each question as its own prompt; a low token tariff or one HTTP request is therefore insufficient to establish a cheaper replacement. GLiNER2.5 extraction and an incrementally updated, source-linked state record also remained untested architecture options.

Limits and decision

These are small offline evaluations of specific prompts and pipelines. The follow-up deliberately concentrates known failure modes. Repeats measure consistency on a case; they are not independent samples. Ten of the twelve real replays in the Luna evaluation have incomplete history.

Semantic grades came from automated reviewers, with disputed findings checked against sources. They are not human acceptance results. ScaleDown's final audits were not blinded; later model comparisons withheld configuration labels during initial review. Pipeline latencies sum measured stages, including reused stages; ScaleDown's prompt table reports single-request latency. Costs use token-usage estimates or OpenRouter’s reported usage cost, excluding review work and account funding fees.

The current Jev 1.13 pipeline remains the preferred configuration among the tested alternatives under these acceptance criteria. It still has fidelity and suggested-reply errors. Prompt tuning, extraction, verification and generative briefs each improved some results, but none established a replacement that met the relevant quality requirements. That leaves the untested models and state-record designs open; it does not make them measured losers.

If you work with several coding agents, try Farero to keep their conversations, workspaces and code reviews in one place.