A weekend with Jev made my coding agents up to 31% faster

Jev is an AI model that scores things instead of writing text. I pointed it at code search for coding agents to see if it would make them faster.

Jev is a model from TypeSafe AI, and it does not write text. You give it some context and a question, and it returns a decision: a label or a score, with a probability attached. TypeSafe calls it a System One model. Their examples finish in about a tenth of a second, and it costs $42 per billion input tokens, which is cheap enough to call on every search.

I was playing around with it over the weekend and wanted something real to point it at. Reranking looked like a good fit: hand it a question and 30 pieces of code, and ask which ones answer the question. Coding agents search code all day, so I built a small open-source search tool for them, Oko, and measured whether it made them faster and used less tokens as Jev tokens are significantly cheaper than frontier model ones.

What Oko does

1. It lists the project’s files with ripgrep and splits them into chunks, roughly one per function.
2. It ranks the chunks against the question with BM25, a standard keyword ranking method. It does this twice, once on the plain text and once giving extra weight to function and class names, then merges the two lists and keeps the top 30. Tests use the same words as the code they test, so they would fill the list. Oko saves half of the 30 spots for files that are not tests.
3. It sends those 30 candidates and the question to Jev, which scores how well each one answers the question. One call covers all 30 and takes about half a second.
4. It returns up to three excerpts. Each is the whole function, starting at its comments and decorators, plus any short helper it calls. Every line carries its real line number, because models miscount lines when they have to count them. Other likely places are listed by path.

Steps 1 and 2 run on your machine. Step 3 sends the question and the candidate snippets to TypeSafe. The index is built when the server starts and cached, so the first search does not pay for it. It connects to the agent over MCP.

How I tested it

The benchmark uses three open-source projects pinned to a fixed commit: Astro (TypeScript), HTTPX (Python) and ripgrep (Rust). There are nine tasks, and none of them names the file or function involved.

Six are search tasks. One from HTTPX:

Trace how response Content-Encoding values select decoders, how multiple decoders are combined, and why decoding runs in reverse application order.

Three are small edits. One from ripgrep:

Extend replacement capture-name parsing to accept ASCII hyphens in named references, both $first-name and ${first-name}.

Each agent ran every task three times without Oko, using only its own grep and file reads, and three times with it. The clock covers the whole session, including the model’s thinking and Oko’s search. Search answers are checked against the known locations. Edits are checked by running tests on the changed file.

Every session starts from nothing. The agent gets a fresh copy of the repository and a new conversation, and its memory features are turned off, so it cannot remember a task it saw ten minutes earlier. The benchmark checks this before each run. It tells the agent a secret code, starts a new conversation and asks for the code. If the agent knows it, memory is leaking and the run stops.

The first version saved tokens and no time

Every agent used about a fifth to a quarter fewer tokens with Oko. Claude Code got faster. Codex went from 25.7 to 23.2 seconds, and OpenCode did not move: 22.8 seconds without Oko, 23.0 with it.

So I looked at where the time goes in a session. A grep returns in milliseconds. What's usually slow is the model deciding what to do next, and it decides again after every tool call, which takes 2 to 6 seconds each time. OpenCode averaged 5.8 tool calls per task without Oko. An agent gets faster when it reaches the right code in fewer calls, and Oko was not reducing the calls enough.

In most sessions Oko’s first response already held the code the task needed, and the agent kept making calls anyway. It opened the same file with a wider range, grepped for a function name that was printed in the result, or printed the surrounding lines with sed. Those extra calls cost more than the search saved.

I first put a sentence in the tool description saying the excerpts were exact file contents. It changed nothing I could measure. What worked was putting the message in the project instructions the agent reads at the start of every session (AGENTS.md for Codex and OpenCode, CLAUDE.md for Claude Code), with a rule for when to stop and examples of good and bad follow-ups:

Oko returns exact, current file contents with real line numbers: the same text a file read would print. Treat each excerpt as a file read you have already done.

After an Oko search, if you can name the exact code to cite or change, act: answer or edit. Usually that is one Oko call and at most one follow-up read.

- Good: Oko search, then edit the returned lines.
- Bad: Oko search, then sed, nl, cat, or a read of the same range to verify it.
- Bad: Oko search, then grep for a name the excerpts already show.

I also made the first response more complete, so that stopping is the right call more often. On 169 questions replayed from earlier runs, the share of responses that held everything the task needed went from 67% to 84%.

Results

Measure

Without Oko

With Oko

Average

Best task

Codex

Time

25.7 s

21.0 s

18% faster

54% faster

Tokens

71,923

48,823

32% fewer

69% fewer

Tool calls

3.4

2.0

40% fewer

79% fewer

OpenCode

Time

22.8 s

19.3 s

15% faster

53% faster

Tokens

29,413

17,017

42% fewer

80% fewer

Tool calls

5.8

2.6

56% fewer

87% fewer

Claude Code

Time

10.6 s

7.3 s

31% faster

52% faster

Tokens

23,960

19,133

20% fewer

46% fewer

Tool calls

3.7

1.6

57% fewer

77% fewer

Average is the mean session over all 9 tasks, 3 runs each, 162 sessions in all. Best task is the one task where Oko helped that agent most. “Faster” means that much less session time. Tokens are the agent’s own and do not include the reranking model.

Tool calls fell on every task for every agent. On one ripgrep task OpenCode went from nine calls to two.

Every session was also graded for correctness. With Oko set up, all 81 sessions passed. Without it, 75 of 81 did, and four of those six misses were search tasks. Six misses is too few to call Oko more accurate. It does show that the speed did not cost correct answers.

This is a small benchmark, and time did not improve on every task. On one HTTPX question OpenCode was 33% slower with Oko and used more tokens, and I do not know why yet.

Try it

Oko runs on macOS and Linux. You need a Jev API key from TypeSafe AI, and setup asks for it the first time. Install Oko, then run setup inside a project:

curl -fsSL https://raw.githubusercontent.com/bartlomein/oko/main/install.sh | sh
oko setup --client claude     # or codex, opencode, all

Setup connects the agent and adds the instructions above. Without a key, oko setup --no-jev gives you keyword search only, with no reranking. The benchmark, its tasks and the answer checks are in the repository, and the full results list every task, including the ones that got slower.