In short
- In the test, Jev got 100% of requests for a human agent right; the model that used to do this job got 67.5%.
- Each decision came out 70 times cheaper and 8 times faster.
- The test used 123 hand-written cases in Portuguese. We haven't measured it on real conversations yet, and we say so below.
01
What Jev is
Jev, from TypeSafe AI, is a model that doesn't generate text. It takes a context and typed questions and returns an answer with a probability: yes or no (from 0 to 1), a choice among options, or a score on a scale.
We use Jev through OpenRouter, with the model version pinned, so its behavior doesn't change without notice.
Context
Illustrative example“I tried twice in the app and nothing. I want to talk to a person.”
Typed questions
- Yes or noDid the person ask for an agent?
- ChoiceWhat is the topic?
- ScoreUrgency, from 1 to 5
Doesn't write text. Only answers.
Answers with probability
- Yes or noyes · 0.94
- ChoiceSupport · 0.81
- Score4 of 5
Calibration
When it says 0.9, the answer is right about 9 times out of 10.
Which is why you can write rules like this
If the chance that the person asked for an agent goes above 0.7, call someone from the team.
02
Why we looked at it
In Jundo's customer service, the same large model that writes the reply also stated, about its own conversation, whether the person wanted to talk to someone, whether they were annoyed, and whether the knowledge base answered the question. That is asking the writer to grade its own work.
In the cases where someone asked for an agent, that judgment was confidently wrong in one out of three cases. A closed question, asked of a model built for it, promised to be cheaper and more reliable.
Large model
- Writes the reply
- Judges whether they asked for an agent
- Judges whether the person is annoyed
- Judges whether the base answers it
Confidently wrong in 1 out of 3 requests for an agent.
Jev
- Asked for an agent?
- Annoyed?
- Does the base answer it?
Large model
- Writes the reply
Each does what it does best. The writer doesn't grade its own test.
03
How we tested
We wrote the pass criterion before running the test and built 123 hand-written cases in Portuguese, across four business areas.
We compared Jev with the customer-service contract in use, copied unchanged and running on the large language model that used to do this job.
Written before the test: four conditions on
- Accuracy
- Calibration
- Confident errors
The cases, by type
- 25requests for an agent
- 8traps for the keyword rule
- 20annoyed customers
- 68questions the base answers
- 20questions the base doesn't cover
- 15conversations with no signal at all
04
Results
Jev passed all four conditions. It did not win everywhere: on “is the person annoyed?”, the large model was slightly more accurate. Where it matters most, the request for an agent, the gap was large.
Accuracy per question
JevPrevious model
Did the person ask for an agent?
Jev: 100%Previous model: 67.5%Is the person annoyed?Here the previous model did better
Jev: 97.6%Previous model: 98.4%Does the knowledge base answer it?
Jev: 91.9%Previous model: 71.5%
Time per decision
8×
median, network included
Cost of the 123 cases
70×
same cases, same question
Confident errors, in the 15 conversations with no signal
Stating with confidence something the conversation didn't say.
See the numbers as a table
| Question | Jev | Previous model |
|---|---|---|
| Did the person ask for an agent? | 100% | 67.5% |
| Is the person annoyed? | 97.6% | 98.4% |
| Does the knowledge base answer it? | 91.9% | 71.5% |
| Confident errors, in no-signal cases | 0 of 15 | 8 of 15 |
| Time per decision (median) | 358 ms | 2,908 ms |
| Cost of the 123 cases | US$ 0.0047 | US$ 0.33 |
05
How we turned it on without taking risks
Neither system hands the final decision to Jev. At Jundo it runs in shadow mode; at Estação Clínica, on pseudonymized text, it suggests and the team decides.
Jundo
Shadow mode- 1A single judgment gatewayEvery closed question goes through one place.
- 2Answers, logs and comparesJev decides in parallel and is compared with what happened next. It changes nothing.
- 3When it is actually switched onIt can only add a handoff to a person, never remove one.
- Provider down? A fallback model answers, without making up calibration.
Estação Clínica
Signals for the team- 1Category and urgencyProposed for each WhatsApp message; the team accepts with one click.
- 2Review of automatic repliesIf it raises a flag, the reply becomes a draft for someone to look at.
- 3Knowledge base orderingRight page first in 90 of 90 tests, against 45 for keyword ordering.
- Works on pseudonymized text, always as support for the team.
In both: if Jev doesn't respond, the system works as before.
06
What we don't know yet
- 01
Hand-written cases
The tests used cases written by the same author as the questions. Jev hasn't been measured on real conversations yet; that is what shadow mode is for.
- 02
Portuguese isn't its main language
The model's main language is English, and Portuguese may perform worse in situations the test didn't cover.
- 03
Savings per decision, not on the bill
Today the part of the work Jev handles is small within the AI bill. The 70-fold saving applies per decision, not to the whole invoice.
07
Next steps
- In progress
1. Measure in shadow mode
On real conversations, with the same criterion written in advance.
- Then
2. Turn on one decision at a time
Only when it passes the criterion.
- Then
3. Show it at the monthly meeting
The result of each decision, in numbers.
A fast decision up front, heavy reasoning only when needed.
If the decisions pass one by one, Jev could become the front door of our systems: it understands the request and sends each one down the cheapest useful path. This is a direction, not something switched on today.
- Knowledge basewhen the answer is already written
- Someone from the teamwhen the case needs a person
- Large modelonly when it needs to reason or write
Small, fast, cheap decisions up front. The rest of the system does the work.