Jev vs. OpenAI Decisions
Two decision APIs, the same 1,000 customer messages and the same 7 questions. A small benchmark of accuracy, speed and cost.
Recently, I built a small demo with Jev called Jev Inbox.
While exploring Jev, what caught my attention was the idea of using an LLM to make decisions instead of asking it to write long answers.
Then I came across the OpenAI Decisions API, and the architecture looked very similar to Jev. That gave me an idea: why not test both of them on the exact same data?
Jev and OpenAI Decisions
In my previous post, I introduced Jev through its three primitives: Choice, Noul and Score. The OpenAI Decisions API has almost the same building blocks.
| Question type | Jev | OpenAI |
|---|---|---|
| Which one? | Choice | choice |
| Is it true? | Noul | predicate |
| How much? | Score | score |
- Choice picks one option from a predefined list, in both systems.
- Noul in Jev and predicate in OpenAI both return the probability that a statement is true.
- Score rates an input based on the criteria we define, in both systems.
One notable difference: the OpenAI Decisions API isn't limited to text. It can also evaluate images, and it runs on the gpt-6-luna model.
Both systems share the same goal. Instead of generating open-ended text every time, they return structured decisions that an application can use directly.
But a similar architecture doesn't mean similar results.
So, what sets them apart?
I didn't want to figure this out just by reading the docs. I wanted to run both on the same data and see how they behave in terms of accuracy, speed and cost.
So I built a small benchmark project called Jev vs Decisions.
Preparing the data
I used the Bitext Customer Support dataset from Hugging Face. It contains short customer messages labeled with 27 intents, along with tags for typos, slang, swearing and more.
Instead of using the raw data directly, I removed duplicates and took a fixed sample of 1,000 messages, balanced across intents. Then I turned the dataset's labels into the expected answer for each question.
Here are a few rows from the sample:
| Message | Intent | Typos | Offensive | Keywords only |
|---|---|---|---|---|
| update data on {{Account Category}} cacount | edit_account | yes | no | yes |
| can uhelp me cancel the corporate newsetter subscription | newsletter_subscription | yes | no | no |
| help contacting damn customer support | contact_customer_service | no | yes | no |
| i have to notify of problems with payments | payment_issue | no | no | no |
The models only see the message text. The labels are used only to check their answers.
Setting up both systems
Every message gets the same 7 questions, sent to both APIs in a single request:
What does the customer want? → Choice (27 options)
Which topic is it? → Choice (11 options)
How polite is it? → Score (rude / neutral / polite)
Is it offensive? → Noul / predicate
Does it have typos? → Noul / predicate
Is it just keywords? → Noul / predicate
Does it contain a negation? → Noul / predicate
I defined the questions once in a provider-agnostic format, and each client converts them to its own API. Both clients return the answers, latency and cost in the same shape, so comparing them later is simple.
On the Jev side, a request looks like this:
response = await client.system_one(text, QUESTION_SET)
And on the OpenAI side:
response = await client.decisions.create(
model="gpt-6-luna",
input=text,
questions=QUESTION_SET,
)
The whole flow is simply:
data/sample.csv (1,000 messages)
↓
benchmark.py
↙ ↘
Jev OpenAI Decisions
↘ ↙
compare with the labels
↓
terminal summary + RESULTS.md
Each message is sent to both APIs at the same time.
The comparison
Running the benchmark wasn't about declaring a winner. I wanted to understand how each one performs and where the trade-offs are.
One detail matters for the yes/no questions: most messages are not offensive, so a model that always says "no" would look 96% correct. To avoid that, I used balanced accuracy, which is the average of how often the model was right on yes-messages and on no-messages. A model that always says "no" scores 50%.
Accuracy
| Question | Jev | OpenAI |
|---|---|---|
| What does the customer want? (27 options) | 94.9% | 93.4% |
| Which topic is it? (11 options) | 97.3% | 96.5% |
| How polite is it? | 88.9% | 90.2% |
| Is it offensive? | 96.4% | 98.8% |
| Does it have typos? | 84.6% | 90.9% |
| Is it just keywords? | 93.0% | 91.4% |
| Does it contain a negation? | 93.2% | 85.3% |
| Overall (average) | 92.6% | 92.3% |
Overall accuracy is almost a tie. Jev was better at understanding what the customer wants, while OpenAI was better at detecting typos and offensive language.
Speed and cost
| Jev | OpenAI | |
|---|---|---|
| Cost for 1,000 messages | $0.05 | $0.15 |
| Typical response time | 0.30 s | 0.22 s |
| Time to process all 1,000 | 19 s | 16 s |
This is where the clearest difference shows up: Jev was about 2.8× cheaper, and OpenAI was about 1.4× faster per request.
Tricky messages
I also wanted to see if they still understood the intent when the message was messy.
| Messages with… | Count | Jev | OpenAI |
|---|---|---|---|
| typos ("how cna i…") | 178 | 93.8% | 92.7% |
| slang ("can u…") | 348 | 94.5% | 93.7% |
| swearing | 42 | 90.5% | 90.5% |
| only keywords ("track order") | 55 | 96.4% | 87.3% |
The biggest gap was on keyword-only messages like "track order", where Jev was clearly more accurate.
These numbers come from live models, so a new run may differ slightly.
Conclusion
This project was a hands-on way to test two very similar decision APIs side by side, under the same conditions.
In short:
Accuracy → almost tied
Cost → Jev is ~2.8× cheaper
Speed → OpenAI is ~1.4× faster
Which one fits better depends on the workload. If cost matters at scale, Jev has a clear advantage. If every millisecond counts, or if you need image inputs, OpenAI Decisions makes more sense.
The benchmark scripts, setup instructions and full results are on GitHub: devardagur/jev-vs-decisions