📰 Key Takeaways

OpenAI has made its Decisions API available to all users for the first time, stepping into the “fast, cheap intelligence” market. That market was pioneered by TypeSafe, a startup founded by former OpenAI researchers, which is now valued at $7.6 billion thanks to its Jev model. Jev is a kind of “super classifier” deep learning model: given a set of options, it picks one quickly and attaches a confidence score. Developers have found that these decision models (also called System One models) can replace large language models in many workflows, running faster and costing far less.

Nikunj Handa, who leads the OpenAI API product, said the team sprinted to ship their own version as soon as they saw Jev. Rather than training a dedicated, calibrated decision model, they reconfigured ChatGPT Luna to speed up inference and cut costs. He said the biggest breakthrough is speed: response times dropped from one to three seconds down to 100 to 180 milliseconds. The API was announced at the developer day on September 29 and opened to all customers on Thursday. The public version adds voice and vision modalities, and places inference resources geographically to reduce latency around the world.

In early trials with customers like Box, Datadog, Ramp, and Speak, engineers swapped it in for OpenAI’s large language models to get higher throughput. Speak CTO Andrew Hsu said the gap between actions fell from 1.4 seconds to 0.13 seconds, and OpenAI also used it to make its Dots voice assistant more responsive.

That said, what makes a good decision model is still an open question, and the tradeoffs between intelligence, cost, calibration, and speed vary by workload. TypeSafe CEO Diogo Almeida doesn’t trust benchmarks and advises users to test in their own scenarios, yet the company still published its own rankings when it released a new version of Jev last week (adding image support and smart-if statements). On the AIM System One Index and the Artificial Analysis Decision Index, Liquid’s D1 Ultra and Base Lab’s Elephant-1 did better; Jev ranked seventh and third respectively, while OpenAI came in eighth and fourteenth, but TypeSafe is still the most-called on OpenRouter. The latter part of the original article was truncated, so see the original link for the details.

💬 JudyAI Lab Take

This story matters because OpenAI reconfigured ChatGPT Luna into the Decisions API, cutting response times from one to three seconds down to 100 to 180 milliseconds and going straight for the “fast, cheap intelligence” market.

For AI builders, this shows that model choice is becoming layered: not every step needs a large language model. The article notes that developers have found these decision models (also called System One models) can replace large language models in many workflows, running faster and costing far less. Speak’s CTO Andrew Hsu also said the gap between actions dropped from 1.4 seconds to 0.13 seconds. At the same time, the article points out that what makes a good decision model is still an open question, with tradeoffs between intelligence, cost, calibration, and speed varying by workload. TypeSafe CEO Diogo Almeida doesn’t even trust benchmarks, and OpenAI ranked only eighth and fourteenth on the two indexes, yet that hasn’t stopped everyone from rushing into this market. Speed and cost are clearly becoming a competitive dimension of their own.

My suggestion: go through your own workflows and pick out the steps that only need to choose from a fixed set of options. Then test a fast decision model against a large language model on your real data and compare latency and accuracy.

📅 Source Info

🔗 Further Reading