Turn GLM-5.3-Flash into a Jev-like System One model

TL;DR: In this post, we show how an off-the-shelf LLM can make typed decisions in a single forward pass. This approach makes it possible to turn an LLM into a Jev-like decision model.

We evaluate the approach using GLM-5.3-Flash running on Privatemode. Using a benchmark constructed from public data sets, we show that this setup delivers results that are on par with TypeSafe's Jev in terms of decision accuracy/correctness and speed.

As a bonus, the setup with GLM-5.3-Flash enables typed decisions on images, which is not possible with Jev.

Why typed decisions

Much of what software asks an LLM is a decision. "Which team should handle this ticket?", or "Does this contract clause belong in the liability section?".

In such cases, software typically requires that the LLM's response follows a certain format like JSON and that it comes from a pre-defined set like "yes" and "no".

Given the right instructions, LLMs can typically already fulfill this reliably. However, in the basic approach, speed and costs become an issue: For each decision, the LLM needs to write a whole JSON object, and a reasoning model may think for hundreds of tokens before that. Further, you also don't learn the confidence of the model (unless you explicitly ask it). All these aspects can matter a lot in practice and have so far prevented people from employing LLMs for decision making in high-volume/high-throughput scenarios.

Specialized decision models (or "System One" models) like Jev and Laya are designed to address this. You pass in a piece of state and a set of named options, and you get back the chosen option together with a confidence value (i.e., probability) for each one.

Turning an LLM into a decision model

Initially, we asked ourselves if an LLM could be turned into a decision model with Jev-like properties. The short answer is: "yes". In the following, we show how it works.

To understand our approach, it's important to understand how LLMs work:

An LLM never writes text directly. Given a prompt, an LLM outputs a probability distribution over its entire vocabulary of tokens. In text generation, in the simplest case, the token with the highest probability is selected as the next token. The selected token then is appended to the prompt and the whole process repeats. As described above, this is costly and slow if you just want to set a few fields in a JSON object.

Our core insight is that it's unnecessary to have the LLM predict the whole JSON object, as we already know its shape. We're only interested in the LLM's typed judgement for a given input.

We realized that it's possible to craft prompts so that we get the typed judgement in a single run of the LLM — with no fine-tuning, on the model exactly as it ships. This is the difference to Jev and Laya, which are models trained for the purpose. The basic steps are as follows:

1. Number the options. The state, the question, and the output options go into the prompt as JSON, with an index on every option. The instruction asks the model to answer with choice_index: followed by an index.

2. Prefill the answer. The prompt ends with choice_index:. Consequently, the first token the model produces will be an index into the pre-defined options.

3. Evaluate the output. Rather than reading the token the model emits, we read the probabilities it assigned to all option indexes at that single position. Normalized over the options, these give a probability for each answer, and we simply pick the most probable one.

We implemented the above steps for GLM-5.3-Flash running on vLLM (in Privatemode). We use the /chat/completions endpoint with continue_final_message and add_generation_prompt: false, because these let the model continue the prefilled assistant turn from step 2 instead of starting a new one. They also let us pass images next to the text, which is what makes typed decisions on images possible.

For vLLM and GLM-5.3-Flash, we found the following details to matter:

1. vLLM's allowed_token_ids can be used to limit the LLM's output vocabulary only to allowed options. It drops every other token to -inf. We set it, but it is a guardrail rather than a requirement.
2. top_logprobs is not enough for step 3. It reports the distribution before the restriction is applied, so formatting tokens such as a leading space take up the top slots, and some options drop off the list and appear to have a probability of zero. vLLM's logprob_token_ids solves this: it returns the log probability of exactly the token ids you ask for.
3. The token ids of the indexes depend on the model's tokenizer. Digits aren't always single tokens. GLM-5.3-Flash, for example, has a single token for 12. Rather than shipping a model-specific tokenizer, the library gets the token ids from the server, which keeps it simple to use with any model: sending a prompt to /completions with echo returns its exact tokenization by the model that is actually serving.

You can find our implementation in the below repository.

Try it

The playground below runs GLM-5.3-Flash queried with the above setup on Privatemode, directly from your browser. Pick one of the examples, among them a scanned invoice and a question that depends on your local time, or write your own questions and add images. Each answer comes back as a distribution over its options, typically within a few hundred milliseconds.

The distribution is often very useful, e. g., to decide whether to include a human-in-the-loop. The model solves most classic trick questions, but not all of them.

Benchmark results

We evaluated our approach using a custom benchmark, which is available in the below repository.

We compared three systems on 29 public, labeled datasets: GLM-5.3-Flash hosted on Privatemode and queried with the technique above, TypeSafe's Jev, and Convai's Laya.

The datasets have between 2 and 151 options and cover intent routing, sentiment, topic classification, moderation, entailment, question answering, legal text, and scanned documents. Both English and German text is included in the corpus. All three systems receive the same state, the same option names in the same order, and the same instruction.

We ran each dataset twice. Even at temperature 0, our GLM-5.3-Flash and Jev changed up to 3.5% of their answers between identical runs: temperature 0 removes the randomness from sampling, but batching and floating-point arithmetic still keep a forward pass on a busy server from being bit-reproducible. We therefore treat smaller differences as noise.

We ran Jev and Laya with their default settings and did not tune our prompt on these datasets.

Accuracy

Compared across the 28 text datasets, GLM-5.3-Flash and Jev are on par. Each is more accurate on 10 datasets; on the remaining 8, the two are within one percentage point of each other. The median gap is 0.7 percentage points in Jev's favor, which is not statistically significant (p = 0.64). Laya, a model with 421 million parameters that we ran locally, scores lower than both on most datasets. Its median gap is 13 to 15 percentage points, which is statistically significant (p < 0.001).

The number of options has a larger effect on accuracy than the choice between Jev and GLM-5.3-Flash.

Across datasets, the number of options changes along with everything else about the task. Three datasets, however, label the same questions twice, once coarsely and once finely, so the task stays the same and only the number of options changes. On TREC, going from 6 to 42 options, Jev drops from 92.1% to 85.6%, GLM-5.3-Flash from 91.2% to 79.6%, and Laya from 88.4% to 51.2%. On MASSIVE, going from 18 scenarios to 59 intents raises the scores of Jev and GLM-5.3-Flash in both languages, while Laya's score drops. The number of options alone doesn't determine how hard a task is.

Latency and cost

We measured latency in separate runs with one request at a time, because timings taken under load measure the queue rather than the model.

As Privatemode is hosted in the EU and Jev is hosted in the US, we ran four of the datasets from Germany and from the US at the same time. From Germany, Privatemode answered in 180 ms and Jev in 264 ms. From the US, the order reverses: 164 ms for Jev against 299 ms for Privatemode.

On cost, Jev is cheaper. One million decisions cost about EUR 62 with GLM-5.3-Flash and about EUR 16 with Jev, at each service's list prices.

Most of the difference comes from the price per input token, and some from how each system packages a question. Jev adds roughly 270 tokens of fixed overhead and about 10 tokens per option. The GLM-5.3-Flash prompt adds about 55 tokens of fixed overhead and about 20 per option. Below about 21 options, it sends fewer tokens than Jev; above that, it sends more.

Laya runs locally, so there is no comparable latency or price per decision for it.

Multimodal decisions

The state doesn't have to be text. GLM-5.3-Flash is a vision-capable model, so a question can come with images, such as a scanned invoice, a photo of a damaged parcel, or a screenshot. The image goes into the same prompt, and the answer is still a single token with a probability for every option. The playground's Scanned document example shows this; you can also paste or drop in your own image.

According to its documentation, Jev works on text, and Laya is a text encoder, so neither takes images as input. On RVL-CDIP, a set of 1,600 scanned business documents in 16 classes, GLM-5.3-Flash reaches an accuracy of 70.2% and is the only one of the three that can answer. A document costs more than a sentence: the image adds about 1,350 input tokens, so a million document decisions cost about EUR 270.

Capabilities

With many options, each system hits a limit. Laya's option names share a budget of 192 tokens, which is enough for the 77 intents of banking77 but not for the 151 intents of CLINC150. Privatemode's deployment of GLM-5.3-Flash reports at most 128 entries in logprob_token_ids, while the mask in allowed_token_ids takes every option. So the library sends a question with 151 options twice, identically, and reads the probabilities of 128 options from the first response and of the other 23 from the second. Both requests run the same forward pass, so the merged result is the distribution a single request would return, up to the run-to-run noise above. On CLINC150, GLM-5.3-Flash reaches 87.5% this way, against 78.4% for Jev. The second request and the long option list cost time: a decision takes 719 ms, against 249 ms for Jev. Past that, the ceiling is the 191 option indexes that GLM-5.3-Flash spells as a single token.

Some tasks only one or two of the systems can handle at all.

Further findings

Some of the remaining errors are in the labels. When most systems agree on an answer and the dataset's label disagrees, the label is often one of two defensible answers. banking77 has many such pairs, for example get_physical_card and order_physical_card, or declined_transfer and failed_transfer. About 17% of banking77's examples fall into this category, so the highest score any system could reach there is about 85% rather than 100%.

Reasoning helps, at a price. As a control, we let the same GLM-5.3-Flash reason before it answers, on all 29 datasets. It is more accurate in every band of option counts, from 89.9% against 85.5% with two options to 82.0% against 79.2% between 21 and 80. It also writes hundreds of tokens per decision instead of one and costs about EUR 350 per million decisions, against EUR 62. At the other end, plain embedding similarity, which picks the option closest to the text with no decision model at all, reaches between 45.9% and 72.8% depending on the band.

Renaming the options affects the systems differently. We re-ran every dataset with each option replaced by a synonym and nothing else changed. On boolq, where true and false became correct and wrong, GLM-5.3-Flash lost 20 points, while the other two lost less than three. Because this renaming also changes the meaning of the question, it doesn't isolate memorization. The results for every dataset are in the benchmark repository.

Build it yourself

Our library is written in Python and works with any vLLM-backed endpoint; it relies on vLLM's extensions to the OpenAI API, such as allowed_token_ids and logprob_token_ids. Its README explains how to set it up with Privatemode, and an AGENTS.md file tells coding agents what an implementation has to get right. The benchmark repository contains the methodology, the dataset specifications, the test harness, the aggregation, and every raw run as a download, so every number in this post can be reproduced without running anything again. If you know a setting that serves any of the three systems better, find a mistake, or want to add a dataset or another system, we welcome pull requests.

With Privatemode, these decisions are protected by confidential computing: your data stays encrypted in memory even during processing, and the client verifies the deployment's attestation report before sending anything. You can read more on our security page.