Running SemIf locally: a System 1 model on your own machine

A lot of software is made of small judgement calls that a person gets right instantly and a programmer can barely write down. Is this support email about billing or a bug? Does this answer actually address the question? The agent says it “successfully updated the record” — does the log agree? Does this document belong to the migration project or the compliance one?

You don’t think about any of these. You glance, you know, you move on, and you save your real attention for the one case that’s actually ambiguous. Kahneman called that reflex System 1: fast, automatic, pattern-matched, confident, and now and then wrong in a way that a moment’s thought would have caught. System 2 is the slow, deliberate mode you switch into when the reflex isn’t enough.

Almost everything we build on large language models is System 2 by design. You send a prompt, the model generates tokens one at a time, maybe reasons its way through a chain of thought, and hands you back prose that you then have to parse. That’s a great way to handle the genuinely hard case. It’s a silly way to answer “billing or bug?” You’re paying for deliberation you didn’t need, waiting on a network round trip for a reflex, and getting a paragraph when you wanted an enum.

A semantic if is the System 1 version. You give a model some text, a question, and the list of answers you’ll accept, and it gives you back a probability for each of those answers. No generated tokens, no reasoning trace, nothing to parse. One forward pass, a few numbers, done.

Jev is the best-known hosted product that works this way: a commercial decision service you call over the network. SemIf (“Semantic If On-Device,” previously branded OpenJev) is an independent research project that does the same job with open weights. SemIf is what this post is about.

Why bother running it yourself?

The hosted version is easier, so running your own needs a specific justification. Here are four, and they stack.

It can live in a hot path, as long as the input is short. A network round trip is a bad place to put a reflex you call on every request. Locally, one comparison against a short premise (about 20 tokens) takes around 58 ms in a batch, or 114 ms by itself. A handful of options costs a couple of hundred milliseconds — roughly one network round trip, minus the network, the rate limit and the bill.

That number grows with how much text you feed it and how many options you score, though. The email-tagging job later in this post has fifty possible labels, narrows them to sixteen, and ends up at about six seconds per message. That’s a background-queue job, not a hot path. Both situations are real, and which one you’re in is a design choice. There’s a cost table near the end to help you make it on purpose.

You’ll call it constantly. That’s the whole point of a cheap decision. Paying per call is fine when you classify a thousand things a day. It’s ruinous once the classifier sits on every message, every retrieval and every agent step.

The input is usually the sensitive part. The text you most want to route automatically is email, tickets, documents and transcripts. If the decision runs on your own machine, the thing being judged never leaves it, and a procurement conversation turns into a pip install.

It has to be there. Anything that can rate-limit you, deprecate a model out from under you, or put you on a waitlist is a dependency in your critical path.

All of that only matters if the answer to one question is yes: is the open-weight model actually good enough? The project’s published comparison puts hosted Jev at 88.3% on a shared public subset and SemIf’s own 4B browser model at 84.5%. SemIf is refreshingly upfront that “none is claimed to match Jev.” (I couldn’t find the underlying data or the definition of that subset, so treat both as the project’s self-reported figures rather than something I’ve checked.)

Most write-ups stop at a gap like that. I think that’s the wrong place to stop, because a single accuracy number quietly lumps together four unrelated things:

• Capability — what the model can actually do.
• Precision — how much numeric precision you threw away to make it fit somewhere convenient.
• Setup — how well you framed the problem for it.
• Data — the state of whatever you pointed it at.

Each has a different fix. A capability gap needs a bigger model. A precision gap means you should stop quantizing so hard. A setup problem won’t be solved by any model. And a data problem won’t be solved by any amount of work on the other three.

To be clear about scope: nothing here re-measures the 88.3-versus-84.5 comparison. I bring it up to introduce those four confounds, because they’re what you’ll actually run into. The rest of the post takes one system apart against them. I get it running on a laptop, benchmark it carefully, and then point it at a real classification job with about fifty possible answers. In my case, setup turned out to be the dominant factor. Figuring that out meant explaining how the same model scores 93.7% on a public benchmark and 40% on a real task.

By the end you should be able to:

• get a classification-head model running natively on Apple Silicon when the usual tools refuse it,
• measure what quantization costs a classifier, instead of assuming it behaves like it does for a chatbot,
• recognize three failure modes — about phrasing, vocabulary, and over-helpful descriptions — that look like model limitations but aren’t,
• work through a short diagnostic checklist on your own system instead of guessing.

Everything was measured on an M4 Pro with 48 GB of memory, in one sitting.

One practical note before any commands: scripts like eval_mlx.py and benchmark_capability.py come from a companion repo that isn’t published yet. The model is public and every install command works today, but you can’t clone my harness scripts at the time of writing. So I’ve put the important code inline rather than behind a dead link, and I give enough detail for every number that you can check the arithmetic even where you can’t re-run the measurement.

---

How a semantic if works

Under the hood, this is a cross-encoder doing natural language inference (NLI), and both ideas matter for everything that follows.

An NLI model takes two pieces of text: a premise (something stated) and a hypothesis (a claim that may or may not follow from it). It then classifies the relationship as entailment, contradiction or neutral. A cross-encoder reads the two texts together in a single pass, instead of embedding each one separately and comparing the vectors. That’s slower per comparison, but much more accurate, because the model can pay attention to how the two texts relate.

In practice, the model reads one packed string and returns three numbers:

Premise: The research vessel departed the harbour before dawn.
Hypothesis: The vessel left port in the early morning.
→ [contradiction: 0.005, entailment: 0.984, neutral: 0.011]

A little vocabulary, since it comes up repeatedly. The model’s raw outputs are logits: one unnormalised score per class. Softmax turns them into probabilities that sum to 1, which is what’s printed above. Argmax just means “the index of the biggest one,” i.e. the predicted class. When I report top-1 or top-3 accuracy later, I mean the right answer was ranked first, or somewhere in the first three, among whatever candidates were scored.

That’s the entire interface, and it turns out to be more useful than it looks. Anything you can phrase as “does this claim follow from this text?” becomes a scored decision:

• Reranking search results: does the question entail this candidate answer?
• Grading a response: does the reference answer entail this one?
• Checking an agent’s work: does the claimed outcome follow from what was observed?
• Classifying a document: does the document entail “this is about billing”?

The checkpoints I use are built on Qwen3.5, Alibaba’s family of open-weight models, which is why “Qwen” shows up in the class names. I’ll say backbone for the big pretrained transformer doing the language understanding, and head for the small layer on top that turns its output into class scores.

Two things called SemIf

This is where you can lose a day by picking the wrong road, because SemIf comes in two forms that share a name and very little else.

flowchart TD
    A["SemIf — semantic ifs from open weights"] --> B["Browser build<br/>openjev.com"]
    A --> C["Cross-encoder checkpoints<br/>AlexWortega/openjev on HF"]

    B --> B1["Decoder LLMs:<br/>Qwen3 0.6B / MiniCPM5 2B / Qwen3.5 4B"]
    B1 --> B2["Reads choice logits over<br/>the options you supply"]
    B2 --> B3["Pinned GGUF via wllama<br/>(llama.cpp in WebAssembly)"]
    B3 --> B4["Zero install · quantized ·<br/>max 20 options"]

    C --> C1["Qwen3_5ForSequenceClassification:<br/>backbone + 3-class head"]
    C1 --> C2["Emits contradiction /<br/>entailment / neutral"]
    C2 --> C3["safetensors — no GGUF"]
    C3 --> C4["This piece: ported to MLX,<br/>full precision, native Metal"]

    classDef default fill:#dbe4ee,stroke:#52657a,color:#1f2933
    classDef muted fill:#e3e1dd,stroke:#8a8a8a,color:#1f2933
    classDef accent fill:#bfe3dd,stroke:#1f7a70,color:#0d3833
    class B4 muted
    class C4 accent

Figure 1. The browser build is the zero-install demo. The cross-encoder checkpoints are a different mechanism, and the only one with a real NLI head. This post takes the right-hand path.

A few formats appear in that diagram. safetensors is Hugging Face’s plain weight format — basically a dictionary of tensors in a file. GGUF is the container that llama.cpp and its relatives use, designed around quantization: storing weights at lower numeric precision so downloads are smaller and the math is cheaper, at some cost in accuracy. Metal is Apple’s GPU API, which anything fast on a Mac ultimately runs on.

The browser build uses ordinary decoder models — the kind that write text one token at a time. It turns them into classifiers with a choice-logit trick: instead of generating anything, it reads the model’s scores for just the tokens that spell your allowed options and normalises across those. That works, but it isn’t the same as a head trained to output class scores, and the difference matters later.

The browser build really is the right place to start. There’s nothing to install, no backend, no waitlist, and your inputs never leave the browser tab. What it gives up is precision, and the site says so: “browser quantization may change model accuracy.” In Part 2 I measure what 4-bit quantization costs this cross-encoder — which is a different question from what the browser build loses, because it runs different models. Keep that distinction in mind; it’s the difference between a measurement and a guess.

The checkpoints are the other road. There are seven at the time of writing, all Qwen3_5ForSequenceClassification, MIT licensed and ungated. The directory name is what you pass to --include, so here they are exactly:

directory	size
qwen3.5-0.8b-nli-v2s-long	1.7 GB
qwen3.5-0.8b-nli-v5	1.7 GB
qwen3.5-2b-nli-v5	4.5 GB
qwen3.5-4b-nli	9.1 GB
qwen3.5-4b-nli-v2	9.1 GB
qwen3.5-4b-nli-v5	9.1 GB
qwen3.5-35b-a3b-nli	69 GB

The last one is a mixture-of-experts model: 35 billion parameters in total, but only about 3 billion active per token. That’s why it runs faster than its size suggests, and also why it still needs all the memory its size implies.

Everything below was measured on qwen3.5-4b-nli-v2. The model card now recommends v5, which came out after I took these measurements. I checked whether that breaks anything, and it doesn’t: the two share the same class, the same id2label ordering, the same nli_template and the same problem_type. So v5 is a drop-in directory swap and all the code here works unchanged. The accuracy numbers, though, are v2 numbers. If you’re starting fresh, use v5 and expect your results to differ from mine.

---

Part 1 — Getting it to run at all

The classification head here is a single 2560×3 matrix that turns the backbone’s output into the three scores above. It’s the whole difference between this model and a chatbot, and it’s why the usual local runtimes can’t serve it: a token-generation API has nowhere to put three class scores. (There’s a short sidebar on this below if you’re curious. You don’t need it.)

So Ollama and LM Studio are out. Start with Hugging Face Transformers, because it works immediately and gives you a reference to check against, then move to MLX — Apple’s array framework, which compiles to Metal — for speed.

# uv is a fast pip/venv replacement (brew install uv); plain `python -m venv` works too
uv venv --python 3.12
uv pip install --python .venv/bin/python \
    "torch>=2.14" "transformers>=5.17" accelerate huggingface_hub \
    mlx "mlx-lm==0.31.3"        # mlx-lm pinned: see the version note below
 
# two separate commands, deliberately -- see the warning underneath
.venv/bin/hf download AlexWortega/openjev --include "qwen3.5-4b-nli-v2/*" --local-dir openjev
.venv/bin/hf download AlexWortega/openjev --include "modeling_openjev.py" --local-dir openjev

Two version pins are doing real work here. transformers needs 5.17 or later, because that’s the first release with Qwen3_5ForSequenceClassification; older versions fail with an error that doesn’t tell you why. mlx-lm is pinned to 0.31.3 because the MLX code later reaches into its internals, which have since moved on main.

Keep the two download lines separate. It’s tempting to combine them:

# DO NOT do this
.venv/bin/hf download AlexWortega/openjev --include "qwen3.5-4b-nli-v2/*" modeling_openjev.py --local-dir openjev

That fails silently. hf download treats modeling_openjev.py as an explicit file to fetch, and once it has one it ignores --include completely. It prints a warning somewhere among the progress bars and exits successfully. You end up with one small file, no weights, and every sign that it worked.

(The second file, modeling_openjev.py, is the repo’s own reference wrapper. The MLX path doesn’t use it, but it’s handy to compare against.)

Transformers ships the model class natively, so it runs on MPS — PyTorch’s Apple-GPU backend, the mps device you use where you’d normally write cuda — with no extra work. It will print a couple of lines worth actually reading:

`causal_conv1d_fn` is falling back to its reference PyTorch implementation
`chunk_gated_delta_rule` is falling back to its reference PyTorch implementation

Qwen3.5 is a hybrid architecture. Of its 32 layers, 8 use full attention and 24 use gated delta net linear attention, a recurrent, Mamba-like mechanism that scales better with sequence length than standard attention. Both warnings refer to CUDA-only kernels. On a Mac, you’re running the slow reference implementation for three quarters of the model, and on this path that will never change — nobody is going to ship a Metal build of a CUDA kernel. MLX is the way out, and we’ll get there shortly.

Checkpoint: does it work?

The Transformers version is only a few lines:

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
 
# NB: a local directory (the one --local-dir just created), not a Hub id, despite
# looking like one. Run this from the parent of openjev/ or use an absolute path.
CKPT = "openjev/qwen3.5-4b-nli-v2"
tok = AutoTokenizer.from_pretrained(CKPT)
# bfloat16 ("bf16") is 16-bit brain-float: the full-precision default here, as
# distinct from the quantized 8-bit/4-bit variants measured in Part 2
model = AutoModelForSequenceClassification.from_pretrained(CKPT, dtype=torch.bfloat16).to("mps").eval()
 
text = model.config.nli_template.format(
    premise="A man is playing a guitar.", hypothesis="Someone is making music.")
with torch.no_grad():
    probs = model(**tok(text, return_tensors="pt").to("mps")).logits.float().softmax(-1)
print(dict(zip(["contradiction", "entailment", "neutral"], probs[0].tolist())))

You should see entailment ≈ 0.936, with contradiction close to zero. Change the hypothesis to “Nobody is making any sound.” and contradiction should go above 0.99. Change it to “The man is a professional musician.” and neutral should win. Three pairs, one working model.

If the labels are right but it’s slow, that’s the kernel fallback, not something you did wrong. If the labels are wrong, there are three quite different ways that shows up:

• All three probabilities hover around 0.33. The head didn’t load, and you’re reading an untrained score layer.
• Confident, but the wrong label. The class order isn’t what you assumed. Don’t settle this by reading config files — score an obvious contradiction (“The cat is asleep.” / “The cat is awake.”) and see which index goes high. That’s more reliable than id2label, and more reliable than this post. Part 2 covers a published checkpoint whose stated order didn’t match its real outputs.
• An exception on load. Usually the weights only partly downloaded, which is exactly what the hf download trap above does to you.

Sidebar — why won’t Ollama or LM Studio run it?

Isn’t it just missing a GGUF? Partly. These checkpoints are safetensors and both runtimes want GGUF. But converting wouldn’t help.

Why not? Both runtimes exist to generate tokens. This model doesn’t generate anything. It outputs three logits from a linear head on the backbone’s final hidden state, and a token-generation API has nowhere to put them.

Then how does openjev.com run in a browser? It uses a different mechanism: ordinary decoder models read through wllama’s choice-logit path. That’s why those can be GGUF and these can’t.

The MLX port is smaller than you’d think

Before you start writing Metal kernels for gated delta net, check whether someone already has — and check the version you actually installed, not the project’s main branch:

ls "$(.venv/bin/python -c 'import mlx_lm,os;print(os.path.join(os.path.dirname(mlx_lm.__file__),"models"))')" | grep -E 'qwen3_5|gated_delta'
# qwen3_5.py
# qwen3_5_moe.py
# gated_delta.py

Both qwen3_5.py and gated_delta.py are in 0.31.3. The hard part — a correct, Metal-accelerated implementation of the hybrid attention stack — has already been written and is maintained by someone else. What mlx-lm doesn’t give you is the one piece you need. Its wrapper is a causal language model, so it throws away the classification head as irrelevant.

That makes the port mostly a matter of reassembly. Here’s the whole thing — 92 lines, 72 of them code — because three disconnected fragments would leave you guessing at the glue:

"""SemIf cross-encoder on MLX. Save as semif_mlx.py"""
import json, pathlib
import mlx.core as mx
import mlx.nn as nn                       # note: mlx.nn, not torch.nn
import numpy as np
from mlx.utils import tree_unflatten
from mlx_lm.models import qwen3_5
from transformers import AutoTokenizer
 
LABELS = ["contradiction", "entailment", "neutral"]   # the head's output order
 
class SemIfMLX(nn.Module):
    """Qwen3.5 backbone plus the 3-class head mlx-lm's causal-LM wrapper drops."""
    def __init__(self, config: dict):
        super().__init__()
        self.backbone = qwen3_5.Model(qwen3_5.ModelArgs.from_dict(config))
        self.score = nn.Linear(config["text_config"]["hidden_size"], 3, bias=False)
 
    def __call__(self, input_ids: mx.array, lengths: mx.array) -> mx.array:
        # .language_model.model is Qwen3_5TextModel, whose __call__ returns
        # self.norm(hidden_states) -- exactly HF's `last_hidden_state`.
        hidden = self.backbone.language_model.model(input_ids)
        pooled = hidden[mx.arange(hidden.shape[0]), lengths - 1]   # last non-pad token
        return self.score(pooled)
 
class SemIfCrossEncoder:
    """Load once, then .predict(pairs) -> (n, 3) probabilities."""
    def __init__(self, path, bits: int | None = None, bs: int = 8, max_len: int = 4096):
        path = pathlib.Path(path)
        self.config = json.loads((path / "config.json").read_text())
        self.template = self.config["nli_template"]   # "Premise: {premise}\nHypothesis: {hypothesis}"
        self.pad_id = self.config["pad_token_id"]
        self.bs, self.max_len = bs, max_len
 
        self.model = SemIfMLX(self.config)
        weights = {}
        for shard in sorted(path.glob("*.safetensors")):
            weights.update(mx.load(str(shard)))
        score_w = weights.pop("score.weight")          # pull the head out BEFORE sanitize
        weights = self.model.backbone.sanitize(weights)  # drops model.visual*, remaps names
        params = {f"backbone.{k}": v for k, v in weights.items()}
        params["score.weight"] = score_w
 
        self.model.update(tree_unflatten(list(params.items())))
        mx.eval(self.model.parameters())                 # load full precision FIRST
        if bits is not None:
            # group_size=64: blocks of 64 weights share one scale. Backbone only --
            # the 3x2560 head stays full precision, where rounding is all downside.
            nn.quantize(self.model.backbone, group_size=64, bits=bits)
            mx.eval(self.model.parameters())
        self.model.eval()
 
        self.tok = AutoTokenizer.from_pretrained(str(path))
        # predict() pads by hand below, so this is belt-and-braces: it only matters if you
        # later let the tokenizer pad, and then it matters a lot -- the head pools the LAST
        # non-pad token, so padding must go after it, never before.
        self.tok.padding_side = "right"
 
    def predict(self, pairs):
        out = []
        for i in range(0, len(pairs), self.bs):
            chunk = pairs[i:i + self.bs]
            texts = [self.template.format(premise=p.strip(), hypothesis=h.strip())
                     for p, h in chunk]
            # Truncate the PREMISE, never the packed string. The template puts the
            # hypothesis last, so tail truncation silently deletes it and leaves the
            # head pooling a cut-off premise -- confident, meaningless output.
            seqs = []
            for (premise, hypothesis), t in zip(chunk, texts):
                ids = self.tok(t)["input_ids"]
                if len(ids) > self.max_len:
                    hyp_only = self.template.format(premise="", hypothesis=hypothesis)
                    budget = self.max_len - len(self.tok(hyp_only)["input_ids"])
                    if budget < 1:
                        raise ValueError("hypothesis alone exceeds max_len")
                    kept = self.tok.decode(self.tok(premise)["input_ids"][:budget])
                    ids = self.tok(self.template.format(premise=kept, hypothesis=hypothesis)
                                   )["input_ids"][:self.max_len]
                seqs.append(ids)
            width = max(len(s) for s in seqs)
            ids = np.full((len(seqs), width), self.pad_id, dtype=np.int32)
            for row, s in enumerate(seqs):
                ids[row, :len(s)] = s          # right-padded; see the note below
            logits = self.model(mx.array(ids), mx.array([len(s) for s in seqs]))
            out.append(np.array(mx.softmax(logits, axis=-1).astype(mx.float32)))
        return np.concatenate(out, 0)
 
 
if __name__ == "__main__":
    ce = SemIfCrossEncoder("openjev/qwen3.5-4b-nli-v2")
    probs = ce.predict([("A man is playing a guitar.", "Someone is making music.")])
    print({k: round(float(v), 4) for k, v in zip(LABELS, probs[0])})

Run it and you get entailment = 0.9336 on the guitar/music pair, compared with 0.9360 from Transformers. A 0.0024 difference is bf16 rounding, not a bug.

A few details in there are easy to get wrong, so they’re worth spelling out.

Truncate the premise, never the whole string. The obvious approach is self.tok(t, truncation=True, max_length=...) on the packed string. But the tokenizer trims from the end, and the end is where the hypothesis lives. Give it a long enough document and the hypothesis disappears entirely. The head then pools the last token of a chopped-off premise and returns three confident, meaningless numbers, with no error and no warning. I checked this: at max_len=256 with a 3,000-character premise, the naive version loses the hypothesis, while the version above still labels both an entailment and a contradiction correctly.

The input is one packed string. The premise and hypothesis go through config["nli_template"], not a sentence-pair call. No text_pair argument, no manual [SEP]. While you’re in that config, note that hidden_size sits under config["text_config"] while pad_token_id and nli_template are at the top level. These are multimodal configs, so the text settings live in their own sub-object next to a vision_config.

sanitize() does useful work for free. It drops model.visual* — the vision tower these checkpoints carry but never use for text — and remaps model.language_model.* into the layout mlx-lm expects.

Pull the head out first. It lives under score.weight, and it has to come out before sanitize() runs. Sanitize doesn’t know about it and would prefix it into the wrong place.

Quantize after loading, not before. If you call nn.quantize before update(), you quantize the random initial weights and then overwrite them with unquantized values. The result is neither one thing nor the other, and it degrades quietly instead of failing loudly.

Mind the mlx-lm version. The 0.31.3 release on PyPI has no Model.model property, even though GitHub main does, which is why the code reaches through .language_model.model. Pin the version or check for the attribute, or you’ll get a bare AttributeError with no hint about the cause.

One more property makes batching safe, and it’s worth understanding rather than just copying. Right-padding works because every layer in this model is causal — full attention, the recurrent linear-attention state, and the conv1d. A pad token placed after the position you pool from can’t affect that position in any of them. That’s not luck; it’s why the reference implementation also sets padding_side="right".

Checkpoint: did the port survive?

Never trust a port you haven’t checked against the thing it replaced. Once you have both paths, the comparison is short. Run this in the same session as the Transformers snippet, so model, tok and CKPT still exist:

import numpy as np, torch
from semif_mlx import SemIfCrossEncoder
 
PAIRS = [
    ("A man is playing a guitar.", "Someone is making music."),                 # entailment
    ("A man is playing a guitar.", "Nobody is making any sound."),              # contradiction
    ("A man is playing a guitar.", "The man is a professional musician."),      # neutral
    ("The bird is 0.05 below the centre of the gap.", "The bird is below the centre of the gap."),
    ("Paris is the capital of France.", "France has a capital city."),
    ("The cat sat on the mat.", "The dog sat on the mat."),
]
 
mlx_probs = SemIfCrossEncoder(CKPT).predict(PAIRS)        # already (6, 3) numpy
 
rows = []                                                  # the Transformers path, looped
for premise, hypothesis in PAIRS:
    text = model.config.nli_template.format(premise=premise, hypothesis=hypothesis)
    with torch.no_grad():
        rows.append(model(**tok(text, return_tensors="pt").to("mps")).logits.float().softmax(-1))
torch_probs = torch.cat(rows).cpu().numpy()
 
agree = (mlx_probs.argmax(-1) == torch_probs.argmax(-1)).sum()
print(f"label agreement vs torch : {agree}/{len(PAIRS)}")
print(f"prob correlation         : {np.corrcoef(mlx_probs.ravel(), torch_probs.ravel())[0, 1]:.6f}")
print(f"max abs prob difference  : {np.abs(mlx_probs - torch_probs).max():.4f}")

label agreement vs torch : 6/6
prob correlation         : 0.999997
max abs prob difference  : 0.0033

All six labels agree, the correlation is five nines, and the biggest probability difference is 0.0033. Now for speed. These are bf16 timings at batch size 8. The sequence lengths are the tokenized lengths the harness reports, which won’t exactly match the rough target you ask for on the command line, so trust the reported number.

sequence length	torch / MPS	MLX bf16
71 tokens	352 ms/pair	137 ms/pair
449 tokens	1747 ms/pair	989 ms/pair
1745 tokens	6663 ms/pair	3985 ms/pair

Runs vary by a few percent (a repeat of the middle row came in at 953 ms), so read these as ratios rather than exact figures.

MLX is 1.7x to 2.6x faster, because its Metal kernels replace the reference PyTorch implementation of the gated delta rule. The ratio shrinks as sequences get longer (2.57x, 1.77x, 1.67x), but the absolute saving grows twelvefold (215 ms, 758 ms, 2678 ms). If the difference were just fixed per-call overhead, the absolute saving would stay flat. It doesn’t, so MLX is cheaper both per call and per token, with the biggest relative win on short inputs.

You’ll notice Part 2 reports a slower per-pair time (186 ms at batch 16) despite shorter inputs than this table’s first row. Two things differ: this table uses inputs of equal length, while Part 2’s batches are ragged and get padded to their longest member, and the batch sizes aren’t the same. I break that down properly near the end.

So it runs, it matches, and it’s meaningfully faster. That earns us the next question, but doesn’t answer it: fast and self-consistent isn’t the same as accurate.

---

Part 2 — Measuring it, and finding out precision is a choice

The model repo on Hugging Face ships an evaluation harness in its code/ directory, and reading it is the quickest way to understand what this kind of model is for. The commands in Part 1 didn’t fetch it, so grab it now:

.venv/bin/hf download AlexWortega/openjev --include "code/*" --local-dir openjev

eval.py implements two standard ways of using an NLI cross-encoder, both taken from a post on the dleemiller Hugging Face blog (huggingface.co/blog/dleemiller/nli-xenc-ways-to-use):

• Reranking: put the question in the premise and a candidate answer in the hypothesis as “The correct answer is: {option}”, then pick the option with the highest entailment probability.
• Grading: put the question plus a reference answer in the premise and the candidate in the hypothesis as “Answer: {option}”, then treat entailment as “correct.”

The same three numbers serve both jobs; only the framing changes. That’s really the whole trick, and once you see it, you start spotting problems that fit the shape everywhere.

Get the label order right first

One thing needs sorting out before you run anything, because getting it wrong silently inverts every result. This checkpoint’s head outputs [contradiction, entailment, neutral] — index 0 is contradiction, just as id2label in its config.json says. But MNLI, the benchmark we’ll use, orders its dataset labels differently: 0 = entailment, 1 = neutral, 2 = contradiction. So the harness carries a remap ({0: ENT, 1: NEU, 2: CON}) to translate dataset labels into the model’s order.

That’s two different orderings with a translation between them, which is fine as long as you know about it. It’s worth being careful, though: eval.py notes that at least one public NLI checkpoint ships a config.json whose stated order doesn’t match its real outputs. Trusting it drops MNLI accuracy to 3%. That’s far worse than the 33% you’d get by guessing, because a wrong label permutation isn’t random — it’s systematically anti-correlated.

There’s a second obstacle. Upstream eval.py is written for CUDA (device="cuda", torch.cuda.synchronize()), so as shipped, none of the published numbers are reproducible on a Mac. My eval_mlx.py is a native port that keeps the task semantics identical: the same label order and MNLI remap, the same prompt construction for both modes, and the full set of metrics. That includes rerank_margin_acc, which ranks options by P(entailment) − P(contradiction) instead of entailment alone and sometimes separates them better, and the two-class collapse SciTail needs, since it only labels entailment versus not-entailment.

It covers thirteen tasks: MNLI; six harder NLI sets (ANLI r1–r3, WANLI, SciTail and ConTRoL); and six multiple-choice sets (ARC-Easy and ARC-Challenge separately, HellaSwag, WinoGrande, MMLU and GPQA-diamond). You don’t need to know them individually. They’re standard public benchmarks, here only to confirm the local harness agrees with published results before it measures anything new.

# companion repo, not yet public -- the runnable version is a few paragraphs down
.venv/bin/python eval_mlx.py --tasks mnli --n 150
.venv/bin/python eval_mlx.py --tasks arc_easy hellaswag gpqa --mc-n 60

The baseline

MNLI is the standard benchmark for this task: about 400,000 sentence pairs, each labelled entailment, contradiction or neutral by human annotators, drawn from ten genres of text. It has two validation splits. Matched uses genres that appear in the training data; mismatched uses genres that don’t, so it tests generalisation. A model that scores about the same on both — or a little better on mismatched — is behaving sensibly, not suspiciously.

On the 4B checkpoint at bf16, MNLI comes back at 0.933 matched and 0.940 mismatched, 150 examples each. Their average, 0.937, is the figure I use for “MNLI accuracy” everywhere else in this post.

The model card reports about 0.91. With 300 examples, the 95% interval around 0.937 is roughly [0.909, 0.964], so 0.91 sits right at its lower edge. That’s consistency in the weak sense of “not contradicted,” not a replication. The card’s figure is presumably on the full validation split, where the interval is much tighter, so don’t read this as the checkpoint beating its own card.

ARC-Easy reranking, as a sanity check on a different kind of task, lands at 0.900 over 60 questions. Treat all of these as smoke tests, not leaderboard entries.

The experiment worth running yourself

You don’t need my harness for this one. With semif_mlx.py from Part 1 exactly as printed, the whole measurement is about twenty lines, and the only extra dependency is datasets:

"""What does quantization cost this classifier on MNLI?  pip install datasets"""
import time
import numpy as np
from datasets import load_dataset
from semif_mlx import SemIfCrossEncoder             # the class from Part 1
 
CKPT, N = "openjev/qwen3.5-4b-nli-v2", 150
MNLI_TO_OURS = {0: 1, 1: 2, 2: 0}                   # MNLI order -> [con, ent, neu]
 
ds = load_dataset("nyu-mll/multi_nli", split="validation_matched")
ds = ds.filter(lambda x: x["label"] in (0, 1, 2)).shuffle(seed=0).select(range(N))
pairs = list(zip(ds["premise"], ds["hypothesis"]))
gold = np.array([MNLI_TO_OURS[l] for l in ds["label"]])
 
for bits in (None, 8, 4):
    ce = SemIfCrossEncoder(CKPT, bits=bits, bs=16)
    ce.predict(pairs[:2])                                    # warm up the kernels
    t0 = time.perf_counter()
    probs = ce.predict(pairs)
    dt = time.perf_counter() - t0
    acc = (probs.argmax(-1) == gold).mean()
    print(f"{'bf16' if bits is None else f'{bits}-bit':>6}  acc={acc:.4f}  "
          f"{N/dt:5.2f} pairs/sec  ({dt/N*1000:.0f} ms/pair)")

On this checkpoint it prints:

  bf16  acc=0.9333   5.37 pairs/sec  (186 ms/pair)
 8-bit  acc=0.9333   4.88 pairs/sec  (205 ms/pair)
 4-bit  acc=0.8867   3.73 pairs/sec  (268 ms/pair)

MNLI_TO_OURS is the remap from above. Leave it out and you’ll measure something near chance and conclude the model is broken. For the mismatched split, change one word: split="validation_mismatched". Your absolute throughput will differ from mine; the ordering is what matters.

Here are both splits together. (The fuller harness sweeps more tasks, but everything important is in the script above.)

# companion repo, not yet public -- the inline script above does the same measurement
.venv/bin/python eval_mlx.py --tasks mnli --compare-quant

	bf16	8-bit	4-bit	Δ 8-bit	Δ 4-bit
MNLI matched	0.9333	0.9333	0.8867	+0.0000	−0.0467
MNLI mismatched	0.9400	0.9400	0.8800	+0.0000	−0.0600
MNLI mean	0.9367	0.9367	0.8833	+0.0000	−0.0533
throughput (pairs/sec)	5.37	4.88	3.73	−9%	−31%

(If you’re checking my arithmetic — and you should — the deltas are computed from unrounded values, so subtracting the four-decimal figures shown can be off by 0.0001.)

8-bit first, because it’s the boring result, and boring is the point: no change on either split. Matching scores on their own could hide flips that cancel out — ten wrong one way, ten wrong the other — so compare the predictions directly. Add preds = {} before the loop, preds[bits] = probs.argmax(-1) inside it, and (preds[None] != preds[8]).sum() at the end. On the matched split it’s 0: all 150 predictions identical. Not just the same score — the same answers.

4-bit costs 5.3 points of MNLI accuracy, and that matters more here than it would for a chatbot. When you over-quantize a generative model, the damage shows up as slightly worse writing, and a sampling step and a human reader soak up some of it. A classifier has neither buffer. You’re reading three numbers and thresholding them directly, so a degraded logit isn’t a slightly worse sentence; it’s a flipped label, delivered with full confidence and no hint that anything went wrong. When I re-ran Part 1’s six-pair check at 4-bit, one contradiction came back as neutral. A decision service that quietly takes the wrong branch is worse than one that admits it doesn’t know.

The table also corrected two assumptions of mine. 8-bit isn’t faster. On a tiny smoke test it looked slightly quicker, but measured properly it loses about 9% of throughput, because dequantizing weights for every matrix multiply costs more than the smaller weights save at these lengths. And 4-bit is the slowest of all, which makes it strictly worse: least accurate and lowest throughput. Its only remaining advantage is size — about 2.3 GB versus 9.1 GB for bf16, with 8-bit around 4.6 GB. On a Mac with unified memory, that’s a trade almost nobody should make.

So the rule is simple: use bf16 by default, 8-bit only if nine gigabytes of weights is genuinely your limiting factor, and never 4-bit for a classifier.

It’s tempting to apply this straight to the 88.3-versus-84.5 gap from the introduction. Resist that, because the logic doesn’t hold. The browser build runs different models — decoders read through a choice-logit path, not these cross-encoders — and the site doesn’t say which quantization level it uses. “4-bit costs this cross-encoder 5.3 points” says nothing about that gap; it’s evidence about a different architecture on a different task.

What it does show is narrower but still useful: precision is a variable people routinely leave uncontrolled when they compare local models with hosted ones. If you benchmark anything quantized against an API, you’re measuring the model and the compression together and reporting them as one number. Running natively takes that confound out of your own system, which is the only one you can actually check.

That’s two of the four confounds dealt with. We’ve partly measured capability, and we can now control precision. That leaves setup and data — and for those, we need a real task.

---

Part 3 — A real task, and three failures that look like the model’s fault

Benchmarks tell you a model can do what benchmarks measure. The more interesting question is whether it does useful work. So here’s a real job: given an email or meeting invitation, assign it a project and a business unit from an organization’s own taxonomy of about fifty projects. Tag it automatically when the model is confident. When it isn’t, send it to a person along with a ranked shortlist instead of a guess. Email triage is one of the examples in SemIf’s own browser demo, so this is squarely what it’s meant for.

There’s one structural difference from the demo that shapes everything else. The demo accepts up to twenty options. Fifty projects is too many to just hand the model every choice. At roughly 400 ms per comparison on a realistic premise, scoring all fifty would take about twenty seconds per message, versus six for a sixteen-deep shortlist. That factor of three decides whether a nightly queue finishes. So the pipeline filters cheaply first and reranks expensively second, which is exactly the division of labour a cross-encoder is built for.

flowchart LR
    M["Message:<br/>subject · body ·<br/>participants · existing tags"] --> P["Build premise<br/>(compact digest)"]
    P --> F["Lexical prefilter<br/>~50 projects → 16"]
    F --> X["Cross-encoder reranks<br/>the 16 survivors"]
    X --> D{"Confident?"}
    D -->|"p ≥ 0.60<br/>top1−top2 ≥ 0.30"| A["Auto-tag"]
    D -->|"plausible,<br/>not confident"| R["Human queue +<br/>top-5 ranked options"]
    D -->|"nothing fits"| N["Flag as<br/>possible new project"]

    classDef default fill:#dbe4ee,stroke:#52657a,color:#1f2933
    classDef accent fill:#bfe3dd,stroke:#1f7a70,color:#0d3833
    classDef warn fill:#f0dfae,stroke:#a07d1c,color:#4a3a10
    class A accent
    class R warn

Figure 2. Filter cheaply, rerank expensively, and route by confidence instead of forcing a decision. “p” is P(entailment) for the winning project, and the second gate is the gap between the top two candidates. (That’s a different quantity from Part 2’s P(ent) − P(con) margin, despite the shared word.) The three-way exit is what makes it safe to deploy: it’s allowed to abstain.

Since the rest of this section is about how sensitive the model is to exact wording, the two boxes on the left are worth making concrete.

The premise is a compact digest, with the most informative fields first and the body last, because the body is what gets cut:

Email subject: Spindle beta onboarding checklist
From: someone@example.com
Participants: a person, b person
Existing tags: MER: Spindle
Body: Following up on the Spindle private beta. We need the customer tenant allowlisted
before Friday so their team can publish components to the registry...

Quoted reply history and signatures are stripped before the body is trimmed to about 700 characters, because they drag the classification toward whatever the earlier thread was about. That budget is why premises come out around 200 tokens, which matters for the cost table later.

The prefilter is deliberately crude, because all it has to do is not lose the right answer. It builds one bag of tokens from the subject, the first 2,000 characters of the body, the sender and any existing tags. Then, for each project:

• It scores each name separately — the canonical name and each alternate — as matched_tokens / form_tokens, and keeps the best one. Scoring names separately instead of pooling them means it knows which name actually appeared, so that name can be reused in the hypothesis later.
• It gives half credit to any token longer than four characters that shows up as a substring rather than a whole word, so “ledger” still fires on “LedgerModernisation”.
• It scores the description the same way at weight 0.6, and any extra search-only terms at 0.9, then takes the maximum against the name score. These can rescue recall, but they never get to phrase the hypothesis.
• It adds a big bonus if the project’s tag is already on the thread, and a tiny prior based on how often the project has been used before.

Then it keeps the top 16. There’s no BM25 or other ranking function, and no embeddings. The cross-encoder handles precision; the prefilter’s only job is not to drop the right answer before it gets there.

Sixteen was measured, not picked. Over 297 real filed messages whose correct label still existed in the taxonomy, recall was 77.1% keeping 8 candidates, 88.2% at 12, 95.6% at 16, and 99.7% at 24. Keeping eight would have silently thrown away 23% of the right answers. Twenty-four would recover another four points, but every extra candidate is another cross-encoder call, and at about 400 ms each, 24 would push a message from about six seconds to nearly ten. So it’s recall against wall-clock time, and where you land depends on whether a missed project or a slow queue hurts you more. Sweep your own K: the recall curve depends entirely on how distinctive your labels are. On the companion repo’s fictional taxonomy, recall is 100% at every K — which says more about the test data than about the method.

The two confidence thresholds were swept too. On 300 real filed messages, raising p from 0.60 to 0.90 (with the gap fixed at 0.30) shrank the auto-tagged set from 25 messages to 23, while precision stayed at 95.7–96.0%. The one wrong tag is never among the messages the stricter bar drops, so raising p just costs you two correct auto-filings and keeps the error. Be honest about the granularity: with 25 items, one message is four percentage points, so “about 96%” really means “exactly one wrong.” I didn’t try anything below 0.60 or vary the gap, so treat both values as where I stopped, not as proven optimums. The right settings for you depend on how much a wrong tag costs compared with a trip through the review queue.

The first version of this pipeline scored 7 out of 10 on a labelled set. That looks like a threshold problem, or a shortlist problem, or proof that a 4B model can’t do fifty-way classification. It was none of those.

Failure 1: the question you ask is the answer you get

The natural way to phrase the hypothesis is as a statement about the message: “This message is about the Atlas project.” It’s also nearly useless, and the reason is a property of NLI, not of the model.

NLI asks whether a hypothesis follows from a premise. A premise made of a subject line and a body never says anything about its own status as a message concerning a project. So the model, quite correctly, won’t call it entailed. Swap in a direct claim about the content and everything changes:

hypothesis form	top-1	mean p(correct)	mean p(others)
This message is about the {name} project.	7/10	0.205	0.005
The email discusses {name}.	10/10	0.989	0.002

That’s ten labelled messages, each scored against the same fixed eight-project shortlist (smaller than the pipeline’s sixteen, so every phrasing faces an identical, easy-to-inspect decision). mean p(correct) is the average entailment probability given to the right label; mean p(others) is the average across the seventy wrong ones.

The technical term for that phrasing is the verbalizer: the template that turns a label into a natural-language claim the model can evaluate. Sweeping yours takes minutes, and here it was worth more than every other change combined. Do it before you touch a threshold, a shortlist size or a model choice.

Two caveats, both of which make the finding more useful. First, ten messages is a small sample, so read 7/10 versus 10/10 as illustrative. The probability columns are the real evidence: 0.205 against 0.005 is a signal you have to squint at, while 0.989 against 0.002 is one you can easily threshold.

Second, how much this matters depends on how easily your labels get confused with each other. This taxonomy has projects whose names overlap — several involve licensing, pricing or agreements — so absolute confidence is what separates them. On the companion repo’s fictional taxonomy, where the names are distinctive made-up words, every phrasing scores 10/10, and even the “about the message” version reaches 0.890 against 0.985 for “discusses.” Both are perfectly usable there.

That isn’t the mechanism contradicting itself. The meta-claim is always the weaker question; what changes is whether weakness costs you anything. When a single word identifies a single label, a weak question still ranks things correctly. When labels compete for the same words, its low confidence is exactly what stops you telling the winner from the runner-up. So phrasing doesn’t become more important as labels get confusable — it becomes load-bearing.

Failure 2: the model can’t recognize a name you never gave it

Real projects rarely have just one name. There’s the code name from when the work started, the internal name the owning team uses, the shorthand people actually type in subject lines, and sometimes a public release name — all in use at the same time. The taxonomy holds one of them. The email uses another.

Here’s a message that only mentions a project by its code name, scored against the code name and then against the official taxonomy entry:

hypothesis	p(entailment)
The email discusses Project Hydra. (the name the text actually uses)	0.996
The email discusses Atlas Migration. (the canonical taxonomy name)	0.000

Zero. Not degraded, not borderline — invisible at any threshold, with nothing anywhere to warn you that a whole category of mail is being systematically missed.

The fix follows directly, and the order of operations keeps it cheap. The lexical prefilter, not the model, checks each of a project’s known names against the text and records which one matched. The cross-encoder is then called once per project, with the hypothesis phrased using that matched name. So aliases cost nothing extra at inference time: you were already scoring each shortlisted project once, and you’ve just stopped asking about a word that isn’t there. (Running every alias through the model would mean N calls per project, and that’s not what this does.)

Failure 3: a description is not a name

The third version of the same problem is the one most likely to catch you, because it looks like being helpful. Every project in the taxonomy has a short description, so adding it to the hypothesis seems like an obvious improvement — more context for the model. Here’s what actually happens:

hypothesis	p(entailment)
The email discusses Spindle.	0.996
The email discusses Spindle (private beta of the component registry).	0.549

The extra clarification knocks off 45%, enough to push confident cases below a 0.60 threshold and lose the auto-file. The description is true and relevant, but it’s still the wrong thing to put in the question. The premise mentions publishing components to a registry, but it never says that Spindle is “the component registry,” and that’s exactly what the parenthetical claims. You’ve handed the model a compound claim, and it correctly hesitates on the half it can’t verify. Descriptions belong in the prefilter, where extra vocabulary can only help.

So those are three failures, and really one lesson at different scales. The model answers the question you ask, in the words you give it, and every word you add is one more thing it has to find support for. None of these was a capability limit, and a bigger model wouldn’t have fixed any of them.

---

Two practical asides

Sidebar — mining aliases from your own history

Every message you’ve already filed pairs real wording with a known-correct label. A term that shows up almost only under one label is probably a name for it, so it’s worth harvesting — with two constraints that aren’t obvious until you measure them.

Keep the two jobs in separate fields. Only genuine alternate names should get to phrase the hypothesis. Everything else can widen the prefilter’s net, and nothing more. The mined terms that look safest — ones that contain the canonical name — are actually the worst. They add no coverage the canonical name didn’t already give you, and if you let them phrase the question, a subject-line fragment replaces a proper project name. Mixing the two jobs cost 3.3 points of top-1 accuracy over 120 real filed messages, with no recall gain in return.

Keep a person in the approval step. Automatic acceptance fails in both directions. Score by “rare single word” and the miner approves city names and out-of-office abbreviations as code names. Score by “distinctive multi-word phrase” and it approves “search tool” and “office hours variant.” Propose candidates with everything rejected by default, and let a human switch on the real ones, because a wrong alias is worse than a missing one. A missing alias costs recall on one project. A wrong one creates false positives at every threshold, forever.

Sidebar — when not to use the model at all

Thread continuity. A reply that says “numbers attached, nothing to flag” has nothing to classify. When a thin reply lands on a thread that’s already tagged, just use the existing tag — a simple rule beats inference that has nothing to work with. This only applies to thin replies. A substantive message on a tagged thread still gets scored, which is why the example premise earlier shows an existing tag and gets a model score, and why the prefilter treats an existing tag as a strong signal rather than an automatic answer.

Anything structural rather than topical. Assigning a business unit looks like the same job as assigning a project, but it isn’t. A unit isn’t a topic, and names like “Platform” or “CIO” never appear in the text, so scoring them against the message barely works. A unit is really a property of who’s on the thread. Resolving it from the participant list against a learned directory works well, as long as you weight each person by the square of how consistently they map to one unit.

Concretely: for each person p seen on past threads, let dist(p) be their distribution over units and purity(p) = max(dist(p)). Give them weight w(p) = min(1, n(p)/3) · purity(p)², where n(p) is how many past threads they’ve appeared on (so anyone seen three or more times is fully trusted). Score each unit u as Σ_p w(p) · dist(p)[u], normalised across units. Auto-assign when the top unit has at least a 0.65 share and a 0.25 lead over second place.

thread composition	result
one member who reliably maps to one unit	1.00 → auto-assign
that member plus the exec who is on everything	0.95 → auto-assign
members from two different units	0.50 / 0.50 → ask a human
only the exec who is on everything	0.28 → ask a human

To be fair, squaring doesn’t change any verdict in that table. Row two passes either way (0.85 without squaring, 0.95 with), and row four is decided by the share threshold, since with only one person the weight cancels out. What squaring buys is headroom. An exec spread evenly across four units has purity = 0.25, and 0.25² ≈ 0.06 cuts their influence to about a sixteenth of a decisive member’s. With one such person on a thread that’s cosmetic. With five — which is common — their combined unsquared weight is what would otherwise outvote the one person who actually indicates the unit.

Either way, the practical benefit is that people spread across many units automatically count for almost nothing, with no exclusion list to maintain as roles change. A 4B model would be an expensive and worse way to do a weighted lookup.

---

So, can it do the job?

Yes, on clean inputs. But that turned out not to be the interesting question.

The honest way to find out is to change one factor at a time and see which one moves the number. Same model, same taxonomy, four conditions. The samples are small, so read these as directional: 21 hand-written messages for the first three rows, and 16 for the stale-taxonomy row, because that condition removes five projects along with their messages. (Those five aren’t thrown away — I score them separately below, to see whether the system abstains on them.) One message is worth 4.8 points out of 21 and 6.3 out of 16.

condition	top-1	top-3	auto-tagged	precision when auto-tagged
full message, aliases populated	90.5%	100%	90.5%	94.7%
subject line only	85.7%	100%	61.9%	100.0%
full message, no aliases	76.2%	90.5%	76.2%	93.8%
full message, stale taxonomy	87.5%	100%	87.5%	92.9%

On clean inputs, the model gets 90.5% top-1, and every correct answer lands in the top three. With 21 messages, “100% top-3” means “no misses in 21 tries,” not a guarantee. But it’s enough to say the model isn’t the bottleneck — and it wasn’t at 7 out of 10 either.

The subject-only row is more interesting than its headline number. Top-1 barely drops, to 85.7%, but the auto-tag rate falls from 90.5% to 61.9% while precision rises to 100%. Take away context and the model doesn’t start getting things wrong; it starts declining to commit. If there’s a human queue behind it, that’s exactly the right way to fail, and it’s the single most reassuring result in this whole exercise.

The stale-taxonomy row needs a caveat of its own, because it compares two different message sets: 19/21 against 14/16. I didn’t measure the baseline on just those same 16 messages, so the 3.0-point “staleness” effect is the loosest number in the table. If the restricted baseline were 15/16, the real effect would be 6.3 points; if it were 14/16, the effect would be zero. Since the full baseline had exactly two errors in 21, 14/16 is at least as likely as 15/16.

The 40% question

Now for the number I’ve been putting off. Run the same pipeline on the real historical mail and it scores 40% top-1, not 90.5%. Both numbers are real, and explaining the gap is messier than I’d like.

The three factors I can measure are subject-only premises (4.8 points), a missing taxonomy entry (3.0, possibly zero), and missing aliases (14.3). Even adding them up naively — and they’re not really additive, because they overlap — gets you 22 points, which lands at 68%, not 40%. On top of that, those deltas came from the 21 hand-written messages, while the 40% comes from 120 real ones. Subtracting effects measured on one set from a baseline on another is loose by construction, and that’s part of why the arithmetic doesn’t close.

That leaves roughly 28 points unexplained. The likely culprits, none of which I’ve isolated:

• Label noise. Historical tags filed by a busy person aren’t a gold standard.
• A long tail of projects with only two or three examples each.
• Records whose correct label no longer exists in any form. That was 3 of 300 on the sweep set, so probably a similar rate here, though I didn’t check this set.

The ablation doesn’t fully add up, and I’d rather say so than round it off. What the four conditions do show is that the model reaches 90.5% on clean inputs, and that each factor moved the number in the direction the gap predicts — with the caveat that the staleness effect might really be zero.

That forces an honest reframing of what I built. On live data, it’s a strong ranked suggester and a weak auto-tagger. Over the same 120 real messages, top-5 accuracy is 80.8% against 40% top-1, so a five-option shortlist contains the right answer four times out of five.

And here’s the number a deployment decision actually hinges on: on the 300-message sweep set, only about 8% of messages cleared the bar for auto-tagging at all — those are the 25 messages behind the ~96% precision I mentioned earlier. So 96% precision and 40% top-1 are both true, and they don’t conflict. Confidence gating separates them by abstaining on the other 92% instead of guessing. That’s genuinely useful — a ranked shortlist is most of what triage needs — but it isn’t the “90% automated” story the clean benchmark would tempt me to tell. The difference between those two claims is the difference between a system people trust and one they switch off.

The catch-all trap

One result from the staleness experiment is a deployment hazard rather than an accuracy number, and it’s what I’d most want you to take away. I removed five projects from the taxonomy and ran their messages through. Four went safely to human review. The fifth was confidently auto-tagged into the generic catch-all category. A fallback bucket is where mail about missing labels quietly goes to die, because nobody reviews a confident decision.

That result is fragile in a way that reveals the mechanism. Re-run the same experiment on the companion repo’s fictional taxonomy and all five go safely to review — no silent mis-tag at all. The difference is that the fictional projects have distinctive made-up names, while the real ones share vocabulary with each other and with the catch-all’s description. So the hazard isn’t “catch-all buckets are dangerous” in general. It’s that a catch-all whose description overlaps your real labels will soak up orphaned mail, and how badly depends on how distinctive your names are. Either way, audit that category regularly, and treat a rise in its volume as a sign your label list has drifted, not as a category that happens to be growing.

---

What it costs to run

The MLX timings in this post range nearly seventy-fold, from 58 ms to 3,985 ms per pair (and the torch path reaches 6,663 ms), which makes it easy to cherry-pick. So here’s one consistent slice — same machine, bf16, warmed up — across the premise lengths that matter for document work. Here the spread is about sevenfold.

premise length	batch 1	batch 8	batch 16
~21 tokens	114 ms	58 ms	58 ms
~57 tokens	218 ms	127 ms	120 ms
~201 tokens	437 ms	389 ms	380 ms

Three things fall out of this, and the third is the one to act on.

Cost rises with premise length. Roughly linearly in the batched columns, though not at batch size 1, where per-token cost more than halves (from 5.4 to 2.2 ms) as the sequence gets longer. Either way, the fast numbers in this post come from short inputs.

Batching stops helping as premises get longer. At 21 tokens it nearly halves the cost per pair, because a short sequence leaves the GPU underused and a batch fills it up. At 201 tokens the gain is only 13%, because one sequence already keeps the GPU busy. So if you’re reranking realistic documents, batch for convenience, not speed.

A batch costs as much as its longest member. The table above uses equal-length inputs, so it’s the best case. Real batches are ragged: every sequence gets padded to the longest one, and the way to see the waste is to count the tokens the model actually processes, not the tokens you gave it.

Part 2’s matched MNLI split is 150 pairs averaging 43 tokens — 6,512 real tokens. Shuffled into batches of 16, padding inflates that to 13,956 processed tokens. Sort the same 150 pairs by length first, and it only inflates to 7,508. Timed back to back in the same run, that’s 178 ms per pair shuffled versus 100 ms per pair sorted — a 44% saving from one argsort. (That 178 ms is the same measurement Part 2 reported as 186 ms; the ~4% difference is run-to-run variance, which is why I quote the saving from a single paired run.)

The ratios line up: 7,508/13,956 is 0.54, and the time ratio is 0.56, with the difference coming from fixed per-call overhead. Don’t reason from average padded width here — it’s 90 versus 59, which predicts a smaller gain — because 150 pairs is nine full batches plus one of six, and sorting pushes the long tail into that last small batch. Count padded tokens instead. If you’re batching anything with mixed lengths, bucket by length first. It’s the cheapest speedup available here, and I nearly missed it.

Which brings us to the number that matters for sizing: a fifty-project taxonomy, reranked sixteen deep, against a ~200-token email premise, measured end to end, comes to about 6 seconds per message, and batching doesn’t rescue it at this length. That’s fine for a queue that runs on a schedule, but it’s not an interactive latency budget.

---

A diagnostic order

If your own numbers disappoint, work through these in order. They’re sorted by how cheap they are to check versus how likely they are to pay off, which is roughly the opposite of where instinct sends you.

1. Check the verbalizer. Print the exact hypothesis strings you’re sending. If any of them is a statement about the input rather than a claim about its content, stop and fix that first. Here, it was worth more than everything else combined.
2. Check that the name you’re asking about actually appears in the text. Score one known-correct example against the canonical label and against whatever the text actually calls it. If the canonical version scores near zero, you have a vocabulary problem, not a model problem.
3. Check what else is in the question. Strip descriptions, glosses and parentheticals down to the bare name and re-score. Every extra clause is something the model has to find support for.
4. Check that the premise contains the signal — and that the hypothesis survived. Log the exact string you tokenized, decoded back from the token IDs. If the body was truncated, stripped or empty, your ceiling is subject-line accuracy. Worse, if your template puts the hypothesis last and you truncate the whole packed string, the hypothesis is the first thing to go, and the failure is completely silent.
5. Check numeric precision — the data type, not the classification metric with the same name. If you’re quantized at all, re-run one benchmark at bf16 before you believe any accuracy number. Here, 4-bit changed labels and 8-bit cost throughput.
6. Check whether the right answer is even in the option list. Measure prefilter recall separately from ranking accuracy. They have different fixes, and it’s easy to tune one while the other is the real constraint.
7. Check that your labels are trustworthy and your classes have enough examples. Historical tags filed by a busy person aren’t a gold standard, and a class with two examples can’t be learned from or fairly evaluated. These are my leading suspects for the 28-point residual in this project — untested, but nothing in steps 1–6 touches them.
8. Only then consider a bigger model. Everything above is cheaper.

---

What it all adds up to

Let’s go back to the four confounds from the introduction. Each one turned out to be measurable on a system I could actually inspect, which is the useful version of this exercise.

Capability was never the constraint. At full precision, the model scored 0.937 on MNLI and 90.5% top-1 on a fifty-way task with clean inputs. That’s better than a lot of production classification jobs need.

Precision is a choice, not a property, and it’s a cheap one to get right. Here, 4-bit cost 5.3 points of MNLI accuracy, 8-bit cost nothing measurable in accuracy at this sample size (though about 9% in throughput), and 4-bit wasn’t even faster. That result is about this cross-encoder on this task and doesn’t transfer to anyone else’s numbers. What it does give you is a variable you can control: run natively, and compression stops being mixed into whatever you’re measuring.

Setup was the dominant factor here. Phrasing the hypothesis as a claim about content rather than about the message took top-1 from 7/10 to 10/10. Giving the model the names people actually use moved it 14 points, and for the three messages that used only a code name, it was the difference between none of them landing and all three. Neither of those is a model capability, but both look exactly like one from the outside — which is what makes them dangerous. A team that hit 7 out of 10 and decided “we need a bigger model” would have spent money to fix a template.

Data is the one no model can make up for. A taxonomy six months out of date doesn’t just lose accuracy on the projects it’s missing; it can confidently misfile them into whatever generic category is left, and nobody notices. That’s the failure worth building a monitor for, and it isn’t a modelling problem at all.

So the conclusion isn’t the one I expected when I started. The interesting question was never “is the open-weight model good enough?” On this evidence — full precision, on a laptop — it comfortably is. The interesting question is whether you’re willing to give it full precision, a current vocabulary, and a question phrased the way it actually reads. In every case I measured, the gap between 40% on real filed mail and 90.5% on clean inputs was on my side of the interface. Those two numbers come from different sets — 120 real messages versus 21 written to be unambiguous — so treat the span as an illustration of how much the inputs matter, not a prediction of what you’ll get.

If you only do one thing from this post, run the twenty-line quantization measurement from Part 2. It needs nothing but the class from Part 1 and datasets, it takes a few minutes, and it’s the result most likely to change how you deploy a classifier.

---

Appendix A: which numbers come from which set

Ten sample sets and subsets appear in this post, and mixing them up is the easiest way to misread the results. Here they are in one place.

set	size	what it measures
hand-written A	10	the verbalizer sweep
hand-written B	21	the four-condition table
— its subset	16	the stale-taxonomy row, after removing five projects
— the remainder	5	the orphaned messages, scored for the catch-all hazard
filed mail A	120	top-1 and top-5 figures, and the alias-mining cost
filed mail B	300	the threshold sweep and its ~96% auto-tag precision
— its usable subset	297	prefilter recall (the 3 whose gold label no longer exists are dropped)
MNLI	300	the quantization table, 150 per split
fictional fixture	21	the companion-repo condition table
— its sweep subset	10	the fixture’s verbalizer re-run

Two coincidences worth flagging so they don’t trip you up. The MNLI sample and filed-mail set B are both 300 and completely unrelated. And the fictional fixture also has 21 messages, which is why its alias effect exactly matches the real one at +14.3 points: both move three messages, not because the datasets agree.

Treat the fixture numbers differently from the rest. “95.2% top-1,” “recall is 100% at every K,” “every verbalizer scores 10/10” and “five out of five routed safely” all describe synthetic data, and I’ve labelled them as such where they appear. Any number in the post without a stated set comes from hand-written set B.

Appendix B: the companion repo

Everything you need to get a classifier running and measure what quantization costs it is inline above. That’s deliberate, since the companion repo isn’t public yet. What the repo adds is breadth rather than anything essential: the thirteen-task harness, the four-condition ablation, the verbalizer sweep, and a fictional taxonomy of 52 projects and 10 business units so the Part 3 pipeline runs without private data.

The numbers in this post were measured against a real taxonomy and real mail, so the fixture can’t reproduce them exactly — it’s a different dataset. What it does reproduce, closely, is the shape of every finding. Aliases are worth +14.3 points in both. Messages that use only an alias score 100% with alternate names and 0% without, identically. The clean condition reaches 95.2% top-1 with 100% auto-file precision. Where the fixture differs, it’s instructive rather than broken, as with the staleness result above.

Here’s what the repo will contain — listed so you can see the evidence behind Part 3, not as commands you can run today:

# Part 1 — port and verify
.venv/bin/python run_semif_mlx.py --smoke      # it works
.venv/bin/python compare_mlx_torch.py            # it matches torch
.venv/bin/python bench.py mlx 512 8              # it is faster
 
# Part 2 — benchmark and quantization
.venv/bin/python eval_mlx.py --tasks mnli --n 150
.venv/bin/python eval_mlx.py --tasks mnli arc_easy --compare-quant
 
# Part 3 — the failures, reproduced
.venv/bin/python sweep_templates.py              # verbalizer effect
.venv/bin/python benchmark_capability.py         # the four-condition table