Cross-Primitive Consistency in Jev

Cross-Primitive Consistency in Jev. By Xu Liu and three others, from Zhejiang University and vivo. Published on the twenty-eighth of October.

Jev supports efficient decision-making with large language models through typed querying primitives. The same semantic state can be put to the model in different formats, for example as a binary question or as a multiple-choice one. A natural expectation follows: if the underlying question has not changed, the decision should not change either. Whether decisions really stay consistent across primitives has not been systematically evaluated.

PrimaCons is a benchmark built to measure exactly that, cross-primitive consistency. It assesses consistency along four axes, and it does so while preserving the semantics of the query, so any difference in the answer comes from the format and not from a change in meaning.

The evaluation runs across three sizes of Jev. Beyond the benchmark itself, the authors propose a remedy, the Cross-Primitive Residual Calibrator, or CPRC. It attaches a lightweight residual network to a frozen backbone and trains it to optimize two things jointly: predictive accuracy and cross-primitive consistency. CPRC is tested under a leave-one-domain-out protocol, which holds out each domain in turn.

Consistency violations appear on all four axes, at every size tested. They are not random noise. The analysis shows they follow systematic patterns, and that discrepancies between primitives predict when a decision will fail, with a ROC-AUC of zero point nine one. As for the remedy, CPRC reduces cross-primitive inconsistency by fifty-eight point five seven percent and improves accuracy by two point nine seven percent. The authors report it as effective both in-domain and out of domain.

The study turns an informal expectation into something measurable. Asking the same question in a different format should not change the answer, and PrimaCons shows that in Jev it does, in patterned ways. That disagreement also carries signal, since it flags decisions likely to fail. The authors argue that cross-primitive consistency should be a standard dimension for evaluating Jev, and they offer CPRC as a practical first step toward more reliable decisions.

Cross-Primitive Consistency in Jev. By Xu Liu and three others, from Zhejiang University and vivo. Published on the twenty-eighth of October. This is the long version: the whole paper rather than the abstract.

Jev is a decision model built for speed. Instead of generating text and parsing it, it answers through typed querying primitives. Noul poses a binary question, true or false. Choice offers a set of candidate answers. They are meant to be two interfaces onto one judgment. The paper's first figure shows how that fails. A context puts the Eiffel Tower in Paris, and a response puts it in Berlin. Asked through Noul whether the response is hallucinated, the model answers false. Asked through Choice, it picks hallucinated. Only the interface changed, so the contradiction cannot be blamed on a different question. Earlier work measured Jev one primitive at a time, which leaves exactly this failure out of view.

The benchmark, PrimaCons, holds the semantic state fixed and varies only the primitive or its presentation, along four axes. The first uses HaluEval and asks one proposition three ways: a positive Noul asking whether a response is hallucinated, a reversed Noul asking whether it is factual, and a binary Choice. The second uses unsafe samples from PKU-SafeRLHF, whose twelve fine safety categories pair into six medium parents and then one coarse label, so fine probabilities ought to add up to their parent. The third, from AbstentionBench, sets a Choice that includes an unable to answer option against a Noul asking whether the question is answerable. The fourth uses the four AG News topics. Does a neutral description on each label move the Choice distribution? And do four topic Nouls, normalized, recover what Choice says? Every disagreement score is zero when the views agree perfectly.

The proposed fix is the Cross-Primitive Residual Calibrator. The Jev backbone stays frozen. A lightweight multilayer perceptron receives the hidden state from the layer just before the last, compressed with PCA, along with the raw probabilities and view-level metadata such as predictive entropy. Separate binary and multiclass heads output a logit correction, bounded by a hyperbolic tangent and simply added to the original logit. Training joins cross-entropy against the gold labels with a weighted consistency penalty, and that penalty is just each axis's own disagreement measure turned into a loss. Accuracy and agreement are pushed together. To test generalization, the calibrator trains on the calibration splits of the first three axes and is tested on the fourth, which it never sees, with hyperparameters chosen on calibration data only.

The diagnosis first. Every axis breaks, at all three sizes: two, four and nine billion parameters. In the four billion model, the positive and reversed Nouls disagree by point one three six on average. The most useful finding is what that gap predicts. Ranking samples by it separates failed decisions from correct ones with a ROC-AUC of point nine one. On the hierarchy, Choice and Noul nearly agree at the coarse level but diverge to point six three at twelve classes, and every medium category's probability sits below the sum of its two children: the model inflates probability mass as labels get finer instead of dividing it. On abstention, the two primitives reach opposite hard decisions on twenty-six point seven percent of questions.

Then the calibrator. In domain, the two billion model's gap between its Nouls roughly halves, and its hierarchy accuracy climbs from seventy-seven point six seven percent to ninety-eight point four eight. The comparison that most directly tests the thesis is the held-out axis, where memorizing a domain cannot help. There, the two billion model's divergence between Choice and Noul falls by fifty-eight point six percent, and the nine billion model's by fifty-eight point four. The added latency is three quarters of a percent.

The paper has no limitations section, so these come from its own tables. The out-of-domain claim rests on a single held-out axis: only the fourth ever plays that role. Not every metric improves. On the abstention axis, the two billion model's probability gap rises under the calibrator, from point one eight eight to point two two six, even while its hard conflict rate halves. Out of domain, the shift caused by label descriptions grows slightly for the two and nine billion models. All three models come from one family, and no error bars or repeated runs are reported.

The idea to carry away is that, for a decision model, asking the same question two ways is a cheap audit. Disagreement across primitives predicts failure well enough to rank errors, and it is systematic: it grows with label granularity, depends on which primitive is asking, and worsens with surface text. A small residual on a frozen backbone, trained to be both right and consistent, recovers much of that agreement, including on an axis it never trained on.