Judging by the Cover
Binary-choice “truth” benchmarks ask a model to pick the correct answer over an incorrect one. But if the two answers differ systematically in surface form — a negation lead, hedging words, length — a model can score above chance without doing any of the intended reasoning. We show this failure mode is easy to detect, measurable across many popular benchmarks, and fixable.
The problem. Could you ace a truth quiz without reading a single question? On TruthfulQA — a benchmark the AI world uses to decide which models can be trusted — the answer turns out to be yes: the "true" answers simply look different, hedged a little more, phrased a little longer, and far more often opening with a flat “No” (a negation lead on 25% of true answers versus 6% of false ones). That is a quiet crisis, because these scores feed leaderboards, safety claims, and deployment decisions — and a model that can pass on style alone turns every one of those decisions into a number that may measure nothing.
How I solved it. First we caught the shortcut in the act: SURFACE6, a six-feature checklist of pure style cues, picks the correct answer at AUC 0.715 without ever seeing a question. Then we turned the audit into the cure — Audit-Prune removes exactly the pairs that feed the shortcut until the style signal collapses to near chance (AUC 0.528), while the ranking of real models barely moves (Spearman ρ = 0.915). The result, TruthfulQA-476, is a truth benchmark that has to be passed the honest way.
Audit-Prune in overview. Left: four TruthfulQA pairs with their cue words highlighted and their SURFACE6 feature vectors. Middle: iterative scoring, removal, refitting, and add-back. Right: the retained pairs form TruthfulQA-476.
The shortcut problem
A benchmark score only means something if it tests what it claims to. In binary-choice evaluations this can fail silently: even a label-balanced set is exploitable if one side of each pair is consistently written differently. Worse, training on such a benchmark can push scores up without any matching gain in capability — and the shortcut is invisible to ordinary accuracy-based evaluation; it surfaces only when the answer texts themselves are probed.
SURFACE6: a small, interpretable audit
We define SURFACE6 — six text-only features: a negation-lead indicator, negation count, hedge rate, word count, average token length, and type–token ratio. No embeddings, no pretrained models. A plain logistic regression on these six features (with grouped cross-validation) separates correct from incorrect answers on the binary-choice TruthfulQA at 68.9% accuracy (AUC 0.715) — on a task where chance is 50%, without the model ever seeing the question. Negation carries most of the signal.
The six SURFACE6 signals: a negation lead and negation count, a hedging rate, and three length/regularity measures (word count, average token length, and type–token ratio). All six are computed from the answer string alone.
It is not just TruthfulQA
Applying the same six features to 13 further benchmarks — 15 datasets in all, counting TruthfulQA and TruthfulQA-476 — shows the shortcut is not TruthfulQA-specific. Two of them, HaluEval QA (AUC 0.973, driven by length) and MedHallu, leak more than TruthfulQA itself; MultiNLI, SNLI, MultiRC, SelfCheckGPT, and FEVER show a non-trivial surface signal; BoolQ (0.525) and PIQA (0.509) sit at the edge of detectability.
Surface-form audit (SURFACE6) AUC across 15 datasets, sorted by AUC — 13 external benchmarks plus TruthfulQA and the released TruthfulQA-476 (starred), which drops to near chance.
Audit-Prune: cleaning the leakage
Given a paired dataset, Audit-Prune greedily removes the pairs that most reinforce the surface asymmetry the audit classifier exploits — refitting the standardization, classifier weights, and audit score at each step — until the audit AUC is at or below a target threshold (θ = 0.53 for the released set), then adds back any removed pair whose return keeps the AUC under that threshold. It needs no knowledge of how the dataset was built, only whether statistical leakage remains.
A fixed-prefix baseline that removes pairs in one precomputed order bottoms out at audit AUC 0.583 and never reaches that threshold; refitting at every step is what lets Audit-Prune reach 0.528 while keeping 476 of the 790 pairs (60.3%).
Threshold sweep on TruthfulQA: pairs retained (top) and audit accuracy of the retained set (bottom) for Audit-Prune and the fixed-prefix baseline. The dotted line is the baseline's minimum attainable audit AUC; the star is the released TruthfulQA-476.
TruthfulQA-476
We release TruthfulQA-476, a cleaned 476-pair subset of binary-choice TruthfulQA (60.3% of the 790 pairs): the surface-audit accuracy falls from 0.689 to 0.522 (AUC 0.528, 95% CI [0.503, 0.557], permutation p = 0.048 — at the edge of statistical detectability), while model-ranking agreement with the full benchmark is preserved (Spearman ρ = 0.915, 95% CI [0.81, 0.99]; Kendall τ = 0.827).
To test whether the cleaning matters downstream, we built two held-out 131-pair cohorts in the same paired format: SurfaceFlipped-131, misconception questions whose false answer is written with the “true-looking” cues (a negation lead, hedging, extra length) and whose true answer is a bare assertion, and Natural-131, open-ended factual questions across twelve neutral domains written without regard to surface form. Each was generated by a frontier LLM, checked by an independent LLM judge, then verified by hand. Across nine frozen-representation classifier families, training on TruthfulQA-476 instead of the full benchmark raises accuracy on the surface-flipped set for five of the nine families (Llama-3.2-3B 0.27 → 0.45, SmolLM2-1.7B 0.43 → 0.63, ModernBERT-base 0.62 → 0.73; McNemar p < 0.05) while leaving accuracy on the natural set unchanged within noise.
TruthfulQA-476 is public on Hugging Face under Apache-2.0 — foadnamjoo/audit-prune — together with the two held-out evaluation cohorts, SurfaceFlipped-131 and Natural-131. The audit and Audit-Prune code is on GitHub; the paper is on arXiv (September 2026).
This audit grew out of our broader work on steering-dataset corruption — see Understanding and Mitigating Dataset Corruption in LLM Steering (2026) and the contrast datasets it draws on.