Judging by the Cover

Cleaning truth benchmarks to avoid surface-level feature leakage — when models score above chance from answer surface form, not the intended reasoning.

Binary-choice “truth” benchmarks ask a model to pick the correct answer over an incorrect one. But if the two answers differ systematically in surface form — a negation lead, hedging words, length — a model can score above chance without doing any of the intended reasoning. We show this failure mode is easy to detect, measurable across many popular benchmarks, and fixable.

The problem

Could you ace a truth quiz without reading a single question? On TruthfulQA — a benchmark the AI world uses to decide which models can be trusted — the answer turns out to be yes: the "true" answers simply look different, hedged a little more, phrased a little longer, rarely opening with a negation. That is a quiet crisis, because these scores feed leaderboards, safety claims, and deployment decisions — and a model that can pass on style alone turns every one of those decisions into a number that may measure nothing.

How I solved it

First we caught the shortcut in the act: SURFACE6, a six-feature checklist of pure style cues, picks the correct answer at AUC 0.714 without ever seeing a question. Then we turned the audit into the cure — Audit-Prune removes exactly the pairs that feed the shortcut until the style signal collapses to a coin flip (AUC 0.528), while the ranking of real models barely moves (Spearman ρ = 0.915). The result, TruthfulQA-476, is a truth benchmark that has to be passed the honest way.

SURFACE6 and the Audit-Prune pipeline

Audit-Prune in overview: each answer pair is scored by its SURFACE6 features, and the pairs that most reinforce the surface asymmetry are removed until the audit can no longer separate correct from incorrect answers.

The shortcut problem

A benchmark score only means something if it tests what it claims to. In binary-choice evaluations this can fail silently: even a label-balanced set is exploitable if one side of each pair is consistently written differently. Worse, training on such a benchmark can push scores up without any matching gain in capability — and the shortcut is invisible to ordinary cross-validation within the dataset.

SURFACE6: a small, interpretable audit

We define SURFACE6 — six text-only features: a negation-lead indicator, negation count, hedge rate, word count, average token length, and type–token ratio. No embeddings, no pretrained models. A plain logistic regression on these six features (with grouped cross-validation) separates correct from incorrect answers on the binary-choice TruthfulQA at 68.9% accuracy (AUC 0.714) — higher than the second-best model on a public TruthfulQA leaderboard at the time of writing. Negation carries most of the signal.

The six SURFACE6 features

The six SURFACE6 signals: a negation lead and negation count, a hedging rate, and three length/regularity measures (word count, average token length, and type–token ratio).

It is not just TruthfulQA

Applying the same six features across 15 benchmarks shows that several carry similar or larger leakage. HaluEval QA is the most extreme (AUC 0.973, driven by length), while MultiNLI, SNLI, MultiRC, SelfCheckGPT, and FEVER all show a non-trivial surface signal.

Cross-dataset surface-form audit AUC across 15 benchmarks

Surface-form audit (SURFACE6) AUC across 15 benchmarks. Several show substantial leakage — led by HaluEval QA — while the cleaned TruthfulQA-476 drops to near chance.

Audit-Prune: cleaning the leakage

Given a paired dataset, Audit-Prune greedily removes the pairs that most reinforce the surface asymmetry the audit classifier exploits — refitting the standardization, classifier weights, and audit score at each step — until the audit AUC drops below a target threshold, then adds back any pairs that can be returned without raising it. It needs no knowledge of how the dataset was built, only whether statistical leakage remains.

TruthfulQA-476

We release TruthfulQA-476, a cleaned 476-pair subset of binary-choice TruthfulQA: the surface-audit accuracy falls from 0.689 to 0.522 (AUC 0.528 — essentially chance), while model-ranking agreement with the full benchmark is preserved (Spearman ρ = 0.915, Kendall τ = 0.827). On an adversarial test where the false answer is restyled to carry “true-looking” surface cues, classifiers trained on the cleaned subset are markedly more robust than those trained on the full benchmark.

TruthfulQA-476 is available on request, and will be released publicly upon publication.

Paper · under review (arXiv on publication)

This audit grew out of our broader work on steering-dataset corruption — see Understanding and Mitigating Dataset Corruption in LLM Steering (2026) and the contrast datasets it draws on.