Kernsignal Evaluation Benchmark · Exploratory
Which questions will a synthetic panel get wrong?
The benchmark measures whether a model can rank, from synthetic answers alone, which survey questions an LLM-simulated panel will answer substantially differently from real people, before any human answers for those questions exist.
The average says how much. Not where.
On Twin-2K-500, GPT-4.1-mini’s simulated answers sit a mean total-variation distance of 0.267 from what the real respondents said. The same humans, asked again in a later survey wave, differ by only 0.032 (study-specific, not a general noise estimate).
Across held-out cells, the most common answer differs in 26.9% of cells. A single fidelity score tells you the panel drifts. It cannot tell you which question to stop trusting.
The vertical rule marks TVD ≥ 0.15, the threshold the benchmark uses to call a cell substantially divergent.
Ranked before anyone answers.
Respondents and whole question families were held out. For each held-out cell, the task is to predict whether the synthetic panel will diverge substantially from humans (TVD ≥ 0.15). Kernsignal’s risk model sees only synthetic answers; the rows below it show what a little human data adds.
Full scorecard · Twin-2K-500 · target: TVD ≥ 0.15
| Approach | Brier (lower is better) |
ROC AUC | Lift in riskiest 20% |
Needs human answers for that question? |
|---|---|---|---|---|
| Coin flip (0.5) | 0.250 | 0.50 | 1.00× | No |
| Prevalence prior | 0.260 | 0.42 | 0.81× | No |
| Synthetic-only (Kernsignal) | 0.190 | 0.80 | 1.33× | No |
| + 10% human audit | 0.166 | 0.88 | 1.55× | Yes small audit |
| + historical human answers | 0.141 | 0.92 | 1.55× | Yes prior waves |
Top-20% lift is capped at 1.55× by the base rate, so lift cannot separate the last two rows.
The ranking carries over. The probabilities don’t yet.
We froze the model trained on Twin-2K only, with no refit, and scored an independent study: GSS 2016–24 attitude items, answered by Claude Haiku 4.5, Claude Sonnet 4.6, Llama 3.1-8B and Llama 3.3-70B. The analysis plan was fixed before any GSS outcome was computed, though the GSS data was already available to the author. Nothing was registered with a third party.
110 cells from 10 question families; 63% of cells diverge substantially. The risk ranking carries over to a new study; the calibrated probabilities do not yet.
Ranking · transfers
CI 0.55–0.89
CI 1.22–2.17
Probabilities · not yet
CI −0.028 to 0.106
0.63 observed
Clone. Install. One command.
The command downloads a pinned revision of the public Twin-2K-500 dataset to local storage and runs the held-out benchmark end to end.
$ git clone https://github.com/essentialols/kernsignal-benchmark $ cd kernsignal-benchmark $ pip install -r requirements-dev.txt \ 'huggingface_hub[hf_xet]>=0.30,<2' $ python -m scripts.twin2k_benchmark \ --revision f883165a3026fde855dfd448e0cd16443ab257b6
Public data. Stated limits.
Twin-2K-500
- License
- CC BY 4.0 (published dataset)
- Compared
- Existing GPT-4.1-mini simulation outputs vs. actual human survey responses
- Used for
- Training and held-out evaluation; respondents and whole question families held out
GSS 2016–24 attitude items
- Source
- Public replication package; reuse under its own terms
- Models
- Claude Haiku 4.5, Claude Sonnet 4.6, Llama 3.1-8B, Llama 3.3-70B
- Used for
- Frozen-model transfer test only, no refit; analysis plan fixed before outcomes were computed
Limits
- L.1One primary dataset plus one transfer study.
- L.2Academic replication data, not commercial vendor panels.
- L.3Models may have seen these public survey items.
- L.4An exploratory benchmark, not a certification.
Testing a synthetic panel?
Let’s find which of your questions still need human answers.
Engagements typically start with a two-week validation sprint on your own use cases and human data.