01/10/2026
Your LLM says it's 95% confident. The token probabilities agree. They're both wrong.
This is one of the most underappreciated problems in production AI systems β and new research just cracked it wide open.
The problem with standard confidence measurement:
Current methods estimate LLM confidence using token probabilities β how likely was each word the model chose? Sounds reasonable. But research shows you can replace those probabilities with random noise and still get similar calibration. That's a red flag.
Meet DTC β Divergent Token Confidence:
Instead of asking "how probable was this token?" β DTC asks "do two models AGREE on this token?"
Here's exactly how it works:
π£ Step 1 β Run the same chain-of-thought through TWO models simultaneously
π΅ Step 2 β Measure Jensen-Shannon divergence at each token position
π’ Step 3 β Count tokens where models strongly disagree (divergent tokens)
π Step 4 β More disagreements = lower confidence signal
The core insight: where models disagree is where the model is actually uncertain. That count of divergent tokens is more honest than any probability score.
The results across 6 math benchmarks:
π Standard white-box methods: 32.7%β42.4% calibration error
β
DTC white-box: 13.0% calibration error
π Baseline black-box (DeepSeek-V3.2): 32.1%β40.2% calibration error
β
DTC black-box: 13.7%β16.3% calibration error
That's a 60%+ reduction in calibration error β without changing the model, without retraining, without touching the generation process.
Three reasons this matters for builders:
Works on frozen chain-of-thought β no inference overhead
Supports both white-box AND black-box settings β works even when you can't access model internals
Training-free β drop it into any existing pipeline today.
How are you currently measuring LLM confidence in your production systems? π