DecodeShift

The decoder is part of the measurement

A reported failure rate (unsafe code, refusals, jailbreak success) depends on the model and on the temperature / top-p / top-k that sampled it. The effect is largest on the prompts where failures are rare, and a single benchmark average can hide it.

0.51–0.67

rare-regime rate ratio

T=0.7 vs untruncated, bandit code weaknesses on low-rate prompts, three models (0.22 at HIGH severity)

1.02 [0.95, 1.09]

same comparison, all prompts

the full-benchmark ratio shows no change

30 / 32

transfers agree with direct sampling

40k audit completions reweighted to 15 decoders vs independent 20k-draw runs

+0.0002

median log-error of the cost predictor

exp(−D̂) from logits alone vs measured ESS/N, 122 cells, two model families

28 / 28

admitted transfers all agree

the logit-only screen; a measured-ESS screen admits the bad cell and rejects 17 good ones

4.5 → 483

ESS with prefix weighting

for an event decided in the first 16 tokens of a 128-token generation

Left: estimates from one audit match direct sampling. Middle: predicted vs measured efficiency. Right: rare-regime ratios vs the full-benchmark ratio. — **Left:** one audit sample estimates the event rate under every target decoder; filled points pass the logit-only screen. **Middle:** the effective sample size is predicted before any target sampling. **Right:** on rare-event prompts, lowering the temperature cuts the weakness rate by a third to a half. The full-benchmark ratio (green band) shows no change.

How it works

Temperature, top-k, top-p and greedy are deterministic maps from the model's next-token logits to a distribution. Two decoders on the same model share those logits, so one teacher-forced pass over stored completions gives the exact likelihood ratio for any target.

AuditSample N completions once under a broad decoder, e.g. untruncated T=1.

ScoreOne teacher-forced pass stores the logits of every completion.

Check supportA truncated audit cannot speak for a less-truncated target. D₂ = ∞ flags it.

PredictD̂ = Σ_t E[D₂] from logits → expected ESS ≈ N·e^−D̂. Screen at ≥ 100.

EstimateReweight the event; report the estimate, interval and ESS. Weight only the prefix if the event is decided early.

w(y) = ∏_t f_c(z_t)[y_t] / f_c′(z_t)[y_t] P̂_c(A) = (1/N) Σ_i w(y_i) · 1{y_i ∈ A}, y_i ~ c′

Quickstart

The core needs only numpy; decodeshift.hf adds Hugging Face models.

# install
git clone https://github.com/ziwaa-se/decodeshift.git
cd decodeshift
pip install -e ".[hf]"

# CPU demos with exact ground truth
python examples/01_audit_once_evaluate_many.py
python examples/02_support_barrier.py
python examples/03_prefix_weighting.py

# a real model
python examples/04_huggingface_quickstart.py

from decodeshift import UNTRUNCATED, Decoder, hf

model, tok = hf.load("Qwen/Qwen2.5-0.5B-Instruct")
audit = hf.sample(model, tok, prompts,
                  UNTRUNCATED, n_per_prompt=64)
targets = [Decoder(temperature=0.7),
           Decoder(top_p=0.9), Decoder(top_k=50)]
scores = hf.score(model, audit, targets)  # one pass

event = [detector(t) for t in audit.texts]
for t in targets:
    est = hf.estimate(scores, event, t)
    print(t.name, est.estimate, est.interval())

More results

All of these are regenerated from the shipped result files by reproduce/scripts/make_outputs.sh.

Rate ratios per model and pooled rates per decoder — Rare-regime rate ratios for T=0.7 and top-p=0.95 across models, next to pooled weakness rates.

Predicted vs measured efficiency, and per-token D2 by model family — The efficiency predictor over lengths 8–128 and two model families, with per-token D₂ per decoder gap.

Risk of at least one weakness over repeated attempts — Risk of at least one weakness in n attempts. Greedy stays at 0; untruncated rises from 0.0074 to 0.0998 by n = 25.

Share of generations that are not valid Python per decoder — Parseability changes with the decoder too, so rates are reported conditional on parseable output.

Know the limits

Support comes first. A top-k=50 audit reweighted to an untruncated target returns 1.4×10⁻³ against a true 1.0×10⁻², with a tight interval and weight ESS of 0.989 N. The D₂ check catches it; ESS does not.
The predictor is not a bound. exp(−D̂) is an empirical approximation, used to decide whether a transfer is worth attempting.
Unsampled mass is invisible. Sample-based diagnostics can miss it, so results across a family of decoders are estimates, not a certificate.

Reproduce

Command	What it does
`make_outputs.sh`	every table and figure from the shipped CSVs · CPU, ~1 min
`run_core.sh`	re-run transfer, barrier, efficiency, prefix, safety, bounds · 1 GPU
`run_variants.sh`	other models (Qwen2.5-7B, Mistral-7B, SmolLM2-1.7B), severity, seeds
`pytest`	22 tests: exact identity by enumeration, HF `generate` parity

About 36 GPU-hours in total, mostly independent direct sampling used for validation.