Credible Evaluation — how n=20 paired stats overturned two loss "improvements"

A methodology upgrade for the tactile world model's decoded evaluation, and the two results it changed. Companion video page: the per-config rollouts. Loss code: docs/CONTACT_AWARE_LOSS_IMPL.md.

The problem

The contact-aware loss weight was first "decided" on n=3 decoded 40s rollouts: Lc=1.0 looked best (contact MSE 58.8 vs 59.99). But n=3 with two near-duplicate val episodes (005/006, same day) gives no confidence intervals and lets a single lucky window flip the ranking. Event metrics (onset, spearman, F1) were pure noise.

The eval upgrade

• Per-sample metrics emitted by both metric scripts → enables paired analysis.
• Paired bootstrap + Wilcoxon (tools/paired_metric_analysis.py): same decoded

windows across configs, per-window deltas, 95% CI + signed-rank p-value + n. This cancels window-difficulty variance — far more powerful than comparing means.

• onset-err t0=0 fix: only count events with a genuine onset (contact begins

after the first future frame). Removes the pathology where "already in contact" reads as perfect 0 ms timing.

• 6-episode stratified cross-day val (05-10 / 05-11 / 05-19) replacing 2

same-day episodes → 5.6× more val windows and a real drift test.

Result 1 — the contact-weight decision was a false positive

Paired Wilcoxon, n=20 val windows, baseline = Lc=0.25 (shipped v3_contact):

candidate	contact MSE ↓	mask IoU ↑	tactile-L LPIPS ↓	view LPIPS ↓
Lc=1.0	+3.3 p=0.011 (worse)	−0.037 p=0.024 (worse)	+0.017 p=7e-4 (worse)	−0.003 p=0.06
Lc=1 + L_false(m0.05)	+2.5 p=0.008 (worse)	−0.014 p=0.12	+0.010 p=0.006 (worse)	−0.0003 p=0.57
no aux	+0.9 p=0.87	−0.024 p=0.08	+0.010 p=0.003 (worse)	+0.033 p=2e-6 (worse)

Lc=1.0 is significantly WORSE than Lc=0.25 on the primary metric — the n=3 "Lc=1 best" was noise. no aux is significantly worse on view LPIPS → the aux helps, but only at the low weight 0.25. onset err: 0 genuine events (the fix exposes that all prior onset numbers were t0=0 pathology).

Result 2 — the energy-match loss idea does not help

We designed an energy-map matching aux term (ā·|e_pred − e_gt| over the whole tactile field) to target localization (mask IoU), where every config was weak. Two from-scratch runs on the stratified val, paired n=20, baseline = Lc=0.25:

candidate	mask IoU ↑ (target)	contact MSE ↓	tactile-L LPIPS ↓	view LPIPS ↓
+ energy-match 0.5	−0.019 p=0.29 (no gain)	−2.3 p=0.29	+0.006 p=0.048 (worse)	+0.0005 p=0.55

Energy-match fails at its own target (IoU: no improvement) and is significantly worse on tactile LPIPS. Rejected.

Corrected recommendation

Keep the shipped tactile_contact_loss_weight: 0.25. Do not raise it, do not activate L_false, do not add energy-match — all three are rejected by paired stats. The contact aux is near its ceiling.

The bigger finding

On the harder stratified val, contact MSE jumps 34.7 → 77 for the same recipe. The real bottleneck is cross-session drift generalization (the 05-19 session), not the ~1%-of-loss contact aux. Future effort should go to data / drift, not more aux-loss variants.

Reproduce

• scripts/_credible_eval_sbatch.sh, scripts/_v4_credible_eval_sbatch.sh
• tools/paired_metric_analysis.py
• reports: figures/credible_eval/paired_report.md, figures/v4_credible_eval/paired_report.md