2026-06-14DegradedResolved~9 min · 1 public asset
A new freeze detector briefly mislabelled JupSOL as Unknown
- What happened
- We shipped an engine 'freeze detector' meant to publish UNKNOWN when a price basis stops moving (a real failure mode we'd just found on another asset). Its threshold — 20 consecutive identical samples, about 3 minutes — was too low for thinly-traded liquid-staking tokens, whose price ratio is naturally constant for minutes between trades. Six such assets, including the public JupSOL, were briefly and wrongly marked Unknown.
- How we caught it
- Caught minutes later by our own post-deploy full-state verification — assets had gone Unknown with no corroborating market move.
- Root cause
- The pre-deploy validation measured the total number of distinct values over 6 hours, not the maximum consecutive-identical run the detector actually keys on. So it missed that a healthy but illiquid asset routinely sits identical for a few minutes. A threshold tuned to minutes cannot tell 'frozen for hours' (the real failure) from 'no trade for a few minutes' (normal).
- Impact
- ~9 minutes. JupSOL (a public asset) emitted a false Pegged→Unknown→Pegged alert; the signal then self-corrected. Five other affected assets were monitoring-only, so the egress gate kept them off the public feed. No asset was actually off its peg.
- The fix
- Reverted the freeze detector and redeployed the engine; every state recovered within ~9 minutes. The separate containment of the genuinely-frozen asset (hyloSOL+) was unaffected. The detector will be redesigned around a multi-hour window — or to detect the upstream source freeze directly — and re-validated on the exact metric it uses, before any redeploy.
Shipped inee7301a7aadf34