Investigating an outage.
Tracked 0·Alerts 24h 0·API ·Postgres ·Delivery Subscribe to updates

Per-source uptime. Last 90 days.

One bar per day, color-coded by the state Pegana observed. Only today is measured so far — earlier days show as no-data (grey) until the per-day uptime rollup ships. We never paint history we didn’t record.

Live channel
Engine → /v1/ws heartbeat
·Outage
90 days agoToday
Alert delivery
Telegram/webhook dispatch health from /v1/stats
waiting for stats·No data
90 days agoToday
Sanctum
LST intrinsic from on-chain stake-pool value
15s·Outage
90 days agoToday
Jupiter
Market quote (best on-chain route)
30s·Outage
90 days agoToday
Pyth
Live price oracle
~1s·Outage
90 days agoToday
Hylo
On-chain CR + Stability Pool
30s·Outage
90 days agoToday

Alert activity. Rolling 24h.

The engine has been quiet — no alerts fired in the last 24 hours.

pegged
0
drift
0
depeg
0
critical
0

Calibration. Are the bands still right?

Each asset’s trailing-8d p95 spread measured against its owndrift / depeg bands — an oracle that flags when it’s drifting out of calibration.

Loading…

Incidents. Permanent log.

1 user-facing incident. We still publish every issue our own monitoring caught internally — a watcher you can’t audit isn’t a watcher. Full post-mortems below.

2026-06-14DegradedResolved~9 min · 1 public asset

A new freeze detector briefly mislabelled JupSOL as Unknown

What happened
We shipped an engine 'freeze detector' meant to publish UNKNOWN when a price basis stops moving (a real failure mode we'd just found on another asset). Its threshold — 20 consecutive identical samples, about 3 minutes — was too low for thinly-traded liquid-staking tokens, whose price ratio is naturally constant for minutes between trades. Six such assets, including the public JupSOL, were briefly and wrongly marked Unknown.
How we caught it
Caught minutes later by our own post-deploy full-state verification — assets had gone Unknown with no corroborating market move.
Root cause
The pre-deploy validation measured the total number of distinct values over 6 hours, not the maximum consecutive-identical run the detector actually keys on. So it missed that a healthy but illiquid asset routinely sits identical for a few minutes. A threshold tuned to minutes cannot tell 'frozen for hours' (the real failure) from 'no trade for a few minutes' (normal).
Impact
~9 minutes. JupSOL (a public asset) emitted a false Pegged→Unknown→Pegged alert; the signal then self-corrected. Five other affected assets were monitoring-only, so the egress gate kept them off the public feed. No asset was actually off its peg.
The fix
Reverted the freeze detector and redeployed the engine; every state recovered within ~9 minutes. The separate containment of the genuinely-frozen asset (hyloSOL+) was unaffected. The detector will be redesigned around a multi-hour window — or to detect the upstream source freeze directly — and re-validated on the exact metric it uses, before any redeploy.
Shipped inee7301a7aadf34
2026-06-120 users affectedResolved~7.5 min · internal only

JupUSD briefly read CRITICAL on a single bad market quote

What happened
A single Jupiter market quote returned a zero output amount, so the market price computed as $0 and the discount pinned to exactly 1.0 (−100%). That one bad sample poisoned the smoothing filter (EWMA, α=0.3) and JupUSD briefly classified CRITICAL even though the market had not moved.
How we caught it
Flagged by our own post-launch T+24h health scorecard — a CRITICAL on an otherwise calm asset with no corroborating market move stood out immediately.
Root cause
Two systemic gaps: the plausibility check accepted an impossible reading — a market price of exactly $0 — instead of rejecting it, and the source adapter had no floor to drop a zero / near-zero quote before it entered the pipeline.
Impact
0 users affected — JupUSD was a brand-new monitoring-only asset with no subscribers, and this was not a real depeg. (We have since added an egress gate so a monitoring-only asset can never reach the public feed at all — see the next entry.)
The fix
The indexer now skips zero and near-zero Jupiter output amounts and applies a NAV floor at the source, and the plausibility guard was tightened to strictly < 1.0 so a fully-zeroed price can never reach the smoothing filter again.
Shipped in05245c16b7bb53
2026-06-120 users affectedResolvedfound pre-impact

Uncalibrated assets could have reached the public alert feed

What happened
A calibration downgrade marked 11 uncalibrated assets as monitoring-only, but the flag was display-only: the engine and dispatcher had no awareness of it, so those assets could in principle have emitted alerts on the public feed.
How we caught it
Found during a pre-promotion review of the full alert egress path, before any uncalibrated asset transitioned.
Root cause
The monitoring-only status was stored as metadata with no corresponding enforcement at the public egress surfaces — a status flag without a gate.
Impact
0 users affected. Caught and gated before any uncalibrated asset emitted a public alert.
The fix
An egress gate now filters monitoring-only assets at every public surface: the alerts list, the audit ledger (index and CSV), the stats counters, the dispatcher fan-out, and the engine's on-chain commit. Internal rows are still recorded for the calibration retro.
Shipped in7e4bf8f