Most monitoring tells you when something happened. The harder question is when something stopped happening - the nightly job that didn't run, the feed that went quiet, the agent that hasn't published in a day. Kythene answers that with expectations: you declare what should keep arriving, and it raises an alert when the silence goes on too long.
Expectations
An expectation says: an event carrying this tag must arrive at least every this interval. Set one on the thing whose regular arrival matters - a nightly benchmark, a sync job, an agent's heartbeat.
A sweep runs every few minutes and compares each expectation against the last matching event. If the gap has grown past the interval, an alert fires. When a matching event arrives again, the alert auto-resolves - you don't clear it by hand, and a job that recovers on its own closes its own alert.
The simplest way to feed one is to have the job publish a short line with kythe ingest and set an expectation on that tag. You hear it when it runs, and you're alerted by its silence.
Alerts
An alert is a stateful object, not a one-off email. It moves through firing → acknowledged → resolved, and repeated raises of the same underlying fault update the one alert (bumping its last-seen time and count) rather than piling up duplicates.
Each alert carries a severity:
| Severity | Use |
|---|---|
info | Worth noting, not urgent. |
warn | Something needs a look. Expectation misses raise at this level. |
error | Acting up now. |
Where an alert goes
- The inbox. When an alert first fires, it lands in every workspace member's inbox, one line per fault, with the severity carried across (
warnshows as a warning,erroras critical). - Notification rules. A notification rule that opts in to alert triggers delivers firing alerts over email or webhook, alongside the publishes it already carries. So the channel a team already watches carries its alerts too.
On-call, without an account
The person who needs to acknowledge a 2 a.m. alert often isn't signed in - and may not even have an account. Space admins hand out on-call share codes at Settings → On-call (/settings/oncall). The holder opens /oncall/{code} and can see the live alerts and acknowledge or resolve them without signing in. The code in the URL is the credential, so treat it like one and revoke it when someone rotates off the rota.
What this is not
Expectations watch for the absence of an event you declared. They don't infer health you never told them about, and they don't watch your infrastructure - point a real infrastructure monitor at the host for CPU, disk and process liveness. Kythene's job is the layer above: the work that was supposed to land, and didn't.