Hold, Then Veto: Building a Notification Pipeline That Fails Toward Noise

ViewPane’s push relay recently learned to suppress person alerts when Frigate’s face recognition identifies someone enrolled in the face library. The user-facing story — why your phone shouldn’t buzz when you take out the trash — is over on the ViewPane blog. This post is about the part that made it worth writing up as engineering: we shipped a feature whose entire job is to drop security alerts, and the design work was proving to ourselves that it could never drop the wrong one.

The race you inherit

Frigate publishes object events over MQTT with a simple lifecycle: new, some number of updates, then end. The relay’s job has always been to turn the earliest useful state into a push notification, fast.

Face recognition breaks the “earliest” part. Detection fires the moment a person-shaped thing appears; recognition needs the person closer, facing the camera, in decent light. The recognized name arrives as a sub_label on a later event state — often seconds after the state you’d normally push on. So the data you want to filter on doesn’t exist yet at the moment you’d normally act.

Push immediately and the face library is decorative. Wait for end and your “real-time” alert arrives after the event is over. The pipeline has to make a decision at a moment when the deciding fact is still in flight.

Hold, then veto

The resolution is a bounded hold with a cancellation path:

1. A person event arrives. Instead of pushing, the relay parks it in a pendingSends map with a timer — the grace window (FACE_GRACE_SEC, default 10 seconds).
2. If any later state for that event carries a recognized, enrolled sub_label, the timer is cancelled and the event id goes into a suppressedEventIds set — no future state for that id can ever push, no matter how many updates follow.
3. If the window expires first, the held push fires.
4. Non-person labels (car, dog, package) skip the hold entirely. Face identity doesn’t apply, so there’s no reason to tax their latency.

There’s one subtle case: the event ends while its push is still held. You now have complete information, so use it — recognized face at end means suppress; otherwise fire immediately if the event was real (snapshot persisted), and drop it silently if it never materialized into anything. That last branch is the only silent drop in the system, and it only ever discards events Frigate itself decided weren’t worth keeping.

Alongside all of this, one invariant from the pre-existing pipeline still holds: one push per event id, period. The suppression set and the notified set are both keyed on event id, so no ordering of new/update/end states can double-fire or resurrect a vetoed alert.

Every failure path gets a direction

Here’s the actual design work. A suppression feature has a worst case that a normal notification feature doesn’t: silence about a stranger. For most products a dropped notification is a minor bug. For a surveillance tool it’s the entire failure mode — worse than the product not existing, because the user believes they’re covered.

So we wrote the rule down before the code: every failure degrades toward an extra push, never toward a missing one. Then we enumerated the ways this thing could fail and forced each one to point in that direction:

• The known-faces list can’t refresh. The relay pulls enrolled names from Frigate’s /api/faces every 10 minutes so new enrollments work without a restart. If the fetch fails — Frigate restarting, auth hiccup, network blip — it logs and keeps the last good list. Stale list, working suppression.
• There’s no list at all. Empty set means nothing matches, means nothing is suppressed, means every person event pushes. Fresh install behaves exactly like the feature doesn’t exist until you enroll someone.
• Recognition is late or never comes. The grace window expires and the push fires. A missed recognition costs the user one unnecessary buzz.
• Recognition is wrong. A false non-match is an extra buzz (fine). A false match — a stranger recognized as you — is the one failure the relay can’t defend against, which is exactly why the veto only honors names that are actually enrolled, and why the grace window is short enough that we’re only ever acting on high-confidence recognitions Frigate committed to.

Notice what “fail open” concretely meant here: the refresh function keeps a cache instead of clearing it, the match check requires membership in a set that defaults to empty, the timer fires unless cancelled rather than the reverse. Fail-open isn’t a comment block; it’s which side of each branch does the work when something’s wrong.

Verification matched the stakes

We tested the direction of failure, not just the feature. A synthetic MQTT event with an enrolled sub_label verified the hold-then-suppress path end to end — held, vetoed, zero pushes, correct log line. The device-side check on a real phone was the inverted one, and it’s the test that actually matters: strangers still generate pushes. An unenrolled person triggers a notification a few seconds after detection, every time. The suppression feature earns its keep by what it doesn’t do — so the release test is proving what it still does.

The deploy also surfaced a classic bonus: the relay’s stored Frigate credentials had gone stale after a same-day password rotation, which the post-deploy verification caught immediately. Verify-after-deploy has paid for itself every single time we’ve done it, and this was no exception.

The transferable rule

If you’re building anything that filters alerts — spam folders for pages, dedup for on-call, smart-home suppression logic — the design question isn’t “does the filter work?” It’s “when the filter breaks, which way does it break?” Decide that first, out loud, and then check every branch against it. The system is allowed to be noisy. It is not allowed to be confidently, silently wrong.