Failure modes

Every alerting system has a point where it stops working. Here is where Crit Alarm’s is.

When your host goes down

In Crit Alarm v1, the repeat loop originates from your self-hosted server:

Monitoring Tool -> Self-Hosted Server -> Push Relay -> APNs/FCM -> Phone
                         |
                 (30s repeat loop)

What Happens If the Host Server Dies?

If the physical machine or virtual server hosting your Crit Alarm instance loses power, network connectivity, or crashes:

1. No new alerts can be received: Any incoming webhooks from your monitoring tools will fail to connect.
2. Existing repeat loops stop: If an incident is open and repeating every 30 seconds, those repeated sirens will cease immediately upon host failure.

Co-locating Monitored Services and Crit Alarm

If you host Crit Alarm on the exact same machine as your primary database or web application:

• If only your database process crashes, Crit Alarm still functions and successfully rings your phone.
• If the entire machine crashes or suffers a kernel panic, Crit Alarm goes down with it. The initial webhook will never be delivered, and your phone will never ring.

Where to run it

To keep alerts working when a machine dies:

• Run Crit Alarm on a separate host: run it on an inexpensive cloud VPS (such as a 5-dollar DigitalOcean Droplet, Hetzner Cloud instance, or AWS Lightsail) or a separate physical machine on a distinct network from your primary workloads.
• Future Roadmap: A relay-side watchdog and repeat heartbeat is planned for v2.

---

What breaks and what happens

Failure Scenario	Effect	Mitigation in Crit Alarm
User’s server crashes	Nothing new is sent; open incidents stop repeating.	Accepted for v1. Mitigation: run Crit Alarm on a separate VPS from the systems it watches.
Push relay service unreachable	Self-hosted pushes cannot reach APNs or FCM.	The relay runs behind Cloudflare with automated health checks and fast failover.
APNs rejects device push token	The handset never rings.	The relay marks the device registration stale upon receiving HTTP 410 from Apple. The mobile app refreshes its token on launch, and the “Ring me now” test button surfaces token issues immediately.
Reverse proxy drops HTTP headers	Webhook publishes return 401 Unauthorized or ignore priority.	The startup log outputs all five configuration values, behind-proxy handles forwarding headers, and documentation provides exact Caddy, Traefik, and Nginx configurations.
iOS cannot play the alarm	Priority 5 alerts arrive as Time-Sensitive notifications rather than loud sirens.	Same code path and same push payload. The app shows what iOS will do with a critical topic during onboarding and in Settings.
Handset offline / Airplane mode	Push notifications are held in transit by Apple or Google.	The server sets apns-expiration and FCM TTL equal to the incident max_ring_duration so stale alarms do not sound hours later.

---

What survives a restart

Timers live in the database

All incident repeat timers, max ring timeouts, and desk timers are persisted as rows in the SQLite database (timers table).

They are never kept exclusively in process memory:

• If the Crit Alarm container restarts, it executes a database scan on boot:

  SELECT * FROM timers WHERE fire_at <= datetime('now');

• A container crash loses at most 5 seconds of the evaluation cycle. Active incidents, acknowledgment states, and desk timers survive intact.

Flapping Guard

When a service flips between healthy and failing, a monitor can fire dozens of alerts in seconds.

Crit Alarm enforces a single active incident per topic:

• If a priority 5 alert arrives while an incident sits in the open or acked state, the message is attached to the existing incident and updates last_message_at.
• It does not create a second incident, does not reset your acknowledgment, and does not cause multiple alarms to overlap.