A reference architecture for operating a system that runs continuously. The focus is not which tool is best, but what to monitor and when an alert genuinely justifies waking someone up.
This is a reference architecture, not a client case study. No client names or figures from a specific project appear on this page.
The problem
A system serving external customers, where downtime costs revenue and reputation. A small engineering team, too small to staff three shifts. Currently the most common way incidents are discovered is a customer phoning in.
Four layers of monitoring
Layer 1 — Is it alive
Checked from outside, every minute, from several geographic locations. This is the absolute minimum. Multiple locations let you distinguish a real outage from a local network problem.
Layer 2 — Server resources
CPU, memory, disk space, database connections. A full disk causes incidents surprisingly often, and it is also the easiest to prevent — alert at 80%.
Layer 3 — Application behaviour
Response latency, error rate, request volume. What matters is the 95th and 99th percentile, not the average. Averages hide problems: if 95% of users are served in 200 milliseconds while 5% wait 8 seconds, the average still looks fine while 5% of your customers are having a bad time.
Layer 4 — Business metrics
This layer is usually skipped and often detects incidents first. Orders per hour, successful logins, payment transactions. If orders suddenly drop to zero while every technical metric is green, something is broken in a way technical monitoring cannot see.
Business metrics catch what technical monitoring misses
Servers healthy, database fast, no errors in the logs — but the checkout button has a rendering fault and nobody can click it. Every light is green while revenue is zero. Only a business metric catches that.
Alerting: the real problem is too many alerts
A system sending 50 alerts a day will soon have all of them ignored, including the real ones. This is the most common way monitoring fails — not missing tools, but too much noise.
| Severity | Condition | Action |
|---|---|---|
| Critical | Service unavailable, or data at risk | Phone call, wake someone regardless of hour |
| Warning | Degraded but serving, or approaching a threshold | Message, handle in working hours |
| Info | Notable anomaly | Log to a dashboard, review weekly |
The rule: only things requiring immediate action may wake a person. Everything else waits until morning. If an alert fires repeatedly and never requires action, fix the threshold or delete the alert.
Backups — the most important item
The practical rule: three copies of the data, on two different media, with one held in a different physical location.
But more important than the backup configuration is testing restoration regularly. A backup that has never been restored is not a backup, it is a belief. Many businesses discover their backups are broken on the day they need them.
Minimum schedule: a full restore test once a quarter into an isolated environment, recording the actual restore time. That number is how long the business would be down in the worst case.
Runbooks
For each common incident type, one short page answering: how to recognise it, how to check quickly, what steps to take, and when to escalate.
The standard for a good runbook: someone who did not build the system, reading it at three in the morning, can still resolve the incident. If only the author can follow it, it has not done its job.
Delivery sequence
- Backups and restore testing. First, no argument. It is the only thing that saves you in the worst case.
- External uptime monitoring. Cheap, fast, immediately valuable.
- Resource monitoring and basic alerts. Disk, memory, connections.
- Centralised logging. So incidents do not require logging into each machine.
- Business metrics and dashboards. For both engineering and commercial staff.
- Runbooks. Added to after every real incident.
What you would need to provide
- A list of systems to monitor, ranked by revenue impact
- Your threshold: how long an outage must run before it counts as serious
- Who receives alerts out of hours, and the escalation order if the first person does not answer
- A log of previous incidents, if one exists
Written from the team's hands-on experience delivering infrastructure and software. No sponsorship, no product placement.