Your Monitor Says Down. The Site Is Up. Both Are Telling the Truth.

A monitoring system that cries wolf is worse than no monitoring at all. People stop reading the alerts, and then the one that matters arrives in a folder nobody opens.

We recently spent a day tracing why a perfectly healthy website was reported as down dozens of times overnight. The site never went down. Every alert was correct about what it observed and wrong about what it concluded. This is what we found, and the four things we changed.

A check is an opinion from one place

Here is the assumption buried in almost every uptime tool, including ours: that the internet gives everyone the same answer. It does not, and it has not for a long time.

The same request to the same URL at the same moment can return different results depending on which network it comes from. Not occasionally — by design:

• Bot protection challenges traffic it finds suspicious. Datacentre IP ranges are suspicious almost by definition.
• Geo-blocking serves 403 to entire continents, usually for licensing or compliance reasons.
• CDNs and anycast route you to the nearest edge. Different edge, different IP, sometimes different behaviour.
• Rate limiting triggers when requests cluster on one source address — which is exactly what a monitoring probe looks like.

None of those mean the site is down. All of them will fail a naive HTTP check.

The failure we traced

A monitor on a healthy site reported roughly 50% uptime over 24 hours and sent dozens of alert emails overnight. Looking at the checks grouped by which probe ran them, the pattern was immediate:

probe location      up    down   status codes
Europe A            92     135   200, 403
Europe B            52       2   200, 403
US East              7       0   200
US West              3       0   200
Asia Pacific         6       0   200

Every US and Asian probe got a clean 200. The European ones mostly got 403. The response headers explained it in one line:

cf-mitigated: challenge
server: cloudflare
<title>Just a moment...</title>

Cloudflare was serving an interactive bot challenge. Not an error, not an outage — a "prove you are human" page that no automated probe can pass. The site was up the entire time, cheerfully serving real visitors.

Meanwhile a plain ICMP ping monitor on the same host was at 100% uptime throughout, which should have been the clue.

Why it flapped instead of just failing

The worse detail is why the monitor alternated between up and down rather than settling. Our scheduler put single-location checks on a shared queue that any worker could pick up. So each check landed wherever a worker happened to grab it first.

That turns a geographic difference into apparent instability. Check from the US, get 200, monitor recovers. Check from Europe, get challenged, monitor goes down. Repeat every few minutes, all night, with an email each time.

Worse, the monitor had a location configured. It was being ignored: the code only honoured that setting for multi-region monitors. A sample of supposedly location-pinned monitors turned out to be running from five different places.

Defence 1: one monitor, one vantage point

A single-location check must run from the same place every time. Otherwise its history is not a time series, it is a sample of different questions.

We now resolve exactly one location per monitor: the configured one if it has a healthy probe, otherwise one derived from the monitor's own ID — stable for that monitor, evenly spread across monitors. Deliberately not random. Stability is the whole point.

The trap we nearly walked into

Pinning work to a location is only safe if you know that location is actually working. We found out the hard way that a heartbeat does not prove that.

One of our probes had been reporting itself healthy every minute for 59 hours while processing exactly zero checks. Its heartbeat travelled over one connection and its work queue over another. The second had been down since a maintenance window; the first never noticed.

Had we pinned monitors to that location on heartbeat evidence alone, those monitors would have stopped being checked — silently, which is the worst way for monitoring to fail. The scheduler now requires a location's work queue to be visibly draining before it will send anything there.

The general lesson is worth stating plainly: liveness checks should measure work completed, not messages received. A process can be fully alive and fully useless.

Defence 2: confirm from somewhere else

We already had a confirmation step. A single-location monitor does not open an incident on first failure; it rechecks 30 seconds later and only alerts if that fails too.

It did not help, for a reason that took a while to see. The confirmation went onto the same shared queue — and that queue is not fairly distributed. Probes that sit closer to the queue server win more jobs. In this case one location was taking 78% of the work.

So the original check and its confirmation both ran from the same blocked vantage point, and agreed with each other. Two observations, one blind spot, zero new information.

A confirmation is only evidence if it is independent. Confirmations now deliberately run from a different location than the one that failed. If no other location is available it falls back to the old behaviour, because a confirmation that never runs would suppress a real outage — and that is a far worse bug than the one we were fixing.

Defence 3: a challenge is not an outage

When a response is a bot challenge, saying "down" is wrong twice: the site is up, and the real problem — your WAF is blocking the monitoring you asked for — never gets mentioned.

We now detect challenge responses from the major vendors and report them as a distinct state that shows in the dashboard but does not page anyone, with a message that says what to do about it.

The detection is deliberately narrow. A plain 403 is not treated as a challenge — plenty of sites return 403 for ordinary authorisation failures, and misclassifying those would hide genuine outages. Every rule requires a vendor-specific header or an unmistakable marker in the response body. Our test matrix explicitly includes a plain 403, a basic-auth prompt, a 404, and a real 503 behind a CDN; none are classified as challenges.

This is the trade-off to be careful with. Every false-positive fix is a potential false negative. The right instinct is to be conservative about what you suppress and loud about what you cannot classify.

Defence 4: sensible defaults, visibly

The deeper cause was that "automatic" location selection stored an empty value, and empty meant "anywhere". An empty setting that silently changes behaviour is a bug waiting for a customer to find.

New monitors now resolve and store a real location, chosen near the account's own region, which the user can see and change. "Automatic" should tell you what it chose.

One non-obvious detail: our fallback is a US location rather than a geographically central one. Bot protection vendors challenge traffic from some hosting regions far more readily than others. A default that gets challenged manufactures false outages, so the safest fallback is the one least likely to be blocked — not the one closest to the middle of the map.

How to tell, on any tool

If a monitor is alerting and you suspect the site is fine, three checks will usually settle it:

1. Group the recent results by probe location. Mixed results across locations means vantage-dependence, not an outage. Consistent failure everywhere means it is real.
2. Compare check types on the same host. Ping green while HTTP is red points at the application layer or a WAF, not the server.
3. Look at the response headers. cf-mitigated, x-datadome, x-iinfo and similar are conclusive. So is a body containing a CAPTCHA.

If it is bot protection, the fix is on your side: allowlist your monitoring provider's probe addresses, or exclude the monitored path from the challenge rule. A monitor that cannot reach your site is not telling you anything useful about it.

What we would say to anyone building this

Three things we would tell our past selves:

Treat every check as an observation, not a verdict. It is one sample from one place. Design the alerting around the possibility that the sample is unrepresentative, because sometimes it is.

Make confirmations independent or do not bother. A retry from the same place is not corroboration. It is the same question asked twice, and it will agree with itself every time.

Measure work, not heartbeats. Our 59-hour silent failure passed every liveness check we had, because the liveness check and the work travelled different paths. Ask "when did this last finish a job", not "did this say hello recently".

Every one of these bugs was ours. But none of them are unique to us — they come from an assumption the internet quietly stopped honouring years ago, and the tools have been slow to catch up.