dnsmonitoringengineeringsecurity

Why DNS Change Monitoring False-Alarms on CDN Domains

M

Monitorion

Engineering Team

||8 min read
Share:

Recently one of our DNS monitors told a customer their records had changed, opened an incident, and emailed them twice. The records had not changed. The domain resolved correctly on every single check, and was never once down.

This is a write-up of what went wrong, what we learned about how DNS actually behaves behind a CDN, and what we changed. The bug was ours, but the thing that makes it easy to write affects every DNS change monitor. That part is worth understanding whichever tool you use.

What DNS change monitoring is for

A DNS change monitor resolves your domain on a schedule, compares the answer against a stored baseline, and tells you when something moved. It is not an uptime check. It is a security and configuration check, and the records that matter most are rarely the ones people think about:

  • NS — if your nameservers change and you did not change them, your domain has been hijacked at the registrar.
  • MX — if your mail exchangers change, someone may be intercepting your email.
  • TXT — SPF and DKIM records live here. Quietly editing them is how spoofing campaigns get started.
  • A / AAAA — where your traffic actually goes.

The naive implementation is obvious: store the last answer, compare the next one, alert on any difference. It works perfectly until the domain sits behind a CDN, which today means most domains.

Problem one: the same zone answers differently depending on where you ask

We run probes in several regions. A check for one customer domain happened to run from a different region than it usually did, and got back completely different A records — a different set of Cloudflare edge addresses.

Nothing had changed. Cloudflare steers answers by the location of the resolver asking. The giveaway was in the same response: the MX, NS and TXT records were byte-identical from both locations. Only the CDN-proxied address records differed, which is exactly what geo-steering looks like.

So the first lesson is that a DNS answer is not a property of a domain. It is a property of a domain and a vantage point. Comparing an answer from one place against a baseline recorded somewhere else is comparing two different questions.

Problem two: some CDNs never give you a stable answer at all

You might think the fix is to always ask from the same place. For anycast CDNs, that works. For others it does not come close.

There are two fundamentally different CDN architectures. Anycast CDNs announce the same small set of IPs from every location and let network routing pick the nearest edge. DNS-steered CDNs hand out different addresses per query and use very short TTLs to stay reactive.

A 2025 measurement study of CDN and resolver behaviour puts numbers on it:

CDNDistinct IPv4 per siteTTLArchitecture
Akamaihundreds to thousands20sDNS-steered
Fastly2–5 via most resolvers, 53 via Quad930sanycast, resolver-dependent
Cloudflare2–5 (four of five sites: exactly 2)300sanycast
Edgecast4, shared across sites3600sanycast

CloudFront is not in that study, but it is unicast by default — which is precisely why AWS now sells Anycast Static IPs as a separate product for customers who need stable addresses on an apex record.

Read the Akamai row again. Hundreds to thousands of addresses, rotating on a 20-second TTL. If your monitor checks hourly and compares exact IPs, every single check sees a different answer. Pinning the probe to one location changes nothing. There is no stable value to compare against, because the CDN is deliberately not providing one.

What we measured on our own data

Before changing anything we went back through our own check history, because an argument about CDN behaviour is worth less than a measurement of it.

MeasurementResult
All DNS monitors, 24h, every region0 variance within any single region
One Cloudflare zone, 92 days, 2,126 checks, one region1 distinct address set — zero drift
Same zone, 92 days, all five regions1 distinct address set
NS / MX / TXT across every monitor1 variant each

Two useful conclusions. First, Cloudflare assigns a zone a stable pair of anycast addresses and keeps it — the fear that CDNs constantly rotate your IPs is, for anycast providers, unfounded. Second, the records that carry the security signal never moved at all. They are the reliable part.

What the usual advice is

The standard recommendation, including from tools that specialise in DNS monitoring, is essentially: you cannot fix this, so exclude those records. Stop watching A and AAAA for CDN-fronted domains and watch the CNAME instead.

That advice is not wrong. It is just manual, and it asks the customer to understand their CDN's DNS architecture before they can trust their monitoring. Most people setting up a monitor do not know whether their provider is anycast or DNS-steered, and they should not have to.

What we built instead

Baselines are stored per vantage point. A check compares only against an answer gathered from the same region. A geo-steered reply from somewhere else cannot look like a change, because it is never compared against the wrong thing. Crucially, checks still go to a shared queue, so any worker can run any job and a failed worker's job is picked up by another within seconds. Correctness did not cost us redundancy.

Volatility is measured, not assumed. For each monitor we track how many distinct value-sets each record type has produced recently. A type that keeps changing is reported but not compared. This is what handles Akamai and CloudFront, and it does so without us maintaining a list of CDN IP ranges or trying to recognise providers by name.

Security records are never auto-suppressed. NS, MX, TXT, CAA, CNAME and SOA are always compared, whatever the measurement says. If one of those goes volatile, that is the alert, not noise to be filtered. We verified against live DNS that with address records excluded, a nameserver change is still caught — that is the hijack case, and losing it would defeat the point.

A suspected change must be confirmed before it alerts. One reading is an observation. Two agreeing readings are a change. This alone would have prevented the incident that started all of this.

A new monitor watches before it judges. Until a region has gathered enough history, it records and compares nothing. Otherwise a brand-new monitor on a DNS-steered CDN would alert on its second ever check, before there was any history to recognise the volatility from.

Two things we deliberately did not do

We did not compare by provider IP range. The tempting shortcut is to treat "still inside Cloudflare's published ranges" as "no change". It would have removed the false positives in one line. It also masks a real hijack: an attacker who repoints your domain at their own Cloudflare account stays comfortably inside those ranges. For the use case that matters, range matching is strictly worse than doing nothing.

We did not detect CDNs by their nameservers. Checking whether NS points at a known CDN looks like a clean way to decide which records to trust. Our own data killed it: two of the domains we monitor are Cloudflare-fronted and returned exactly one address set across 92 days and five regions. Treating every Cloudflare zone as unmonitorable would throw away perfectly good hijack detection on domains that never vary. Behaviour is measurable; provider identity is a guess about behaviour.

If you monitor DNS anywhere

Whatever tool you use, these hold:

  • Know which records you are actually watching. For a CDN-fronted domain, A and AAAA are the noisiest and least meaningful. NS, MX and TXT are where hijacks and mail interception show up.
  • Treat a single differing reading as a question, not an answer. Anything that alerts on one observation will cry wolf.
  • Be suspicious of "no change" as much as "changed". A monitor that suppresses too aggressively is worse than no monitor, because you believe it.
  • If answers differ by location, that is information. It means the zone is geo-steered, which is usually intentional — but if you did not intend it, you have just learned something.

Why we wrote this up

We shipped a false alarm to a customer and told them their DNS was down when it was not. The honest response to that is not a quieter alert threshold; it is understanding why the comparison was wrong in the first place, and fixing the thing that made it easy to get wrong.

The bug was a day's work to fix. Understanding it properly took measuring 92 days of our own history and reading the research on how four different CDNs answer the same question differently. That part was worth more than the fix.

Share:

Enjoyed this post?

Get monitoring tips and product updates delivered to your inbox.


Related Posts