What “ready” means for a standby server

Plenty of hosting setups with a second server describe it in one word: standby. It sounds reassuring. If the main server goes down, traffic moves to the other one and the site stays up. The question nobody asks until it's too late is whether the other one has the same site on it.

A standby server drifts. A plugin gets updated on the main server and not the other. A database copy stops running because a password changed. A certificate renews on one machine. Last week's uploads never made it across. None of that shows while traffic is on the main server. All of it shows the moment you fail over.

When we built the failover monitor for the hosting we run, the problem was simple to state: nobody could say which sites were actually in sync. So we changed the definition.

Ready means the same page

A site is ready to fail over when both servers serve the same page. Not when the second server is switched on. Not when the last copy job reported success. When you ask each server for the page, directly, and get the same answer.

That definition is useful because you can check it. “Is the server up?” is a weak question. “Would this server show a visitor the right page?” is the one that matters, and the only way to answer it is to ask for the page.

Check every minute, from both sides

The failover monitor probes every hosted site once a minute: through the CDN, the content delivery network in front of the sites, the way a visitor would see it, and directly on each server, going around the CDN. The results fill a matrix of sites by server, in green, amber or red.

Checking through the CDN tells you what visitors see right now. Checking each server directly tells you what they'd see after a failover. You need both. A site can look fine through the CDN because it's serving a cached copy while the server behind it is broken.

Every minute sounds like a lot until you think about when drift happens. It happens after a deploy, after an update, after someone fixes something on one server late at night. Checking once a day means you find out tomorrow. Checking every minute means the cell turns amber while the person who caused it is still at the keyboard.

What a sync check looks like

Comparing two pages byte for byte doesn't work. Pages carry timestamps, security tokens and headers that differ on every request. So a sync check compares what should match and ignores what shouldn't. Roughly:

1. 1.Request the page from each server directly, using the site's real host name, the way the CDN would.
2. 2.Check the status code. Both should answer with the page, not a redirect to a login or an error.
3. 3.Check the certificate is valid on both, and not about to expire on one of them.
4. 4.Strip out the parts that change per request and compare the rest, or compare a marker the site prints, like the release it's running.
5. 5.Check something that comes from the database, like the newest post or product, so a stale copy can't pass on code alone.
6. 6.Record the result, and if it fails, record why.

The last step is what makes it useful. A red cell that only says failed sends someone digging. The failover monitor writes the next step into each check, for example: the database copy is a day behind, the release on the second server is older, the certificate on the first expires soon. Anything that isn't ready goes on the to-do list with the fix already named.

Small, on purpose

The failover monitor has no dependencies, so it runs anywhere, including somewhere that has nothing to do with the servers it watches. A monitor that falls over along with the thing it monitors isn't much use, so the fewer moving parts it has, the better.

Next to the matrix sits a list of server issues: things like a disk filling up or a service that keeps restarting. They aren't sync problems yet. They're the reason a server will be out of sync next week.

Ask your host one question

If you pay for a standby server, or for hosting that promises failover, ask which of your sites would serve the right page from the second server right now. Not in general. Right now, and how they know.

If the answer is a screen with your sites on it and a time next to each check, good. If it's a description of the copy schedule, that's a plan, not a check. Ask for the check.