Checking out their machines showed that their DNS server addresses were being changed to an address on the wrong subnet, and their domain being changed to “mshome.net”. That last part’s a red flag: the thing that does that is Windows’ Internet Connection Sharing, which means someone had that enabled on an interface and we basically had a rogue DHCP server.

Rogue DHCP servers are a pain to track down because without a monitoring port on the switch, all you have to go by is broadcast traffic, and then all you get is the address the DHCP server thinks it’s at — which, we know, is on the wrong subnet anyhow — and its MAC address. And we’re a small shop but we still don’t have a handy list of MAC addresses lying around. I did know that the MAC address’s vendor ID was Dell.

So the first thing I did when I found the problem was to check the MAC addresses of all of the wired and wireless interfaces of the Dell computers in the office, and none of them matched! I puzzled over this for a while, had people double-check, and eventually something clicked and Saul remembered that Sunir had enabled ICS during their road trip.

I took a second look at Saul’s laptop, and there was the MAC address — on a disabled wireless broadband interface. Turns out that if you have ICS on, the DHCP server keeps running even when the shared network interface is down. Disable it, problem went away.

But the strange part was that Saul’s been back for a week and the problem just came up today.

I scratched my head about that for a bit and then it hit me: before today, the switch in the wiring closet was in the Linksys router that also served DHCP:

[client]----[switch + dhcp server]----[saul's PC]

After today, both Saul’s network segment and the new DHCP server were both connected to a separate switch:

[client]-----------[switch]-----------[saul's PC]

                       |

                       |

                 [dhcp server]

DHCP is designed to handle multiple (cooperating) DHCP servers on a segment; when a client sends a request, any DHCP servers can respond, and the client chooses one of the responses and informs the DHCP server that sent it that it will use that one. The usual client implementation is to accept the first response.

So before today, a client on one segment would make a DHCP request, but the legitimate DHCP server (at the switch) would be located one Ethernet segment closer to the client than the rogue DHCP server, so it would always win. As of today, the legitimate DHCP server was now the same distance from the client as the rogue one, so part of the time it’d lose, which is exactly what was happening — not every DHCP lease was broken, just the occasional one.

Sometimes it’s easy to forget that actual electrons need to move around for this stuff to work — which in turn reminded me of Trey Harris’s 500-mile email.

But a single ADSL link is a little bit fragile. If it’s down for a day, that’s a day without access to our servers at Rackspace, our tech support email, our marketing reports, everything. Everyone might as well go home save for one person to answer the phones, not that that person can do much when people call.

So in case the DSL line goes down, we’ve got a backup connection using Rogers’ WiMax^? network which they call “Portable Internet“. It’s portable in the sense of the old “portable computers”, in that the router/radio we have is a wall-powered RSU that acts as an Ethernet bridge; there’s also a PC card^? version which would provide a real mobile everywhere-in-the-city network.

Unfortunately it’s a bit slow, especially for the price: $49.95/mo gets you 1.5Mbps down and 256kbps up, or $24.95/mo for 256kbps down and 64kbps (!?) up. I’m not quite sure why it’s as slow as it is, since the real-world maximums for WiMax are in the 10Mbps range, but speed tests confirm that it’s right where they say it is.

But it works! I wanted to have some idea of how it worked before we needed to use it, so I brought the RSU home tonight, and I’m connected to it now. In the window I get five bars of signal, although five feet inside our plaster walls that’s down to one or two bars, although a weak signal seems to affect latency more than speed. And it couldn’t be much easier — plug the laptop into the RSU’s ethernet jack and it grabs a DHCP lease and it’s online.

The speed will make it a bit painful if we ever have to rely on it to get the office connected, but it’ll be a lot less painful than falling off the net completely, and a lot more straightforward than provisioning some sort of permanent redundancy to our little office. (And with a new box arriving early next week to be put to use as a Linux-based firewall and router, I think I might look into automating failover, too. Incidentally, on that subject, I’m torn between trying IPCop and just going with a straight CentOS install and managing iptables and so on by hand. Thoughts?)