I want one.

Update: Another coworker, Taavi, points out that it’s also available as desktop wallpaper.

We moved the printer across the room, and following that it wouldn’t print; it’d just sit there at “Printing…”, and the client print progress thing would stay at 0%… until you disconnected the network cable. Then it’d print whatever you’d sent. Weirder still, the same thing would happen with internal print jobs. Print a configuration page? “Printing…” until you disconnect the network cable.

It was still under warranty, so I gave Dell a call. He walked through some obvious things, and then had me flash the firmware on the printer — oops, wait, that’s over the network. Ok, bring the printer over to my desk and… you need Windows to flash it? Ok, over to Levi’s desk, and flash it. No problem. Plug it back in; no luck.

So the Dell guy gives up, they’re just going to send us another printer. Great! But it took a while to figure out whether or not it was in stock, but while it was waiting, a page came out.

Wait, what?

And then, five minutes later, another page. Now, “takes five minutes to print a page” is a very different problem than before! But at this point the replacement printer was being dispatched and the Dell guy didn’t want to do more troubleshooting. But once I got off the phone, I did, and I’m glad.

The first thing I noticed is that the switch lights were blinking like crazy. I tracked that back through two more switches to our Samba server. Aha! Run tcpdump there, and:

14:06:13.459933 IP 192.168.1.151.137 > 192.168.1.150.137: NBT UDP PACKET(137):
REGISTRATION; REQUEST; UNICAST
14:06:13.459933 IP 192.168.1.150.137 > 192.168.1.151.137: NBT UDP PACKET(137):
REGISTRATION; NEGATIVE; RESPONSE; UNICAST
14:06:13.463931 IP 192.168.1.151.137 > 192.168.1.150.137: NBT UDP PACKET(137):
REGISTRATION; REQUEST; UNICAST
14:06:13.463931 IP 192.168.1.150.137 > 192.168.1.151.137: NBT UDP PACKET(137):
REGISTRATION; NEGATIVE; RESPONSE; UNICAST

And as you can see on the timestamps there, both ends were talking as fast as they can — the printer sending NBT registration requests, and the Samba server sending errors back, over and over, hundreds of times per second. Tell the printer to forget about its Samba server, and voila, printing’s back to normal.

So what happened? As best as I can tell, one of two things: Either moving the printer made it get a DHCP configuration for the first time in over a month, since we rolled out a new DHCP server in the meantime; or it’s been slow all along, and moving it to the same switch as the Samba server, instead of two switches away, made it marginally busier, enough for it to not print at all instead of just printing slowly.

Still, I could think of better things to have spent an afternoon on.

We’re no longer allowing (new) FTP/SHELL users to have an email address associated with them.
[…]
Fortunately, this change should be more or less invisible to everybody! The only thing lost is the ability to see and manipulate your mail files via FTP/Shell… (and even that is only for new users from now on). Whoop-dee-do, I say!

Right, why would anyone want to use their own SpamAssassin, procmail, or a Unix mail client? I never had a problem with overselling at Dreamhost — in fact, I’d go so far as to say that I’m happy to take advantage of it — but I don’t think that’s their problem. I think they’ve just let themselves grow until they’re deep over their heads.

(And yes, that doesn’t affect existing shell accounts there, but I imagine that’s just a matter of time, because it’s not like running two parallel mail architectures is going to help them much.)

For $20/mo, I get a virtual server (using Xen, which is conceptually like VMware if you’ve heard of one but not the other) with 360MB of RAM, 10GB of disk, 200GB of monthly bandwidth, a true remote console, and full root access. There’s no CPU or I/O limiter; you’re expected to play nicely but you can burst to the capacity of the hardware (which in my case is a dual quad-core Xeon shared with 39 other Linodes; the bigger Linodes have fewer neighbours). You choose your data centre from three options, too — I’m in Dallas, 2.6 ms from FreshBooks’ servers. And they don’t oversell: there’s often a waiting list for a particular size virtual server, because if the current servers are full they just don’t sell any until they get more servers.

When I moved to Dreamhost, I’d been a sysadmin on a communal coloed box hosted by a friend, and that eventually turned into a drag due to unreliable hardware and unreliable users. I’d decided that I sysadminned enough during the day and that someone else could be my sysadmin. But I was never really happy with that; the web side of things was okaaaay, but not having control over the mail server was a pain, and having hardly any visibility of what MySQL was doing was annoying.

That’s solved now! I’ve moved all of our sites except the whistle forum to the Linode, and my and Candice’s mail is there too. It’s crazy fast compared to Dreamhost (especially IMAP), and I’ve got the flexibility to play with things; one weekend I installed four or five alternative webservers and load–balancers and switched between them, just to get used to their quirks before trying them out at the office, and then back to Apache again.

But what really won me over at Linode was service. It’s a small shop — there can’t be more than five or six employees, support tickets are addressed in minutes instead of days, the userbase is friendly to each other on the forums, and a bunch of senior staff including the owner all hang out on the support IRC channel. I ran into a weird issue once and was sharing my Munin graphs with him minutes later. Even though we never tracked down exactly what happened I’m completely confident in these guys.

They offer virtual servers from my little $20/mo one up to an $80 1.4GB-40GB-800GB/mo plan. They’ve got no referral programs or discount codes; just great performance and great service, and are a great place to dip your toes into system administration, finally get that personal colo box, or even set up a remote monitoring box for critical work-related services.

Ages ago I was doubtful about virtual servers, but that was when $20 only got you 60MB of RAM; now that you can run pretty much anything you’d want to, it’s working out great.

Checking out their machines showed that their DNS server addresses were being changed to an address on the wrong subnet, and their domain being changed to “mshome.net”. That last part’s a red flag: the thing that does that is Windows’ Internet Connection Sharing, which means someone had that enabled on an interface and we basically had a rogue DHCP server.

Rogue DHCP servers are a pain to track down because without a monitoring port on the switch, all you have to go by is broadcast traffic, and then all you get is the address the DHCP server thinks it’s at — which, we know, is on the wrong subnet anyhow — and its MAC address. And we’re a small shop but we still don’t have a handy list of MAC addresses lying around. I did know that the MAC address’s vendor ID was Dell.

So the first thing I did when I found the problem was to check the MAC addresses of all of the wired and wireless interfaces of the Dell computers in the office, and none of them matched! I puzzled over this for a while, had people double-check, and eventually something clicked and Saul remembered that Sunir had enabled ICS during their road trip.

I took a second look at Saul’s laptop, and there was the MAC address — on a disabled wireless broadband interface. Turns out that if you have ICS on, the DHCP server keeps running even when the shared network interface is down. Disable it, problem went away.

But the strange part was that Saul’s been back for a week and the problem just came up today.

I scratched my head about that for a bit and then it hit me: before today, the switch in the wiring closet was in the Linksys router that also served DHCP:

[client]----[switch + dhcp server]----[saul's PC]

After today, both Saul’s network segment and the new DHCP server were both connected to a separate switch:

[client]-----------[switch]-----------[saul's PC]

                       |

                       |

                 [dhcp server]

DHCP is designed to handle multiple (cooperating) DHCP servers on a segment; when a client sends a request, any DHCP servers can respond, and the client chooses one of the responses and informs the DHCP server that sent it that it will use that one. The usual client implementation is to accept the first response.

So before today, a client on one segment would make a DHCP request, but the legitimate DHCP server (at the switch) would be located one Ethernet segment closer to the client than the rogue DHCP server, so it would always win. As of today, the legitimate DHCP server was now the same distance from the client as the rogue one, so part of the time it’d lose, which is exactly what was happening — not every DHCP lease was broken, just the occasional one.

Sometimes it’s easy to forget that actual electrons need to move around for this stuff to work — which in turn reminded me of Trey Harris’s 500-mile email.

I don’t know why I’m so tempted, I’ve already got a little laptop, my 12″ Dell D400 — I think it’s that it’s so little and not “I’m carrying a laptop today” fragile that you could carry it more like a book. It’s decently kitted out, too, with built-in wifi and wired Ethernet, external VGA, an MMC/SD reader, and a webcam.

I think I’ll probably hold off until they start showing up secondhand when their early adopters get tired of them. (I’m glad I’m now in a city where people use Craigslist.)

Any of you guys pick one of these up yet? (deviant-, I’m looking at you!)

Also, until the Air costs $400, you are not permitted to suggest a MacBook Air instead.

I forgot that I wouldn’t have a router right away, so I ended up getting a Speedtouch 516 DSL modem configured as a bridge, so I’m using Windows’ own PPPOE client until the move.

Next up is VOIP. I’m leaning towards Unlimitel because Babytel never replied to my email sent to their support address (pilot error: sent to .com instead of .ca), but there’s still a little shopping to do there. New blog theme/approach coming soon too, although that’s just fitting in the quiet moments of which there have been few lately.

Work is going awesome. I’m moving support over to RT any day now, have a huge set of projects and to-dos to drill through, and have really been given pretty much full ownership over my areas of responsibility already. (And everyone’s so great to work with!)

Off to Dan’s tonight for pizza, beer and conversation. (We tried this last week but I made it a block before I decided it was time to take a rain check and get the car in for a brake job right away. I lucked out in that a great garage happens to be exactly one block away from me, and Frank at Master Mechanic on Dupont and Concord took it in on no notice and did good work at a good price, which was nice considering I needed a garage on short notice in a strange city.)

Movers are scheduled: pack on the 25th, load on the 28th, unload a couple of days later. Final stretch!

It is not like the geek world where top-posting is uniformly evil and >-quoting is normal. Top-posting is unremarkable, and in fact in Outlook it is difficult to not top-post when responding to HTML email. I am at least using Outlook-QuoteFix to >-quote when dealing with plain text, and new email I send is in plain text too.

But still.

(HTML email itself is not hard to get used to because I am using Outlook.)

One thing I’ve noticed is that >-quoting lends itself to taking apart someone’s argument point by point or line by line. It turns out that often this isn’t a very productive approach!

It’s also hard to get used to not being surrounded by geeks. One of the things I did in the weeks I had off between work and school was install Skype and plan on buying a headset, because obviously my team at school was going to use Skype for conference calling.

In fact, none of my group regularly use instant messaging, even. It’s remarkable to some classmates that I have a blog.

I’ve been working hard at not becoming “the IT guy” at school, although there are enough people on Vista that I didn’t really have to worry about that. However, I’ve converted three or four people in class to taking notes in OneNote. (It doesn’t hurt that it’s free on MSDNAA.)

I am, strangely enough, becoming “the financial accounting guy”. Unexpected!