Comments on: This is the most annoying hardware problem ever.

By: brianenigma

brianenigma — Mon, 05 Dec 2005 23:05:37 +0000

One place that I worked at had a pretty crappy server room with crappy building wiring. Every Thursday night at roughly the same time, several severs in the rack would force-reboot (without saving anything, possibly tweaking the non-journaled filesyste). Not all of the servers did this, just a few. It took a very long time to realize that one of the outlets in the room (which, I think, was actually a broom closet) was linked to a mysterious light switch two rooms away that nobody ever thought to mess with (“it’s broken; it doesn’t connect to anything.”) When the cleaning people came in on Thursdays after everyone left, they would jiggle the switch, thinking it went to the overhead lights, causing the servers to switch off and on.

By: ptomblin_lj

ptomblin_lj — Mon, 05 Dec 2005 16:17:57 +0000

Maybe the hang is some exact number of minutes after the last reboot. Like the old Windows 47 day thing?

By: dossy

dossy — Mon, 05 Dec 2005 16:08:16 +0000

Duuuuuuuude, it’s so obvious, maaaaaan.

Your Ultra 10’s stoned. I mean, how long after 4:20 would it take you to just say “aw, fuck it” and stop working?

:-)

By: rahaeli

rahaeli — Mon, 05 Dec 2005 08:25:23 +0000

*snerk* Yes, well. We managers are like that. ;)

By: Rich

Rich — Mon, 05 Dec 2005 08:22:59 +0000

I loved that story, until a former manager liked it enough to tell it lots, having forgotten every time that he'd told it before. :-)

By: rahaeli

rahaeli — Mon, 05 Dec 2005 08:15:42 +0000

Still. A bizarre question, on the order of the 500 mile email.

By: Rich

Rich — Mon, 05 Dec 2005 08:12:15 +0000

I have no idea why the word “Exactly” is there at the beginning.

By: Rich

Rich — Mon, 05 Dec 2005 08:11:08 +0000

Exactly. Doubtful — if the cleaning people were that consistent then sometimes they’d empty the garbage at my desk when it needs it. None of the other machines’ crontabs have anything odd (:27? That’d be weird.) It’s in a data centre, not a back room, so power ought to be stable, or unstable power ought to be setting off alarms. It does have a lot of network interfaces (pair of qfes) so there are a lot of possible dependencies, but that’s the whole problem, figuring out what dependency is doing it :-)

By: crschmidt

crschmidt — Mon, 05 Dec 2005 07:21:03 +0000

I found out this morning, looking back through my support ticket history, that at 4am on the 4th of every month, my colocated machine fails. Nothing in syslog, it just *dies*.

I knew it had happened a couple times, but I only noticed the timing today. Quite annoying, really. Makes me wonder if it’s something that’s not actually on my end. Left a non-urgent ticket in their support queue, so we’ll see what they say.

Just felt like sharing a similar anecdote.

By: rahaeli

rahaeli — Mon, 05 Dec 2005 05:41:03 +0000

Is it always at exactly 4:27? Might it be the cleaning people hitting some kind of interference with the vacuum or something?

*ponder*

Is there anything in the crontab of the machines racked with it that might cause the machine above/below it to wig out at that time and cause radio interference?