Memory-Lane Monday: Never let a software guy near hardware

This pilot fish gets an early-evening call from his business’s security company, telling him that the computer room is sending out a “high thermal event” alarm.

There’s no answer after several tries to contact the tech who’s on call, so fish makes the 45-minute drive to the office.

“I assumed the office must have lost power and the IT systems were still running on battery backup,” says fish.

Wrong.

“While the general office space was fine and showed no evidence of power loss, the computer room felt like Arizona — the thermometer read over 90 degrees — and the two redundant wall-mounted air-handling systems weren’t running.”

Some of the older hard drives are making horrible screeching sounds as fish gets to work. He props open the computer room doors and then begins to shut down all the servers, arrays and network hardware as fast and as safely as he can.

Once that’s done, he puts some fans in the doorways to draw cooler air in from the main office space. Then fish turns his attention to the pair of air handlers, wondering how two systems running on separate electrical circuits could have died at the same time.

He pushes the power button on one. It roars to life, pumping out cold air. He flips the switch on the other air handler. It starts right up too.

A few minutes later, the on-call tech arrives with one of the company’s software developers — and an explanation.

“The two had been out golfing at a course about two miles from the office after work,” fish says. “The tech put the on-call cellphone, which was set to vibrate, in his golf bag. He never noticed until they were finished with the round that he had voice mails from me.”

Then fish describes what he found in the computer room — the Arizona-like heat, the screaming disk drives, and the air handlers that were switched off.

That’s when the developer groans. Turns out he had been in the computer room a few hours earlier, troubleshooting a problem over the phone.

But between the noise from the servers, disk arrays and air handlers, he was having a hard time hearing the person on the other end of the line.

“Figuring it would only be a few moments, he turned both air handlers off to quiet things down,” says fish. “But he forgot to turn them back on when he left.

“Fortunately, the damage was minimal — we lost one drive in a RAID array and missed our nightly backup cycle, but all of our systems were working, communications to the remote offices were back up and our data was safe.

“And I got movie passes and a gift card to a restaurant from the developer as an apology.”