Disasters happen….

At Tim’s request, here is a disaster recovery story I entered as part of another discussion. It is reposted here as a new blog in relation to the current podcast.

I worked at Apple when the big earthquake of 1989 hit, which completed destroyed the insides of the building I was in (DeAnza 3, across from the current campus). The earthquake was so bad to that building, that a MAJOR water main broke open on the fourth floor and filled the building with water, as much as 4 to 5 feet on each floor after several hours.

But what is worse was that the Herman Miller cubicles that we all used basically fell apart, the horizontal surfaces in a number of cubes dropping off at a 45 degree angle dumping most of our equipment on (or near) the floor BEFORE the rush of water filled the building.

I was one of the first people out of the building from my office on the third floor (another story.) However, my keys and briefcase were not. So an hour after the quake, I re-entered the building, and was VERY happy that the power was off, because as I went up a stairwell, it looked like a ride from Disneyland, with a rushing waterfall coming down. On the third floor was about 2 feet of water now, and water raining down from the ceiling above through the light fixtures. Computers, drives, monitors, and other equipment dangling into the water supported from the work surfaces by their power cords could be seen everwhere! When I found my ‘office’ (or what was left of it,) all the horizontal surfaces were now tilted at about 45 degrees, and much of my equipment was either hanging by power cords just above the water level, or now completely under water! I also had a small 20-gallon salt-water aquarium that had dumped 50% of its salt-water onto my two external drives sitting next to it, strangely enough, all three still sitting on the tilted surface. But, being in an extreme hurry to get back out, I just grabbed my briefcase, which lucky for me was sitting on my chair (only a few inches above the water level now) and quickly left the building. We did not get to go back to Apple for a number of days, and were not let back into that building for a very long time.

They called for, and I volunteered for the equipment recovery team after the quake. Volunteers collected equipment from around the buildings that was damaged, or thought damaged, and from the now condemned DeAnza 3 building as well, and brought it to the team I was on, ‘discover and recover.’ Unfortunately, given the need to do this quickly, much of what came from DeAnza 3 was not marked as to who owned what, the goal to see what could still be used, and what was trash. Thousands of Macs, monitors, keyboards, mice, printers, and hard drives poured (literally) into our triage center, where we were to evaluate their condition and recover what we could. Believe it or not, opening and drying out the equipment with high-pressure air and using the hardware test chamber oven to ‘cook’ out the water, we were able to recover, I am guessing, more than 80% of the equipment. Most that did not make it had cracked cases (one monitor fell out a third story window!) or, in the worst cases, the lithium battery powered a CPU logic board long enough for the water to short out and erode away traces, making the board useless. But even on many of those computers we were able to boot by applying ‘start’ power through the ADB keyboard connector (which is really all that the battery did, keep the keyboard and buttons alive so the system could read the start button’¦and yes, non-volatile memory settings too.)

But my favorite items were when my own two external hard drives hit the table. Nothing was really labeled as to who owned what, but here came an Apple 80 and 160 MB (very large for the time) external drives caked with salt, and still wet with salt water inside. That HAD to be mine, and it was (I discovered later.) I immediately dunked them both into a bucket of clean water several times to eliminate the salt, and then opened them up and did that again. Dropping a hard drive into a bucket of water was a strange feeling at best. I then blew dry the units with the high pressure air, and set them in the test oven for a while to completely dry. They actually looked slightly rusted from the salt water, but being the optimist I was, I put them back into the cases and powered them up. After all, what did I have to loose but a fuse or circuit breaker! Imagine my surprise when both drives spun up immediately! But could I read them?

Ok now to my computer, I believe it was called a ‘zone 5’, a prototype of the IIfx close to completion, which was only slightly wet (it managed to stay on the desk they told me), and my color monitor, which was also only slightly wet. I dried them as well, plugged it in and connected it to the hard drives, and hit power. It booted immediately from the EXTERNAL 160 drive (how I had it set last time I used it) and mounted all three drives (2 external and 1 internal) as if it were simply the next day of work! I did, of course, move all the data off to a new drive, but today, almost 15 years later, I still have that IIfx computer AND the one of the two drives as well, and they still work! The other 80 Meg drive I removed from the external case and then removed the top and bottom plates of the driave and let it sit exposed, RUNNING, on the window sill of my new office. I used it as a scratch drive so I did not put anything on it important, but it was connected and ran non-stop in this exposed manner for almost 2 years. It died only because some bozo tossed a nerf football at it, and knocked the running drive on its side! Parts flew EVERWHERE!

So when I see or hear a recovery story, I realize that Apple still builds them as strong and solid as ever. We recovered a majority of the computers damaged that day, and those that did not were usually because of physical damage to the case or insides from a long fall. It is worth noting that we recovered almost all the information on the damaged machines as well.

As for my salt-water fish? They were not as recoverable. All three were actually found still barely alive about a week later, in half a tank of cool water (they need warm water and food daily typically), and although I tried to save them, they died, unfortunately, shortly after being moved to a new tank.