11-30-2007, 05:38 AM
Karl Wrote:Was it just the forums that were down? I could've sworn the entire site was down for a good month or so (everytime I'd try to go onto the site, the page would just never finish loading). Before that, I had noticed that you couldn't join the forums but I thought that was just a technical glitch.
The forum was hijacked by someone with a .ru email address, who used an exploit in the phpBB code that allowed him to have administrator rights to the forums, essentially commandeering the controls here. The forum software itself was old and didn't have the latest security patches (phpBB is probably the most hacked forum software due to its popularity.) We've now installed the latest version, so hopefully the exploits have been closed. This version has a better update facility as well.
Something else shortly thereafter, I'm not sure what, caused the disk to fail. We were running on a DEC Alpha server previously. After the crash, it refused to boot up to the operating system. Something either overwrote the boot sector or the BIOS itself. Whether that was a hardware failure, or malicious code is hard to tell without forensically restoring the hard drive and hoping to find something in the system logs indicating what happened just prior.
The past couple of months have been frustrating as backups of certain components (Interlinear, Forum, Lexical Database, etc) were all over the place. Software compatibility issues hampered progress as well (it's not fun trying to migrate the forum's database tables to a format several versions newer). In addition to all of that, new hardware had to be purchased, installed, configured, etc.
Thankfully most of the important stuff is still intact. The forum lost a lot of posts, anything posted basically after mid-2005.
Moral of the story: Backup frequently.
We are now running on a state-of-the art server with hardware-based RAID units (if one disk fails, the other is a mirror), hardware-based firewall and we are now backing up hourly to an external server that happens to be offshore. The most data we will ever lose in the future will be a mere hour's worth.