It seems like Markus was doing something similar to me this weekend. We both were dealing with unwelcomed (and scary) nightmares. His ordeal was related to PostgreSQL and MySQL, mine was a bit scarier. In other words, while he knew exactly what went wrong from the start, I did not have that luxury.
It happened right about midnight. I was working on finalizing Lucene integration with my new community project when suddenly the server went down for automatic reboot due to heavy load (someone was trying a brute force attack). It wouldn't come back up.
After hours of communicating with my unmanaged hosting provider, the server was still not back up.
The issue as they reported was that my hard drive was mounted read-only and there were a lot of /dev/null related errors. They (my hosting company) told me that my hard drive may have gone bad.
Unfortunately my entire project was on the hard drive. *sweat*
All night I couldn't sleep.
Today, I was told it could take many weeks to have a systems support specialist investigate the issue.
I researched furiously without KVM or direct access and was able to find some pointers which I communicated to my hosting company. Basically I asked them to recreate the /dev/null properly after removing it. To my horror, they told me that after recreating /dev/null, the system won't come back on.
As I sat in a state of shock I posted on the hosting company's support forum where a different SSS came to my rescue and within minutes he had the server back online (he recreated /dev/null properly this time).
So the lesson here is that if you are working on a project of any kind, keep backups on different drives. Normally I am pretty paranoid about such things but since this project was in development I hadn't backed it up. All my data and MySQL database schema were at risk.
In the end I didn't have enough words to thank the SSS. Without him, I would be crying.
In related notes:
Arjen had a homework for everyone about checking Horde's schema but before I could comment he posted the answer. He also has a post differentiating relation from relationship, something that many people often get wrong.
Rumors are that Google may be planning to buy Sun Microsystems. I have commented earlier that Yahoo should consider jumping in.
sun MySQL community backup database Google
No comments:
Post a Comment