Welcome to MekTrix 4.0

I don’t know if I’ve ranted on this last time, but it happened again with another site, so I shall rant again. For context, I’m the technical admin of a certain large site, which I won’t even name due to NSFWness — I’m just mentioning it here so that visitors from that site know who I am :) A brief timeline of events:

• 2 days ago, I move the database to a separate server, as MySQL grinds to a halt when the database is too big to fit in RAM.
• At around 11pm yesterday, the site suddenly dies, printing “500 Internal Server Error” in response to all requests
• Restarting the web server fixes it, but the old PHP processes don’t die, and the new ones lock up pretty quickly too
• PHP isn’t actually printing any error messages, it’s just locking up; LigHTTPD only prints “All PHP processes are busy, try again later”; MySQL logs show no error messages.
• It seems that the site is fine for as long as nobody tries to post new content.
• I try stopping and restarting the database, it says “Can’t stop database”
• I finally realise what happened — the disk with the database tables filled up, then MySQL crashed and died horribly, corrupting chunks of the tables. Then when PHP asks the database what’s up, and the database says “error”, it doesn’t print “there’s a database error”, it just dies horribly and locks everything up, causing ISE’s and using up all the ram until everything around it dies too.

In contrast, my preferred database server, Postgres, ran fine throughout the incident — read-only requests were served as normal, and if anything tried to add data it would be told “the disk is full”; no crashing, no corrupt data, no problem. Also, it seems to scale better, currently dealing with a database of 8 * the size of RAM quite happily.

It’s 7am, so I’m going to bed now; first thing I do when I wake up will be start work on giving this app Postgres support; then I can uninstall mysql since nothing else needs it \o/

This entry was posted on Monday, January 7th, 2008 at 12:17 am and is filed under problem, rage, tech. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

1. mr furry Says:
   January 7th, 2008 at 12:45 am

   HELLO thnks for all this you did excelent web you have

2. Rat Says:
   January 7th, 2008 at 3:39 am

   I’m actually kinda glad the site died for a few hours, it gave me time to sleep for once! But I’m also glad it’s back.

3. pl0x Says:
   January 7th, 2008 at 11:21 am

   You did wonderfully, very good job. You deserve the sleep!

4. King1234 Says:
   January 8th, 2008 at 7:17 pm

   At the moment im not getting any images to load

5. Shish Says:
   January 8th, 2008 at 10:48 pm

   That’s known, and unrelated — Having fixed the overloaded database, then fixed the crashed database, dreamhost have cut off both our image servers -_-

6. Fluffy Says:
   January 9th, 2008 at 5:05 pm

   Is that why some of the pictures won’t show on my screen?
   All I get a blank boxes with a red X.

7. Mudkip Says:
   January 10th, 2008 at 12:06 am

   Holy cow, man. Left and right all things MySQL die off and corrupt. Kind of like windows, only less retarded users…
   Nonetheless, thanks for supplying all the NSFW stuff ;)

8. Blarg Says:
   January 10th, 2008 at 11:53 pm

   Upon load of a page it only displays some images, seemingly at random. If I reload, which images process properly and which end up broken is again randomized, if that makes sense. So if I reload enough times, I may just get to see every thumbnail at least once. :(