Just to observe that I'm back home after appearing at Picocon in London and doing a bunch of publicity-related stuff with my UK publishers. No props whatsoever to our hotel who had no internet connectivity of any kind (modem line in room included) and blocked mobile phone signal so efficiently that I couldn't even get online via GPRS.
Yes, I'm aware the site may have been inaccessible; we've been hit by trackback spammers so badly that it took a couple of reboots and some emergency duct tape to get everything working again (which I prefer not to do after a night of being gently bounced off the walls and ceiling of a compartment on a sleeper train).
Normal service will be resumed when I've caught up on my sleep deficit. Meanwhile, the Linux-cognoscenti among you might find the following screenshot amusing:
holy crap. how long did it take for top to come up?
Ahem: I figured out the machine was pingable but probably dos'd, so I requested a reboot. As soon as it came back up, I set top running, because I figured it'd show me what the attacker was hammering. As indeed it did; spot the 40 minute uptime. So I requested another reboot, logged in immediately, and disabled Movable Type trackbacks by the simple expedient of chmod -x $(locate mt-tb.cgi) before the bastards could hose me again. (slocate saves the day, once again.)
Turns out you can throttle comment posting under MT, but not trackback pings. As you'll have noticed, this box is already locked down to only accept one comment per three minutes from any given IP address. I don't really use trackbacks anyway, so gutting the subsystem was no problem.
Wow! The last time I saw a load average that high, the computer crapped out a few seconds later in between machine instructions in a kernel routine. Didn't event get to panic; just stone dead.
Did anyone get the IP of that truck?
Bruce, the machine did indeed stop updating the console at that point.
(Worst ever load average I've seen was a screenshot -- I wasn't there at the time -- of a sun box that had just been fork-bombed. The graph showed a beautiful asymptotic curve and had frozen around 400.)
I run a wordpress blog, so I sympathize with the comment/trackback spam. However, you might use this as inspiration for a new Atrocity Archive short story. Imagine a Mythos tainted terrorist who uses spam tools to spread Mythos knowledge, exposing millions of innocent blogs to flawed summoning formulas...
Am I imagining things, or is someone/a group out to get you, or does this happen to all bloggers?
BTW - I'm seriously out-of-date on this stuff ... I gave up when they replaced FORTRAN IV - well, not long after that, anyway .....
I'd think about putting some pretty tight limits on things like how many concurrent processes the www-data user can have. It won't stop evil-doers from DOSsing the web server, but at least it'll be easier for you to get in and fix the damage.
FWIW, I avoid these problems on my journal by not having trackbacks (I'm not entirely sure what they are even) and by moderating any post that contains the string 'http://'. Dunno if that's helpful in your situation though.
G. Tingey: this is totally normal for anyone who runs a popular blog. Sad to say, we get hit by comment spam every day, trackback spam that amounts to a DOS attack every month or so, and I personally get about 400-500 spam emails per day. (In case you were wondering why I use that annoying feedback form for people to contact me, it's so that there's some hope of seeing your missives, because that way I can bypass the spam filter! Except that periodically spammers discover it and try to mailbomb me ...)
David: Trackbacks were globally disabled in MT on this system, but it doesn't seem to have stopped the trackback spammers using the mechanism to DOS me. Annoying, isn't it?
400 spam a day? I wish - spammers have been forging their spams so it looks like it came from my subdomain for the last couple of weeks, and I'm currently getting 40 000 bounces a day....
AndrewSI: yeah, that's happened to me, too. When they stop doing it, the shit-storm will die down again. The icing on the shit-cake in my case was that the spammer (and the ISP they were using, and the domain they were pointing their junk at) were Turkish (with no English-language contact info in the case of the ISP).
Okay, I'm impressed, holding a load of 100 means you're almost not a toy OS.
400 spam a day? That was an hourly number for me, at least, before I gave up on eriko@mo.net -- which was roughly equivalent to giving up my name.
BTW, that box needs I/O, badly -- 141 load average, but 89% CPU idle? Elf need IO, Badly!
(Spammer shot the food!)
AFOE had part of this experience - our hostco darked the blog over Christmas claiming we had an insecure trackback script. We haven't had trackback enabled for more than a year.
Erik #11: it's a PC. An athlon 2.4GHz box, in fact -- not exactly this year's season -- with a SCSI RAID controller. The hardware's new build from UK2.net's copious spares bin and basically replaces a box I'd been leasing for three years. Yeah, it could use I/O; it could also use SATA and more than 1Gb of RAM. I figure for a cheap 2003-vintage no-name PC server that's not bad performance.
(Hope Boskone went okay ...?)
Here's hoping. In the meantime, if anyone's interested in share tips, I'm getting a load of really hot picks from these bounces!
I've got about 10k spams currently in my gmail spam filter. Apparently a woman in the Bronx named Rhondell Payton is using my email address for various reasons. It's not just spam she's signed me up for it either. I've gotten several 'welcome' emails to various services included with password information. The sad part is, I don't know her real email address so I can't contact her. I tried calling her but I've only gotten an answering machine.
Charlie (#10), use the nice new toy I implemented. It'll make your life much simpler, at least until threaded comments on MT comes along.
Largest system load I've seen on Linux was around 400.
The machine was disconnected from the LAN for a couple of days (lack of Ethernet cables at that table, and a guest needed to connect his laptop). A cron script tried to record some statistics every five minutes, but was blocked by a disconnected NFS mount. When we connected the machine back, cron tried to send out about four thousand error emails at the same time. The machine lived, but was nonresponsive for about 10 minutes, after which I could log in and saw the load figure mentione above.