![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
This afternoon, just as I was about ready to give up and go to lunch, I received an instant message from ![[livejournal.com profile]](https://www.dreamwidth.org/img/external/lj-userinfo.gif)
stormgren, my friendly Colocation provider. He wanted to inform me that there was something horribly, horribly wrong with my server.
I ssh'ed into the box, and to say that it was sluggish to respond is like calling the Sun a ball of fire. No matter what I did, it took several minutes to respond. I hit top to check the load average -- 34+... and > 95% of the CPU's time was being spent in IOWAIT.
Run "netstat -a" and when it manages to get far enough along, I begin to realize just how screwed I am. There are WAAAAAAY more open connections than there should be -- screens and screens of 'em. And almost all of them were from machines (yes, plural) in the clusters (yes, plural) of computers that Yahoo has crawling webpages to index them. I didn't notice many duplicate machine names.
So, I send the command to kill apache and givestormgren an update. He did a little looking, and discovered > 10,000 open connections coming through the firewall, and >75% of those were coming from Yahoo... and going into my server. As a band-aid, he blocked all traffic from Yahoo's crawlers to my server, and it started feeling better immediately. (Yes, the "pkill -9 httpd" was still running when he updated the firewall.)
It shouldn't be too long before Yahoo's Slurp clusters regain some sanity... or at least before their insanity is being directed at some other poor schmuck. In a few days, I'll probably try letting them back in.
stormgren, my friendly Colocation provider. He wanted to inform me that there was something horribly, horribly wrong with my server.I ssh'ed into the box, and to say that it was sluggish to respond is like calling the Sun a ball of fire. No matter what I did, it took several minutes to respond. I hit top to check the load average -- 34+... and > 95% of the CPU's time was being spent in IOWAIT.
Run "netstat -a" and when it manages to get far enough along, I begin to realize just how screwed I am. There are WAAAAAAY more open connections than there should be -- screens and screens of 'em. And almost all of them were from machines (yes, plural) in the clusters (yes, plural) of computers that Yahoo has crawling webpages to index them. I didn't notice many duplicate machine names.
So, I send the command to kill apache and give
stormgren an update. He did a little looking, and discovered > 10,000 open connections coming through the firewall, and >75% of those were coming from Yahoo... and going into my server. As a band-aid, he blocked all traffic from Yahoo's crawlers to my server, and it started feeling better immediately. (Yes, the "pkill -9 httpd" was still running when he updated the firewall.)It shouldn't be too long before Yahoo's Slurp clusters regain some sanity... or at least before their insanity is being directed at some other poor schmuck. In a few days, I'll probably try letting them back in.
