15 June 2007 (25 June Reprise) :( --- Too Many MySQL Connections

Marc

Fully vaccinated are you?
Leader
There is an apparent overload happening this morning, much like I have seen a couple of times in the past. It appears to be an issue with the number of connections so it may be someone trying to flood the server. My apologies for any inconvience as I try to see what is happening.
 

harry

Trusted Information Resource
Re: 15 June 2007 - Server overload

Yes Marc,

I am having some difficulty logging into the main page from this part of the world. From there, logging onto specific threads is even worse. It is happening right now.

Regards.
 

Marc

Fully vaccinated are you?
Leader
Re: 15 June 2007 - Server overload

I'm monitoring processes and making changes to try to limit the connection timeout. Usually only about 40 to 80 processes are stacked. Today, like a week or so ago, there are up to 180 stacked requests. When it reaches 180 stacked http requests everything slows down or stops.

I started this thread to warn people because I have no option at this time other than to try to tweak connection settings. I may have to hard reboot the server at some time this morning to ensure the settings I am changing have fully taken effect. The server supposedly has DDoS protection turned on so I'm not sure why the httpd requests are stacking up like this.
 

Jim Wynne

Leader
Admin
Re: 15 June 2007 - Server overload

I'm monitoring processes and making changes to try to limit the connection timeout. Usually only about 40 to 80 processes are stacked. Today, like a week or so ago, there are up to 180 stacked requests. When it reaches 180 stacked http requests everything slows down or stops.

I started this thread to warn people because I have no option at this time other than to try to tweak connection settings. I may have to hard reboot the server at some time this morning to ensure the settings I am changing have fully taken effect. The server supposedly has DDoS protection turned on so I'm not sure why the httpd requests are stacking up like this.

All seems OK right now (9:10 CDT).
 

Marc

Fully vaccinated are you?
Leader
Re: 15 June 2007 - Server overload

It appears I have the server stabilized for now.

I have identifed one issue and that is when it reaches 180 persistent connections it goes over the 2 gigabytes of memory and starts using the swap file which slows the server to a crawl. Now the question is why so many persistent connections occur.

Right now it's running normally at between 40 and 80 processes.
 
C

chergh - 2008

Re: 15 June 2007 - Server overload

If your routers are getting hit with large quantities of hits per second then even ddos protection isn't going to save you. If they're spoofing their source IP address, depending on how your ddos protection works, your ddos protection may not even work at all as it may just look for high volumes of traffic coming from specific origins.
 

Jim Wynne

Leader
Admin
Re: 15 June 2007 - Server overload

It appears I have the server stabilized for now.

I have identifed one issue and that is when it reaches 180 persistent connections it goes over the 2 gigabytes of memory and starts using the swap file which slows the server to a crawl. Now the question is why so many persistent connections occur.

Right now it's running normally at between 40 and 80 processes.

Sounds like maybe an amateur is screwing around. A real, dedicated DOS attack might involve thousands of connection attempts from a botnet. If I recall, your server is on a remote farm; is there any way you can increase the RAM?
 

harry

Trusted Information Resource
Re: 15 June 2007 - Server overload

Hi Marc,

Just to report that things are back to normal from over here. If I am not mistaken, some slowdown was apparent since noon over here or mid-night over at US.

Thanks for your prompt action.
 

Jim Wynne

Leader
Admin
Re: 15 June 2007 - Server overload

Hi Marc,

Just to report that things are back to normal from over here. If I am not mistaken, some slowdown was apparent since noon over here or mid-night over at US.

Thanks for your prompt action.

Yes--It's good to know that Marc is vigilant and working to keep things flowing. :agree1:
 
C

Craig H.

Re: 15 June 2007 - Server overload

I tried to use "contact Marc" earlier this morning to tell you about the problem and got the "page not found, try refresh" screen several times. I was getting the same thing for several of the threads, with slow response. Now the site appears to be back to its smokin' self.

Thanks, Marc!!!
 
Top Bottom