XFileSharing Pro - (11)Resource temporarily unavailable Error

[ufkabakan]

Hello All,

I have a load problems on FileServers. Im using still Apache on FileServer. My settings for Free/Guest users "dl.cgi" for speed limiting and non resume.

My httpd.conf:

Code: Select all

Timeout 60
KeepAlive Off
MaxKeepAliveRequests 10
KeepAliveTimeout 5

<IfModule prefork.c>
StartServers       30
MinSpareServers    30
MaxSpareServers   75
ServerLimit      4096
MaxClients       4096
MaxRequestsPerChild  20000
</IfModule>

Some peak hours i see this error msg on Apache error log:

Code: Select all

[Wed Aug 01 13:13:31 2012] [error] [client 223.25.10.xxx] (11)Resource temporarily unavailable: couldn't create child process: 11: dl.cgi, referer: http://www.mydomain.com/ssxtutvlxaaa/2011.Guiness.World-Rec.part1.rar
[Wed Aug 01 13:13:31 2012] [error] [client 223.25.10.xxx] (11)Resource temporarily unavailable: couldn't spawn child process: /home/sv9/cgi-bin/dl.cgi, referer: http://www.mydomain.com/sstutvlxpoa/2011.Guiness.World-Rec.part1.rar
[Wed Aug 01 13:13:33 2012] [error] [client 223.25.10.xxx] (11)Resource temporarily unavailable: couldn't create child process: 11: dl.cgi, referer: http://www.mydomain.com/96xtutvlxpoa/ss12GuinessWorldRecords.part1.rar
[Wed Aug 01 13:13:33 2012] [error] [client 223.25.10.xxx] (11)Resource temporarily unavailable: couldn't spawn child process: /home/sv9/cgi-bin/dl.cgi, referer: http://www.mydomain.com/sstutvlxpoa/2011.Guiness.World-Rec.part1.rar
[Wed Aug 01 13:13:33 2012] [error] [client 223.25.10.xxx] (11)Resource temporarily unavailable: couldn't create child process: 11: dl.cgi, referer: http://www.mydomain.com/96xtutvlxpoa/2011.Guiness.World-Rec.part1.rar
[Wed Aug 01 13:13:33 2012] [error] [client 223.25.10.xx] (11)Resource temporarily unavailable: couldn't spawn child process: /home/sv9/cgi-bin/dl.cgi, referer: http://www.mydomain.com/sstutvlxpoa/2011.Guiness.World-Rec.part1.rar
[Wed Aug 01 13:13:35 2012] [error] [client xx.aa.xx.bb] (11)Resource temporarily unavailable: couldn't create child process: 11: api.cgi
[Wed Aug 01 13:13:35 2012] [error] [client xx.aa.xx.bb] (11)Resource temporarily unavailable: couldn't create child process: 11: api.cgi
[Wed Aug 01 13:13:35 2012] [error] [client xx.aa.xx.bb] (11)Resource temporarily unavailable: couldn't spawn child process: /home/sv9/cgi-bin/api.cgi
[Wed Aug 01 13:13:35 2012] [error] [client 210.187.243.bbb] (11)Resource temporarily unavailable: couldn't create child process: 11: dl.cgi, referer: http://www.mydomain.com/aaa8u3uwjd1/WinUtilitiesPro.10.44.Multilingual.rar
[Wed Aug 01 13:13:35 2012] [error] [client 210.187.243.bbb] (11)Resource temporarily unavailable: couldn't spawn child process: /home/sv9/cgi-bin/dl.cgi, referer: http://www.mydomain.com/aaa8u3uwjd1/Win.Utilities.1.34.Multilingual.rar
[Wed Aug 01 13:13:36 2012] [error] [client 202.88.75.ccc] (11)Resource temporarily unavailable: couldn't create child process: 11: dl.cgi, referer: http://www.mydomain.com/aaa8mtsodh58/BarAAtts.rar
[Wed Aug 01 13:13:36 2012] [error] [client 202.88.75.ccc] (11)Resource temporarily unavailable: couldn't spawn child process: /home/sv9/cgi-bin/dl.cgi, referer: http://www.mydomain.com/aaamtsodh58/BarAAtts.rar
[Wed Aug 01 13:13:37 2012] [error] [client 203.117.37.ccc] (11)Resource temporarily unavailable: couldn't create child process: 11: dl.cgi, referer: http://www.mydomain.com/aaauwdwts7l/X-Men.1.part2.rar
[Wed Aug 01 13:13:37 2012] [error] [client 203.117.37.ccc] (11)Resource temporarily unavailable: couldn't spawn child process: /home/sv9/cgi-bin/dl.cgi, referer: http://www.mydomain.com/aaahuwdwts7l/X-Men.1.part2.rar
[Wed Aug 01 13:13:38 2012] [error] [client xx.aa.xx.bb] (11)Resource temporarily unavailable: couldn't create child process: 11: api.cgi
[Wed Aug 01 13:13:38 2012] [error] [client xx.aa.xx.bb] (11)Resource temporarily unavailable: couldn't spawn child process: /home/sv9/cgi-bin/api.cgi

On this error, server have total 1500~ connection on Apache, server have low loads like %5, ram usage 2GB~

This servers hardware like:
E3-1230 CPU
8GB DDR3 Ram
4x2TB Disk
1Gbit Port

I think problem about apache config. Because i remember this server can handle easly 4K connection, before reinstalling OS + Apache, 3 months ago.

Any Advice?
Please dont offer nginx mod, this is not a CPU load problem.

[PowerChaos]

Code: Select all

Timeout 30 
KeepAlive On
MaxKeepAliveRequests 5 
KeepAliveTimeout 10 

<IfModule prefork.c> 
StartServers       5 
MinSpareServers  5  
MaxSpareServers   10
ServerLimit      512
MaxClients       350 
MaxRequestsPerChild  10000
</IfModule> 

this should be fine

with keep alive off , every new request will be taken care of

then for the child process error , it says exactly what stands there
that apache is not able to create a child process , that means or apache is flashing or under heavy load
or apache just have not enouf time or to many open connections so it can not create childprocess

eather way , lower the maxrequestperchild and that problem should be solved but load can go up (also ram can go up)

those settings are the same i use on my webserver and here you can see the status of how it is doin (cpanel on it )

only when i get a ddos attack then all requests get filled but the website can still be accesed (slow but working)

http://powerchaos.com/status

Greetings From PowerChaos

[ufkabakan]

I cant stay with 512 Max client.

From my status page, after havea load and restart Apache:

Total accesses: 1389 - Total Traffic: 58.5 GB
CPU Usage: u.03 s.68 cu3.91 cs0 - 1.5% CPU load
4.5 requests/sec - 193.9 MB/second - 43.1 MB/request
1008 requests currently being processed, 51 idle workers

This server must handle 3K connection with Apache, but how?

[Mr_parham]

I cant stay with 512 Max client.

From my status page, after havea load and restart Apache:

Total accesses: 1389 - Total Traffic: 58.5 GB
CPU Usage: u.03 s.68 cu3.91 cs0 - 1.5% CPU load
4.5 requests/sec - 193.9 MB/second - 43.1 MB/request
1008 requests currently being processed, 51 idle workers

This server must handle 3K connection with Apache, but how?

Apache always had problem with high number of connections, i would recommend changing the webserver mate

[PowerChaos]

let me show you a difference ok ??

Total accesses: 1389 - Total Traffic: 58.5 GB
CPU Usage: u.03 s.68 cu3.91 cs0 - 1.5% CPU load
4.5 requests/sec - 193.9 MB/second - 43.1 MB/request
1008 requests currently being processed, 51 idle workers

Current Time: Sunday, 05-Aug-2012 12:56:42 CEST
Restart Time: Thursday, 02-Aug-2012 12:02:41 CEST
Parent Server Generation: 41
Server uptime: 3 days 54 minutes 1 second
Total accesses: 1926008 - Total Traffic: 495.4 GB
CPU Usage: u273.22 s85.27 cu17.02 cs0 - .143% CPU load
7.34 requests/sec - 1.9 MB/second - 269.7 kB/request
21 requests currently being processed, 18 idle workers

Do you see the request per second ??
i got more requests per second then you have but a lot less requests that are beein processed at the same time

That means a few things
or you are having a ddos attack ( mass connection at the same moment)
or your server does not close connections like it should be

You need to tweak apache to close connections after a certain moment
You can also use keep alive to limit those connections of the same users , a keep alive let you run mass connections to 1 single connection ( like 10 requests to the same process )

i do not know if your webserver is serperated from your file server
but if that is not the case , then i definaly would do that to spare some connections (so file server only serve downloads and webserver take care of users with a fast close connection )

hopely this helps you

here is the current status of a dedicated server that only serve a certain type of forum and that got a lot of visitors at the same moment

average users online is 30 at all times (whois.among.us) with peaks of 60 users

Current Time: Sunday, 05-Aug-2012 13:05:35 CEST
Restart Time: Thursday, 05-Jul-2012 22:06:44 CEST
Parent Server Generation: 468
Server uptime: 30 days 14 hours 58 minutes 51 seconds
Total accesses: 13269078 - Total Traffic: 73.0 GB
CPU Usage: u42.96 s4.64 cu.01 cs0 - .0018% CPU load
5.01 requests/sec - 28.9 kB/second - 5.8 kB/request
12 requests currently being processed, 8 idle workers

ofcourse a file server is a little differend , but it shows that it depends how you tweak the things and that it does mather how you setup it

many requests is never good , so they need to stay low

and my final example is apache itself
http://www.apache.org/server-status

Greetings From PowerChaos

[ufkabakan]

I fixed this problem.

Solution: Kernel Update and Optimize Kernel configs.