XFileSharing Pro - Nginx strange behavior of download limit

[traleto]

Greetings,

We own couple of file storage server and recently we've experienced low disk performance on some of them. We investigated and discovered that the problem was into the disks so we replaced them and now disks are preforming very well. Here are some stats about disk performance:

Before disk replacment:

Code: Select all

----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw
  1   2  89   8   0   1|  20M  714k|   0     0 |   0     0 |2696  1936
  0   2  41  55   0   2|  30M    0 |   0     0 |   0     0 |  16k 1136
  1   2  36  59   0   2|  41M    0 |2136k   87M|   0     0 |  19k 1584
  0   2  61  34   0   2|  37M    0 |   0     0 |   0     0 |  20k 1511
  0   2  60  35   0   2|  37M    0 |2382k  103M|   0     0 |  19k 1675
  1   2  57  39   0   1|  33M 2080k|   0     0 |   0     0 |  19k 1335
  0   2  50  46   0   2|  39M    0 |2374k  100M|   0     0 |  20k 1375

And disk transfer reads at peak usage:

Code: Select all

 Timing cached reads:   7448 MB in  2.00 seconds = 3727.51 MB/sec
 Timing buffered disk reads:   60 MB in  3.12 seconds =  19.22 MB/sec

After disk replacment:

Code: Select all

----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw
  6   5  58  27   0   4| 158M 2048B|   0     0 |   0     0 |  23k 2660
  1   4  63  28   0   3| 188M   32k|2997k  139M|   0     0 |  23k 2669
  1   5  59  31   0   3| 195M    0 |   0     0 |   0     0 |  23k 2342
  1   5  61  30   0   3| 230M    0 |3052k  136M|   0     0 |  22k 2378
  1   4  55  36   0   3| 192M  120k|   0     0 |   0     0 |  22k 2581
  1   5  56  35   0   3| 235M   16k|2931k  128M|   0     0 |  22k 2329

And disk transfer reads at peak usage:

Code: Select all

Timing cached reads:   11230 MB in  2.00 seconds = 5623.07 MB/sec
 Timing buffered disk reads:  668 MB in  3.14 seconds = 212.54 MB/sec

The problem we are facing now is that our premium users are limited on 1.5MB/s but they can't achieve that speed. The strange thing is that he load of the serves is low less than 1 and IO wait is around 20-30% so we've tried to download file directly from the server via Apache and we've reached download speed around 7-8MB/s. Obviously the server is capable to serve all those clients so then we've decided to increase the download limit for premium users and voila they were able to download with around 1.5MB/s but the limit was set 2MB/s. So currently we've managed to achieve this speed by cheating the download speed but that affected our other servers that are working properly. So my question is how we can solve this problem is it Nginx configuration problem or some kind of server problem.

Thanks