We own couple of file storage server and recently we've experienced low disk performance on some of them. We investigated and discovered that the problem was into the disks so we replaced them and now disks are preforming very well. Here are some stats about disk performance:
Before disk replacment:
Code: Select all
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ| recv send| in out | int csw
1 2 89 8 0 1| 20M 714k| 0 0 | 0 0 |2696 1936
0 2 41 55 0 2| 30M 0 | 0 0 | 0 0 | 16k 1136
1 2 36 59 0 2| 41M 0 |2136k 87M| 0 0 | 19k 1584
0 2 61 34 0 2| 37M 0 | 0 0 | 0 0 | 20k 1511
0 2 60 35 0 2| 37M 0 |2382k 103M| 0 0 | 19k 1675
1 2 57 39 0 1| 33M 2080k| 0 0 | 0 0 | 19k 1335
0 2 50 46 0 2| 39M 0 |2374k 100M| 0 0 | 20k 1375
And disk transfer reads at peak usage:
Code: Select all
Timing cached reads: 7448 MB in 2.00 seconds = 3727.51 MB/sec
Timing buffered disk reads: 60 MB in 3.12 seconds = 19.22 MB/sec
After disk replacment:
Code: Select all
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ| recv send| in out | int csw
6 5 58 27 0 4| 158M 2048B| 0 0 | 0 0 | 23k 2660
1 4 63 28 0 3| 188M 32k|2997k 139M| 0 0 | 23k 2669
1 5 59 31 0 3| 195M 0 | 0 0 | 0 0 | 23k 2342
1 5 61 30 0 3| 230M 0 |3052k 136M| 0 0 | 22k 2378
1 4 55 36 0 3| 192M 120k| 0 0 | 0 0 | 22k 2581
1 5 56 35 0 3| 235M 16k|2931k 128M| 0 0 | 22k 2329
And disk transfer reads at peak usage:
Code: Select all
Timing cached reads: 11230 MB in 2.00 seconds = 5623.07 MB/sec
Timing buffered disk reads: 668 MB in 3.14 seconds = 212.54 MB/sec
The problem we are facing now is that our premium users are limited on 1.5MB/s but they can't achieve that speed. The strange thing is that he load of the serves is low less than 1 and IO wait is around 20-30% so we've tried to download file directly from the server via Apache and we've reached download speed around 7-8MB/s. Obviously the server is capable to serve all those clients so then we've decided to increase the download limit for premium users and voila they were able to download with around 1.5MB/s but the limit was set 2MB/s. So currently we've managed to achieve this speed by cheating the download speed but that affected our other servers that are working properly. So my question is how we can solve this problem is it Nginx configuration problem or some kind of server problem.
Thanks