Re: Where is the performance bottleneck?

[Michael Tokarev <mjt@tls.msk.ru>]

Holger Kiehl wrote:
> On Wed, 31 Aug 2005, Jens Axboe wrote:
> 
>> On Wed, Aug 31 2005, Holger Kiehl wrote:
>>
[]
>>> I used the following command reading from all 8 disks in parallel:
>>>
>>>    dd if=/dev/sd?1 of=/dev/null bs=256k count=78125
>>>
>>> Here vmstat output (I just cut something out in the middle):
>>>
>>> procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
>>>  r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
>>>  3  7   4348  42640 7799984   9612    0    0 322816     0 3532  4987 0 22 0 78
>>>  1  7   4348  42136 7800624   9584    0    0 322176     0 3526  4987 0 23 4 74
>>>  0  8   4348  39912 7802648   9668    0    0 322176     0 3525  4955 0 22 12 66
>>
>> Ok, so that's somewhat better than the writes but still off from what
>> the individual drives can do in total.
>>
>>>> You might want to try the same with direct io, just to eliminate the
>>>> costly user copy. I don't expect it to make much of a difference though,
>>>> feels like the problem is elsewhere (driver, most likely).
>>>>
>>> Sorry, I don't know how to do this. Do you mean using a C program
>>> that sets some flag to do direct io, or how can I do that?
>>
>> I've attached a little sample for you, just run ala
>>
>> # ./oread /dev/sdX
>>
>> and it will read 128k chunks direct from that device. Run on the same
>> drives as above, reply with the vmstat info again.
>>
> Using kernel 2.6.12.5 again, here the results:
> 
> procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
>  r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
>  0  8      0 8005380   4816  41084    0    0 318848    32 3511  4672  0 1 75 24
>  0  8      0 8005380   4816  41084    0    0 320640     0 3512  4877  0 2 75 23
>  0  8      0 8005380   4816  41084    0    0 322944     0 3533  5047  0 2 75 24
>  0  8      0 8005380   4816  41084    0    0 322816     0 3531  5053  0 1 75 24
>  0  8      0 8005380   4816  41084    0    0 322944     0 3531  5048  0 2 75 23
>  0  8      0 8005380   4816  41084    0    0 322944     0 3529  5043  0 1 75 24
>  0  0      0 8008360   4816  41084    0    0 266880     0 3112  4224  0 2 78 20

I went on and did similar tests on our box, which is:

 dual Xeon 2.44GHz with HT (so it's like 4 logical CPUs)
 dual-channel AIC-7902 U320 controller
 8 SEAGATE ST336607LW drives attached to the 2 channels of the
  controller, sd[abcd] to channel 0 and sd[efgh] to channel 1

Each drive is capable to get about 60 megabytes/sec.
The kernel is 2.6.13 from kernel.org.

With direct-reading:
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  8     12     87    471   1839    0    0 455296    84 1936  3739  0  3 47 50
 1  7     12     87    471   1839    0    0 456704    80 1941  3744  0  4 48 48
 1  7     12     87    471   1839    0    0 446464    82 1914  3648  0  2 48 50
 0  8     12     87    471   1839    0    0 454016    94 1944  3765  0  2 47 50
 0  8     12     87    471   1839    0    0 458752    60 1944  3746  0  2 48 50

Without direct:
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 8  6     12     80    470   1839    0    0 359966   124 1726  2270  1 89  0 10
 2  7     12     80    470   1839    0    0 352813   113 1741  2124  1 88  1 10
 8  4     12     80    471   1839    0    0 358990    34 1669  1934  1 94  0  5
 7  5     12     79    471   1839    0    0 354065   157 1761  2128  1 90  1  8
 6  5     12     80    471   1839    0    0 358062    44 1686  1911  1 93  0  6

So the difference direct vs "indirect" is quite.. significant.  And with
direct-reading, all 8 drives are up to their real speed.  Note the CPU usage
in case of "indirect" reading too - it's about 90%...

And here's an idle system as well:
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0     12     89    471   1839    0    0     0    58  151   358  0  0 100  0
 0  0     12     89    471   1839    0    0     0    66  157   167  0  0 99  0

Too bad I can't perform write tests on this system...

/mjt