Re: Raid-5 long write wait while reading

[Bill Davidsen <davidsen@tmr.com>]

tj wrote:
> Bill Davidsen wrote:
>> tj wrote:
>>> Thomas Jager wrote:
>>>> Hi list.
>>>>
>>>> I run a file server on MD raid-5.
>>>> If a client reads one big file and at the same time another client 
>>>> tries to write a file, the thread writing just sits in 
>>>> uninterruptible sleep until the reader has finished. Only very 
>>>> small amount of writes get trough while the reader is still working.
>>>> I'm having some trouble pinpointing the problem.
>>>> It's not consistent either sometimes it works as expected both the 
>>>> reader and writer gets some transactions. On huge reads I've seen 
>>>> the writer blocked for 30-40 minutes without any significant writes 
>>>> happening (Maybe a few megabytes, of several gigs waiting). It 
>>>> happens with NFS, SMB and FTP, and local with dd. And seems to be 
>>>> connected to raid-5. This does not happen on block devices without 
>>>> raid-5. I'm also wondering if it can have anything to do with 
>>>> loop-aes? I use loop-aes on top of the md, but then again i have 
>>>> not observed this problem on loop-devices with disk backend. I do 
>>>> know that loop-aes degrades performance but i didn't think it would 
>>>> do something like this?
>>>>
>>>> I've seen this problem in 2.6.16-2.6.21
>>>>
>>>> All disks in the array is connected to a controller with a SiI 3114 
>>>> chip.
>>>
>>> I just noticed something else. A couple of slow readers where 
>>> running on my raid-5 array. Then i started a copy from another local 
>>> disk to the array. Then i got the extremely long wait. I noticed 
>>> something in iostat:
>>>
>>> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>>>           3.90    0.00   48.05   31.93    0.00   16.12
>>>
>>> Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
>>> ....
>>> sdg               0.80        25.55         0.00        128          0
>>> sdh             154.89       632.34         0.00       3168          0
>>> sdi               0.20        12.77         0.00         64          0
>>> sdj               0.40        25.55         0.00        128          0
>>> sdk               0.40        25.55         0.00        128          0
>>> sdl               0.80        25.55         0.00        128          0
>>> sdm               0.80        25.55         0.00        128          0
>>> sdn               0.60        23.95         0.00        120          0
>>> md0             199.20       796.81         0.00       3992          0
>>>
>>> All disks are member of the same raid array (md0). One of the disks 
>>> has a ton of transactions compared to the other disks. Read 
>>> operations as far as i can tell. Why? May be connected with my problem? 
>> Two thoughts on that, if you are doing a lot of directory operations, 
>> it's possible that the inodes being used most are all in one chunk.
> Hi thanks for the reply.
>
> It's not directory operations AFAIK. Reading a few files (3 in this 
> case) and writing one.
>>
>> The other possibility is that these a journal writes and reflect 
>> updates to the atime. The way to see if this is in some way  related 
>> is to mount (remount) with noatime: "mount -o remount,noatime 
>> /dev/md0 /wherever" and retest. If this is journal activity you can 
>> do several things to reduce the problem, which I'll go into (a) if it 
>> seems to be the problem, and (b) if someone else doesn't point you to 
>> an existing document or old post on the topic. Oh, you could also try 
>> mounting the filesystem as etc2, assuming that it's ext3 now. I 
>> wouldn't run that way, but it's useful as a diagnostic tool.
> I don't use ext3 i use ReiserFS. ( It seemed like a good idea at the 
> time. ) It's mounted with  -o  noatime.
> I've done some more testing and i seems like it might be connected to 
> mount --bind. If i write to a binded mount i get the slow writes. But 
> if i write directly to the real mount i don't. It might just be a 
> random occurrence, as the problem always has been inconsistent. Thoughts? 

I don't beat on the bind mounts, let me do a test and get back.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979