Re: slow sequential read on partitioned raid6

[Nicolae Mihalache <mache@abcpages.com>]

I created a second 100GB partition on all the disks and then made a
normal /dev/md1 raid6 array out of them, and the results I get:
bacula:~# dd if=/dev/zero of=/mnt1/test-file bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 72.6303 s, 144 MB/s

bacula:~# dd if=/mnt1/test-file of=/dev/null bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 29.1241 s, 360 MB/s

I really believe it's something with the partitioned array.
/proc/devices shows:

Block devices:
  ...
  9 md
  ...
253 mdp


All the md_d1 partitions have major number 253. I don't know if this
means something but maybe there is a bug in the mdp driver (or whatever
is called).

nicolae
Daniel Reurich wrote:
> On Wed, 2010-03-17 at 00:16 +0100, Nicolae Mihalache wrote:
>   
>> On 03/16/2010 11:22 PM, Neil Brown wrote:
>>     
>>> On Tue, 16 Mar 2010 20:05:45 +0100
>>> Nicolae Mihalache <mache@abcpages.com> wrote:
>>>   
>>>       
>>>> Hello,
>>>>
>>>> I have created a partitioned raid6 array over 6x1TB SATA disks using the
>>>> command (from memory): mdadm --create --auto=mdp --level=6
>>>> --raid-devices /dev/md_d1 /dev/sd[b-g].
>>>>
>>>> When I run a sequential read test using
>>>> dd if=/dev/md_d1p1 of=/dev/null bs=1M
>>>> I get low read speeds of around 80MB/s but only when the partition is
>>>> mounted.
>>>>
>>>> If I unmount, the speed is around 350MB/s. The filesystems I tried are
>>>> ext3 and xfs.
>>>>     
>>>>         
>>> Thanks for reporting this.
>>>
>>> I just did some testing and I get the reverse!!
>>>
>>> When a filesystem is mounted I get 135MB/s.  When it isn't mounted
>>> I get 64MB/s.
>>>
>>> I cannot think what could cause this.  I will have to explore.
>>> Can you please double check you results and confirm that it definitely
>>> is  faster then unmounted.
>>>   
>>>       
>> I'm positive that it's slow when mounted, that's how I discovered the
>> problem. See below (I recreated the array over 1/10 of the original
>> disks to be able to test easier).
>> In fact the highest speed I get when accessing directly the entire disk
>> even when one partition is mounted.
>>
>>
>> bacula:~# cat /proc/mdstat
>> Personalities : [raid1] [raid6] [raid5] [raid4]
>> md_d1 : active raid6 sdg1[5] sdf1[4] sde1[3] sdd1[2] sdc1[1] sdb1[0]
>>       390668288 blocks level 6, 64k chunk, algorithm 2 [6/6] [UUUUUU]
>>
>> md2 : active raid1 sdi1[0] sdj1[1]
>>       1462750272 blocks [2/2] [UU]
>>
>> unused devices: <none>
>>
>> bacula:~# parted  /dev/md_d1
>> GNU Parted 1.8.8
>> Using /dev/md_d1
>> Welcome to GNU Parted! Type 'help' to view a list of commands.
>> (parted) print
>> Model: Unknown (unknown)
>> Disk /dev/md_d1: 400GB
>> Sector size (logical/physical): 512B/512B
>> Partition Table: gpt
>>
>> Number  Start   End     Size    File system  Name     Flags
>>  1      17.4kB  50.0GB  50.0GB  ext3         primary
>>
>> (parted) quit
>>
>> bacula:~# umount /dev/md_d1p1
>> umount: /dev/md_d1p1: not mounted
>>
>> bacula:~# dd if=/dev/md_d1p1 of=/dev/null bs=1M count=10000
>> 10000+0 records in
>> 10000+0 records out
>> 10485760000 bytes (10 GB) copied, 37.4938 s, 280 MB/s
>>
>> bacula:~# mount /dev/md_d1p1 /mnt
>>
>> bacula:~# dd if=/dev/md_d1p1 of=/dev/null bs=1M count=10000
>> 10000+0 records in
>> 10000+0 records out
>> 10485760000 bytes (10 GB) copied, 132.894 s, 78.9 MB/s
>>
>> bacula:~# dd if=/dev/md_d1 of=/dev/null bs=1M count=10000
>> 10000+0 records in
>> 10000+0 records out
>> 10485760000 bytes (10 GB) copied, 28.222 s, 372 MB/s
>>     
>
> Why are you trying directly from the block devices when they contain a
> mounted filesystem?  Surely the fs layer would be holding a locks on the
> block device causing it to slow down raw layer access.
>
> Might I suggest you should be reading files that are located within the
> mounted filesystem.
>
> I suggest you try this in the mounted filesystem:
>
> dd if=/dev/zero of=/mnt/test-file bs=1M count=10000
> dd if=/mnt/test-file of=/dev/null bs=1M
> rm /mnt/test-file
>
> I hope this helps.
>