md raid6: slow resync if fs is mounted rw

md raid6: slow resync if fs is mounted rw

[Jakob Unterwurzacher]

Dear list,

I am seeing a strange performance issue on a 4 x 4TB md raid6 on kernel
3.12. Initial resync goes nice and fast at 130MB/s:

> # cat /proc/mdstat
> Personalities : [raid1] [raid6] [raid5] [raid4]
> md1 : active raid6 sda3[0] sdd3[3] sdc3[2] sdb3[1]
>       7751269376 blocks super 1.2 level 6, 512k chunk, algorithm 2 [4/4] [UUUU]
>       [=>...................]  resync =  7.5% (293925504/3875634688) finish=458.4min speed=130210K/sec
> 
> md0 : active raid1 sda2[0] sdd2[3] sdc2[2] sdb2[1]
>       29280128 blocks super 1.2 [4/4] [UUUU]
> 
> unused devices: <none>

...unless the ext4 filesystem on the array is mounted read-write!
Then, speed plummets to 1 - 7MB/s. The filesystem is empty and totally
idle; a simple "mount -o remount,ro" restores the speed to 130MB/s.

I dumped the /sys/block/md1/md tree with the filesystem mounted ro and
rw, the diff looks like this:

> # diff -u md.rw md.ro
> [...]
> --- md.rw/stripe_cache_active   2014-02-25 17:33:31.250019324 +0100
> +++ md.ro/stripe_cache_active   2014-02-25 17:33:31.250019324 +0100
> @@ -1 +1 @@
> -0
> +256
> diff -u md.rw/sync_speed md.ro/sync_speed
> --- md.rw/sync_speed    2014-02-25 17:33:31.250019324 +0100
> +++ md.ro/sync_speed    2014-02-25 17:33:31.250019324 +0100
> @@ -1 +1 @@
> -1052
> +126244

So, stripe_cache_active is zero if the fs is mounted rw. Hmm.

Interestingly, if I increase sync_speed_min to 100MB/s, I do get 100MB/s.

Any idea what is going on?

Best regards,
Jakob Unterwurzacher

Re: md raid6: slow resync if fs is mounted rw

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 2047 bytes --]

On Tue, 25 Feb 2014 23:55:15 +0100 Jakob Unterwurzacher <jakobunt@gmail.com>
wrote:

> Dear list,
> 
> I am seeing a strange performance issue on a 4 x 4TB md raid6 on kernel
> 3.12. Initial resync goes nice and fast at 130MB/s:
> 
> > # cat /proc/mdstat
> > Personalities : [raid1] [raid6] [raid5] [raid4]
> > md1 : active raid6 sda3[0] sdd3[3] sdc3[2] sdb3[1]
> >       7751269376 blocks super 1.2 level 6, 512k chunk, algorithm 2 [4/4] [UUUU]
> >       [=>...................]  resync =  7.5% (293925504/3875634688) finish=458.4min speed=130210K/sec
> > 
> > md0 : active raid1 sda2[0] sdd2[3] sdc2[2] sdb2[1]
> >       29280128 blocks super 1.2 [4/4] [UUUU]
> > 
> > unused devices: <none>
> 
> ...unless the ext4 filesystem on the array is mounted read-write!
> Then, speed plummets to 1 - 7MB/s. The filesystem is empty and totally
> idle; a simple "mount -o remount,ro" restores the speed to 130MB/s.
> 
> I dumped the /sys/block/md1/md tree with the filesystem mounted ro and
> rw, the diff looks like this:
> 
> > # diff -u md.rw md.ro
> > [...]
> > --- md.rw/stripe_cache_active   2014-02-25 17:33:31.250019324 +0100
> > +++ md.ro/stripe_cache_active   2014-02-25 17:33:31.250019324 +0100
> > @@ -1 +1 @@
> > -0
> > +256
> > diff -u md.rw/sync_speed md.ro/sync_speed
> > --- md.rw/sync_speed    2014-02-25 17:33:31.250019324 +0100
> > +++ md.ro/sync_speed    2014-02-25 17:33:31.250019324 +0100
> > @@ -1 +1 @@
> > -1052
> > +126244
> 
> So, stripe_cache_active is zero if the fs is mounted rw. Hmm.
> 
> Interestingly, if I increase sync_speed_min to 100MB/s, I do get 100MB/s.
> 
> Any idea what is going on?
> 

Simplest explanation would be that ext4 is writing to the array
occasionally.  Maybe 1 block per second.  That would be enough for md to
decide that the array is busy and to back-off the resync.

Try
  blktrace -w 20 /dev/md1

while the filesystem is mounted read-write, then
  blkparse md1.blktrace*

do we if anything writes are going to the array.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: md raid6: slow resync if fs is mounted rw

[Brad Campbell]

On 26/02/14 06:55, Jakob Unterwurzacher wrote:
> Dear list,
>
> I am seeing a strange performance issue on a 4 x 4TB md raid6 on kernel
> 3.12. Initial resync goes nice and fast at 130MB/s:
>
>> # cat /proc/mdstat
>> Personalities : [raid1] [raid6] [raid5] [raid4]
>> md1 : active raid6 sda3[0] sdd3[3] sdc3[2] sdb3[1]
>>        7751269376 blocks super 1.2 level 6, 512k chunk, algorithm 2 [4/4] [UUUU]
>>        [=>...................]  resync =  7.5% (293925504/3875634688) finish=458.4min speed=130210K/sec
>>
>> md0 : active raid1 sda2[0] sdd2[3] sdc2[2] sdb2[1]
>>        29280128 blocks super 1.2 [4/4] [UUUU]
>>
>> unused devices: <none>
>
> ...unless the ext4 filesystem on the array is mounted read-write!
> Then, speed plummets to 1 - 7MB/s. The filesystem is empty and totally
> idle; a simple "mount -o remount,ro" restores the speed to 130MB/s.
>

You are probably expeririencing an effect of this :

http://www.thomas-krenn.com/en/wiki/Ext4_Filesystem#Lazy_Initialization

Regards,
Brad

Re: md raid6: slow resync if fs is mounted rw

[Jakob Unterwurzacher]

On Wed, Feb 26, 2014 at 3:31 AM, Brad Campbell
<lists2009@fnarfbargle.com> wrote:
> You are probably expeririencing an effect of this :
>
> http://www.thomas-krenn.com/en/wiki/Ext4_Filesystem#Lazy_Initialization

That's it. ext4lazyinit shows up in the blktrace and also flashes up
every now and then in iotop - I did not notice it before.
I reformatted with "mkfs.ext4 -E
lazy_itable_init=0,lazy_journal_init=0" and the issue is gone. Of
course, just waiting a bit with the filesystem mounted rw would have
done it, too.

Thanks a lot Neil and Brad!
Jakob