Re: raid6check extremely slow ?

[Piergiorgio Sartor <piergiorgio.sartor@nexgo.de>]

On Sun, May 10, 2020 at 02:07:25PM +0200, Wolfgang Denk wrote:
> Hi,
> 
> I'm running raid6check on a 12 TB (8 x 2 TB harddisks)
> RAID6 array and wonder why it is so extremely slow...
> It seems to be reading the disks only a about 400 kB/s,
> which results in an estimated time of some 57 days!!!
> to complete checking the array.  The system is basically idle, there
> is neither any significant CPU load nor any other I/o (no to the
> tested array, nor to any other storage on this system).
> 
> Am I doing something wrong?
> 
> 
> The command I'm running is simply:
> 
> # raid6check /dev/md0 0 0
> 
> This is with mdadm-4.1 on a Fedora 32 system (mdadm-4.1-4.fc32.x86_64).
> 
> The array data:
> 
> # mdadm --detail /dev/md0
> /dev/md0:
>            Version : 1.2
>      Creation Time : Thu Nov  7 19:30:03 2013
>         Raid Level : raid6
>         Array Size : 11720301024 (11177.35 GiB 12001.59 GB)
>      Used Dev Size : 1953383504 (1862.89 GiB 2000.26 GB)
>       Raid Devices : 8
>      Total Devices : 8
>        Persistence : Superblock is persistent
> 
>        Update Time : Mon May  4 22:12:02 2020
>              State : active
>     Active Devices : 8
>    Working Devices : 8
>     Failed Devices : 0
>      Spare Devices : 0
> 
>             Layout : left-symmetric
>         Chunk Size : 16K
> 
> Consistency Policy : resync
> 
>               Name : atlas.denx.de:0  (local to host atlas.denx.de)
>               UUID : 4df90724:87913791:1700bb31:773735d0
>             Events : 181544
> 
>     Number   Major   Minor   RaidDevice State
>       12       8       64        0      active sync   /dev/sde
>       11       8       80        1      active sync   /dev/sdf
>       13       8      112        2      active sync   /dev/sdh
>        8       8      128        3      active sync   /dev/sdi
>        9       8      144        4      active sync   /dev/sdj
>       10       8      160        5      active sync   /dev/sdk
>       14       8      176        6      active sync   /dev/sdl
>       15       8      192        7      active sync   /dev/sdm
> 
> # iostat /dev/sd[efhijklm]
> Linux 5.6.8-300.fc32.x86_64 (atlas.denx.de)     2020-05-07      _x86_64_        (8 CPU)
> 
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>            0.18    0.01    1.11    0.21    0.00   98.49
> 
> Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
> sde              19.23       388.93         0.09         0.00  158440224      35218          0
> sdf              19.20       388.94         0.09         0.00  158447574      34894          0
> sdh              19.23       388.89         0.08         0.00  158425596      34178          0
> sdi              19.23       388.99         0.09         0.00  158466326      34690          0
> sdj              20.18       388.93         0.09         0.00  158439780      34766          0
> sdk              19.23       388.88         0.09         0.00  158419988      35366          0
> sdl              19.20       388.97         0.08         0.00  158457352      34426          0
> sdm              19.21       388.92         0.08         0.00  158435748      34566          0
> 
> 
> top - 09:08:19 up 4 days, 17:10,  3 users,  load average: 1.00, 1.00, 1.00
> Tasks: 243 total,   1 running, 242 sleeping,   0 stopped,   0 zombie
> %Cpu(s):  0.2 us,  0.5 sy,  0.0 ni, 98.5 id,  0.1 wa,  0.6 hi,  0.1 si,  0.0 st
> MiB Mem :  24034.6 total,  11198.4 free,   1871.8 used,  10964.3 buff/cache
> MiB Swap:   7828.5 total,   7828.5 free,      0.0 used.  21767.6 avail Mem
> 
>     PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
>   19719 root      20   0    2852   2820   2020 D   5.1   0.0 285:40.07 raid6check
>    1123 root      20   0       0      0      0 S   0.3   0.0  25:47.54 md0_raid6
>   37816 root      20   0       0      0      0 I   0.3   0.0   0:00.08 kworker/3:1-events
>   37903 root      20   0  219680   4540   3716 R   0.3   0.0   0:00.02 top
> ...
> 
> 
> HDD in use:
> 
> /dev/sde : ST2000NM0033-9ZM175
> /dev/sdf : ST2000NM0033-9ZM175
> /dev/sdh : ST2000NM0033-9ZM175
> /dev/sdi : ST2000NM0033-9ZM175
> /dev/sdj : ST2000NM0033-9ZM175
> /dev/sdk : ST2000NM0033-9ZM175
> /dev/sdl : ST2000NM0033-9ZM175
> /dev/sdm : ST2000NM0008-2F3100
> 
> 
> 3 days later:
> 
> # iostat /dev/sd[efhijklm]
> Linux 5.6.8-300.fc32.x86_64 (atlas.denx.de)     2020-05-10      _x86_64_        (8 CPU)
> 
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>            0.18    0.00    1.07    0.17    0.00   98.57
> 
> Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
> sde              20.15       370.73         0.10         0.00  253186948      68154          0
> sdf              20.13       370.74         0.10         0.00  253194646      68138          0
> sdh              20.15       370.71         0.10         0.00  253172656      67738          0
> sdi              20.15       370.77         0.10         0.00  253213854      68158          0
> sdj              20.72       370.73         0.10         0.00  253187084      68066          0
> sdk              20.15       370.70         0.10         0.00  253166960      69286          0
> sdl              20.13       370.76         0.10         0.00  253204572      68070          0
> sdm              20.14       370.73         0.10         0.00  253182964      68070          0
> 
> 
> I've tried playing with speed_limit_min/speed_limit_max, but this
> didn't change anything:
> 
> # cat /proc/sys/dev/raid/speed_limit_max
> 2000000
> cat /proc/sys/dev/raid/speed_limit_min
> 10000
> 
> Any ideas welcome!

Difficult to say.

raid6check is CPU bounded, no vector optimization
and no multithread.

Nevertheless, if you see no CPU load (single core
load), then something else is not OK, but I've no
idea what it could be.

Please check if one core is up 100%, if this is
the case, then there is the limit.
If not, sorry, I cannot help.

bye,

-- 

piergiorgio