async/X threads and rebuild slowdown

async/X threads and rebuild slowdown

[Jeremy Sanders]

Hi - It seems our MD raid rebuild speeds are very slow with the new Fedora 
2.6.32.9-70 kernel. We're only getting ~13MB/s:

[root@xback2 ~]# cat /proc/mdstat 
Personalities : [raid6] [raid5] [raid4] 
md0 : active raid5 sdb1[0] sdj1[10](S) sdk1[9] sdl1[8] sdi1[7] sdh1[6] 
sdg1[5] sdf1[4] sde1[3] sdd1[2] sdc1[1]
      8788959360 blocks level 5, 32k chunk, algorithm 2 [10/10] [UUUUUUUUUU]
      [=====>...............]  resync = 29.3% (286397320/976551040) 
finish=897.2min speed=12819K/sec
      
The maximum rebuild speed is at the default 200000K/s. The drives are 
connected to a 3ware 9650SE-16ML controller on PCI-Express.

There seem to be hundreds of async/X threads, where X goes up to 129 at 
least. The md0_raid5 process is using at least 80% of the CPU on this dual 
core Athlon 7750 system.

Has anything changed recently in the kernel. I don't remember md0_raid5 
being so processor inefficient, slow at rebuilding or creating hundreds of 
kernel threads.

Are these async/X threads documented anywhere?

Thanks

Jeremy

Re: async/X threads and rebuild slowdown

[Tao Guo]

I also meet this problem in several of my machines. If I change back
to kernel 2.6.18, it is
OK, but in 2.6.32 kernel, the performance is really poor.

At last I found a solution, set CONFIG_MULTICORE_RAID456=n when
compiling raid456 module,
then everything goes normal. My machines do have multicore cpu, so it
is weird :<
Maybe there is something wrong with the CONFIG_MULTICORE_RAID456 enabled code?

On Tue, Mar 16, 2010 at 8:41 PM, Jeremy Sanders
<jeremy@jeremysanders.net> wrote:
> Hi - It seems our MD raid rebuild speeds are very slow with the new Fedora
> 2.6.32.9-70 kernel. We're only getting ~13MB/s:
>
> [root@xback2 ~]# cat /proc/mdstat
> Personalities : [raid6] [raid5] [raid4]
> md0 : active raid5 sdb1[0] sdj1[10](S) sdk1[9] sdl1[8] sdi1[7] sdh1[6]
> sdg1[5] sdf1[4] sde1[3] sdd1[2] sdc1[1]
>      8788959360 blocks level 5, 32k chunk, algorithm 2 [10/10] [UUUUUUUUUU]
>      [=====>...............]  resync = 29.3% (286397320/976551040)
> finish=897.2min speed=12819K/sec
>
> The maximum rebuild speed is at the default 200000K/s. The drives are
> connected to a 3ware 9650SE-16ML controller on PCI-Express.
>
> There seem to be hundreds of async/X threads, where X goes up to 129 at
> least. The md0_raid5 process is using at least 80% of the CPU on this dual
> core Athlon 7750 system.
>
> Has anything changed recently in the kernel. I don't remember md0_raid5
> being so processor inefficient, slow at rebuilding or creating hundreds of
> kernel threads.
>
> Are these async/X threads documented anywhere?
>
> Thanks
>
> Jeremy
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
tao.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: async/X threads and rebuild slowdown

[John Robinson]

On 16/03/2010 13:24, Tao Guo wrote:
> I also meet this problem in several of my machines. If I change back
> to kernel 2.6.18, it is
> OK, but in 2.6.32 kernel, the performance is really poor.
> 
> At last I found a solution, set CONFIG_MULTICORE_RAID456=n when
> compiling raid456 module,
> then everything goes normal. My machines do have multicore cpu, so it
> is weird :<
> Maybe there is something wrong with the CONFIG_MULTICORE_RAID456 enabled code?

I remember it being discussed on this list, so yes I think there is or was.

Cheers,

John.

Re: async/X threads and rebuild slowdown

[Dan Williams]

On Tue, Mar 16, 2010 at 6:49 AM, John Robinson
<john.robinson@anonymous.org.uk> wrote:
> On 16/03/2010 13:24, Tao Guo wrote:
>>
>> I also meet this problem in several of my machines. If I change back
>> to kernel 2.6.18, it is
>> OK, but in 2.6.32 kernel, the performance is really poor.
>>
>> At last I found a solution, set CONFIG_MULTICORE_RAID456=n when
>> compiling raid456 module,
>> then everything goes normal. My machines do have multicore cpu, so it
>> is weird :<
>> Maybe there is something wrong with the CONFIG_MULTICORE_RAID456 enabled
>> code?
>
> I remember it being discussed on this list, so yes I think there is or was.
>

fyi,

https://bugzilla.redhat.com/show_bug.cgi?id=575897

--
Dan