RAID resync stalled at 99.7% ?

RAID resync stalled at 99.7% ?

[Christopher Smith]

In doing some benchmarking, I've found a curious problem - after
creating an array the resync has stalled at 99.7%:

[root@justinstalled ~]# cat /proc/mdstat
Personalities : [raid6]
md0 : active raid6 sdm1[11] sdl1[10] sdk1[9] sdj1[8] sdi1[7] sdh1[6]
sdg1[5] sdf1[4] sde1[3] sdd1[2] sdc1[1] sdb1[0]
      4963200 blocks level 6, 32k chunk, algorithm 2 [12/12]
[UUUUUUUUUUUU]
      [===================>.]  resync = 99.7% (496320/496349)
finish=0.0min speed=628K/sec
unused devices: <none>
[root@justinstalled ~]#                                     

It's been sitting like this for some time now, and since the resync up
until this point progress at about 15M/sec, I can't see any reason to
think it will suddenly finish.

mdadm -S /dev/md0 simply hangs.

This problem is reproducible as well - if I reboot the machine the
resync will complete successfully, then if I delete it and try to create
another array, exactly the same thing will happen.

It's also not a problem with, for example, bad sectors on one of the
components, as creating a larger array stalls right near the end as well
(the exact percentage varies, but it's always around the 99% part).

Does anyone have any ideas ?



Some relevant info:

The command used to create the aray was:

mdadm -C /dev/md0 -l6 -n12 -c 32 -z 496349 /dev/sd[b-m]1

It's a Fedora Core 4 box:

[root@justinstalled ~]# uname -a
Linux justinstalled.syd.nighthawkrad.net 2.6.12-1.1398_FC4smp #1 SMP Fri
Jul 15 01:30:13 EDT 2005 i686 i686 i386 GNU/Linux
[root@justinstalled
~]#                                                             

The components are 12x400GB drives attached to a 3ware 9500s-12
controller.  They are configured as "single disks" on the controller,
ie: no hardware RAID is involved.

Regards,
Chris Smith

Re: RAID resync stalled at 99.7% ?

[Neil Brown]

On Friday September 2, csmith@nighthawkrad.net wrote:
> In doing some benchmarking, I've found a curious problem - after
> creating an array the resync has stalled at 99.7%:
> 
> [root@justinstalled ~]# cat /proc/mdstat
> Personalities : [raid6]
> md0 : active raid6 sdm1[11] sdl1[10] sdk1[9] sdj1[8] sdi1[7] sdh1[6]
> sdg1[5] sdf1[4] sde1[3] sdd1[2] sdc1[1] sdb1[0]
>       4963200 blocks level 6, 32k chunk, algorithm 2 [12/12]
> [UUUUUUUUUUUU]
>       [===================>.]  resync = 99.7% (496320/496349)
> finish=0.0min speed=628K/sec
> unused devices: <none>

$ bc -l 
bc 1.06
Copyright 1991-1994, 1997, 1998, 2000 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'. 
496349/32
15510.90625000000000000000
15510*32
496320


Get the picture?

I suspect you are using mdadm-2.0 - yes?  This is a bug triggered by
they way mdadm 2.0 creates arrays (differently to 1.x).  It is fixed
in 2.6.13.

> The command used to create the aray was:
> 
> mdadm -C /dev/md0 -l6 -n12 -c 32 -z 496349 /dev/sd[b-m]1
> 

If you replace 496349 with 496320, it should work fine.

NeilBrown

3ware RAID (was Re: RAID resync stalled at 99.7% ?)

[Daniel Pittman]

Christopher Smith <csmith@nighthawkrad.net> writes:

[...]

> The components are 12x400GB drives attached to a 3ware 9500s-12
> controller.  They are configured as "single disks" on the controller,
> ie: no hardware RAID is involved.

A quick question for you, because I have a client looking at 3ware RAID
hardware at the moment:

Why are you running this as software RAID, rather than using the
hardware on the 3ware card?

From everything I have heard, 3ware hardware RAID doesn't suck, so if
you have a good reason to avoid it I would love to hear it (and possibly
save my client trouble and/or money...)

     Daniel

Re: 3ware RAID (was Re: RAID resync stalled at 99.7% ?)

[Christopher Smith]

Daniel Pittman wrote:
> Christopher Smith <csmith@nighthawkrad.net> writes:
> 
> [...]
> 
> 
>>The components are 12x400GB drives attached to a 3ware 9500s-12
>>controller.  They are configured as "single disks" on the controller,
>>ie: no hardware RAID is involved.
> 
> 
> A quick question for you, because I have a client looking at 3ware RAID
> hardware at the moment:
> 
> Why are you running this as software RAID, rather than using the
> hardware on the 3ware card?

Because after doing some preliminary benchmarks, I've found Linux's 
software RAID to be significantly faster than 3ware's hardware RAID (at 
the sacrifice of higher CPU usage, but since the machine has a fairly 
fast CPU and doesn't do anything else, that's a sacrifice I'm happy to 
make).

I have some iozone and bonnie++ results, but they're at work and I'm at 
home - I'll post them tomorrow.

CS

Re: 3ware RAID (was Re: RAID resync stalled at 99.7% ?)

[Brad Dameron]

On Fri, 2005-09-02 at 20:38 +1000, Daniel Pittman wrote:
> Christopher Smith <csmith@nighthawkrad.net> writes:
> 
> [...]
> 
> > The components are 12x400GB drives attached to a 3ware 9500s-12
> > controller.  They are configured as "single disks" on the controller,
> > ie: no hardware RAID is involved.
> 
> A quick question for you, because I have a client looking at 3ware RAID
> hardware at the moment:

I would also look at the Areca line of cards. They are much faster than
the 3ware and support up to 24 port if needed. Also do RAID6 now.

http://www.areca.com.tw/index/html/index.htm


Brad Dameron
SeaTab Software
www.seatab.com

Re: 3ware RAID (was Re: RAID resync stalled at 99.7% ?)

[berk walker]

Brad Dameron wrote:
>On Fri, 2005-09-02 at 20:38 +1000, Daniel Pittman wrote:
>  
>>Christopher Smith <csmith@nighthawkrad.net> writes:
>>
>>[...]
>>
>>    
>>>The components are 12x400GB drives attached to a 3ware 9500s-12
>>>controller.  They are configured as "single disks" on the controller,
>>>ie: no hardware RAID is involved.
>>>      
>>A quick question for you, because I have a client looking at 3ware RAID
>>hardware at the moment:
>>    
>
>I would also look at the Areca line of cards. They are much faster than
>the 3ware and support up to 24 port if needed. Also do RAID6 now.
>
>http://www.areca.com.tw/index/html/index.htm
>
>
>Brad Dameron
>SeaTab Software
>www.seatab.com
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>  
I guess if we were all wholesalers with a nice long lead time, that
would be great, Brad.  But where, and for how much might one purchase these?

b-

Re: 3ware RAID (was Re: RAID resync stalled at 99.7% ?)

[Brad Dameron]

On Thu, 2005-09-01 at 13:50 -0400, berk walker wrote:
> I guess if we were all wholesalers with a nice long lead time, that
> would be great, Brad.  But where, and for how much might one purchase these?
> 
> b-
> 

http://www.topmicrousa.com/controllers--tekram.html
http://www.rackmountpro.com/productsearch.php?catid=199
http://www.pc-pitstop.com/sata_raid_controllers/

Just for starters. Google found those by the way. And yes they are a
little more than 3ware. But I can say they do twice the performance of
the 3ware cards. Mainly due to their 800Mhz processor and faster memory.
Their driver support is also more upt to date.

Brad Dameron
SeaTab Software
www.seatab.com

Re: 3ware RAID (was Re: RAID resync stalled at 99.7% ?)

[Ming Zhang]

On Fri, 2005-09-02 at 11:09 -0700, Brad Dameron wrote:
> On Thu, 2005-09-01 at 13:50 -0400, berk walker wrote:
> > I guess if we were all wholesalers with a nice long lead time, that
> > would be great, Brad.  But where, and for how much might one purchase these?
> > 
> > b-
> > 
> 
> http://www.topmicrousa.com/controllers--tekram.html
> http://www.rackmountpro.com/productsearch.php?catid=199
> http://www.pc-pitstop.com/sata_raid_controllers/
> 
> Just for starters. Google found those by the way. And yes they are a
> little more than 3ware. But I can say they do twice the performance of
> the 3ware cards. Mainly due to their 800Mhz processor and faster memory.
> Their driver support is also more upt to date.
> 

when u talk about 2 x performance, do u have any performance data to
back u claim?



> Brad Dameron
> SeaTab Software
> www.seatab.com
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 3ware RAID (was Re: RAID resync stalled at 99.7% ?)

[Joshua Baker-LePain]

On Fri, 2 Sep 2005 at 11:09am, Brad Dameron wrote

> On Thu, 2005-09-01 at 13:50 -0400, berk walker wrote:
> > I guess if we were all wholesalers with a nice long lead time, that
> > would be great, Brad.  But where, and for how much might one purchase these?
> > 
> > b-
> > 
> 
> http://www.topmicrousa.com/controllers--tekram.html
> http://www.rackmountpro.com/productsearch.php?catid=199
> http://www.pc-pitstop.com/sata_raid_controllers/
> 
> Just for starters. Google found those by the way. And yes they are a
> little more than 3ware. But I can say they do twice the performance of
> the 3ware cards. Mainly due to their 800Mhz processor and faster memory.
> Their driver support is also more upt to date.

But has it made it into the mainline kernel yet?  3w-xxxx has been in 
mainline for a *long* time.  3w-9xxx is still settling a bit, that's true, 
but it's doing so in mainline.  Personally, I wouldn't put important data 
on an Areca controller until the drivers have been through wider testing 
than the -mm series (where I believe they're currently percolating).

-- 
Joshua Baker-LePain
Department of Biomedical Engineering
Duke University