checking md device parity (forced resync) - is it necessary?

checking md device parity (forced resync) - is it necessary?

[Tomasz Chmielewski]

Lately I installed Debian on a Thecus n4100 machine.
It's a 600 MHz ARM storage device, and has 4 x 400 GB drives.

I made Linux software RAID on these drives:
- RAID-1  - ~1 GB for system (/)
- RAID-1  - ~1 GB for swap
- RAID-10 - ~798 GB for iSCSI storage


I noticed that each day the device slows down; a quick investigation 
discovered that Debian runs a "checkarray" script each night at 1 am 
(via cron). The essence of "checkarray" script is basically this:

echo check > /sys/block/$dev/md/sync_action

Which starts a resync of drives. As one can imagine, resync of 800 GB on 
a rather slow device (600 MHz ARM) can take 12 hours or so...


So my question is: is this "daily forced resync" necessary?

Perhaps in some cases, yes, because someone wrote that tool which does 
it daily.

On the other hand, if we consider Linux software RAID stable, such a 
resync would be only needed in some rare situations.

When can one need to run a "daily forced resync", and in which 
circumstances?


-- 
Tomasz Chmielewski

Re: checking md device parity (forced resync) - is it necessary?

[Neil Brown]

On Tuesday September 5, mangoo@wpkg.org wrote:
> Lately I installed Debian on a Thecus n4100 machine.
> It's a 600 MHz ARM storage device, and has 4 x 400 GB drives.
> 
> I made Linux software RAID on these drives:
> - RAID-1  - ~1 GB for system (/)
> - RAID-1  - ~1 GB for swap
> - RAID-10 - ~798 GB for iSCSI storage
> 
> 
> I noticed that each day the device slows down; a quick investigation 
> discovered that Debian runs a "checkarray" script each night at 1 am 
> (via cron). The essence of "checkarray" script is basically this:
> 
> echo check > /sys/block/$dev/md/sync_action
> 
> Which starts a resync of drives. As one can imagine, resync of 800 GB on 
> a rather slow device (600 MHz ARM) can take 12 hours or so...
> 

I believe that was intended to be once a month, not once a day.
Slight error in crontab.

> 
> So my question is: is this "daily forced resync" necessary?

Daily is probably excessive, certainly on an array that size.

Monthly is good.  Weekly might be justified on cheap (i.e. unreliable)
drives and very critical data.

With RAID, sleeping bad blocks can be bad.  If you hit one while
recovering a failed drive, you have to put the piece back together by
hand.
A regular check can wake up those sleeping bad blocks.

> 
> Perhaps in some cases, yes, because someone wrote that tool which does 
> it daily.
> 
> On the other hand, if we consider Linux software RAID stable, such a 
> resync would be only needed in some rare situations.

It has little to do with the stability of Linux software RAID and a
lot to do with stability of modern disk drives.

> 
> When can one need to run a "daily forced resync", and in which 
> circumstances?

As I said, I think the 'daily' is an error.  What exactly do you have
in crontab??

NeilBrown

Re: checking md device parity (forced resync) - is it necessary?

[Tomasz Chmielewski]

Neil Brown wrote:

(...)

>> Which starts a resync of drives. As one can imagine, resync of 800 GB on 
>> a rather slow device (600 MHz ARM) can take 12 hours or so...
>>
> 
> I believe that was intended to be once a month, not once a day.
> Slight error in crontab.
> 
>> So my question is: is this "daily forced resync" necessary?
> 
> Daily is probably excessive, certainly on an array that size.
> 
> Monthly is good.  Weekly might be justified on cheap (i.e. unreliable)
> drives and very critical data.
> 
> With RAID, sleeping bad blocks can be bad.  If you hit one while
> recovering a failed drive, you have to put the piece back together by
> hand.
> A regular check can wake up those sleeping bad blocks.

Thanks a lot for clarification.


>> When can one need to run a "daily forced resync", and in which 
>> circumstances?
> 
> As I said, I think the 'daily' is an error.  What exactly do you have
> in crontab??

Indeed, the crontab entry is wrong:

# by default, run at 01:06 on the first Sunday of each month.
6 1 1-7 * 7 root [ -x /usr/share/mdadm/checkarray ] && 
/usr/share/mdadm/checkarray --cron --all --quiet


However, it will run at 01:06, on 1st-7th day of each month, and on 
Sundays (Debian etch).


-- 
Tomasz Chmielewski
http://wpkg.org

Re: checking md device parity (forced resync) - is it necessary?

[Luca Berra]

On Tue, Sep 05, 2006 at 02:00:03PM +0200, Tomasz Chmielewski wrote:
># by default, run at 01:06 on the first Sunday of each month.
>6 1 1-7 * 7 root [ -x /usr/share/mdadm/checkarray ] && 
>/usr/share/mdadm/checkarray --cron --all --quiet
>
>However, it will run at 01:06, on 1st-7th day of each month, and on 
>Sundays (Debian etch).
hihihi
monthday and weekday are or-ed in crontab 

L.

-- 
Luca Berra -- bluca@comedia.it
        Communication Media & Services S.r.l.
 /"\
 \ /     ASCII RIBBON CAMPAIGN
  X        AGAINST HTML MAIL
 / \

Re: checking md device parity (forced resync) - is it necessary?

[Mario 'BitKoenig' Holbe]

Tomasz Chmielewski <mangoo@wpkg.org> wrote:
> # by default, run at 01:06 on the first Sunday of each month.
> 6 1 1-7 * 7 root [ -x /usr/share/mdadm/checkarray ] && 

You have a quite old version of mdadm. This issue has been fixed in
mdadm (2.5.2-8) unstable from 27 Jul 2006. Current version is mdadm
(2.5.3.git200608202239-2) unstable from 23 Aug 2006.

Since the feature did just appear in mdadm (2.5.2-3) UNRELEASED from
7 Jul 2006 (and even just went to unstable with mdadm (2.5.2-7) unstable
from 20 Jul 2006), it seems you were simply in quite bad luck :)


regards
   Mario
-- 
It is practically impossible to teach good programming style to students
that have had prior exposure to BASIC: as potential programmers they are
mentally mutilated beyond hope of regeneration.  -- Dijkstra

Re: checking md device parity (forced resync) - is it necessary?

[Tomasz Chmielewski]

Mario 'BitKoenig' Holbe wrote:
> Tomasz Chmielewski <mangoo@wpkg.org> wrote:
>> # by default, run at 01:06 on the first Sunday of each month.
>> 6 1 1-7 * 7 root [ -x /usr/share/mdadm/checkarray ] && 
> 
> You have a quite old version of mdadm. This issue has been fixed in
> mdadm (2.5.2-8) unstable from 27 Jul 2006. Current version is mdadm
> (2.5.3.git200608202239-2) unstable from 23 Aug 2006.
> 
> Since the feature did just appear in mdadm (2.5.2-3) UNRELEASED from
> 7 Jul 2006 (and even just went to unstable with mdadm (2.5.2-7) unstable
> from 20 Jul 2006), it seems you were simply in quite bad luck :)

Yes, I saw somewhere in the changelogs that the issue is fixed.

Still, I can't find 2.5.2-8 anywhere on the mirrors, perhaps it's just a 
matter of time...


-- 
Tomasz Chmielewski
http://wpkg.org

Re: checking md device parity (forced resync) - is it necessary?

[John Stoffel]

Tomasz> Lately I installed Debian on a Thecus n4100 machine.  It's a
Tomasz> 600 MHz ARM storage device, and has 4 x 400 GB drives.

Interesting box... how quiet is it?  I'm thinking of one of these for
home use, but I'll probably go with an EPIA box so I can actually
setup a NFS/CiFS/backup/http/mysql server...

I've looked at the Irfant and the Buffalo Logic ones as well... all
tempting.  But backups are the killer.  :]

Thanks,
John

Re: checking md device parity (forced resync) - is it necessary?

[Tomasz Chmielewski]

John Stoffel wrote:
> Tomasz> Lately I installed Debian on a Thecus n4100 machine.  It's a
> Tomasz> 600 MHz ARM storage device, and has 4 x 400 GB drives.
> 
> Interesting box... how quiet is it?  

I can't tell you if it's quiet or not, it's in the server room right 
now, and serves as a backup and iSCSI storage for Xen.

But as I still configured it and it was on my desk, it seemed a bit 
louder than a normal PC.
After all, it has 4 drives (which you have to buy separately).


> I'm thinking of one of these for
> home use, but I'll probably go with an EPIA box so I can actually
> setup a NFS/CiFS/backup/http/mysql server...

Although it has two gigabit network cards, you can't fully use them - 
the IXP3xx ARM CPU (600 MHz) allows you to only send about 25 MB/s of 
packets (measured on a loopback interface, so iSCSI LAN speeds are a bit 
lower).


> I've looked at the Irfant and the Buffalo Logic ones as well... all
> tempting.  But backups are the killer.  :]

There's also Thecus n5200, 800 MHz mobile Celeron, which you can supply 
with 5 drives... :)


-- 
Tomasz Chmielewski
http://wpkg.org