Scrub?

Scrub?

[Derek Listmail Acct]

Many of the hardware raid controllers I've used have the ability to run a
'scrub' on an array.  Is there a way to do this on a linux software raid 5
array?

RE: Scrub?

[Salyzyn, Mark]

dd if=/dev/zero of=/dev/md0 bs=200b

-----Original Message-----
From: linux-raid-owner@vger.kernel.org
[mailto:linux-raid-owner@vger.kernel.org] On Behalf Of Derek Listmail
Acct
Sent: Friday, August 06, 2004 2:07 PM
To: linux-raid@vger.kernel.org
Subject: Scrub?

Many of the hardware raid controllers I've used have the ability to run
a
'scrub' on an array.  Is there a way to do this on a linux software raid
5
array?


-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Scrub?

[Kanoa Withington]

Eh, that would delete the contents of the array, including the
filesystem. Is that what you meant by "scrub"? I thought "scrubbing"
meant looking for unreadable blocks and pro-actively replacing them
from parity. The latter would be very useful for a software raid 5
array since there is currently no facility in software raid for doing
this on-the-fly. I imagine it would be possible to write such a
utility in user space. I, too, wonder if anyone out there has
something that works.

-Kanoa

On Fri, 6 Aug 2004, Salyzyn, Mark wrote:

> dd if=/dev/zero of=/dev/md0 bs=200b
>
> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org
> [mailto:linux-raid-owner@vger.kernel.org] On Behalf Of Derek Listmail
> Acct
> Sent: Friday, August 06, 2004 2:07 PM
> To: linux-raid@vger.kernel.org
> Subject: Scrub?
>
> Many of the hardware raid controllers I've used have the ability to run
> a
> 'scrub' on an array.  Is there a way to do this on a linux software raid
> 5
> array?
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

RE: Scrub?

[Derek Listmail Acct]

> dd if=/dev/zero of=/dev/md0 bs=200b
>

This will fill the array with zeros, which isn't what I want to do.  I'm
trying to verify the parity data for my raid 5 array is correct.  I feel
one of my drives may be failing, and want to make sure the array is
consistent.

Re: Scrub?

[Mark Watts]

On Friday 06 Aug 2004 8:07 pm, Derek Listmail Acct wrote:
> > dd if=/dev/zero of=/dev/md0 bs=200b
>
> This will fill the array with zeros, which isn't what I want to do.  I'm
> trying to verify the parity data for my raid 5 array is correct.  I feel
> one of my drives may be failing, and want to make sure the array is
> consistent.

Thats array verification, not scrubbing (which is erasing the disk)

RE: Scrub?

[Salyzyn, Mark]

We have two RAID cards of different heritage over here with differing
definitions. One defines scrub as a means of erasing any content and
ensuring all blocks have correct parity or redundancy information; the
other defines it solely for RAID-5 to rebuild all the parity blocks. The
action in the second case resolves itself as a result of reading the bad
block.

Just reading the entire array should correct the bad blocks, so reverse
the sense of the dd:

	dd if=/dev/md0 of=/dev/null bs=200b

to find and replace the bad blocks (making the assumption that md works
like the H/W RAID cards).

Sincerely -- Mark Salyzyn

-----Original Message-----
From: Kanoa Withington [mailto:kanoa@cfht.hawaii.edu] 
Sent: Friday, August 06, 2004 3:03 PM
To: Salyzyn, Mark
Cc: Derek Listmail Acct; linux-raid@vger.kernel.org
Subject: RE: Scrub?

Eh, that would delete the contents of the array, including the
filesystem. Is that what you meant by "scrub"? I thought "scrubbing"
meant looking for unreadable blocks and pro-actively replacing them
from parity. The latter would be very useful for a software raid 5
array since there is currently no facility in software raid for doing
this on-the-fly. I imagine it would be possible to write such a
utility in user space. I, too, wonder if anyone out there has
something that works.

-Kanoa

On Fri, 6 Aug 2004, Salyzyn, Mark wrote:

> dd if=/dev/zero of=/dev/md0 bs=200b
>
> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org
> [mailto:linux-raid-owner@vger.kernel.org] On Behalf Of Derek Listmail
> Acct
> Sent: Friday, August 06, 2004 2:07 PM
> To: linux-raid@vger.kernel.org
> Subject: Scrub?
>
> Many of the hardware raid controllers I've used have the ability to
run
> a
> 'scrub' on an array.  Is there a way to do this on a linux software
raid
> 5
> array?
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid"
in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid"
in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Scrub?

[Mark Watts]

On Friday 06 Aug 2004 8:24 pm, Salyzyn, Mark wrote:
> We have two RAID cards of different heritage over here with differing
> definitions. One defines scrub as a means of erasing any content and
> ensuring all blocks have correct parity or redundancy information; the
> other defines it solely for RAID-5 to rebuild all the parity blocks. The
> action in the second case resolves itself as a result of reading the bad
> block.

Well I've only ever seen the phrase 'scrub' used when refering to wiping 
disks, usually in a secure way (multiple passes with different bit paterns).
Thats on this side of the pond anyway...

RE: Scrub?

[Kanoa Withington]

On Fri, 6 Aug 2004, Salyzyn, Mark wrote:
> Just reading the entire array should correct the bad blocks, so reverse
> the sense of the dd:
>
> 	dd if=/dev/md0 of=/dev/null bs=200b
>
> to find and replace the bad blocks (making the assumption that md works
> like the H/W RAID cards).

In this case software RAID does not work like the H/W cards. Finding
an unreadable block that way in a software array would cause it to go
into a degraded state.

-Kanoa

Re: Scrub?

[Mikael Abrahamsson]

On Fri, 6 Aug 2004, Mark Watts wrote:

> Well I've only ever seen the phrase 'scrub' used when refering to wiping 
> disks, usually in a secure way (multiple passes with different bit paterns).
> Thats on this side of the pond anyway...

http://www.nexsan.com/products/FAQATAboy.pdf

There is a sentense:

"Array is verified before being put into production;  automatic "parity
scrub" (array verify) is performed as a background task every 24 hours
after"

So some use it in the sense of "verify" anyway. I prefer verify as well, 
it seems a more adequate word. There are numerous referrals to the word 
"scrub" as verify when doing a google search for "+raid +scrub".

I set my 3ware volumes to verify every week, it makes me more comfortable 
with the volumes being checked and excercised on a regular basis so I know 
all blocks can be read from all drives.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: Scrub?

[Tim Moore]

No.  Try 'e2fsck -c -c' if unmounted or 'badblocks -nsv' if mounted on the 
individual physical devices.

Derek Listmail Acct wrote:
> Many of the hardware raid controllers I've used have the ability to run a
> 'scrub' on an array.  Is there a way to do this on a linux software raid 5
> array?
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 

RE: Scrub?

[dean gaudet]

On Fri, 6 Aug 2004, Kanoa Withington wrote:

> On Fri, 6 Aug 2004, Salyzyn, Mark wrote:
> > Just reading the entire array should correct the bad blocks, so reverse
> > the sense of the dd:
> >
> > 	dd if=/dev/md0 of=/dev/null bs=200b
> >
> > to find and replace the bad blocks (making the assumption that md works
> > like the H/W RAID cards).
>
> In this case software RAID does not work like the H/W cards. Finding
> an unreadable block that way in a software array would cause it to go
> into a degraded state.

if the disks support SMART (i.e. they're less than a few years old) then
try running the smart long selftest... it can be done online and on many
disks it will force sector reallocation (and produce a SMART log event so
you know it happenned).

get smartmontools and run "smartctl -a" to see info on your drive, and
"smartctl -t long" to launch the long test.  man page has more examples.

i run smart long tests on each my disks once a week (staggerred over many
nights)... see /etc/smartd.conf.

-dean

RE: Scrub?

[Salyzyn, Mark]

Problem with running the relocation is that the RAID-5 will now be
corrupt. The RAID-5 algorithm needs to be in-touch with disk block
relocation so that it can correct the parity and the data.

Sincerely -- Mark Salyzyn

-----Original Message-----
From: dean gaudet [mailto:dean-list-linux-raid@arctic.org] 
Sent: Friday, August 06, 2004 5:59 PM
To: Kanoa Withington
Cc: Salyzyn, Mark; Derek Listmail Acct; linux-raid@vger.kernel.org
Subject: RE: Scrub?

On Fri, 6 Aug 2004, Kanoa Withington wrote:

> On Fri, 6 Aug 2004, Salyzyn, Mark wrote:
> > Just reading the entire array should correct the bad blocks, so
reverse
> > the sense of the dd:
> >
> > 	dd if=/dev/md0 of=/dev/null bs=200b
> >
> > to find and replace the bad blocks (making the assumption that md
works
> > like the H/W RAID cards).
>
> In this case software RAID does not work like the H/W cards. Finding
> an unreadable block that way in a software array would cause it to go
> into a degraded state.

if the disks support SMART (i.e. they're less than a few years old) then
try running the smart long selftest... it can be done online and on many
disks it will force sector reallocation (and produce a SMART log event
so
you know it happenned).

get smartmontools and run "smartctl -a" to see info on your drive, and
"smartctl -t long" to launch the long test.  man page has more examples.

i run smart long tests on each my disks once a week (staggerred over
many
nights)... see /etc/smartd.conf.

-dean

RE: Scrub?

[Kanoa Withington]

Furthermore, drives (including those with SMART support) only
reallocate bad blocks following write errors. A scrub/verify tool
would be looking for read errors and would need to actively replace
the unreadable block with rudundant data from a _different_ drive.

Actually, I guess it would be kind of complicated. A scan for
unreadable blocks would need to occur outside of the md layer (to
prevent the errors from triggering a degraded state), yet the block
replacement would need to occur through the md layer, since only md
knows how to recreate the block from parity. It would probably be a
easier for a RAID 1 array where the disks are symetric and you could
guess the location of the redundant block on the second volume. You
would probably need to use md on a RAID 5 array.

-Kanoa

On Sat, 7 Aug 2004, Salyzyn, Mark wrote:

> Problem with running the relocation is that the RAID-5 will now be
> corrupt. The RAID-5 algorithm needs to be in-touch with disk block
> relocation so that it can correct the parity and the data.
>
> Sincerely -- Mark Salyzyn
>
> -----Original Message-----
> From: dean gaudet [mailto:dean-list-linux-raid@arctic.org]
> Sent: Friday, August 06, 2004 5:59 PM
> To: Kanoa Withington
> Cc: Salyzyn, Mark; Derek Listmail Acct; linux-raid@vger.kernel.org
> Subject: RE: Scrub?
>
> On Fri, 6 Aug 2004, Kanoa Withington wrote:
>
> > On Fri, 6 Aug 2004, Salyzyn, Mark wrote:
> > > Just reading the entire array should correct the bad blocks, so
> reverse
> > > the sense of the dd:
> > >
> > > 	dd if=/dev/md0 of=/dev/null bs=200b
> > >
> > > to find and replace the bad blocks (making the assumption that md
> works
> > > like the H/W RAID cards).
> >
> > In this case software RAID does not work like the H/W cards. Finding
> > an unreadable block that way in a software array would cause it to go
> > into a degraded state.
>
> if the disks support SMART (i.e. they're less than a few years old) then
> try running the smart long selftest... it can be done online and on many
> disks it will force sector reallocation (and produce a SMART log event
> so
> you know it happenned).
>
> get smartmontools and run "smartctl -a" to see info on your drive, and
> "smartctl -t long" to launch the long test.  man page has more examples.
>
> i run smart long tests on each my disks once a week (staggerred over
> many
> nights)... see /etc/smartd.conf.
>
> -dean
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

RE: Scrub?

[dean gaudet]

no, that's not how it works.  i'm referring to the hard disk itself
relocating a sector -- it's transparent to the host/raid.  the only thing
the raid software might see is that the disk will be less snappy while
it's running the SMART long test.  (mind you i do this on live busy
systems and i don't really tend to notice it -- although on particularly
busy weeks, some disks can take several days to complete their self test
in the few spare cycles they find.)

-dean

On Sat, 7 Aug 2004, Salyzyn, Mark wrote:

> Problem with running the relocation is that the RAID-5 will now be
> corrupt. The RAID-5 algorithm needs to be in-touch with disk block
> relocation so that it can correct the parity and the data.
>
> Sincerely -- Mark Salyzyn
>
> -----Original Message-----
> From: dean gaudet [mailto:dean-list-linux-raid@arctic.org]
> Sent: Friday, August 06, 2004 5:59 PM
> To: Kanoa Withington
> Cc: Salyzyn, Mark; Derek Listmail Acct; linux-raid@vger.kernel.org
> Subject: RE: Scrub?
>
> On Fri, 6 Aug 2004, Kanoa Withington wrote:
>
> > On Fri, 6 Aug 2004, Salyzyn, Mark wrote:
> > > Just reading the entire array should correct the bad blocks, so
> reverse
> > > the sense of the dd:
> > >
> > > 	dd if=/dev/md0 of=/dev/null bs=200b
> > >
> > > to find and replace the bad blocks (making the assumption that md
> works
> > > like the H/W RAID cards).
> >
> > In this case software RAID does not work like the H/W cards. Finding
> > an unreadable block that way in a software array would cause it to go
> > into a degraded state.
>
> if the disks support SMART (i.e. they're less than a few years old) then
> try running the smart long selftest... it can be done online and on many
> disks it will force sector reallocation (and produce a SMART log event
> so
> you know it happenned).
>
> get smartmontools and run "smartctl -a" to see info on your drive, and
> "smartctl -t long" to launch the long test.  man page has more examples.
>
> i run smart long tests on each my disks once a week (staggerred over
> many
> nights)... see /etc/smartd.conf.
>
> -dean
>

RE: Scrub?

[dean gaudet]

at least two of my older maxtor disks (model 4G120J6) reallocated a sector
during a long self test.  or maybe i didn't run enough of a controlled
experiment... but smartctl -a before and after diffs showed
"Reallocated_Sector_Ct"  increase by 1, and one of the two disks wasn't in
any use at the time.  (the other was in light use, and may have tripped up
the normal write-reallocate path.)

i have a cronjob run smartctl -a every day, and diff the output with the
previous day; and smartd runs the self tests once a week.

not that i'm disagreeing with the need for improvements to md in this
area... see <http://arctic.org/~dean/raid-wishlist.html> :)

-dean

On Sat, 7 Aug 2004, Kanoa Withington wrote:

> Furthermore, drives (including those with SMART support) only
> reallocate bad blocks following write errors. A scrub/verify tool
> would be looking for read errors and would need to actively replace
> the unreadable block with rudundant data from a _different_ drive.
>
> Actually, I guess it would be kind of complicated. A scan for
> unreadable blocks would need to occur outside of the md layer (to
> prevent the errors from triggering a degraded state), yet the block
> replacement would need to occur through the md layer, since only md
> knows how to recreate the block from parity. It would probably be a
> easier for a RAID 1 array where the disks are symetric and you could
> guess the location of the redundant block on the second volume. You
> would probably need to use md on a RAID 5 array.
>
> -Kanoa
>
> On Sat, 7 Aug 2004, Salyzyn, Mark wrote:
>
> > Problem with running the relocation is that the RAID-5 will now be
> > corrupt. The RAID-5 algorithm needs to be in-touch with disk block
> > relocation so that it can correct the parity and the data.
> >
> > Sincerely -- Mark Salyzyn
> >
> > -----Original Message-----
> > From: dean gaudet [mailto:dean-list-linux-raid@arctic.org]
> > Sent: Friday, August 06, 2004 5:59 PM
> > To: Kanoa Withington
> > Cc: Salyzyn, Mark; Derek Listmail Acct; linux-raid@vger.kernel.org
> > Subject: RE: Scrub?
> >
> > On Fri, 6 Aug 2004, Kanoa Withington wrote:
> >
> > > On Fri, 6 Aug 2004, Salyzyn, Mark wrote:
> > > > Just reading the entire array should correct the bad blocks, so
> > reverse
> > > > the sense of the dd:
> > > >
> > > > 	dd if=/dev/md0 of=/dev/null bs=200b
> > > >
> > > > to find and replace the bad blocks (making the assumption that md
> > works
> > > > like the H/W RAID cards).
> > >
> > > In this case software RAID does not work like the H/W cards. Finding
> > > an unreadable block that way in a software array would cause it to go
> > > into a degraded state.
> >
> > if the disks support SMART (i.e. they're less than a few years old) then
> > try running the smart long selftest... it can be done online and on many
> > disks it will force sector reallocation (and produce a SMART log event
> > so
> > you know it happenned).
> >
> > get smartmontools and run "smartctl -a" to see info on your drive, and
> > "smartctl -t long" to launch the long test.  man page has more examples.
> >
> > i run smart long tests on each my disks once a week (staggerred over
> > many
> > nights)... see /etc/smartd.conf.
> >
> > -dean
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
>

RE: Scrub?

[Salyzyn, Mark]

As long as the relocation contains the original information; which is
only the case if the relocation is done on a write to the disk.

A RAID-5 generated relocation allows the data to be reconstructed on a
media failure on erad, and the relocation (triggered by a re-write)
would contain the correct information preventing corruption.

Sincerely -- Mark Salyzyn

-----Original Message-----
From: dean gaudet [mailto:dean-list-linux-raid@arctic.org] 
Sent: Saturday, August 07, 2004 4:13 PM
To: Salyzyn, Mark
Cc: Kanoa Withington; Derek Listmail Acct; linux-raid@vger.kernel.org
Subject: RE: Scrub?

no, that's not how it works.  i'm referring to the hard disk itself
relocating a sector -- it's transparent to the host/raid.  the only
thing
the raid software might see is that the disk will be less snappy while
it's running the SMART long test.  (mind you i do this on live busy
systems and i don't really tend to notice it -- although on particularly
busy weeks, some disks can take several days to complete their self test
in the few spare cycles they find.)

-dean

On Sat, 7 Aug 2004, Salyzyn, Mark wrote:

> Problem with running the relocation is that the RAID-5 will now be
> corrupt. The RAID-5 algorithm needs to be in-touch with disk block
> relocation so that it can correct the parity and the data.
>
> Sincerely -- Mark Salyzyn
>
> -----Original Message-----
> From: dean gaudet [mailto:dean-list-linux-raid@arctic.org]
> Sent: Friday, August 06, 2004 5:59 PM
> To: Kanoa Withington
> Cc: Salyzyn, Mark; Derek Listmail Acct; linux-raid@vger.kernel.org
> Subject: RE: Scrub?
>
> On Fri, 6 Aug 2004, Kanoa Withington wrote:
>
> > On Fri, 6 Aug 2004, Salyzyn, Mark wrote:
> > > Just reading the entire array should correct the bad blocks, so
> reverse
> > > the sense of the dd:
> > >
> > > 	dd if=/dev/md0 of=/dev/null bs=200b
> > >
> > > to find and replace the bad blocks (making the assumption that md
> works
> > > like the H/W RAID cards).
> >
> > In this case software RAID does not work like the H/W cards. Finding
> > an unreadable block that way in a software array would cause it to
go
> > into a degraded state.
>
> if the disks support SMART (i.e. they're less than a few years old)
then
> try running the smart long selftest... it can be done online and on
many
> disks it will force sector reallocation (and produce a SMART log event
> so
> you know it happenned).
>
> get smartmontools and run "smartctl -a" to see info on your drive, and
> "smartctl -t long" to launch the long test.  man page has more
examples.
>
> i run smart long tests on each my disks once a week (staggerred over
> many
> nights)... see /etc/smartd.conf.
>
> -dean
>

RE: Scrub?

[dean gaudet]

aha, now i see what you're saying -- yeah i have no evidence either way as
to what happens in a long self-test in that regard.

-dean

On Sat, 7 Aug 2004, Salyzyn, Mark wrote:

> As long as the relocation contains the original information; which is
> only the case if the relocation is done on a write to the disk.
>
> A RAID-5 generated relocation allows the data to be reconstructed on a
> media failure on erad, and the relocation (triggered by a re-write)
> would contain the correct information preventing corruption.
>
> Sincerely -- Mark Salyzyn
>
> -----Original Message-----
> From: dean gaudet [mailto:dean-list-linux-raid@arctic.org]
> Sent: Saturday, August 07, 2004 4:13 PM
> To: Salyzyn, Mark
> Cc: Kanoa Withington; Derek Listmail Acct; linux-raid@vger.kernel.org
> Subject: RE: Scrub?
>
> no, that's not how it works.  i'm referring to the hard disk itself
> relocating a sector -- it's transparent to the host/raid.  the only
> thing
> the raid software might see is that the disk will be less snappy while
> it's running the SMART long test.  (mind you i do this on live busy
> systems and i don't really tend to notice it -- although on particularly
> busy weeks, some disks can take several days to complete their self test
> in the few spare cycles they find.)
>
> -dean
>
> On Sat, 7 Aug 2004, Salyzyn, Mark wrote:
>
> > Problem with running the relocation is that the RAID-5 will now be
> > corrupt. The RAID-5 algorithm needs to be in-touch with disk block
> > relocation so that it can correct the parity and the data.
> >
> > Sincerely -- Mark Salyzyn
> >
> > -----Original Message-----
> > From: dean gaudet [mailto:dean-list-linux-raid@arctic.org]
> > Sent: Friday, August 06, 2004 5:59 PM
> > To: Kanoa Withington
> > Cc: Salyzyn, Mark; Derek Listmail Acct; linux-raid@vger.kernel.org
> > Subject: RE: Scrub?
> >
> > On Fri, 6 Aug 2004, Kanoa Withington wrote:
> >
> > > On Fri, 6 Aug 2004, Salyzyn, Mark wrote:
> > > > Just reading the entire array should correct the bad blocks, so
> > reverse
> > > > the sense of the dd:
> > > >
> > > > 	dd if=/dev/md0 of=/dev/null bs=200b
> > > >
> > > > to find and replace the bad blocks (making the assumption that md
> > works
> > > > like the H/W RAID cards).
> > >
> > > In this case software RAID does not work like the H/W cards. Finding
> > > an unreadable block that way in a software array would cause it to
> go
> > > into a degraded state.
> >
> > if the disks support SMART (i.e. they're less than a few years old)
> then
> > try running the smart long selftest... it can be done online and on
> many
> > disks it will force sector reallocation (and produce a SMART log event
> > so
> > you know it happenned).
> >
> > get smartmontools and run "smartctl -a" to see info on your drive, and
> > "smartctl -t long" to launch the long test.  man page has more
> examples.
> >
> > i run smart long tests on each my disks once a week (staggerred over
> > many
> > nights)... see /etc/smartd.conf.
> >
> > -dean
> >
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>