raid5+hotspare: request for recommended procedure

raid5+hotspare: request for recommended procedure

[Stefan G. Weichinger]

Greets, raid-users,

I would like to ask for hints on how to proceed.

I have a customers server ~500kms away ... running 2 raid5-arrays w/
hotspare:

# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid6] [raid5] [raid4]
md1 : active raid1 sdb1[1] sda1[0]
      104320 blocks [2/2] [UU]

md3 : active raid5 sdd3[3](S) sdc3[2] sdb3[1] sda3[0]
      15647104 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]

md4 : active raid5 sdd4[3](S) sdc4[2] sdb4[1] sda4[0]
      471941376 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]

unused devices: <none>


sdb shows errors:

197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always
      -       13
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age
Offline      -       13

--

The customer would now take the server with him and bring it to a fellow
technician who could take out sdb, clone it to a new hdd and re-insert it.

This would be plan A.

Plan B would be that I mark sdb failed now and let the raids rebuild. I
fear that a second hdd might fail when doing this.

All Seagate-drives:

sda, sdb: ST3250310NS
sdc, sdd: ST3250621NS

I also ran a "echo check > /sys/block/mdX/md/sync_action" because this
had helped to remove those errors at another server, unfortunately it
did not help here.

Could you please advise what the better and safer alternative would be?

Thanks a lot, Stefan

Re: raid5+hotspare: request for recommended procedure

[Tim Small]

On 29/09/10 10:03, Stefan G. Weichinger wrote:
> I also ran a "echo check>  /sys/block/mdX/md/sync_action" because this
> had helped to remove those errors at another server, unfortunately it
> did not help here.
>
> Could you please advise what the better and safer alternative would be?
>
>    

Run smartctl -t long, and see what the LBA of the errors are (smartctl 
-a when finished, and look at the error log) - it may be that it is 
outside of the used space (e.g. right at the end of the drive - I've 
seen this in the past) if this is the case, you can just dd over it.

If the test completes without error then the unreadable sector is either 
not user-addressable, or doesn't really exist, in which case I'd ignore 
it (or perhaps a security-erase will get the drive back into a sensible 
state).  Some of the earlier firmwares for the ST3250310NS were a little 
buggy in my experience, so you might want to look at upgrading it 
(latest Seagate non-OEM firmware is SN06), which can be done with hdparm.

Tim.

-- 
South East Open Source Solutions Limited
Registered in England and Wales with company number 06134732.
Registered Office: 2 Powell Gardens, Redhill, Surrey, RH1 1TQ
VAT number: 900 6633 53  http://seoss.co.uk/ +44-(0)1273-808309

Re: raid5+hotspare: request for recommended procedure

[Tim Small]

On 29/09/10 11:03, Tim Small wrote:
> earlier firmwares for the ST3250310NS were a little buggy in my 
> experience, so you might want to look at upgrading it (latest Seagate 
> non-OEM firmware is SN06), which can be done with hdparm.

I should say that I've "cross-flashed" Dell's MA0D on SN0x drives too, 
and that seems to give good stability.

Tim.

-- 
South East Open Source Solutions Limited
Registered in England and Wales with company number 06134732.
Registered Office: 2 Powell Gardens, Redhill, Surrey, RH1 1TQ
VAT number: 900 6633 53  http://seoss.co.uk/ +44-(0)1273-808309

Re: raid5+hotspare: request for recommended procedure

[Stefan G. Weichinger]

Am 2010-09-29 12:03, schrieb Tim Small:

> Run smartctl -t long, and see what the LBA of the errors are (smartctl
> -a when finished, and look at the error log) - it may be that it is
> outside of the used space (e.g. right at the end of the drive - I've
> seen this in the past) if this is the case, you can just dd over it.

I will do that maybe after the repair-action is through ...

Could you tell me in detail what you mean by "dd over it" ? Write zeros
there or what ...?

> If the test completes without error then the unreadable sector is either
> not user-addressable, or doesn't really exist, in which case I'd ignore
> it (or perhaps a security-erase will get the drive back into a sensible
> state).  Some of the earlier firmwares for the ST3250310NS were a little
> buggy in my experience, so you might want to look at upgrading it
> (latest Seagate non-OEM firmware is SN06), which can be done with hdparm.

You know I really hesitate with such low-level-stuff as the server is
far away and I won't be able to just boot from a live-cd or something
... so I am very cautious with my steps.

Thanks so far, Stefan

Re: raid5+hotspare: request for recommended procedure

[Stefan G. Weichinger]

Am 2010-09-29 12:03, schrieb Tim Small:

> Run smartctl -t long, and see what the LBA of the errors are (smartctl
> -a when finished, and look at the error log) - it may be that it is
> outside of the used space (e.g. right at the end of the drive - I've
> seen this in the past) if this is the case, you can just dd over it.
> 
> If the test completes without error then the unreadable sector is either
> not user-addressable, or doesn't really exist, in which case I'd ignore
> it (or perhaps a security-erase will get the drive back into a sensible
> state).  Some of the earlier firmwares for the ST3250310NS were a little
> buggy in my experience, so you might want to look at upgrading it
> (latest Seagate non-OEM firmware is SN06), which can be done with hdparm.

Yep, it finished without error!

Does that mean I could ignore that safely?

Or should I rebuild the raids onto the spare-disk and then swap drive?

S

Re: raid5+hotspare: request for recommended procedure

[Jon Hardcastle]

--- On Wed, 29/9/10, Stefan G. Weichinger <lists@xunil.at> wrote:

> From: Stefan G. Weichinger <lists@xunil.at>
> Subject: Re: raid5+hotspare: request for recommended procedure
> To: "linux-raid@vger.kernel.org" <linux-raid@vger.kernel.org>
> Date: Wednesday, 29 September, 2010, 15:25
> Am 2010-09-29 12:03, schrieb Tim
> Small:
> 
> > Run smartctl -t long, and see what the LBA of the
> errors are (smartctl
> > -a when finished, and look at the error log) - it may
> be that it is
> > outside of the used space (e.g. right at the end of
> the drive - I've
> > seen this in the past) if this is the case, you can
> just dd over it.
> > 
> > If the test completes without error then the
> unreadable sector is either
> > not user-addressable, or doesn't really exist, in
> which case I'd ignore
> > it (or perhaps a security-erase will get the drive
> back into a sensible
> > state).  Some of the earlier firmwares for the
> ST3250310NS were a little
> > buggy in my experience, so you might want to look at
> upgrading it
> > (latest Seagate non-OEM firmware is SN06), which can
> be done with hdparm.
> 
> Yep, it finished without error!
> 
> Does that mean I could ignore that safely?
> 
> Or should I rebuild the raids onto the spare-disk and then
> swap drive?
> 
> S
> --

Personally I would trust nothing less than a dammed good thrashing from badblocks -svw ideally or -svn if the drive has data on it. Then an smart offline check to be sure the pending/reallocated sector counts have no increased.

Before commissioning a new drive a do a badblocks -svw atleast 3 times without any counters increasing (something there are bad sectors I just want to flush them out early) then I do some short, long and offline smart tests.

Then I add to the array, then I run check a couple of times.


      
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: raid5+hotspare: request for recommended procedure

[Stefan G. Weichinger]

Am 2010-09-29 16:29, schrieb Jon Hardcastle:

> Personally I would trust nothing less than a dammed good thrashing 
> from badblocks -svw ideally or -svn if the drive has data on it.
> Then an smart offline check to be sure the pending/reallocated
> sector counts have no increased.
> 
> Before commissioning a new drive a do a badblocks -svw atleast 3 
> times without any counters increasing (something there are bad 
> sectors I just want to flush them out early) then I do some short, 
> long and offline smart tests.
> 
> Then I add to the array, then I run check a couple of times.

To be explicit:

This would mean to run "badblocks -svn" on md3 or sdb or sdb3 ?

just to make sure I understand you correctly ...

Does that step take long?

Stefan

Re: raid5+hotspare: request for recommended procedure

[Jon Hardcastle]

--- On Wed, 29/9/10, Stefan G. Weichinger <lists@xunil.at> wrote:

> From: Stefan G. Weichinger <lists@xunil.at>
> Subject: Re: raid5+hotspare: request for recommended procedure
> To: "linux-raid@vger.kernel.org" <linux-raid@vger.kernel.org>
> Cc: Jon@eHardcastle.com, "Jon Hardcastle" <jd_hardcastle@yahoo.com>
> Date: Wednesday, 29 September, 2010, 16:08
> Am 2010-09-29 16:29, schrieb Jon
> Hardcastle:
> 
> > Personally I would trust nothing less than a dammed
> good thrashing 
> > from badblocks -svw ideally or -svn if the drive has
> data on it.
> > Then an smart offline check to be sure the
> pending/reallocated
> > sector counts have no increased.
> > 
> > Before commissioning a new drive a do a badblocks -svw
> atleast 3 
> > times without any counters increasing (something there
> are bad 
> > sectors I just want to flush them out early) then I do
> some short, 
> > long and offline smart tests.
> > 
> > Then I add to the array, then I run check a couple of
> times.
> 
> To be explicit:
> 
> This would mean to run "badblocks -svn" on md3 or sdb or
> sdb3 ?
> 
> just to make sure I understand you correctly ...
> 
> Does that step take long?
> 
> Stefan
> 

This step is run on the drives that form the array so sdb... and it can take a long time.. depends on how many times you run it through and how the drive is connected...


      

Re: raid5+hotspare: request for recommended procedure

[Stefan G. Weichinger]

Am 29.09.2010 17:39, schrieb Jon Hardcastle:
>> Does that step take long?
> 
> This step is run on the drives that form the array so sdb... and it
> can take a long time.. depends on how many times you run it through
> and how the drive is connected...

Thanks for the info.

Right now the status is that the server didn't come up anymore after I
issued a reboot ...

And they don't even have a monitor to attach *sigh*

They will have to transport it now ... and my saturday will contain
raid-related work ...

Thanks all, Stefan

Re: raid5+hotspare: request for recommended procedure

[Stefan G. Weichinger]

Am 30.09.2010 11:50, schrieb Stefan G. Weichinger:

> Right now the status is that the server didn't come up anymore after I
> issued a reboot ...
> 
> And they don't even have a monitor to attach *sigh*
> 
> They will have to transport it now ... and my saturday will contain
> raid-related work ...

update: the transported server simply booted ... hmm.
All the raids valid, nothing suspicious.

I took the opportunity and upgraded the kernel now. Before I decide how
to proceed with that hdd I pull some backup to tape. They didn't swap
tapes for 3 weeks ... oh my.

Stefan

Re: raid5+hotspare: request for recommended procedure

[Stefan G. Weichinger]

Am 01.10.2010 11:00, schrieb Stefan G. Weichinger:

> update: the transported server simply booted ... hmm.
> All the raids valid, nothing suspicious.
> 
> I took the opportunity and upgraded the kernel now. Before I decide how
> to proceed with that hdd I pull some backup to tape. They didn't swap
> tapes for 3 weeks ... oh my.

The current plan is to wait for the backup to finish, after that I will
mark the suspicious sdb-partitions as failed and let the raid rebuild
onto the spare.

After having run the sync_action repair stuff I think I can trust all
the hdds. I issue a "smartctl -t long" now in advance ...

pls correct me if needed ...

Stefan

Re: raid5+hotspare: request for recommended procedure

[Stefan G. Weichinger]

Am 2010-10-01 11:59, schrieb Stefan G. Weichinger:

> The current plan is to wait for the backup to finish, after that I will
> mark the suspicious sdb-partitions as failed and let the raid rebuild
> onto the spare.
> 
> After having run the sync_action repair stuff I think I can trust all
> the hdds. I issue a "smartctl -t long" now in advance ...
> 
> pls correct me if needed ...

Let me add this:
I also like some "yep, go for it, you are doing the right thing" ;-)

Greets, Stefan

Re: raid5+hotspare: request for recommended procedure

[Michael Tokarev]

29.09.2010 13:03, Stefan G. Weichinger wrote:
> 
> Greets, raid-users,
> 
> I would like to ask for hints on how to proceed.
> 
> I have a customers server ~500kms away ... running 2 raid5-arrays w/
> hotspare:
> 
[]
> sdb shows errors:
> 
> 197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always
>       -       13
> 198 Offline_Uncorrectable   0x0010   100   100   000    Old_age
> Offline      -       13

I'd run the repair procedure for the raids first.  The procedure
reads all blocks from all raid drives, comparing them while it
goes.  If any block is unreadable, it tries to re-write it based
on the data reconstructed from other raid disks.  This way, your
sdb may become good again, after remapping the 13 bad sectors
(which is very small amount for nowadays high-dencity drives).

> The customer would now take the server with him and bring it to a fellow
> technician who could take out sdb, clone it to a new hdd and re-insert it.
> 
> This would be plan A.
> 
> Plan B would be that I mark sdb failed now and let the raids rebuild. I
> fear that a second hdd might fail when doing this.

Yes, at least run smart selftests on the remaining drives first,
to ensure they wont find badblocks.

> All Seagate-drives:
> 
> sda, sdb: ST3250310NS
> sdc, sdd: ST3250621NS
> 
> I also ran a "echo check > /sys/block/mdX/md/sync_action" because this
> had helped to remove those errors at another server, unfortunately it
> did not help here.

It should be repair, not check - check merely reads stuff, repair tries
to fix it.

> Could you please advise what the better and safer alternative would be?

There is - unfortunately - no good procedure for this case, even if
it is the most frequent usage case with failed drives (i think).
What should be needed is to let raid arrays to pick spare drive,
copy data to it from the drive being replaced (or, failing that,
from other raid drives), thus making a sort of raid1 (mirror)
between the spare drive and the one being replaced, and when the
mirror is finished, switch the roles, so you can remove the "new
spare".  But there's no such code, and any attempt to do that
manually does not work as good as the scenario I outlined.

Note you can't just copy all good data from failing to spare
outside the raid5 array: the bad blocks will be unreadable, and
you'll have to skip them somehow, but when you add the replacement
back to the array you can't tell raid code which blocks were
skipped.  So your array will be corrupt most likely, -- the only
way to get the original data is to reconstruct the unreadable
blocks from other raid disks, which is difficult to do manually.
That's actually what I said above - any attempt to recover it
outside the original arrays is worse than the ideal procedure
which does not exist.

So I'd do this:

 1) run repair.  If it fixes everything, you may as well just
  keep the "failing" drive, since I'm not really sure it is
  really bad.
 2) if you still want to replace it, after running repair you'll
  know your other drives work fine, so you can do either replace,
  or copy+replace - the latter is if the "failing" drive will be
  repaired in the first step.  Sure thing you've more luck
  (as in: two attempts instead of just one) attempting  to clone
  the "failing" drive - while your array is stopped; if that does
  not work, use the replace way.

Just imho :)

/mjt

Re: raid5+hotspare: request for recommended procedure

[Tim Small]

On 29/09/10 13:01, Michael Tokarev wrote:
>
>> I also ran a "echo check>  /sys/block/mdX/md/sync_action" because this
>> had helped to remove those errors at another server, unfortunately it
>> did not help here.
>>      
> It should be repair, not check - check merely reads stuff, repair tries
> to fix it.
>
>    

No it shouldn't, AFAIK - check will read from all sectors on all drives, 
and will trigger a rewrite on unreadable blocks (which is what the  OP has).

Tim.

-- 
South East Open Source Solutions Limited
Registered in England and Wales with company number 06134732.
Registered Office: 2 Powell Gardens, Redhill, Surrey, RH1 1TQ
VAT number: 900 6633 53  http://seoss.co.uk/ +44-(0)1273-808309

Re: raid5+hotspare: request for recommended procedure

[Stefan G. Weichinger]

Am 2010-09-29 14:21, schrieb Tim Small:

>> It should be repair, not check - check merely reads stuff, repair tries
>> to fix it.
> 
> No it shouldn't, AFAIK - check will read from all sectors on all drives,
> and will trigger a rewrite on unreadable blocks (which is what the  OP
> has).

I ran a repair on md3 now, currently I do it for md4.

kernel-docs say:

   sync_action
     a text file that can be used to monitor and control the rebuild
     process.  It contains one word which can be one of:
       resync        - redundancy is being recalculated after unclean
                       shutdown or creation
       recover       - a hot spare is being built to replace a
                       failed/missing device
       idle          - nothing is happening
       check         - A full check of redundancy was requested and is
                       happening.  This reads all block and checks
                       them. A repair may also happen for some raid
                       levels.
       repair        - A full check and repair is happening.  This is
                       similar to 'resync', but was requested by the
                       user, and the write-intent bitmap is NOT used to
                       optimise the process.

thanks, Stefan