Re: Raid check didn't fix Current_Pending_Sector, but badblocks -nsv did

Re: Raid check didn't fix Current_Pending_Sector, but badblocks -nsv did

[Marc MERLIN]

Howdy, I have a raid 5 where one drive reported this:
197 Current_Pending_Sector  0x0032   200   199   000    Old_age   Always -       29

So I did this:
myth:~# echo check > /sys/block/md5/md/sync_action
[173947.749761] md: data-check of RAID array md5
(...)
[370316.769230] md: md5: data-check done.

My understanding was that it was supposed to read every block of every
drive, and if some blocks were unreadable, use parity to rewrite them on
some fresh backup blocks.
If a block returned garbage instead, md5 cannot fix this not knowing which
block is wrong, but I'm assuming the check would have failed with an error.

However after the check is over, I still have 29 Current_Pending_Sector on
that drive.

Since raid check succeeded, I'm going to assume that the sectors were
readable and did not return garbage, or I'd have gotten a parity mismatch
error.
Should then assume that either
1) the smart counter/logic is wrong?
2) the pending sectors started returning correct data again, so linux md has
no idea those blocks are "weak" and I have no easy way to forcibly remap
them.
3) the bad blocks did get remapped somehow, but the smart counter did not get
reset due to a firmware bug
4) other

After 2 days of testing with badblocks, it seems that it's #2, and I'm
not sure if there is anything raid check could have done (probably not)

Since raid check didn't do the job I was hoping for, I ran this instead:
myth:~# badblocks -nsv /dev/sdg  
Checking for bad blocks in non-destructive read-write mode  
From block 0 to 3907018583  
Checking for bad blocks (non-destructive read-write test)
Testing with random pattern:  57.87% done, 22:24:15 elapsed. (0/0/0 errors))

And this worked:
197 Current_Pending_Sector  0x0032   200   199   000    Old_age   Always       -       0
strangely, I also have:
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0

I guess this means that my drive's auto reallocation logic is faulty and
that it will not re-allocate blocks that are weak, even after it was
able to read them.
Does that sound correct?


More drive details from before I ran badblocks  (not an SMR drive):
Device Model:     WDC WD40EFRX-68WT0N0
Serial Number:    WD-WCC4E0642444
LU WWN Device Id: 5 0014ee 2b437e9a6
Firmware Version: 80.00A80
User Capacity:    4,000,787,030,016 bytes [4.00 TB]

  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       1617
  3 Spin_Up_Time            0x0027   175   173   021    Pre-fail  Always       -       8250
  4 Start_Stop_Count        0x0032   094   094   000    Old_age   Always       -       6773
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   074   074   000    Old_age   Always       -       19092
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       158
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       91
193 Load_Cycle_Count        0x0032   182   182   000    Old_age   Always       -       54642
194 Temperature_Celsius     0x0022   121   103   000    Old_age   Always       -       31
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   199   000    Old_age   Always       -       29
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   199   000    Old_age   Always       -       2
200 Multi_Zone_Error_Rate   0x0008   200   189   000    Old_age   Offline      -       0

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     19081         -
# 2  Short offline       Completed without error       00%     19057         -
# 3  Short offline       Completed without error       00%     19035         -
# 4  Short offline       Completed without error       00%     18984         -
# 5  Extended offline    Completed without error       00%     18974         -

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: Raid check didn't fix Current_Pending_Sector, but badblocks -nsv did

[Sarah Newman]

On 06/06/2016 10:41 AM, Marc MERLIN wrote:
> Howdy, I have a raid 5 where one drive reported this:
> 197 Current_Pending_Sector  0x0032   200   199   000    Old_age   Always -       29
> 
> So I did this:
> myth:~# echo check > /sys/block/md5/md/sync_action
> [173947.749761] md: data-check of RAID array md5
> (...)
> [370316.769230] md: md5: data-check done.
> 
> My understanding was that it was supposed to read every block of every
> drive, and if some blocks were unreadable, use parity to rewrite them on
> some fresh backup blocks.
> If a block returned garbage instead, md5 cannot fix this not knowing which
> block is wrong, but I'm assuming the check would have failed with an error.

https://www.kernel.org/doc/Documentation/md.txt shows for sync_action

       check         - A full check of redundancy was requested and is
                       happening.  This reads all blocks and checks
                       them. A repair may also happen for some raid
                       levels.
       repair        - A full check and repair is happening.  This is
                       similar to 'resync', but was requested by the
                       user, and the write-intent bitmap is NOT used to
		       optimise the process.

I think you wanted 'repair' not 'check'.

--Sarah

Re: Raid check didn't fix Current_Pending_Sector, but badblocks -nsv did

[Marc MERLIN]

On Mon, Jun 06, 2016 at 12:10:23PM -0700, Sarah Newman wrote:
> https://www.kernel.org/doc/Documentation/md.txt shows for sync_action
> 
>        check         - A full check of redundancy was requested and is
>                        happening.  This reads all blocks and checks
>                        them. A repair may also happen for some raid
>                        levels.
>        repair        - A full check and repair is happening.  This is
>                        similar to 'resync', but was requested by the
>                        user, and the write-intent bitmap is NOT used to
> 		       optimise the process.
> 
> I think you wanted 'repair' not 'check'.

From what I understand, the only difference between the 2 is that repair
does not use the write-intent bitmap, but both will repair an error if
found.
https://www.thomas-krenn.com/en/wiki/Mdadm_checkarray#Check_vs._Repair

Or are you saying that check after getting a read error from one drive,
would not rewrite the bad block on that drive? I thought it did...

Either way, it seems that neither would have worked because while those
blocks were marked as "need to be reallocated" by the drive, I think the
kernel was actually able to read them without problem, so the md layer never
saw anything and therefore never did anything either.

Whereas badblocks forced an unconditional rewrite of all blocks, and forced
the drive to re-allocate those "weak" blocks, even though it was able to
read them.

Does that sound about right?

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: Raid check didn't fix Current_Pending_Sector, but badblocks -nsv did

[Phil Turmel]

On 06/06/2016 06:44 PM, Marc MERLIN wrote:

> From what I understand, the only difference between the 2 is that repair
> does not use the write-intent bitmap, but both will repair an error if
> found.

Repair doesn't even read the parity (and Q syndrome) blocks if it
doesn't need them.  It unconditionally recomputes and writes parity (and
Q) blocks.

Both processes will compute and rewrite a block that reports a read error.

> https://www.thomas-krenn.com/en/wiki/Mdadm_checkarray#Check_vs._Repair
> 
> Or are you saying that check after getting a read error from one drive,
> would not rewrite the bad block on that drive? I thought it did...

They both do.

> Either way, it seems that neither would have worked because while those
> blocks were marked as "need to be reallocated" by the drive, I think the
> kernel was actually able to read them without problem, so the md layer never
> saw anything and therefore never did anything either.

Both processes only apply to the actual data area of each member device,
which is usually substantially less than the whole disk.  Both processes
will read or write (or both) every data area sector.

Sectors that haven't ever been deliberately written and aren't read by
normal mdadm processes are often the first to pop up in disk
self-diagnostics.  If you look at the sector offsets in the SMART data,
I bet you find they're all outside the area mdadm is using.

Magnetism does slowly decay on drive platters, faster in the spots where
manufacturing didn't maintain the intended oxide thickness.  Completely
normal and is the basis for URE specifications.

> Whereas badblocks forced an unconditional rewrite of all blocks, and forced
> the drive to re-allocate those "weak" blocks, even though it was able to
> read them.

Probably not.  It got 'em because it wasn't limited to the data area.

Phil

Re: Raid check didn't fix Current_Pending_Sector, but badblocks -nsv did

[Marc MERLIN]

On Mon, Jun 06, 2016 at 08:54:16PM -0400, Phil Turmel wrote:
> Repair doesn't even read the parity (and Q syndrome) blocks if it
> doesn't need them.  It unconditionally recomputes and writes parity (and
> Q) blocks.
 
I see. So if it happens to rewrite a parity block that was unreadable
but that it didn't have to read, I won't even know it was unreadable to
start with (but only one change out of n of that happening in raid5)

> Both processes will compute and rewrite a block that reports a read error.
> > https://www.thomas-krenn.com/en/wiki/Mdadm_checkarray#Check_vs._Repair
> > 
> > Or are you saying that check after getting a read error from one drive,
> > would not rewrite the bad block on that drive? I thought it did...
> 
> They both do.

Thanks for confirming.

> Both processes only apply to the actual data area of each member device,
> which is usually substantially less than the whole disk.  Both processes
> will read or write (or both) every data area sector.
 
Mmmh, so it keeps track of how much of my block device is used by
filesystem blocks and skips the rest? I didn't know that.

> Sectors that haven't ever been deliberately written and aren't read by
> normal mdadm processes are often the first to pop up in disk
> self-diagnostics.  If you look at the sector offsets in the SMART data,
> I bet you find they're all outside the area mdadm is using.

Got it.

> > Whereas badblocks forced an unconditional rewrite of all blocks, and forced
> > the drive to re-allocate those "weak" blocks, even though it was able to
> > read them.
> 
> Probably not.  It got 'em because it wasn't limited to the data area.

Right, I understand now, good to know.
So I'll use badblocks next time I have this issue.

Thanks for clearing this up for me.
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: Raid check didn't fix Current_Pending_Sector, but badblocks -nsv did

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 1042 bytes --]

On Mon, 6 Jun 2016 10:41:13 -0700
Marc MERLIN <marc@merlins.org> wrote:

> However after the check is over, I still have 29 Current_Pending_Sector on
> that drive.

I had that once, even asked on this list (can't find that now), and it turned
out the pending sector(s) were in the gap between the start of the partition
and the start of the mdadm actual data (which turned out to be surprisingly
large) -- so naturally neither a check nor repair have ever tried to read or
write there.

> More drive details from before I ran badblocks  (not an SMR drive):
> Device Model:     WDC WD40EFRX-68WT0N0
> Serial Number:    WD-WCC4E0642444

At least if it's out of warranty, consider checking the contact pads underside
the drive PCB and cleaning them. AFAIK the WD drives in particular are known to
develop rust there, which can lead to all sorts of problems including transient
"pending" sectors like that.
https://www.youtube.com/watch?v=qmT32VC22PU
https://www.youtube.com/watch?v=tDTt_yjYYQ8

-- 
With respect,
Roman

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: Raid check didn't fix Current_Pending_Sector, but badblocks -nsv did

[Phil Turmel]

On 06/07/2016 12:51 AM, Marc MERLIN wrote:
> On Mon, Jun 06, 2016 at 08:54:16PM -0400, Phil Turmel wrote:

>> Both processes only apply to the actual data area of each member device,
>> which is usually substantially less than the whole disk.  Both processes
>> will read or write (or both) every data area sector.
>  
> Mmmh, so it keeps track of how much of my block device is used by
> filesystem blocks and skips the rest? I didn't know that.

No, it doesn't keep track of anything the filesystem does.  Look at the
mdadm -E report for one of you member devices -- it shows the location
of the superblock, the start location and size of the data area, and
possibly information about the optional bitmap or bad block table.
Areas outside the data area are not touched by check or repair scrubs.

> Right, I understand now, good to know.
> So I'll use badblocks next time I have this issue.

Or just ignore them.  You aren't using them, so they can't hurt you.
However, do look at the sector numbers in the SMART reports to make sure
they aren't in the data area.  (If you aren't using the whole disk for
mdadm, be sure to adjust for the partition start sector.)  If they *are*
in the data area, watch dmesg while scrubbing to see what's really
happening.

Phil

Re: Raid check didn't fix Current_Pending_Sector, but badblocks -nsv did

[Mikael Abrahamsson]

On Tue, 7 Jun 2016, Phil Turmel wrote:

> Or just ignore them.  You aren't using them, so they can't hurt you. 
> However, do look at the sector numbers in the SMART reports to make sure 
> they aren't in the data area.  (If you aren't using the whole disk for 
> mdadm, be sure to adjust for the partition start sector.)  If they *are* 
> in the data area, watch dmesg while scrubbing to see what's really 
> happening.

I had a similar problem a few years back.

I seem to remember that there was a problem that the pending blocks were 
in the superblock area, and these weren't written out when I did "check" 
or "repair". I might be misremembering though, but I seem to remember I 
had some kind of problem that when I tried to add the dive back in again, 
it got a read error when trying to read the superblock.

They might have also been in the offset area. What I did was that I did a 
--replace to another drive, then I dd:ed /dev/zero to the first gigabyte 
of the drive, which removed the pending blocks. I then added the drive 
back in.

Could someone confirm whether the entire (actually valuable portion) of 
the superblock area is written out at any kind of frequency, or when a 
check/repair is issued? If not, I think that would be a valuable thing to 
do?

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: Raid check didn't fix Current_Pending_Sector, but badblocks -nsv did

[Andreas Klauer]

On Mon, Jun 06, 2016 at 10:41:13AM -0700, Marc MERLIN wrote:
> Howdy, I have a raid 5 where one drive reported this:
> 197 Current_Pending_Sector  0x0032   200   199   000    Old_age   Always -       29

RAID survival depends on healthy drives. This one does not look healthy. 
If the data on your RAID is important, I'd replace this drive. Getting 
this count back down to zero doesn't make it any more trustworthy.

Re: Raid check didn't fix Current_Pending_Sector, but badblocks -nsv did

[Marc MERLIN]

On Tue, Jun 07, 2016 at 09:04:58AM -0400, Phil Turmel wrote:
> No, it doesn't keep track of anything the filesystem does.  Look at the
> mdadm -E report for one of you member devices -- it shows the location
> of the superblock, the start location and size of the data area, and
> possibly information about the optional bitmap or bad block table.
> Areas outside the data area are not touched by check or repair scrubs.
 
Ok, that makes more sense. In my case, the md array is the entire drive,
minus partition and boot sector header, which is why I expected scanning
that would be good enough.
But as been pointed out already, it's very possible that the bad
sectors happened either on the parity drive that didn't get scanned, or
the unused space before the md array data actually starts on the drive.

> Or just ignore them.  You aren't using them, so they can't hurt you.
> However, do look at the sector numbers in the SMART reports to make sure
> they aren't in the data area.  (If you aren't using the whole disk for

The smart scan log for that drive didn't seem to show where said bad
sectores were (smartctl -a, log at the end). Either way, now they've
been remapped, so it's fine.

On Tue, Jun 07, 2016 at 03:57:31PM +0200, Andreas Klauer wrote:
> On Mon, Jun 06, 2016 at 10:41:13AM -0700, Marc MERLIN wrote:
> > Howdy, I have a raid 5 where one drive reported this:
> > 197 Current_Pending_Sector  0x0032   200   199   000    Old_age   Always -       29
> 
> RAID survival depends on healthy drives. This one does not look healthy. 
> If the data on your RAID is important, I'd replace this drive. Getting 
> this count back down to zero doesn't make it any more trustworthy.

Yeah, I hear what you're saying here. I usually do this actually. If
this drive grows another single bad sector, I will do this, but at this
time the hassle fo replacing the drive is not worth it yet.
The array is also used to hold offsite backups of backups of personal
data (not work stuff)

Thanks for the replies.
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: Raid check didn't fix Current_Pending_Sector, but badblocks -nsv did

[Phil Turmel]

On 06/07/2016 09:57 AM, Andreas Klauer wrote:
> On Mon, Jun 06, 2016 at 10:41:13AM -0700, Marc MERLIN wrote:
>> Howdy, I have a raid 5 where one drive reported this:
>> 197 Current_Pending_Sector  0x0032   200   199   000    Old_age   Always -       29
> 
> RAID survival depends on healthy drives. This one does not look healthy. 
> If the data on your RAID is important, I'd replace this drive. Getting 
> this count back down to zero doesn't make it any more trustworthy.

That's excessively paranoid.  You missed:

>   5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0

In other words, the totally normal accumulation of weak spots on his
disk aren't being cleaned up by overwriting as expected from raid
scrubbing.  The vast majority of such flaws are successfully overwritten
and are not a problem.  When they are overwritten and the drive firmware
decides to reallocate, then you can start worrying.  I replace drives
when *reallocations* hit double digits, as my experience says they go to
hell quickly between 10 and 100.

FWIW, all of my troubles with drives were consumer-grade, mostly w/
scterc support, @ > 35k hours.  I'll report to the list when my first
big batch of WD reds starts to die.

Re: Raid check didn't fix Current_Pending_Sector, but badblocks -nsv did

[Brad Campbell]

On 07/06/16 21:04, Phil Turmel wrote:
> On 06/07/2016 12:51 AM, Marc MERLIN wrote:

>> Right, I understand now, good to know.
>> So I'll use badblocks next time I have this issue.
>
> Or just ignore them.  You aren't using them, so they can't hurt you.

That's actually not necessarily true.

If you have a dud sector early on the disk (so before the start of the 
RAID data) you will terminate every SMART long test in the first couple 
of meg of the disk. So while a dud down there won't necessarily impact 
your usage from a RAID perspective, it'll knacker your ability to 
regularly check the disks in their entirety. SMART tests abort on the 
first bad read.

It's ugly, but in the single instance I had that happen, I removed the 
drive from the array, wrote zero to the entire disk and then added it 
back. That forced a reallocation in the affected area.

Usually if it is in the RAID zone, a check scrub will clear it up. 
Having said that I've had a very peculiar one here in the last couple of 
days.

A WD 2TB Green drive with TLER set to 7 seconds. The first read would 
error out in 7 seconds (as it should), but a second read succeeded. 
After returning the error, the drive must have kept trying to recover in 
the background and eventually succeeded and cached the result. So 
subsequent reads were ok. After reading and writing enough to other 
parts of the drive to flush the drives cache, the process would repeat.

In this case, it took about 3 check scrubs to actually hit the read 
error and force a re-write.

Re: Raid check didn't fix Current_Pending_Sector, but badblocks -nsv did

[Phil Turmel]

On 06/07/2016 09:39 PM, Brad Campbell wrote:
> On 07/06/16 21:04, Phil Turmel wrote:
>> On 06/07/2016 12:51 AM, Marc MERLIN wrote:
> 
>>> Right, I understand now, good to know.
>>> So I'll use badblocks next time I have this issue.
>>
>> Or just ignore them.  You aren't using them, so they can't hurt you.
> 
> That's actually not necessarily true.
> 
> If you have a dud sector early on the disk (so before the start of the
> RAID data) you will terminate every SMART long test in the first couple
> of meg of the disk. So while a dud down there won't necessarily impact
> your usage from a RAID perspective, it'll knacker your ability to
> regularly check the disks in their entirety. SMART tests abort on the
> first bad read.

Don't bother doing long self-tests on drives participating in an array
-- check scrubs do everything a long self-test does on the area of
interest, plus actually fixing UREs that are found.  And check scrubs
don't abort on a read failure.

My advice stands: ignore the UREs in unused areas of the disk.

> It's ugly, but in the single instance I had that happen, I removed the
> drive from the array, wrote zero to the entire disk and then added it
> back. That forced a reallocation in the affected area.

Completely pointless exercise that opened a window of higher-risk of
failure of your array.  Unless you used --replace with another spare to
maintain redundancy on your array while that disk was out.

> Usually if it is in the RAID zone, a check scrub will clear it up.
> Having said that I've had a very peculiar one here in the last couple of
> days.
> 
> A WD 2TB Green drive with TLER set to 7 seconds. The first read would
> error out in 7 seconds (as it should), but a second read succeeded.
> After returning the error, the drive must have kept trying to recover in
> the background and eventually succeeded and cached the result. So
> subsequent reads were ok. After reading and writing enough to other
> parts of the drive to flush the drives cache, the process would repeat.

Pure speculation.  Unless you can show better evidence that those drives
will cache a read in that case, I would say it was just a mild enough
weak spot that it randomly succeeded more than not.  And if you follow
my advice, it doesn't matter:  if the array is the only process reading
from the disk, the first appearance of the URE would be the last, as the
array would re-write it immediately.  Whether during a scrub or due to
normal access.

Regular long self-tests are highly recommended for stand-alone disks and
for array hot spares.

Phil