* Re: Recovering from an URE on a RAID5 rebuild/resize
2013-01-25 11:14 Recovering from an URE on a RAID5 rebuild/resize Roman Mamedov
@ 2013-01-25 18:34 ` Stan Hoeppner
2013-01-25 20:28 ` Chris Murphy
2013-01-26 23:40 ` Phil Turmel
2 siblings, 0 replies; 6+ messages in thread
From: Stan Hoeppner @ 2013-01-25 18:34 UTC (permalink / raw)
To: Roman Mamedov; +Cc: linux-raid
On 1/25/2013 5:14 AM, Roman Mamedov wrote:
> Hello,
>
> Recently there has been some talk on this list, about probability of seeing an
> URE during a RAID5 rebuild on modern large (e.g. 2TB) drives.
>
> I would like to ask for some advice of what would be the best way to proceed
> when such an URE is encountered. This is mostly theoretical, no real situation
> at hand at the moment.
>
> As I understand, a RAID5 that is being resized or rebuilt, has no redundancy;
> it is essentially as reliable as a RAID0 of total members-1, or even less.
>
> So on an unreadable sector that mdadm needs to read (because it has no
> redundancy to recover it from), mdadm will:
>
> - mark the corresponding array member as "failed";
> - mark the one that was being rebuilt/resized onto as "spare";
> - and the whole array as down and "not enough members to start the array".
>
> Let's assume only a couple of sectors on that member were unreadable, and then
> their readability was restored (either by drive replacement or by overwriting
> them to making the drive remap), and I would be okay with losing data that was
> in those sectors.
>
> What would be the best way to proceed from there?
The best way? Use RAID6 and avoid the situation entirely.
"An ounce of prevention is worth a pound of cure", as they say.
--
Stan
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Recovering from an URE on a RAID5 rebuild/resize
2013-01-25 11:14 Recovering from an URE on a RAID5 rebuild/resize Roman Mamedov
2013-01-25 18:34 ` Stan Hoeppner
@ 2013-01-25 20:28 ` Chris Murphy
2013-01-25 20:43 ` Chris Murphy
2013-01-26 23:40 ` Phil Turmel
2 siblings, 1 reply; 6+ messages in thread
From: Chris Murphy @ 2013-01-25 20:28 UTC (permalink / raw)
To: linux-raid
On Jan 25, 2013, at 4:14 AM, Roman Mamedov <rm@romanrm.ru> wrote:
>
> Let's assume only a couple of sectors on that member were unreadable, and then
> their readability was restored (either by drive replacement or by overwriting
> them to making the drive remap), and I would be okay with losing data that was
> in those sectors.
>
> What would be the best way to proceed from there?
Seems to me if you lose a sector, you've lost a chunk, and if you lost a chunk you've lost a full stripe. So for a 512KB chunk, you've lost a few MB at least. It could be anything, files, directory information, a whole superblock.
I think you'd need a mechanism to ask md to ignore the reconstruction of a stripe containing a bad sector (the URE), and proceed with the next stripe rather than halting. That is, do not replace either data or parity chunks in a stripe which contains a bad sector, simply do not rebuild that full stripe. But, I don't know if "doing nothing" for that stripe is possible, if md can find the next stripe's first chunk and proceed on with the rebuild. Or if the rebuild has sequential dependency.
In any case, you'd have unknown problems above.
What you'd be better off with is a way to reconstruct that 512 bytes, and to do that you'd need to know what's on it. And to do that, well first you need to know the affected LBA (easy), then figure out how to get md to tell you what LBA's are in the affected stripe (no clue how to do this), and then ask the file system what things are stored in those LBAs, and if you can, replace them. Tedious to say the least, you have 1024 LBA's to sift through to find out what's affected by that one sector loss. Because md certainly has no idea what it is. And I bet dollar to donuts you can't look at that one sector (even if the drive let you, which it won't) and know what's there, just the nature of how it's encoded. Maybe you'd have a better chance.
In a way you'd be better off just blowing away that whole stripe, either from the file system level (force the reassemble of the array in degraded mode, and overwrite the affected file system LBAs, which will travel down and zero out that whole stripe and probably then some, and thus "fix" the read error); or if there were a feature in md to just write zeros to a whole stripe (data and parity chunks) affected by a URE. Now you have holes in your file system but maybe there's something to recapture.
Chris Murphy
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Recovering from an URE on a RAID5 rebuild/resize
2013-01-25 20:28 ` Chris Murphy
@ 2013-01-25 20:43 ` Chris Murphy
2013-01-25 20:50 ` Chris Murphy
0 siblings, 1 reply; 6+ messages in thread
From: Chris Murphy @ 2013-01-25 20:43 UTC (permalink / raw)
To: linux-raid
On Jan 25, 2013, at 1:28 PM, Chris Murphy <lists@colorremedies.com> wrote:
>
> In a way you'd be better off just blowing away that whole stripe
http://smartmontools.sourceforge.net/badblockhowto.html
So if you do a smartctl -t long on the affected drive, it will stop the test at the first bad sector and report the LBA. If you write zeros to that LBA only, the drive firmware will "fix" the problem, but now you have zeros for that sector.
Note how this document says part of the data in the file will be overwritten with zeros. Well, this is not a file, it's part of an md chunk, which is part of a stripe, then it's a file system, then it's a file. So by zeroing one sector, you've got ambiguous data in the whole stripe.
But, even if you have a totally wrong reconstruction on a whole stripe, at least the rest of the rebuild has a chance to proceed without further URE. So then it's an open question what was affected. If it's a movie, maybe you lose 5 seconds of video or sound, depending on how it's encoded. Depending the player's tolerances, maybe it crashes. Who knows.
Chris Murphy
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Recovering from an URE on a RAID5 rebuild/resize
2013-01-25 20:43 ` Chris Murphy
@ 2013-01-25 20:50 ` Chris Murphy
0 siblings, 0 replies; 6+ messages in thread
From: Chris Murphy @ 2013-01-25 20:50 UTC (permalink / raw)
To: linux-raid
On Jan 25, 2013, at 1:43 PM, Chris Murphy <lists@colorremedies.com> wrote:
> But, even if you have a totally wrong reconstruction on a whole stripe, at least the rest of the rebuild has a chance to proceed without further URE. So then it's an open question what was affected. If it's a movie, maybe you lose 5 seconds of video or sound, depending on how it's encoded. Depending the player's tolerances, maybe it crashes. Who knows.
And if it's file system, it may or may not be repairable with an fsck. So another unknown. Which means you may need to backup the whole thing (if you even can, which is uncertain if it's a bad file system corruption now); or trash it.
So my tentative conclusion is that if the data is valuable enough to go to this much effort to raise an array basically from the dead, it's worth backing up the data, and just starting it all from scratch and restoring it. If the data is important enough to be available in case of a URE during rebuild, then it needs to be a RAID6 or RAID10.
Chris Murphy
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Recovering from an URE on a RAID5 rebuild/resize
2013-01-25 11:14 Recovering from an URE on a RAID5 rebuild/resize Roman Mamedov
2013-01-25 18:34 ` Stan Hoeppner
2013-01-25 20:28 ` Chris Murphy
@ 2013-01-26 23:40 ` Phil Turmel
2 siblings, 0 replies; 6+ messages in thread
From: Phil Turmel @ 2013-01-26 23:40 UTC (permalink / raw)
To: Roman Mamedov; +Cc: linux-raid
On 01/25/2013 06:14 AM, Roman Mamedov wrote:
> Hello,
>
> Recently there has been some talk on this list, about probability of seeing an
> URE during a RAID5 rebuild on modern large (e.g. 2TB) drives.
>
> I would like to ask for some advice of what would be the best way to proceed
> when such an URE is encountered. This is mostly theoretical, no real situation
> at hand at the moment.
>
> As I understand, a RAID5 that is being resized or rebuilt, has no redundancy;
> it is essentially as reliable as a RAID0 of total members-1, or even less.
>
> So on an unreadable sector that mdadm needs to read (because it has no
> redundancy to recover it from), mdadm will:
>
> - mark the corresponding array member as "failed";
> - mark the one that was being rebuilt/resized onto as "spare";
> - and the whole array as down and "not enough members to start the array".
No. On modern kernels, you have to experience multiple read errors in a
short time (compile time constant 20, if I recall correctly) before the
device is failed. So for a single unrecoverable sector, or a small
number of them, the error will be passed to the filesystem, and possibly
on to the application.
> Let's assume only a couple of sectors on that member were unreadable, and then
> their readability was restored (either by drive replacement or by overwriting
> them to making the drive remap), and I would be okay with losing data that was
> in those sectors.
If you are in this situation, rewriting the files that contain the bad
sectors is an option, if the sectors are in a file at all. If they hold
filesystem metadata, you might lose more.
> What would be the best way to proceed from there?
1) With the array stopped, dd_rescue the array members onto new drives.
Allow bad sectors to be replaced with zeroes, possibly keep a record of
the bad sector locations. Set the original drives aside for later
forensics, if needed.
2) Start up with the new members. Add another new drive and allow the
rebuild to finish. fsck the filesystem and assess the corruption,
possibly rewriting files identified with the bad block data from (1).
3) Take one of the original drives, zero its superblock, add it to the
array, and reshape to raid6.
4) Use regular "check" scrubs with raid6 to never be in the situation again.
HTH,
Phil
^ permalink raw reply [flat|nested] 6+ messages in thread