raid1 narrow_write_error with 4K disks, sd "bad block number requested" messages

raid1 narrow_write_error with 4K disks, sd "bad block number requested" messages

[Nate Dailey]

I'm writing about something that appears to be an issue with raid1's 
narrow_write_error, particular to non-512-byte-sector disks. Here's what 
I'm doing:

- 2 disk raid1, 4K disks, each connected to a different SAS HBA
- mount a filesystem on the raid1, run a test that writes to it
- remove one of the SAS HBAs (echo 1 > 
/sys/bus/pci/devices/0000\:45\:00.0/remove)

At this point, writes fail and narrow_write_error breaks them up and 
retries, one sector at a time. But these are 512-byte sectors, and sd 
doesn't like it:

[ 2645.310517] sd 3:0:1:0: [sde] Bad block number requested
[ 2645.310610] sd 3:0:1:0: [sde] Bad block number requested
[ 2645.310690] sd 3:0:1:0: [sde] Bad block number requested
...

There appears to be no real harm done, but there can be a huge number of 
these messages in the log.

I can avoid this by disabling bad block tracking, but it looks like 
maybe the superblock's bblog_shift is intended to address this exact 
issue. However, I don't see a way to change it. Presumably this is 
something mdadm should be setting up? I don't see bblog_shift ever set 
to anything other than 0.

This is on a RHEL 7.1 kernel, version 3.10.0-221.el7. I took a look at 
upstream sd and md changes and nothing jumps out at me that would have 
affected this (but I have not tested to see if the bad block messages do 
or do not happen on an upstream kernel).

I'd appreciate any advice re: how to handle this. Thanks!

Nate Dailey
Stratus Technologies

Re: raid1 narrow_write_error with 4K disks, sd "bad block number requested" messages

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1942 bytes --]

On Wed, 28 Jan 2015 10:29:46 -0500 Nate Dailey <nate.dailey@stratus.com>
wrote:

> I'm writing about something that appears to be an issue with raid1's 
> narrow_write_error, particular to non-512-byte-sector disks. Here's what 
> I'm doing:
> 
> - 2 disk raid1, 4K disks, each connected to a different SAS HBA
> - mount a filesystem on the raid1, run a test that writes to it
> - remove one of the SAS HBAs (echo 1 > 
> /sys/bus/pci/devices/0000\:45\:00.0/remove)
> 
> At this point, writes fail and narrow_write_error breaks them up and 
> retries, one sector at a time. But these are 512-byte sectors, and sd 
> doesn't like it:
> 
> [ 2645.310517] sd 3:0:1:0: [sde] Bad block number requested
> [ 2645.310610] sd 3:0:1:0: [sde] Bad block number requested
> [ 2645.310690] sd 3:0:1:0: [sde] Bad block number requested
> ...
> 
> There appears to be no real harm done, but there can be a huge number of 
> these messages in the log.
> 
> I can avoid this by disabling bad block tracking, but it looks like 
> maybe the superblock's bblog_shift is intended to address this exact 
> issue. However, I don't see a way to change it. Presumably this is 
> something mdadm should be setting up? I don't see bblog_shift ever set 
> to anything other than 0.
> 
> This is on a RHEL 7.1 kernel, version 3.10.0-221.el7. I took a look at 
> upstream sd and md changes and nothing jumps out at me that would have 
> affected this (but I have not tested to see if the bad block messages do 
> or do not happen on an upstream kernel).
> 
> I'd appreciate any advice re: how to handle this. Thanks!

Thanks for the report.

narrow_write_error() should use bdev_logical_block_size() and round up to
that.
Possibly mdadm should get the same information and set bblog_shift
accordingly when creating a bad block log.

I've made a note to fix that, but I'm happy to review  patches too :-)

thanks,
NeilBrown


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]

Re: raid1 narrow_write_error with 4K disks, sd "bad block number requested" messages

[Nate Dailey]

On 02/04/2015 11:59 PM, NeilBrown wrote:
> On Wed, 28 Jan 2015 10:29:46 -0500 Nate Dailey <nate.dailey@stratus.com>
> wrote:
>
>> I'm writing about something that appears to be an issue with raid1's
>> narrow_write_error, particular to non-512-byte-sector disks. Here's what
>> I'm doing:
>>
>> - 2 disk raid1, 4K disks, each connected to a different SAS HBA
>> - mount a filesystem on the raid1, run a test that writes to it
>> - remove one of the SAS HBAs (echo 1 >
>> /sys/bus/pci/devices/0000\:45\:00.0/remove)
>>
>> At this point, writes fail and narrow_write_error breaks them up and
>> retries, one sector at a time. But these are 512-byte sectors, and sd
>> doesn't like it:
>>
>> [ 2645.310517] sd 3:0:1:0: [sde] Bad block number requested
>> [ 2645.310610] sd 3:0:1:0: [sde] Bad block number requested
>> [ 2645.310690] sd 3:0:1:0: [sde] Bad block number requested
>> ...
>>
>> There appears to be no real harm done, but there can be a huge number of
>> these messages in the log.
>>
>> I can avoid this by disabling bad block tracking, but it looks like
>> maybe the superblock's bblog_shift is intended to address this exact
>> issue. However, I don't see a way to change it. Presumably this is
>> something mdadm should be setting up? I don't see bblog_shift ever set
>> to anything other than 0.
>>
>> This is on a RHEL 7.1 kernel, version 3.10.0-221.el7. I took a look at
>> upstream sd and md changes and nothing jumps out at me that would have
>> affected this (but I have not tested to see if the bad block messages do
>> or do not happen on an upstream kernel).
>>
>> I'd appreciate any advice re: how to handle this. Thanks!
>
> Thanks for the report.
>
> narrow_write_error() should use bdev_logical_block_size() and round up to
> that.
> Possibly mdadm should get the same information and set bblog_shift
> accordingly when creating a bad block log.
>
> I've made a note to fix that, but I'm happy to review  patches too :-)
>
> thanks,
> NeilBrown
>

I will post a narrow_write_error patch shortly.

I did some experimentation with setting the bblog_shift in mdadm, but it 
didn't work out the way I expected. It turns out that the value is only 
loaded from the superblock if:

1453        if ((le32_to_cpu(sb->feature_map) & MD_FEATURE_BAD_BLOCKS) &&
1454            rdev->badblocks.count == 0) {
...
1473                rdev->badblocks.shift = sb->bblog_shift;

And this feature bit is only set if any bad blocks have actually been 
recorded.

It also appears to me that the shift is used when loading the bad blocks 
from the superblock, but not when storing the bad block list in the 
superblock.

Seems like these are bugs, but I'm not certain how the code is supposed 
to work (and am getting in a bit over my head with this).

In any case, it doesn't appear to me that there's any harm in having the 
bblog_shift not match the disk's block size (right?).

Nate Dailey

Re: raid1 narrow_write_error with 4K disks, sd "bad block number requested" messages

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 3813 bytes --]

On Thu, 12 Feb 2015 11:46:21 -0500 Nate Dailey <nate.dailey@stratus.com>
wrote:

> On 02/04/2015 11:59 PM, NeilBrown wrote:
> > On Wed, 28 Jan 2015 10:29:46 -0500 Nate Dailey <nate.dailey@stratus.com>
> > wrote:
> >
> >> I'm writing about something that appears to be an issue with raid1's
> >> narrow_write_error, particular to non-512-byte-sector disks. Here's what
> >> I'm doing:
> >>
> >> - 2 disk raid1, 4K disks, each connected to a different SAS HBA
> >> - mount a filesystem on the raid1, run a test that writes to it
> >> - remove one of the SAS HBAs (echo 1 >
> >> /sys/bus/pci/devices/0000\:45\:00.0/remove)
> >>
> >> At this point, writes fail and narrow_write_error breaks them up and
> >> retries, one sector at a time. But these are 512-byte sectors, and sd
> >> doesn't like it:
> >>
> >> [ 2645.310517] sd 3:0:1:0: [sde] Bad block number requested
> >> [ 2645.310610] sd 3:0:1:0: [sde] Bad block number requested
> >> [ 2645.310690] sd 3:0:1:0: [sde] Bad block number requested
> >> ...
> >>
> >> There appears to be no real harm done, but there can be a huge number of
> >> these messages in the log.
> >>
> >> I can avoid this by disabling bad block tracking, but it looks like
> >> maybe the superblock's bblog_shift is intended to address this exact
> >> issue. However, I don't see a way to change it. Presumably this is
> >> something mdadm should be setting up? I don't see bblog_shift ever set
> >> to anything other than 0.
> >>
> >> This is on a RHEL 7.1 kernel, version 3.10.0-221.el7. I took a look at
> >> upstream sd and md changes and nothing jumps out at me that would have
> >> affected this (but I have not tested to see if the bad block messages do
> >> or do not happen on an upstream kernel).
> >>
> >> I'd appreciate any advice re: how to handle this. Thanks!
> >
> > Thanks for the report.
> >
> > narrow_write_error() should use bdev_logical_block_size() and round up to
> > that.
> > Possibly mdadm should get the same information and set bblog_shift
> > accordingly when creating a bad block log.
> >
> > I've made a note to fix that, but I'm happy to review  patches too :-)
> >
> > thanks,
> > NeilBrown
> >
> 
> I will post a narrow_write_error patch shortly.
> 
> I did some experimentation with setting the bblog_shift in mdadm, but it 
> didn't work out the way I expected. It turns out that the value is only 
> loaded from the superblock if:
> 
> 1453        if ((le32_to_cpu(sb->feature_map) & MD_FEATURE_BAD_BLOCKS) &&
> 1454            rdev->badblocks.count == 0) {
> ...
> 1473                rdev->badblocks.shift = sb->bblog_shift;
> 
> And this feature bit is only set if any bad blocks have actually been 
> recorded.
> 
> It also appears to me that the shift is used when loading the bad blocks 
> from the superblock, but not when storing the bad block list in the 
> superblock.
> 
> Seems like these are bugs, but I'm not certain how the code is supposed 
> to work (and am getting in a bit over my head with this).

Yes, that's probably a bug.
The

	} else if (sb->bblog_offset != 0)
		rdev->badblocks.shift = 0;

should be

	} else if (sb->bblog_offset != 0)
		rdev->badblocks.shift = sb->bblog_shift;

> 
> In any case, it doesn't appear to me that there's any harm in having the 
> bblog_shift not match the disk's block size (right?).

Having the bblog_shift larger than the disk's block size certainly should not
be a problem.  Having it small only causes the problem that you have already
discovered.

NeilBrown


> 
> Nate Dailey
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]