What would happen if the block device driver/firmware found some block of a bio is corrupted?

What would happen if the block device driver/firmware found some block of a bio is corrupted?

[Qu Wenruo]

Hi,

I'm wondering what would happen if we submit a read bio containing 
multiple sectors, while the block disk driver/firmware has internal 
checksum and found just one sector is corrupted (mismatch with its 
internal csum)?

For example, we submit a read bio sized 16KiB, and the device is in 4K 
sector size (like most modern HDD/SSD).
The corruption happens at the 2nd sector of the 16KiB.

My instinct points to either of them:

A) Mark the whole 16KiB bio as BLK_STS_IOERR
    This means even we have 3 good sectors, we have to treat them all as
    errors.

B) Ignore the error mark the bio as BLK_STS_OK
    This means higher layer must have extra ways to verify the contents.

But my concern is, if we go path A), it means after a read bio failure, 
we should try read again with much smaller block size, until we hit a 
failure with one sector.

IIRC VFS would do some retry, but otherwise the FS/driver layer needs to 
do some internal work and hit an error, then they need to do the 
split-and-retry manually.

On the other hand path B) seems more straightforward, but the problem is 
also obvious. Thankfully most fses are already doing checksum for their 
metadata at least.


So what's the common solution in real world for device drivers/firmware?
Path A/B or some other solution?

And should the upper layer do extra split-and-retry by themselves?
I know btrfs scrub code and repair is doing such split-and-retry, but 
not 100% sure if this is really needed or helpful in real world.

Thanks,
Qu

Re: What would happen if the block device driver/firmware found some block of a bio is corrupted?

[Keith Busch]

On Tue, Jan 24, 2023 at 10:31:47AM +0800, Qu Wenruo wrote:
> I'm wondering what would happen if we submit a read bio containing multiple
> sectors, while the block disk driver/firmware has internal checksum and
> found just one sector is corrupted (mismatch with its internal csum)?
> 
> For example, we submit a read bio sized 16KiB, and the device is in 4K
> sector size (like most modern HDD/SSD).
> The corruption happens at the 2nd sector of the 16KiB.
> 
> My instinct points to either of them:
> 
> A) Mark the whole 16KiB bio as BLK_STS_IOERR
>    This means even we have 3 good sectors, we have to treat them all as
>    errors.

I believe BLK_STS_MEDIUM is the appropriate status for this scenario,
not IOERR. The MEDIUM errno is propogated up as ENODATA.

Finding specific failed sectors makes sense if a partial of the
originally requested data is useful. That's application specific, so the
retry logic should probably be driven at a higher level than the block
layer based on seeing a MEDIUM error.

Some protocols can report partial transfers. If you trust the device,
you could know the first unreadable sector and retry from there.

Some protocols like NVMe optionally support querying which sectors are
not readable. We're not making use of that in kernel, but these kinds of
features exist if you need to know which LBAs to exclude for future
retries.

Outside that, you could search for specific unrecoverable LBAs with
split retries till you find them all, divide-and-conquer.

Re: What would happen if the block device driver/firmware found some block of a bio is corrupted?

[Qu Wenruo]

On 2023/1/24 12:44, Keith Busch wrote:
> On Tue, Jan 24, 2023 at 10:31:47AM +0800, Qu Wenruo wrote:
>> I'm wondering what would happen if we submit a read bio containing multiple
>> sectors, while the block disk driver/firmware has internal checksum and
>> found just one sector is corrupted (mismatch with its internal csum)?
>>
>> For example, we submit a read bio sized 16KiB, and the device is in 4K
>> sector size (like most modern HDD/SSD).
>> The corruption happens at the 2nd sector of the 16KiB.
>>
>> My instinct points to either of them:
>>
>> A) Mark the whole 16KiB bio as BLK_STS_IOERR
>>     This means even we have 3 good sectors, we have to treat them all as
>>     errors.
> 
> I believe BLK_STS_MEDIUM is the appropriate status for this scenario,
> not IOERR. The MEDIUM errno is propogated up as ENODATA.
> 
> Finding specific failed sectors makes sense if a partial of the
> originally requested data is useful. That's application specific, so the
> retry logic should probably be driven at a higher level than the block
> layer based on seeing a MEDIUM error.

Thanks a lot, that indeed makes more sense.

The retry for file read is indeed triggered inside VFS, not fs/block/dm 
layer itself.

Thanks,
Qu
> 
> Some protocols can report partial transfers. If you trust the device,
> you could know the first unreadable sector and retry from there.
> 
> Some protocols like NVMe optionally support querying which sectors are
> not readable. We're not making use of that in kernel, but these kinds of
> features exist if you need to know which LBAs to exclude for future
> retries.
> 
> Outside that, you could search for specific unrecoverable LBAs with
> split retries till you find them all, divide-and-conquer.

Re: What would happen if the block device driver/firmware found some block of a bio is corrupted?

[Christoph Hellwig]

On Tue, Jan 24, 2023 at 01:38:41PM +0800, Qu Wenruo wrote:
> The retry for file read is indeed triggered inside VFS, not fs/block/dm
> layer itself.

Well, it's really MM code.  If ->readahead fails, we eventually fall
back to a single-page ->radpage.  That might still be more than one
sector in some cases, but at least nicely narrows down the range.

Re: What would happen if the block device driver/firmware found some block of a bio is corrupted?

[Qu Wenruo]

On 2023/1/24 14:32, Christoph Hellwig wrote:
> On Tue, Jan 24, 2023 at 01:38:41PM +0800, Qu Wenruo wrote:
>> The retry for file read is indeed triggered inside VFS, not fs/block/dm
>> layer itself.
> 
> Well, it's really MM code.  If ->readahead fails, we eventually fall
> back to a single-page ->radpage.  That might still be more than one
> sector in some cases, but at least nicely narrows down the range.

This also means, if some internal work (like btrfs scrub) is not 
triggered by MM, then we have to do the split all by ourselves...

Thanks,
Qu

Re: What would happen if the block device driver/firmware found some block of a bio is corrupted?

[Christoph Hellwig]

On Tue, Jan 24, 2023 at 03:52:38PM +0800, Qu Wenruo wrote:
> This also means, if some internal work (like btrfs scrub) is not triggered
> by MM, then we have to do the split all by ourselves...

Yes.

Re: What would happen if the block device driver/firmware found some block of a bio is corrupted?

[Matthew Wilcox]

On Mon, Jan 23, 2023 at 10:32:06PM -0800, Christoph Hellwig wrote:
> On Tue, Jan 24, 2023 at 01:38:41PM +0800, Qu Wenruo wrote:
> > The retry for file read is indeed triggered inside VFS, not fs/block/dm
> > layer itself.
> 
> Well, it's really MM code.  If ->readahead fails, we eventually fall
> back to a single-page ->radpage.  That might still be more than one
> sector in some cases, but at least nicely narrows down the range.

I had code to split a large folio into single-page folios at one point,
but I don't believe that ever got merged.  At least, I can't find any
trace of it in filemap.c or iomap/buffered_io.c.  So I think we just
retry the ->read_folio() call each time.