Re: [PATCH] [16/19] HWPOISON: Enable .remove_error_page for migration aware file systems

Hidehiro Kawai
2009-08-11T12:00:37Z

Andi Kleen wrote:

>>1. An uncorrected error on a dirty page cache page is detected by
>> memory scrubbing
>>2. Kernel unmaps and truncates the page to recover from the error
>>3. An application reads data from the file location corresponding
>> to the truncated page
>> ==> Old or garbage data will be read into a new page cache page
>
> The problem currently is that the error is not sticky enough and
> doesn't stay around long enough. It gets reported once,
> but not in later IO operations.
>
> However it's a generic problem not unique to hwpoison. Me

Yes, it's a generic problem, and introducing a sticky error flag
is one of the approach to solve the problem. I think it is a good
approach because it doesn't depend on individual filesystems.

> And application
> that doesn't handle current IO errors correctly will also
> not necessarily handle hwpoison correctly (it's not better and not worse)

This is my main concern. I'd like to prevent re-corruption even if
applications don't have good manners.

As for usual I/O error, ext3/4 can now do it by using data=ordered and
data_err=abort mount options. Moreover, if you mount the ext3/4
filesystem with the additional errors=panic option, kernel gets
panic on write error instead of read-only remount. Customers
who regard data integrity is very important require these features.

But this patch (PATCH 16/19) introduce this problem again, because
it doesn't provide a way to shut out further writes to the fs.
Of course, we can do it by setting tolerant level to 0 or
memory_failure_recovery to 0. But it would be overkill.
That is why I suggested this:
>>(2) merge this patch with new panic_on_dirty_page_cache_corruption
>> sysctl

> That is something that could be improved in the VFS -- although I fear
> any improvements here could also break compatibility. I don't think
> it's a blocker on hwpoison for now. It needs more design
> effort and thinking (e.g. likely the address space IO error
> bit should be separated into multiple bits)
>
> Perhaps you're interested in working on this?

Yes. Transient IO errors have a potential for causing re-corruption
problem. Now ext3/4 provide ways to prevent it, but not the other
filesystems. We would need a generic way.

>>4. The application modifies the data and write back it to the disk
>>5. The file will corrurpt!
>>
>>(Yes, the application is wrong to not do the right thing, i.e. fsync,
>> but it's not user's fault!)
>>
>>A similar data corruption can be caused by a write I/O error,
>>because dirty flag is cleared even if the page couldn't be written
>>to the disk.
>>
>>However, we have a way to avoid this kind of data corruption at
>>least for ext3. If we mount an ext3 filesystem with data=ordered
>>and data_err=abort, all I/O errors on file data block belonging to
>>the committing transaction are checked. When I/O error is found,
>>abort journaling and remount the filesystem with read-only to
>>prevent further updates. This kind of feature is very important
>>for mission critical systems.
>
> Well it sounds like a potentially useful enhancement to ext3 (or ext4).
>
> One issue is that the default is not ordered anymore since
> Linus changed the default.

Yes, but what is important is whether the system provides
such feature or not.

> I'm sure other enhancements for IO errors could be done too.
> Some of the file systems also handle them still quite poorly (e.g. btrfs)
>
> But again I don't think it's a blocker for hwpoison.

Unfortunately, it can be a blocker. As I stated, we can block the
possible re-corruption caused by transient IO errors on ext3/4
filesystems. But applying this patch (PATCH 16/19), re-corruption
can happen even if we use data=ordered, data_err=abort and
errors=panic mount options.

So...

>>I think there are three options,
>>
>>(1) drop this patch
>>(2) merge this patch with new panic_on_dirty_page_cache_corruption
>> sysctl
>>(3) implement a more sophisticated error_remove_page function
>
> (4) accept that hwpoison error handling is not better and not worse than normal
> IO error handling.
>
> We opted for (4).

Could you consider adopting (2) or (3)? Fengguang's sticky EIO
approach (http://lkml.org/lkml/2009/6/11/294) is also OK.
I hope HWPOISON patches are merged into 2.6.32. So (2) is the
best answer for me, because it's simple and less intrusive.

Thanks,
--
Hidehiro Kawai
Hitachi, Systems Development Laboratory
Linux Technology Center