Re: [PATCH v2 5/5] dax: handle media errors in dax_do_io

["Verma, Vishal L" <vishal.l.verma@intel.com>]

On Tue, 2016-04-26 at 09:25 +1000, Dave Chinner wrote:
> 
<>

> > 
> > - It checks badblocks and discovers it's files have lost data
> Lots of hand-waving here. How does the application map a bad
> "sector" to a file without scanning the entire filesystem to find
> the owner of the bad sector?

Yes this was hand-wavey, but we talked about this a bit at LSF..
The idea is that a per-block-device badblocks list is available at
/sys/block/<pmemX>/badblocks. The application (or a suitable yet-to-be-
written library function) does a fiemap to figure out the sectors its
files are using, and correlates the two lists.
We can also look into providing an easier-to-use interface from the
kernel, in the form of an fiemap flag to report only the bad sectors, or
a SEEK_BAD flag..
The application doesn't have to scan the entire filesystem, but
presumably it knows what files it 'owns', and does a fiemap for those.

> 
> > 
> > - It write()s those sectors (possibly converted to file offsets
> > using
> > fiemap)
> >     * This triggers the fallback path, but if the application is
> > doing
> > this level of recovery, it will know the sector is bad, and write
> > the
> > entire sector
> Where does the application find the data that was lost to be able to
> rewrite it?

The data that was lost is gone -- this assumes the application has some
ability to recover using a journal/log or other redundancy - yes, at the
application layer. If it doesn't have this sort of capability, the only
option is to restore files from a backup/mirror.

> 
> > 
> > - Or it replaces the entire file from backup also using write() (not
> > mmap+stores)
> >     * This just frees the fs block, and the next time the block is
> > reallocated by the fs, it will likely be zeroed first, and that will
> > be
> > done through the driver and will clear errors
> There's an implicit assumption that applications will keep redundant
> copies of their data at the /application layer/ and be able to
> automatically repair it? And then there's the implicit assumption
> that it will unlink and free the entire file before writing a new
> copy, and that then assumes the the filesystem will zero blocks if
> they get reused to clear errors on that LBA sector mapping before
> they are accessible again to userspace..
> 
> It seems to me that there are a number of assumptions being made
> across multiple layers here. Maybe I've missed something - can you
> point me to the design/architecture description so I can see how
> "app does data recovery itself" dance is supposed to work?

There isn't a document other than the flow in my head :) - but maybe I
could write one up..
I wasn't thinking the application itself maintains and restores from
backup copy of the file.. The application hits either a SIGBUS or EIO
depending on how it accesses the data, and crashes or raises some alarm.
The recovery is then done out-of-band, by a sysadmin or such (i.e.
delete the file, replace with a known good copy, restart application).

To summarize, the two cases we want to handle are:
1. Application has inbuilt recovery:
  - hits badblock
  - figures out it is able to recover the data
  - handles SIGBUS or EIO
  - does a (sector aligned) write() to restore the data
2. Application doesn't have any inbuilt recovery mechanism
  - hits badblock
  - gets SIGBUS (or EIO) and crashes
  - Sysadmin restores file from backup

Case 1 is handled by either a fallback to direct_IO from dax_do_io, or
always _actually_ doing direct_IO when we're opened with O_DIRECT in
spite of dax (what Dan suggested). Currently if we're mounted with dax,
all IO O_DIRECT or otherwise will go through dax_do_io.
Case 2 is handled by patch 4 of the series:
    dax: use sb_issue_zerout instead of calling dax_clear_sectors

> 
> Cheers,
> 
> Dave.