Re: [PATCH v2] generic: disable dmlogwrites tests on XFS

Brian Foster
2020-09-01T12:38:23Z

On Tue, Sep 01, 2020 at 09:25:42AM +0300, Amir Goldstein wrote:
> On Mon, Aug 31, 2020 at 4:37 PM Brian Foster <bfoster@redhat.com> wrote:
> >
> > On Sat, Aug 29, 2020 at 07:48:50AM +0100, Christoph Hellwig wrote:
> > > On Thu, Aug 27, 2020 at 10:53:29AM -0400, Brian Foster wrote:
> > > > Several generic fstests use dm-log-writes to test the filesystem for
> > > > consistency at various crash recovery points. dm-log-writes and the
> > > > associated replay mechanism rely on discard to clear stale blocks
> > > > when moving to various points in time of the fs. If the storage
> > > > doesn't provide discard zeroing or the discard requests exceed the
> > > > hardcoded maximum (128MB) of the fallback solution to physically
> > > > write zeroes, stale blocks are left around in the target fs. This
> > > > causes issues on XFS if recovery observes metadata from a future
> > > > version of an fs that has been replayed to an older point in time.
> > > > This corrupts the filesystem and leads to spurious test failures
> > > > that are nontrivial to diagnose.
> > > >
> > > > Disable the generic dmlogwrites tests on XFS for the time being.
> > > > This is intended to be a temporary change until a solution is found
> > > > that allows these tests to predictably clear stale data while still
> > > > allowing them to run in a reasonable amount of time.
> > >
> > > As said in the other discussion I don't think this is correct. The
> > > intent of the tests is to ensure the data can't be read. You just
> > > happen to trigger over that with XFS, but it also means that tests
> > > don't work correctly on other file systems in that configuration.
> > >
> >
> > Yes, but the goal of this patch is not to completely fix the dmlogwrites
> > infrastructure and set of tests. The goal is to disable a subset of
> > tests that are known to produce spurious corruptions on XFS until that
> > issue can be addressed, so it doesn't result in continued bug reports in
> > the meantime. I don't run these tests routinely on other fs', so it's
> > not really my place to decide that the tradeoff between this problem and
> > the ability of the test to reproduce legitimate bugs justifies disabling
> > the test on those configs.
> >
>
> Brian,
>
> Let's not take this course please.
> Please post patches v1 2/4-4/4 without patch v1 1/4
> The only objection was to patch 1/4 and it is not strictly needed
> to solve the problem you care about.
>

Sure, I'm fine with that approach. I posted this because it was
essentially where I started and there doesn't seem to be much agreement
on a proper fix. We had precedent to use dm-thinp in generic/482, so it
seemed ideal to at least try to keep the tests active in the near term.
IOW, my approach was to either try for an obvious/simple fix to keep the
tests active or otherwise disable the tests (at least on XFS) until a
more involved fix is agreed on, tested and implemented. That more
involved fix could be anything from genericizing the dm-thin approach to
replacing it such that discard zeroing is not a critical component, but
I don't want to gate addressing the spurious corruption problem on a
nontrivial rework of the test mechanism.

> I had a *concern* about pacthes 2-4, but I can live with that
> concern and it is certainly preferred to disabling the tests.
>

Agree.

> I can follow up with fixing the dmlogwrites common helpers
> later when I get the time, so they do not rely on discard for
> correctness of replay.
>
> As I wrote, all it takes is to issue an explicit zero/punch command
> in the beginning of replay halpers. Just need to find the command
> that works correctly and most efficiently with thinp.
>
> If you have the time to do that (since I believe you already tested
> some commands) that would be great. Otherwise, I'll do that later.
>

Well, I was testing some of the zeroing commands Christoph mentioned
moreso than punch, particularly with intent to remove the dependency on
dm-thin. The problem with that is I don't think it helps much for
anybody who is testing on devices without hardware offload. The more
efficient in-kernel zeroing is still pretty slow, so I suppose it
depends on how often it must be invoked in a particular test (once? per
recovery point? on-demand buried down in the recovery code?). An obvious
mitigation, at least for the generic tests, is to reduce the size of the
target devices such that the manual zeroing is less noticeable. I don't
see much impact of using a 100MB fs, for example, and the recovery tool
already does the manual zeroing. The tradeoff is that I think we'd want
some kind of _notrun check in situations where we know the zeroing
doesn't occur.

I don't see much difference with zero/punch on dm-thinp. An
fallocate(FL_PUNCH_HOLE|FL_KEEP_SIZE) doesn't work because it explicitly
requests hardware zeroing, which I don't have.
fallocate(FL_ZERO_RANGE|FL_KEEP_SIZE) works, but takes a minute or two
on my 10G device because it falls back to manual zeroing. There is a
NO_HIDE_STALE variant of PUNCH_HOLE, but I don't seem to have any
userspace tools that define NO_HIDE_STALE and it looks like it just
sends discards anyways. Of course, a 'blkdiscard -o 0 -l 10g <thindev>'
unmaps nearly the entire device in ~1s, but then we're back to the
argument of using discard for zeroing. :P

Brian

> Thanks,
> Amir.
>