Re: [PATCH v3 10/13] fstests: crash consistency fsx test using dm-log-writes

[Josef Bacik <josef@toxicpanda.com>]

On Tue, Nov 28, 2017 at 06:48:43PM +0200, Amir Goldstein wrote:
> On Mon, Nov 27, 2017 at 5:04 PM, Josef Bacik <josef@toxicpanda.com> wrote:
> > On Mon, Nov 27, 2017 at 11:56:58AM +0200, Amir Goldstein wrote:
> >> On Tue, Sep 5, 2017 at 10:11 PM, Amir Goldstein <amir73il@gmail.com> wrote:
> >> > Cherry-picked the test from commit 70d41e17164b
> >> > in Josef Bacik's fstests tree (https://github.com/josefbacik/fstests).
> >> > Quoting from Josef's commit message:
> >> >
> >> >   The test just runs some ops and exits, then finds all of the good buffers
> >> >   in the directory we provided and:
> >> >   - replays up to the mark given
> >> >   - mounts the file system and compares the md5sum
> >> >   - unmounts and fsck's to check for metadata integrity
> >> >
> >> >   dm-log-writes will pretend to do discard and the replay-log tool will
> >> >   replay it properly depending on the underlying device, either by writing
> >> >   0's or actually calling the discard ioctl, so I've enabled discard in the
> >> >   test for maximum fun.
> >> >
> >> > [Amir:]
> >> > - Removed unneeded _test_falloc_support dynamic FSX_OPTS
> >> > - Fold repetitions into for loops
> >> > - Added place holders for using constant random seeds
> >> > - Add pre umount checkpint
> >> > - Add test to new 'replay' group
> >> > - Address review comments by Eryu Guan
> >> >
> >> > Cc: Josef Bacik <jbacik@fb.com>
> >> > Signed-off-by: Amir Goldstein <amir73il@gmail.com>
> >>
> >>
> >> Josef,
> >>
> >> As you know, this test is now merged to xfstest as generic/455.
> >> I have been running the test for a while on xfs and it occasionally
> >> reports inconsistencies which I try to investigate.
> >>
> >> In some of the reports, it appears that dm-log-writes may be exhibiting
> >> a reliability issue (see below).
> >>
> >
> > It's not a reliability issue, its a caching issue.  dm-log-writes is just
> > issuing bio's to the log device, and our destructor waits for all pending io
> > blocks to complete before exiting, so unless I've missed how dm is destroying
> > devices everything should be on disk.
> >
> > However since we replay in userspace we are going through the blockdevice's
> > pagecache, so we could have stale pages left in place which is screwing us up.
> > Will you try this patch and see if it fixes the problem?  Thanks,
> >
> > Josef
> >
> >
> > diff --git a/drivers/md/dm-log-writes.c b/drivers/md/dm-log-writes.c
> > index 8b80a9ce9ea9..1c502930af5e 100644
> > --- a/drivers/md/dm-log-writes.c
> > +++ b/drivers/md/dm-log-writes.c
> > @@ -545,6 +545,8 @@ static void log_writes_dtr(struct dm_target *ti)
> >                    !atomic_read(&lc->pending_blocks));
> >         kthread_stop(lc->log_kthread);
> >
> > +       invalidate_bdev(lc->logdev->bdev);
> > +       invalidate_bdev(lc->dev->bdev);
> >         WARN_ON(!list_empty(&lc->logging_blocks));
> >         WARN_ON(!list_empty(&lc->unflushed_blocks));
> >         dm_put_device(ti, lc->dev);
> 
> Josef,
> 
> With your patch OR with my xfstest patch that adds "sync" I did not yet see
> another problem of garbage fs after _log_writes_remove.
> 
> I did however, encounter this error (failure to verify read data during fsx)
> from scratch/log-writes device (see attached full log).
> 
> I will keep running the test to collect more information.
> 

That failure I'll lay at the feet of whatever fs you are testing ;).  I'm glad
my patch fixed the replay problem, I'll send that up.  Thanks,

Josef