Re: [PATCH v2 00/14] Crash consistency xfstest using dm-log-writes

Re: [PATCH v2 00/14] Crash consistency xfstest using dm-log-writes

[Amir Goldstein]

Sorry noise xfs list, I meant to CC fsdevel

On Wed, Aug 30, 2017 at 5:51 PM, Amir Goldstein <amir73il@gmail.com> wrote:
> Hi all,
>
> This is the 2nd revision of crash consistency patch set.
> The main thing that changed since v1 is my confidence in the failures
> reported by the test, along with some more debugging options for
> running the test tools.
>
> I've collected these patches that have been sitting in Josef Bacik's
> tree for a few years and kicked them a bit into shape.
> The dm-log-writes target has been merged to kernel v4.1, see:
> https://github.com/torvalds/linux/blob/master/Documentation/device-mapper/log-writes.txt
>
> For this posting, I kept the random seeds constant for the test.
> I set these constant seeds after running with random seed for a little
> while and getting failure reports. With the current values in the test
> I was able to reproduce at high probablity failures with xfs, ext4 and btrfs.
> The probablity of reproducing the failure is higher on a spinning disk.
>
> For xfs, I posted a fix for potential data loss post fsync+crash.
> For ext4, I posted a reliable reproducer using dm-flakey.
> For btrfs, I shared the recorded log with Josef.
>
> There is an outstanding problem with the test - when I run it with
> kvm-xfstests, the test halts and I get soft lockup of log_writes_kthread.
> I suppose its a bug in dm-log-writes with some kernel config or with virtio
> I wasn't able to determine the reason and have little time to debug this.
>
> Since dm-log-writes is anyway in upstream kernel, I don't think a bug
> in dm-log-writes for a certain config is a reason to block this xfstest
> from being merged.
> Anyway, I would be glad if someone could take a look at the soft lockup
> issue. Josef?
>
> Thanks,
> Amir.
>
> Amir Goldstein (14):
>   common/rc: convert some egrep to grep
>   common/rc: fix _require_xfs_io_command params check
>   fsx: fixes to random seed
>   fsx: fix path of .fsx* files
>   fsx: fix compile warnings
>   fsx: add support for integrity check with dm-log-writes target
>   fsx: add optional logid prefix to log messages
>   fsx: add support for --record-ops
>   fsx: add support for -g filldata
>   log-writes: add replay-log program to replay dm-log-writes target
>   replay-log: output log replay offset in verbose mode
>   replay-log: add support for replaying ops in target device range
>   fstests: add support for working with dm-log-writes target
>   fstests: add crash consistency fsx test using dm-log-writes
>
>  .gitignore                   |   1 +
>  README                       |   2 +
>  common/dmlogwrites           |  84 +++++++++
>  common/rc                    |  15 +-
>  doc/auxiliary-programs.txt   |   8 +
>  doc/requirement-checking.txt |  20 +++
>  ltp/fsx.c                    | 220 +++++++++++++++++++----
>  src/Makefile                 |   2 +-
>  src/log-writes/Makefile      |  23 +++
>  src/log-writes/SOURCE        |   6 +
>  src/log-writes/log-writes.c  | 410 +++++++++++++++++++++++++++++++++++++++++++
>  src/log-writes/log-writes.h  |  72 ++++++++
>  src/log-writes/replay-log.c  | 381 ++++++++++++++++++++++++++++++++++++++++
>  tests/generic/500            | 138 +++++++++++++++
>  tests/generic/500.out        |   2 +
>  tests/generic/group          |   1 +
>  16 files changed, 1340 insertions(+), 45 deletions(-)
>  create mode 100644 common/dmlogwrites
>  create mode 100644 src/log-writes/Makefile
>  create mode 100644 src/log-writes/SOURCE
>  create mode 100644 src/log-writes/log-writes.c
>  create mode 100644 src/log-writes/log-writes.h
>  create mode 100644 src/log-writes/replay-log.c
>  create mode 100755 tests/generic/500
>  create mode 100644 tests/generic/500.out
>
> --
> 2.7.4
>

Re: [PATCH v2 00/14] Crash consistency xfstest using dm-log-writes

[Josef Bacik]

On Wed, Aug 30, 2017 at 06:04:26PM +0300, Amir Goldstein wrote:
> Sorry noise xfs list, I meant to CC fsdevel
> 
> On Wed, Aug 30, 2017 at 5:51 PM, Amir Goldstein <amir73il@gmail.com> wrote:
> > Hi all,
> >
> > This is the 2nd revision of crash consistency patch set.
> > The main thing that changed since v1 is my confidence in the failures
> > reported by the test, along with some more debugging options for
> > running the test tools.
> >
> > I've collected these patches that have been sitting in Josef Bacik's
> > tree for a few years and kicked them a bit into shape.
> > The dm-log-writes target has been merged to kernel v4.1, see:
> > https://github.com/torvalds/linux/blob/master/Documentation/device-mapper/log-writes.txt
> >
> > For this posting, I kept the random seeds constant for the test.
> > I set these constant seeds after running with random seed for a little
> > while and getting failure reports. With the current values in the test
> > I was able to reproduce at high probablity failures with xfs, ext4 and btrfs.
> > The probablity of reproducing the failure is higher on a spinning disk.
> >

I'd rather we make it as evil as possible.  As long as we're printing out the
seed that was used in the output then we can go in and manually change the test
to use the same seed over and over again if we need to debug a problem.

> > For xfs, I posted a fix for potential data loss post fsync+crash.
> > For ext4, I posted a reliable reproducer using dm-flakey.
> > For btrfs, I shared the recorded log with Josef.
> >

I posted a patch to fix the problem you reported by the way, but my
git-send-email thing isn't set to cc people in the commit, sorry about that.

> > There is an outstanding problem with the test - when I run it with
> > kvm-xfstests, the test halts and I get soft lockup of log_writes_kthread.
> > I suppose its a bug in dm-log-writes with some kernel config or with virtio
> > I wasn't able to determine the reason and have little time to debug this.
> >
> > Since dm-log-writes is anyway in upstream kernel, I don't think a bug
> > in dm-log-writes for a certain config is a reason to block this xfstest
> > from being merged.
> > Anyway, I would be glad if someone could take a look at the soft lockup
> > issue. Josef?
> >

Yeah can you give this a try and see if the soft lockup goes away?

diff --git a/drivers/md/dm-log-writes.c b/drivers/md/dm-log-writes.c
index a1da0eb..b900758 100644
--- a/drivers/md/dm-log-writes.c
+++ b/drivers/md/dm-log-writes.c
@@ -345,6 +345,7 @@ static int log_writes_kthread(void *arg)
 		struct pending_block *block = NULL;
 		int ret;
 
+		cond_resched();
 		spin_lock_irq(&lc->blocks_lock);
 		if (!list_empty(&lc->logging_blocks)) {
 			block = list_first_entry(&lc->logging_blocks,

Re: [PATCH v2 00/14] Crash consistency xfstest using dm-log-writes

[Amir Goldstein]

On Wed, Aug 30, 2017 at 6:23 PM, Josef Bacik <josef@toxicpanda.com> wrote:
> On Wed, Aug 30, 2017 at 06:04:26PM +0300, Amir Goldstein wrote:
>> Sorry noise xfs list, I meant to CC fsdevel
>>
>> On Wed, Aug 30, 2017 at 5:51 PM, Amir Goldstein <amir73il@gmail.com> wrote:
>> > Hi all,
>> >
>> > This is the 2nd revision of crash consistency patch set.
>> > The main thing that changed since v1 is my confidence in the failures
>> > reported by the test, along with some more debugging options for
>> > running the test tools.
>> >
>> > I've collected these patches that have been sitting in Josef Bacik's
>> > tree for a few years and kicked them a bit into shape.
>> > The dm-log-writes target has been merged to kernel v4.1, see:
>> > https://github.com/torvalds/linux/blob/master/Documentation/device-mapper/log-writes.txt
>> >
>> > For this posting, I kept the random seeds constant for the test.
>> > I set these constant seeds after running with random seed for a little
>> > while and getting failure reports. With the current values in the test
>> > I was able to reproduce at high probablity failures with xfs, ext4 and btrfs.
>> > The probablity of reproducing the failure is higher on a spinning disk.
>> >
>
> I'd rather we make it as evil as possible.  As long as we're printing out the
> seed that was used in the output then we can go in and manually change the test
> to use the same seed over and over again if we need to debug a problem.

Yeh that's what I did, but then I found values that reproduce a problem,
so maybe its worth clinging on to these values now until the bugs are fixed in
upstream and then as regression tests.

Anyway, I can keep these presets commented out, or run the test twice,
once with presets and once with random seed, whatever Eryu decides.


>
>> > There is an outstanding problem with the test - when I run it with
>> > kvm-xfstests, the test halts and I get soft lockup of log_writes_kthread.
>> > I suppose its a bug in dm-log-writes with some kernel config or with virtio
>> > I wasn't able to determine the reason and have little time to debug this.
>> >
>> > Since dm-log-writes is anyway in upstream kernel, I don't think a bug
>> > in dm-log-writes for a certain config is a reason to block this xfstest
>> > from being merged.
>> > Anyway, I would be glad if someone could take a look at the soft lockup
>> > issue. Josef?
>> >
>
> Yeah can you give this a try and see if the soft lockup goes away?
>

It does go away. Thanks!
Now something's wrong with the log.
it get corrupted in most of the test runs, something like this:

replaying 17624@158946: sector 8651296, size 4096, flags 0
replaying 17625@158955: sector 0, size 0, flags 0
replaying 17626@158956: sector 72057596591815616, size 103079215104, flags 0
Error allocating buffer 103079215104 entry 17626

I'll look into it

Amir.

Re: [PATCH v2 00/14] Crash consistency xfstest using dm-log-writes

[Josef Bacik]

On Wed, Aug 30, 2017 at 09:39:39PM +0300, Amir Goldstein wrote:
> On Wed, Aug 30, 2017 at 6:23 PM, Josef Bacik <josef@toxicpanda.com> wrote:
> > On Wed, Aug 30, 2017 at 06:04:26PM +0300, Amir Goldstein wrote:
> >> Sorry noise xfs list, I meant to CC fsdevel
> >>
> >> On Wed, Aug 30, 2017 at 5:51 PM, Amir Goldstein <amir73il@gmail.com> wrote:
> >> > Hi all,
> >> >
> >> > This is the 2nd revision of crash consistency patch set.
> >> > The main thing that changed since v1 is my confidence in the failures
> >> > reported by the test, along with some more debugging options for
> >> > running the test tools.
> >> >
> >> > I've collected these patches that have been sitting in Josef Bacik's
> >> > tree for a few years and kicked them a bit into shape.
> >> > The dm-log-writes target has been merged to kernel v4.1, see:
> >> > https://github.com/torvalds/linux/blob/master/Documentation/device-mapper/log-writes.txt
> >> >
> >> > For this posting, I kept the random seeds constant for the test.
> >> > I set these constant seeds after running with random seed for a little
> >> > while and getting failure reports. With the current values in the test
> >> > I was able to reproduce at high probablity failures with xfs, ext4 and btrfs.
> >> > The probablity of reproducing the failure is higher on a spinning disk.
> >> >
> >
> > I'd rather we make it as evil as possible.  As long as we're printing out the
> > seed that was used in the output then we can go in and manually change the test
> > to use the same seed over and over again if we need to debug a problem.
> 
> Yeh that's what I did, but then I found values that reproduce a problem,
> so maybe its worth clinging on to these values now until the bugs are fixed in
> upstream and then as regression tests.
> 
> Anyway, I can keep these presets commented out, or run the test twice,
> once with presets and once with random seed, whatever Eryu decides.
> 
> 
> >
> >> > There is an outstanding problem with the test - when I run it with
> >> > kvm-xfstests, the test halts and I get soft lockup of log_writes_kthread.
> >> > I suppose its a bug in dm-log-writes with some kernel config or with virtio
> >> > I wasn't able to determine the reason and have little time to debug this.
> >> >
> >> > Since dm-log-writes is anyway in upstream kernel, I don't think a bug
> >> > in dm-log-writes for a certain config is a reason to block this xfstest
> >> > from being merged.
> >> > Anyway, I would be glad if someone could take a look at the soft lockup
> >> > issue. Josef?
> >> >
> >
> > Yeah can you give this a try and see if the soft lockup goes away?
> >
> 
> It does go away. Thanks!
> Now something's wrong with the log.
> it get corrupted in most of the test runs, something like this:
> 
> replaying 17624@158946: sector 8651296, size 4096, flags 0
> replaying 17625@158955: sector 0, size 0, flags 0
> replaying 17626@158956: sector 72057596591815616, size 103079215104, flags 0
> Error allocating buffer 103079215104 entry 17626
> 
> I'll look into it

Oh are the devices 4k sectorsize devices?  I fucked up 4k sectorsize support, I
sent some patches to fix it but they haven't been integrated yet, I'll poke
those again.  They are in my dm-log-writes-fixes branch in my btrfs-next tree on
kernel.org.  Thanks,

Josef

Re: [PATCH v2 00/14] Crash consistency xfstest using dm-log-writes

[Amir Goldstein]

On Wed, Aug 30, 2017 at 9:55 PM, Josef Bacik <josef@toxicpanda.com> wrote:
> On Wed, Aug 30, 2017 at 09:39:39PM +0300, Amir Goldstein wrote:
>> On Wed, Aug 30, 2017 at 6:23 PM, Josef Bacik <josef@toxicpanda.com> wrote:
>> > On Wed, Aug 30, 2017 at 06:04:26PM +0300, Amir Goldstein wrote:
>> >> Sorry noise xfs list, I meant to CC fsdevel
>> >>
>> >> On Wed, Aug 30, 2017 at 5:51 PM, Amir Goldstein <amir73il@gmail.com> wrote:
>> >> > Hi all,
>> >> >
>> >> > This is the 2nd revision of crash consistency patch set.
>> >> > The main thing that changed since v1 is my confidence in the failures
>> >> > reported by the test, along with some more debugging options for
>> >> > running the test tools.
>> >> >
>> >> > I've collected these patches that have been sitting in Josef Bacik's
>> >> > tree for a few years and kicked them a bit into shape.
>> >> > The dm-log-writes target has been merged to kernel v4.1, see:
>> >> > https://github.com/torvalds/linux/blob/master/Documentation/device-mapper/log-writes.txt
>> >> >
>> >> > For this posting, I kept the random seeds constant for the test.
>> >> > I set these constant seeds after running with random seed for a little
>> >> > while and getting failure reports. With the current values in the test
>> >> > I was able to reproduce at high probablity failures with xfs, ext4 and btrfs.
>> >> > The probablity of reproducing the failure is higher on a spinning disk.
>> >> >
>> >
>> > I'd rather we make it as evil as possible.  As long as we're printing out the
>> > seed that was used in the output then we can go in and manually change the test
>> > to use the same seed over and over again if we need to debug a problem.
>>
>> Yeh that's what I did, but then I found values that reproduce a problem,
>> so maybe its worth clinging on to these values now until the bugs are fixed in
>> upstream and then as regression tests.
>>
>> Anyway, I can keep these presets commented out, or run the test twice,
>> once with presets and once with random seed, whatever Eryu decides.
>>
>>
>> >
>> >> > There is an outstanding problem with the test - when I run it with
>> >> > kvm-xfstests, the test halts and I get soft lockup of log_writes_kthread.
>> >> > I suppose its a bug in dm-log-writes with some kernel config or with virtio
>> >> > I wasn't able to determine the reason and have little time to debug this.
>> >> >
>> >> > Since dm-log-writes is anyway in upstream kernel, I don't think a bug
>> >> > in dm-log-writes for a certain config is a reason to block this xfstest
>> >> > from being merged.
>> >> > Anyway, I would be glad if someone could take a look at the soft lockup
>> >> > issue. Josef?
>> >> >
>> >
>> > Yeah can you give this a try and see if the soft lockup goes away?
>> >
>>
>> It does go away. Thanks!
>> Now something's wrong with the log.
>> it get corrupted in most of the test runs, something like this:
>>
>> replaying 17624@158946: sector 8651296, size 4096, flags 0
>> replaying 17625@158955: sector 0, size 0, flags 0
>> replaying 17626@158956: sector 72057596591815616, size 103079215104, flags 0
>> Error allocating buffer 103079215104 entry 17626
>>
>> I'll look into it
>
> Oh are the devices 4k sectorsize devices?  I fucked up 4k sectorsize support, I
> sent some patches to fix it but they haven't been integrated yet, I'll poke
> those again.  They are in my dm-log-writes-fixes branch in my btrfs-next tree on
> kernel.org.  Thanks,
>

No they are just virtio devices in kvm reflecting my ssd LV, on whom
the same test works just fine not inside kvm.

Re: [PATCH v2 00/14] Crash consistency xfstest using dm-log-writes

[Eryu Guan]

On Wed, Aug 30, 2017 at 09:39:39PM +0300, Amir Goldstein wrote:
> On Wed, Aug 30, 2017 at 6:23 PM, Josef Bacik <josef@toxicpanda.com> wrote:
> > On Wed, Aug 30, 2017 at 06:04:26PM +0300, Amir Goldstein wrote:
> >> Sorry noise xfs list, I meant to CC fsdevel
> >>
> >> On Wed, Aug 30, 2017 at 5:51 PM, Amir Goldstein <amir73il@gmail.com> wrote:
> >> > Hi all,
> >> >
> >> > This is the 2nd revision of crash consistency patch set.
> >> > The main thing that changed since v1 is my confidence in the failures
> >> > reported by the test, along with some more debugging options for
> >> > running the test tools.
> >> >
> >> > I've collected these patches that have been sitting in Josef Bacik's
> >> > tree for a few years and kicked them a bit into shape.
> >> > The dm-log-writes target has been merged to kernel v4.1, see:
> >> > https://github.com/torvalds/linux/blob/master/Documentation/device-mapper/log-writes.txt
> >> >
> >> > For this posting, I kept the random seeds constant for the test.
> >> > I set these constant seeds after running with random seed for a little
> >> > while and getting failure reports. With the current values in the test
> >> > I was able to reproduce at high probablity failures with xfs, ext4 and btrfs.
> >> > The probablity of reproducing the failure is higher on a spinning disk.
> >> >
> >
> > I'd rather we make it as evil as possible.  As long as we're printing out the
> > seed that was used in the output then we can go in and manually change the test
> > to use the same seed over and over again if we need to debug a problem.
> 
> Yeh that's what I did, but then I found values that reproduce a problem,
> so maybe its worth clinging on to these values now until the bugs are fixed in
> upstream and then as regression tests.
> 
> Anyway, I can keep these presets commented out, or run the test twice,
> once with presets and once with random seed, whatever Eryu decides.

My thought on this with first glance is using random seed, if a specific
seed reproduce something, maybe another targeted regression test can be
added, as what you did for that ext4 corruption?

> 
> 
> >
> >> > There is an outstanding problem with the test - when I run it with
> >> > kvm-xfstests, the test halts and I get soft lockup of log_writes_kthread.
> >> > I suppose its a bug in dm-log-writes with some kernel config or with virtio
> >> > I wasn't able to determine the reason and have little time to debug this.
> >> >
> >> > Since dm-log-writes is anyway in upstream kernel, I don't think a bug
> >> > in dm-log-writes for a certain config is a reason to block this xfstest
> >> > from being merged.
> >> > Anyway, I would be glad if someone could take a look at the soft lockup
> >> > issue. Josef?
> >> >
> >
> > Yeah can you give this a try and see if the soft lockup goes away?
> >
> 
> It does go away. Thanks!
> Now something's wrong with the log.
> it get corrupted in most of the test runs, something like this:
> 
> replaying 17624@158946: sector 8651296, size 4096, flags 0
> replaying 17625@158955: sector 0, size 0, flags 0
> replaying 17626@158956: sector 72057596591815616, size 103079215104, flags 0
> Error allocating buffer 103079215104 entry 17626
> 
> I'll look into it
> 
> Amir.

The first 6 patches are all prepare work and seem fine, so I probably
will push them out this week. But I may need more time to look into all
these log-writes dm target and fsx changes.

But seems that there're still problems not sorted out (e.g. this
log-write bug), I'd prefer, when they get merged, removing the auto
group for now until things settle down a bit.

Thanks,
Eryu

Re: [PATCH v2 00/14] Crash consistency xfstest using dm-log-writes

[Amir Goldstein]

On Thu, Aug 31, 2017 at 6:38 AM, Eryu Guan <eguan@redhat.com> wrote:
> On Wed, Aug 30, 2017 at 09:39:39PM +0300, Amir Goldstein wrote:
...
>> >> > For this posting, I kept the random seeds constant for the test.
>> >> > I set these constant seeds after running with random seed for a little
>> >> > while and getting failure reports. With the current values in the test
>> >> > I was able to reproduce at high probablity failures with xfs, ext4 and btrfs.
>> >> > The probablity of reproducing the failure is higher on a spinning disk.
>> >> >
>> >
>> > I'd rather we make it as evil as possible.  As long as we're printing out the
>> > seed that was used in the output then we can go in and manually change the test
>> > to use the same seed over and over again if we need to debug a problem.
>>
>> Yeh that's what I did, but then I found values that reproduce a problem,
>> so maybe its worth clinging on to these values now until the bugs are fixed in
>> upstream and then as regression tests.
>>
>> Anyway, I can keep these presets commented out, or run the test twice,
>> once with presets and once with random seed, whatever Eryu decides.
>
> My thought on this with first glance is using random seed, if a specific
> seed reproduce something, maybe another targeted regression test can be
> added, as what you did for that ext4 corruption?
>

Sure. Speaking of ext4 corruption, I did not re-post this test with this
series because its quite an ugly black box test. I figured if ext4 guys
would take a look and understand the problem they could write a more
intelligent test. OTOH maybe its better than nothing?

BTW, Josef, did/could you write a more intelligent test to catch the
extent crc bug that you fixed? if not, was it easy to reproduce with the
provided seed presets? and without them?
I am asking to understand if a regression test to that bug is in order
beyond random seed fsx.

BTW2, the xfs bug I found is reproduced with reasonable likelihood
with any random seed. By using the provided presets, I was able to
reduce the test run time and debug cycle considerably. I used
NUM_FILES=2; NUM_OPS=31 to reproduce at > 50% probability
within seconds. So this bug doesn't require a specialized regression test.

...
>
> The first 6 patches are all prepare work and seem fine, so I probably
> will push them out this week. But I may need more time to look into all
> these log-writes dm target and fsx changes.
>
> But seems that there're still problems not sorted out (e.g. this
> log-write bug), I'd prefer, when they get merged, removing the auto
> group for now until things settle down a bit.
>

Good idea. Anyway, I would be happy to see these tests used by N > 1
testers for start.
If some version is merged so people can start pointing this big gun to
their file systems, I imagine more interesting bug will come surface.

Amir.

Re: [PATCH v2 00/14] Crash consistency xfstest using dm-log-writes

[Amir Goldstein]

On Thu, Aug 31, 2017 at 6:38 AM, Eryu Guan <eguan@redhat.com> wrote:
...
> The first 6 patches are all prepare work and seem fine, so I probably
> will push them out this week. But I may need more time to look into all
> these log-writes dm target and fsx changes.
>
> But seems that there're still problems not sorted out (e.g. this
> log-write bug), I'd prefer, when they get merged, removing the auto
> group for now until things settle down a bit.
>

I don't object to removing the auto group, but keep in mind that this test
is opt-in anyway, because it requires to define LOGWRITES_DEV
(well it SHOULD require, I actually forgot to check it...)

For now, it seems that the problem observed with kvm-xfstests
is specific to kvm-qemu aio=threads config, so you shouldn't have any
problems trying out the test on non kvm setup.

Amir.

Re: [PATCH v2 00/14] Crash consistency xfstest using dm-log-writes

[Eryu Guan]

On Fri, Sep 01, 2017 at 10:29:38AM +0300, Amir Goldstein wrote:
> On Thu, Aug 31, 2017 at 6:38 AM, Eryu Guan <eguan@redhat.com> wrote:
> ...
> > The first 6 patches are all prepare work and seem fine, so I probably
> > will push them out this week. But I may need more time to look into all
> > these log-writes dm target and fsx changes.
> >
> > But seems that there're still problems not sorted out (e.g. this
> > log-write bug), I'd prefer, when they get merged, removing the auto
> > group for now until things settle down a bit.
> >
> 
> I don't object to removing the auto group, but keep in mind that this test
> is opt-in anyway, because it requires to define LOGWRITES_DEV

That's a good point.

> (well it SHOULD require, I actually forgot to check it...)
> 
> For now, it seems that the problem observed with kvm-xfstests
> is specific to kvm-qemu aio=threads config, so you shouldn't have any
> problems trying out the test on non kvm setup.

Thanks for the heads-up! I'll run this test and look into the code
closely and see what's the best option.

Thanks,
Eryu