Is is possible to submit binary image as fstest test case?

Is is possible to submit binary image as fstest test case?

[Qu Wenruo]

Hi,

Just as the title says, for some case(OK, btrfs again) we need to catch 
a file system in special timing.

In this specific case, we need to grab a btrfs image undergoing 
balancing, just before the balance finished.

Although we can use flakey to drop all write, we still don't have method 
to catch the timing of the that transaction.


On the other hand, we can tweak our local kernel, adding 
msleep()/message and dump the disk during the sleep.
And the image I dumped can easily trigger btrfs kernel and user-space bug.

So I'm wondering if I can just upload a zipped raw image as part of the 
test case?

Thanks,
Qu

Re: Is is possible to submit binary image as fstest test case?

[Brian Foster]

On Thu, Oct 06, 2016 at 04:12:56PM +0800, Qu Wenruo wrote:
> Hi,
> 
> Just as the title says, for some case(OK, btrfs again) we need to catch a
> file system in special timing.
> 
> In this specific case, we need to grab a btrfs image undergoing balancing,
> just before the balance finished.
> 
> Although we can use flakey to drop all write, we still don't have method to
> catch the timing of the that transaction.
> 
> 
> On the other hand, we can tweak our local kernel, adding msleep()/message
> and dump the disk during the sleep.
> And the image I dumped can easily trigger btrfs kernel and user-space bug.
> 
> So I'm wondering if I can just upload a zipped raw image as part of the test
> case?
> 

Doesn't necessarily bother me one way or the other, but something we've
done with XFS in such situations is introduce a DEBUG mode only sysfs
tunable that delays certain infrastructure (log recovery in our case) to
coordinate with test cases that try to reproduce such timing/racing
problems.

See test xfs/051 for an example..

Brian

> Thanks,
> Qu
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Is is possible to submit binary image as fstest test case?

[Theodore Ts'o]

On Thu, Oct 06, 2016 at 08:29:28AM -0400, Brian Foster wrote:
> Doesn't necessarily bother me one way or the other, but something we've
> done with XFS in such situations is introduce a DEBUG mode only sysfs
> tunable that delays certain infrastructure (log recovery in our case) to
> coordinate with test cases that try to reproduce such timing/racing
> problems.

The other approach the btrfs folks might consider is to have a
sufficiently powerful "debugfs" or "xfs_db" program that can generate
test images with the desired property.

I've found that the time I've invested in creating debugfs has repaid
itself a hundred times over, especially when maintaining a regression
test suite for e2fsck.

Cheers,

					- Ted

Re: Is is possible to submit binary image as fstest test case?

[David Sterba]

On Thu, Oct 06, 2016 at 03:24:52PM -0400, Theodore Ts'o wrote:
> On Thu, Oct 06, 2016 at 08:29:28AM -0400, Brian Foster wrote:
> > Doesn't necessarily bother me one way or the other, but something we've
> > done with XFS in such situations is introduce a DEBUG mode only sysfs
> > tunable that delays certain infrastructure (log recovery in our case) to
> > coordinate with test cases that try to reproduce such timing/racing
> > problems.
> 
> The other approach the btrfs folks might consider is to have a
> sufficiently powerful "debugfs" or "xfs_db" program that can generate
> test images with the desired property.

We can generate certain classes of testing images, eg. by mkfs, adding
files and then corrupting specific keys after unmount. This could be
unreliable for timing-dependent bugs, or catching some intermediate
filesystem state that is created by kernel for free but would be tedious
to recreate manually from scratch in userspace.

> I've found that the time I've invested in creating debugfs has repaid
> itself a hundred times over, especially when maintaining a regression
> test suite for e2fsck.

I agree that's a good thing.

Re: Is is possible to submit binary image as fstest test case?

[Dave Chinner]

On Thu, Oct 06, 2016 at 04:12:56PM +0800, Qu Wenruo wrote:
> Hi,
> 
> Just as the title says, for some case(OK, btrfs again) we need to
> catch a file system in special timing.
> 
> In this specific case, we need to grab a btrfs image undergoing
> balancing, just before the balance finished.
> 
> Although we can use flakey to drop all write, we still don't have
> method to catch the timing of the that transaction.
> 
> 
> On the other hand, we can tweak our local kernel, adding
> msleep()/message and dump the disk during the sleep.
> And the image I dumped can easily trigger btrfs kernel and user-space bug.
> 
> So I'm wondering if I can just upload a zipped raw image as part of
> the test case?

Preferably not. We've managed to avoid pre-built images in xfstests
for 15 years, so there'd have to be a really good reason to start
doing this, especially as once we open that floodgate we'll end up
with everyone wanting to do this and it will blow out the size of
the repository in now time.

If the issue is just timing or being unable to trigger an error
at the right time, this is what error injection frameworks or
debug-only sysfs hooks are for. The XFS kernel code has both,
xfstests use both, and they pretty much do away with the need for
custom binary filesystem images for such testing...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Is is possible to submit binary image as fstest test case?

[Qu Wenruo]

At 10/07/2016 05:18 PM, Dave Chinner wrote:
> On Thu, Oct 06, 2016 at 04:12:56PM +0800, Qu Wenruo wrote:
>> Hi,
>>
>> Just as the title says, for some case(OK, btrfs again) we need to
>> catch a file system in special timing.
>>
>> In this specific case, we need to grab a btrfs image undergoing
>> balancing, just before the balance finished.
>>
>> Although we can use flakey to drop all write, we still don't have
>> method to catch the timing of the that transaction.
>>
>>
>> On the other hand, we can tweak our local kernel, adding
>> msleep()/message and dump the disk during the sleep.
>> And the image I dumped can easily trigger btrfs kernel and user-space bug.
>>
>> So I'm wondering if I can just upload a zipped raw image as part of
>> the test case?
>
> Preferably not. We've managed to avoid pre-built images in xfstests
> for 15 years, so there'd have to be a really good reason to start
> doing this, especially as once we open that floodgate we'll end up
> with everyone wanting to do this and it will blow out the size of
> the repository in now time.

Makes sense.
For btrfs-progs, which includes test images, it already takes about 77M, 
even we have tried our best to reduce image size.

>
> If the issue is just timing or being unable to trigger an error
> at the right time, this is what error injection frameworks or
> debug-only sysfs hooks are for. The XFS kernel code has both,
> xfstests use both, and they pretty much do away with the need for
> custom binary filesystem images for such testing...

So again, btrfs is lacking infrastructure for debug.
It seems that we can only rely on images out of xfstest tree, created by 
some customized kernel for a long time.

Thanks for the reply.
Qu


>
> Cheers,
>
> Dave.
>

Re: Is is possible to submit binary image as fstest test case?

[Dave Chinner]

On Fri, Oct 07, 2016 at 05:26:27PM +0800, Qu Wenruo wrote:
> 
> 
> At 10/07/2016 05:18 PM, Dave Chinner wrote:
> >On Thu, Oct 06, 2016 at 04:12:56PM +0800, Qu Wenruo wrote:
> >>Hi,
> >>
> >>Just as the title says, for some case(OK, btrfs again) we need to
> >>catch a file system in special timing.
> >>
> >>In this specific case, we need to grab a btrfs image undergoing
> >>balancing, just before the balance finished.
> >>
> >>Although we can use flakey to drop all write, we still don't have
> >>method to catch the timing of the that transaction.
> >>
> >>
> >>On the other hand, we can tweak our local kernel, adding
> >>msleep()/message and dump the disk during the sleep.
> >>And the image I dumped can easily trigger btrfs kernel and user-space bug.
> >>
> >>So I'm wondering if I can just upload a zipped raw image as part of
> >>the test case?
> >
> >Preferably not. We've managed to avoid pre-built images in xfstests
> >for 15 years, so there'd have to be a really good reason to start
> >doing this, especially as once we open that floodgate we'll end up
> >with everyone wanting to do this and it will blow out the size of
> >the repository in now time.
> 
> Makes sense.
> For btrfs-progs, which includes test images, it already takes about
> 77M, even we have tried our best to reduce image size.
> 
> >
> >If the issue is just timing or being unable to trigger an error
> >at the right time, this is what error injection frameworks or
> >debug-only sysfs hooks are for. The XFS kernel code has both,
> >xfstests use both, and they pretty much do away with the need for
> >custom binary filesystem images for such testing...
> 
> So again, btrfs is lacking infrastructure for debug.
> It seems that we can only rely on images out of xfstest tree,

That's the /wrong answer/. Go and implement debug infrastructure
that btrfs needs - if you wait for someone else to do it, it will
never get done and btrfs will never stabilise....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Is is possible to submit binary image as fstest test case?

[David Sterba]

On Fri, Oct 07, 2016 at 05:26:27PM +0800, Qu Wenruo wrote:
> > Preferably not. We've managed to avoid pre-built images in xfstests
> > for 15 years, so there'd have to be a really good reason to start
> > doing this, especially as once we open that floodgate we'll end up
> > with everyone wanting to do this and it will blow out the size of
> > the repository in now time.
> 
> Makes sense.
> For btrfs-progs, which includes test images, it already takes about 77M, 
> even we have tried our best to reduce image size.

The number 77M is bogus. Clean checkout of the kernel.org repository is
15M in total, the .git directory is 5M and all the images are 7.3MB in
total. We're not close to any scary numbers yet. Should we need large
images, I'll create a separate repository for that.

> > If the issue is just timing or being unable to trigger an error
> > at the right time, this is what error injection frameworks or
> > debug-only sysfs hooks are for. The XFS kernel code has both,
> > xfstests use both, and they pretty much do away with the need for
> > custom binary filesystem images for such testing...
> 
> So again, btrfs is lacking infrastructure for debug.

Depends what exactly you need. IIRC Filipe once submitted a patch for
transaction failure injection but then realzied that he could achieve
the same using the flaky device. You can try to take a snapshot of the
block device after using the freeze syscall. More specific tasks would
need code support. You definetelly know about the btrfs-corrupt-block
utility so you can use it to creatively damage the filesystem structures.

Re: Is is possible to submit binary image as fstest test case?

[Qu Wenruo]

On 10/08/2016 12:16 AM, David Sterba wrote:
> On Fri, Oct 07, 2016 at 05:26:27PM +0800, Qu Wenruo wrote:
>>> Preferably not. We've managed to avoid pre-built images in xfstests
>>> for 15 years, so there'd have to be a really good reason to start
>>> doing this, especially as once we open that floodgate we'll end up
>>> with everyone wanting to do this and it will blow out the size of
>>> the repository in now time.
>>
>> Makes sense.
>> For btrfs-progs, which includes test images, it already takes about 77M,
>> even we have tried our best to reduce image size.
>
> The number 77M is bogus. Clean checkout of the kernel.org repository is
> 15M in total, the .git directory is 5M and all the images are 7.3MB in
> total. We're not close to any scary numbers yet. Should we need large
> images, I'll create a separate repository for that.
>
>>> If the issue is just timing or being unable to trigger an error
>>> at the right time, this is what error injection frameworks or
>>> debug-only sysfs hooks are for. The XFS kernel code has both,
>>> xfstests use both, and they pretty much do away with the need for
>>> custom binary filesystem images for such testing...
>>
>> So again, btrfs is lacking infrastructure for debug.
>
> Depends what exactly you need. IIRC Filipe once submitted a patch for
> transaction failure injection but then realzied that he could achieve
> the same using the flaky device. You can try to take a snapshot of the
> block device after using the freeze syscall. More specific tasks would
> need code support. You definetelly know about the btrfs-corrupt-block
> utility so you can use it to creatively damage the filesystem structures.

Btrfs-corrupt-block is also my first guess.

But the problem turns out that, I must implement (at least part of) 
relocation code in btrfs-progs, to create the desired image, for my 
specific usage.

That will be a lot of work, at least not something I could do in one or 
two months.

But thanks for all your advice, I'm aware of the necessity of these 
infrastructure now.

Thanks,
Qu

Re: Is is possible to submit binary image as fstest test case?

[David Sterba]

On Fri, Oct 07, 2016 at 08:18:38PM +1100, Dave Chinner wrote:
> On Thu, Oct 06, 2016 at 04:12:56PM +0800, Qu Wenruo wrote:
> > Hi,
> > 
> > Just as the title says, for some case(OK, btrfs again) we need to
> > catch a file system in special timing.
> > 
> > In this specific case, we need to grab a btrfs image undergoing
> > balancing, just before the balance finished.
> > 
> > Although we can use flakey to drop all write, we still don't have
> > method to catch the timing of the that transaction.
> > 
> > 
> > On the other hand, we can tweak our local kernel, adding
> > msleep()/message and dump the disk during the sleep.
> > And the image I dumped can easily trigger btrfs kernel and user-space bug.
> > 
> > So I'm wondering if I can just upload a zipped raw image as part of
> > the test case?
> 
> Preferably not. We've managed to avoid pre-built images in xfstests
> for 15 years, so there'd have to be a really good reason to start
> doing this, especially as once we open that floodgate we'll end up
> with everyone wanting to do this and it will blow out the size of
> the repository in now time.
> 
> If the issue is just timing or being unable to trigger an error
> at the right time, this is what error injection frameworks or
> debug-only sysfs hooks are for. The XFS kernel code has both,
> xfstests use both, and they pretty much do away with the need for
> custom binary filesystem images for such testing...

Error injection framework may not be alwasy available, eg. in kernels
built for production. Yet I'd like to make the test possible.

Also, I find it useful to keep the exact images that are attached to a
report and not necessarily try to recreate it to trigger a bug. If the
images are small, hosting them with the sources makes testing easy.
Big images would probably go to own repository and be linked.

I don't really expect fstests to store crafted images and would advise
Qu to not even try to propose that, because the answer was quite
predictable.

Re: Is is possible to submit binary image as fstest test case?

[Dave Chinner]

On Fri, Oct 07, 2016 at 06:05:51PM +0200, David Sterba wrote:
> On Fri, Oct 07, 2016 at 08:18:38PM +1100, Dave Chinner wrote:
> > On Thu, Oct 06, 2016 at 04:12:56PM +0800, Qu Wenruo wrote:
> > > So I'm wondering if I can just upload a zipped raw image as part of
> > > the test case?
> > 
> > Preferably not. We've managed to avoid pre-built images in xfstests
> > for 15 years, so there'd have to be a really good reason to start
> > doing this, especially as once we open that floodgate we'll end up
> > with everyone wanting to do this and it will blow out the size of
> > the repository in now time.
> > 
> > If the issue is just timing or being unable to trigger an error
> > at the right time, this is what error injection frameworks or
> > debug-only sysfs hooks are for. The XFS kernel code has both,
> > xfstests use both, and they pretty much do away with the need for
> > custom binary filesystem images for such testing...
> 
> Error injection framework may not be alwasy available, eg. in kernels
> built for production. Yet I'd like to make the test possible.

Why would you ever enable error injection on a production kernel?
We simply don't run the error injection tests when the
infrastructure is not there, jsut like we do with all other tests
that are depenent on optional kernel/fs features....

> Also, I find it useful to keep the exact images that are attached to a
> report and not necessarily try to recreate it to trigger a bug. If the
> images are small, hosting them with the sources makes testing easy.
> Big images would probably go to own repository and be linked.
> 
> I don't really expect fstests to store crafted images and would advise
> Qu to not even try to propose that, because the answer was quite
> predictable.

You say that like it's a bad thing?  What's the guarantee that a
specific corrupt image will always be sufficient to trigger the
problem the test is supposed to check? i.e. we could change a
different part of the FS code and that could mask the bug the image
used to trigger. The test then passes, but we haven't actually fix
the bug that the test used to exercise. i.e. we've got a false "we
fixed the problem" report, when all we did is prevent a specific
vector from tripping over it.

Error injection and sysfs hooks into debug code are much more
reliable ways of testing that the root cause of a problem is fixed
and stays fixed.  The reproducing trigger cannot be masked by
changes in other code layers, so we know that if the error
injection/sysfs hook test handles the problem correctly, we have
actually fixed the underlying bug and not just masked it...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Is is possible to submit binary image as fstest test case?

[David Sterba]

On Mon, Oct 10, 2016 at 10:56:12AM +1100, Dave Chinner wrote:
> On Fri, Oct 07, 2016 at 06:05:51PM +0200, David Sterba wrote:
> > On Fri, Oct 07, 2016 at 08:18:38PM +1100, Dave Chinner wrote:
> > > On Thu, Oct 06, 2016 at 04:12:56PM +0800, Qu Wenruo wrote:
> > > > So I'm wondering if I can just upload a zipped raw image as part of
> > > > the test case?
> > > 
> > > Preferably not. We've managed to avoid pre-built images in xfstests
> > > for 15 years, so there'd have to be a really good reason to start
> > > doing this, especially as once we open that floodgate we'll end up
> > > with everyone wanting to do this and it will blow out the size of
> > > the repository in now time.
> > > 
> > > If the issue is just timing or being unable to trigger an error
> > > at the right time, this is what error injection frameworks or
> > > debug-only sysfs hooks are for. The XFS kernel code has both,
> > > xfstests use both, and they pretty much do away with the need for
> > > custom binary filesystem images for such testing...
> > 
> > Error injection framework may not be alwasy available, eg. in kernels
> > built for production. Yet I'd like to make the test possible.
> 
> Why would you ever enable error injection on a production kernel?

That's not what I mean. Let me rephrase. If the test needs the injection
infrastructue it can't be run with production build. A crafted image can
reproduce that.

> We simply don't run the error injection tests when the
> infrastructure is not there, jsut like we do with all other tests
> that are depenent on optional kernel/fs features....

I think this depends on the test and the bug class it's supposed to hit.
The images need some determinism, eg. a traversal of structures, or
existence (or not) of some state, or error handling.

> > Also, I find it useful to keep the exact images that are attached to a
> > report and not necessarily try to recreate it to trigger a bug. If the
> > images are small, hosting them with the sources makes testing easy.
> > Big images would probably go to own repository and be linked.
> > 
> > I don't really expect fstests to store crafted images and would advise
> > Qu to not even try to propose that, because the answer was quite
> > predictable.
> 
> You say that like it's a bad thing?

>From the POV of fstests, it won't work, possibly in some very limited
number of tests. As it's a multi-filesystem project, this won't scale.
You said that earlier and I consider this obvious.

What we can do for fstests, configure an external path with filesystem
images and let tests use it for any purpose. In btrfs-progs we eg. don't
have mount tests of the fuzzed images, but I would like to make it more
accessible for wider testing. In a way that's acceptable for fstests.

> What's the guarantee that a
> specific corrupt image will always be sufficient to trigger the
> problem the test is supposed to check? i.e. we could change a
> different part of the FS code and that could mask the bug the image
> used to trigger. The test then passes, but we haven't actually fix
> the bug that the test used to exercise. i.e. we've got a false "we
> fixed the problem" report, when all we did is prevent a specific
> vector from tripping over it.

Again, I think this depends. I can't be sure the image will always
trigger the bug that was there originally. I've seen that some fuzzed
images required to add early checks that in some cases did not reach the
original point of failure in other images. As long as the test
assumptions are met, the test will not give false sense of fixing. In
case of the fuzzed images, one may not cover all the assumptions, but I
take them more like statistical proof that we don't have new bugs. And
it's convenient to have the images at hand.

> Error injection and sysfs hooks into debug code are much more
> reliable ways of testing that the root cause of a problem is fixed
> and stays fixed.  The reproducing trigger cannot be masked by
> changes in other code layers, so we know that if the error
> injection/sysfs hook test handles the problem correctly, we have
> actually fixed the underlying bug and not just masked it...

Agreed. This applies to the example that Qu asked about, as balance
modifies a global filesystem state, a single image won't cover many
variants of structures (with snapshots, optional mkfs features). The
injection framework will give better testing coverage.