Re: Ubuntu Ext4 regression testing

Re: Ubuntu Ext4 regression testing

[Eric Sandeen]

Hi Brad -

(cc: xfs list too)

On 9/12/12 5:52 PM, Brad Figg wrote:
> 
> The Ubuntu kernel team has been putting some automated testing
> infrastructure in place. We are very interested in engaging with
> the appropriate upstream developers. We have been running the
> xfstests that come as part of the autotest testing framework.
> Some of these tests fail or never complete when run against an
> Ext4 file-system. 

Which ones?  Feel free to file bugs or send mail.  Tests should pass.
Sometimes it's a test bug, though, of course ;)

> Our initial questions are:
> 
> 1. Is this an appropriate set of tests to be run as regression
>    tests?

Yes, that's what it's for!

> 2. Is there a list of the xfstests that are appropriate for
>    Ext4?

Any test which says

_supported_fs generic
or
_supported_fs ext4

can run on ext4, and should in theory pass.  I think there are
about 100 of them by now.

There is a "dangerous" group (see groups file) which contains test nrs that
might stop testing via a hang or panic.  But don't skip those by
default; newer kernels with those bugs fixed _should_ pass them too.

> 3. Are there additional regression tests that would be beneficial
>    to the Linux community for us to be running?

We've been encouraging new tests to be written for xfstests lately,
and it's gotten a good amount of traction.  There certainly may be
other things out there, though.

-Eric


> Test results can be found at:
>     http://kernel.ubuntu.com/testing/index.html
> 
> 
> Thanks
> Brad
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Ubuntu Ext4 regression testing

[Brad Figg]

On 09/12/2012 04:01 PM, Eric Sandeen wrote:
> Hi Brad -
> 
> (cc: xfs list too)
> 
> On 9/12/12 5:52 PM, Brad Figg wrote:
>>
>> The Ubuntu kernel team has been putting some automated testing
>> infrastructure in place. We are very interested in engaging with
>> the appropriate upstream developers. We have been running the
>> xfstests that come as part of the autotest testing framework.
>> Some of these tests fail or never complete when run against an
>> Ext4 file-system. 
> 
> Which ones?  Feel free to file bugs or send mail.  Tests should pass.
> Sometimes it's a test bug, though, of course ;)
> 
>> Our initial questions are:
>>
>> 1. Is this an appropriate set of tests to be run as regression
>>    tests?
> 
> Yes, that's what it's for!
> 
>> 2. Is there a list of the xfstests that are appropriate for
>>    Ext4?
> 
> Any test which says
> 
> _supported_fs generic
> or
> _supported_fs ext4
> 
> can run on ext4, and should in theory pass.  I think there are
> about 100 of them by now.
> 
> There is a "dangerous" group (see groups file) which contains test nrs that
> might stop testing via a hang or panic.  But don't skip those by
> default; newer kernels with those bugs fixed _should_ pass them too.
> 
>> 3. Are there additional regression tests that would be beneficial
>>    to the Linux community for us to be running?
> 
> We've been encouraging new tests to be written for xfstests lately,
> and it's gotten a good amount of traction.  There certainly may be
> other things out there, though.
> 
> -Eric
> 
> 
>> Test results can be found at:
>>     http://kernel.ubuntu.com/testing/index.html
>>
>>
>> Thanks
>> Brad
>>
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

I'm going to be doing some new runs so anything I find will be reported.

Thanks,
Brad
-- 
Brad Figg brad.figg@canonical.com http://www.canonical.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Ubuntu Ext4 regression testing

[Eric Sandeen]

On 9/12/12 6:15 PM, Brad Figg wrote:
> I'm going to be doing some new runs so anything I find will be reported.

Dave Chinner also pointed out that i.e.

http://kernel.ubuntu.com/beta/testing/test-results/statler.2012-09-11_22-42-47/xfstests/default/control

seems to redefine, re-group, exclude etc various tests, and is taking "intelligence" out of the test suite itself.

I'd be wary of that; xfstests is dynamic - things get fixed, tests get added, groups changed, etc.

If you hard code for example "this test is for xfs" somewhere else, you might miss updates which add coverage.

Another example :

	    #'197' : ['xfs'],# This test is only valid on 32 bit machines

but the test handles that gracefully:

bitsperlong=`src/feature -w`
if [ "$bitsperlong" -ne 32 ]; then
        _notrun "This test is only valid on 32 bit machines"
fi

In general any test should be runnable; it may then issue 'not run' for some reason or other, but there's no harm in it - not as much harm as skipping regression tests because some config file got out of date...

and:

	    #'275' : ['generic'] # ext4 fails

but I just fixed that one up, and it should pass now.  Who will update the 3rd party config?

Failing tests absolutely should be run as well.  That information is as valuable as passing tests.  The goal is getting a complete picture, not just a series of "pass" results.  :)

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Ubuntu Ext4 regression testing

[Brad Figg]

On 09/12/2012 05:20 PM, Eric Sandeen wrote:
> On 9/12/12 6:15 PM, Brad Figg wrote:
>> I'm going to be doing some new runs so anything I find will be reported.
> 
> Dave Chinner also pointed out that i.e.
> 
> http://kernel.ubuntu.com/beta/testing/test-results/statler.2012-09-11_22-42-47/xfstests/default/control
> 
> seems to redefine, re-group, exclude etc various tests, and is taking "intelligence" out of the test suite itself.
> 
> I'd be wary of that; xfstests is dynamic - things get fixed, tests get added, groups changed, etc.
> 
> If you hard code for example "this test is for xfs" somewhere else, you might miss updates which add coverage.
> 
> Another example :
> 
> 	    #'197' : ['xfs'],# This test is only valid on 32 bit machines
> 
> but the test handles that gracefully:
> 
> bitsperlong=`src/feature -w`
> if [ "$bitsperlong" -ne 32 ]; then
>         _notrun "This test is only valid on 32 bit machines"
> fi
> 
> In general any test should be runnable; it may then issue 'not run' for some reason or other, but there's no harm in it - not as much harm as skipping regression tests because some config file got out of date...
> 
> and:
> 
> 	    #'275' : ['generic'] # ext4 fails
> 
> but I just fixed that one up, and it should pass now.  Who will update the 3rd party config?
> 
> Failing tests absolutely should be run as well.  That information is as valuable as passing tests.  The goal is getting a complete picture, not just a series of "pass" results.  :)
> 
> -Eric
> 

Eric,

Thanks for taking the time to point this out. We will adjust our testing accordingly.
We initially tried to run xfstest against ext2, ext3, ext4, xfs and btrfs. We are also
trying to get these tests to run on several different kernel versions as you can
see from our test results. We were running into issues on different kernels and various
file-systems while getting our act together, we did this as a band-aid.

I accept that we have some things to learn w.r.t. running this test suite. We will work
to run the xfstests "as is" without any outside "intelligence". We do recognise that
is a dynamic set of tests that people are adding to regularly.

I am not attempting to get just a series of "pass" results. If that were my goal
I could accomplish it much easier and would not have engaged with the community
on the mailing list. We want to help where we can and will accept constructive
criticism.

Brad
-- 
Brad Figg brad.figg@canonical.com http://www.canonical.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Ubuntu Ext4 regression testing

[Eric Sandeen]

On 9/12/12 7:41 PM, Brad Figg wrote:

> Eric,
> 
> Thanks for taking the time to point this out. We will adjust our testing accordingly.
> We initially tried to run xfstest against ext2, ext3, ext4, xfs and btrfs. We are also
> trying to get these tests to run on several different kernel versions as you can
> see from our test results. We were running into issues on different kernels and various
> file-systems while getting our act together, we did this as a band-aid.

I see.

> I accept that we have some things to learn w.r.t. running this test suite. We will work
> to run the xfstests "as is" without any outside "intelligence". We do recognise that
> is a dynamic set of tests that people are adding to regularly.
> 
> I am not attempting to get just a series of "pass" results. If that were my goal
> I could accomplish it much easier and would not have engaged with the community
> on the mailing list. We want to help where we can and will accept constructive
> criticism.

Sorry, it sounds like I came across too strong there - it was just a little worrying to see failing or problematic tests disabled or otherwise artificially restricted.

I'm actually very excited to see you setting up ongoing, public testing using xfstests, I think it'll be a great benefit, especially if there's a way to see a particular test's results across several kernel versions and/or filesystems and/or architectures, so that patterns of failure can emerge.

If you find that xfstests is missing some feature or behavior which would facilitate testing in the automated environment, please do let us know what you need - or send patches.  :)

Thanks,
-Eric

> Brad
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Ubuntu Ext4 regression testing

[Brad Figg]

On 09/12/2012 06:51 PM, Eric Sandeen wrote:
> On 9/12/12 7:41 PM, Brad Figg wrote:
> 
>> Eric,
>>
>> Thanks for taking the time to point this out. We will adjust our testing accordingly.
>> We initially tried to run xfstest against ext2, ext3, ext4, xfs and btrfs. We are also
>> trying to get these tests to run on several different kernel versions as you can
>> see from our test results. We were running into issues on different kernels and various
>> file-systems while getting our act together, we did this as a band-aid.
> 
> I see.
> 
>> I accept that we have some things to learn w.r.t. running this test suite. We will work
>> to run the xfstests "as is" without any outside "intelligence". We do recognise that
>> is a dynamic set of tests that people are adding to regularly.
>>
>> I am not attempting to get just a series of "pass" results. If that were my goal
>> I could accomplish it much easier and would not have engaged with the community
>> on the mailing list. We want to help where we can and will accept constructive
>> criticism.
> 
> Sorry, it sounds like I came across too strong there - it was just a little worrying to see failing or problematic tests disabled or otherwise artificially restricted.
> 
> I'm actually very excited to see you setting up ongoing, public testing using xfstests, I think it'll be a great benefit, especially if there's a way to see a particular test's results across several kernel versions and/or filesystems and/or architectures, so that patterns of failure can emerge.
> 
> If you find that xfstests is missing some feature or behavior which would facilitate testing in the automated environment, please do let us know what you need - or send patches.  :)
> 
> Thanks,
> -Eric
> 
>> Brad
>>
> 

No harm, no foul. We really don't mind constructive criticism. We are also
eager to get this setup and running. We will try to contribute more than
just running tests.

I do want to point out that we are using the xfstests which is a snapshot
in autotest. We do also look at the latest xfstests in the official xfstests
repo and add it in when we see a delta. We will also work with the autotest
maintainers to stay more up-to-date with xfstests.

Thanks,
Brad
-- 
Brad Figg brad.figg@canonical.com http://www.canonical.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Ubuntu Ext4 regression testing

[Eric Sandeen]

On 9/12/12 9:04 PM, Brad Figg wrote:
> On 09/12/2012 06:51 PM, Eric Sandeen wrote:
>> On 9/12/12 7:41 PM, Brad Figg wrote:
>>
>>> Eric,
>>>
>>> Thanks for taking the time to point this out. We will adjust our testing accordingly.
>>> We initially tried to run xfstest against ext2, ext3, ext4, xfs and btrfs. We are also
>>> trying to get these tests to run on several different kernel versions as you can
>>> see from our test results. We were running into issues on different kernels and various
>>> file-systems while getting our act together, we did this as a band-aid.
>>
>> I see.
>>
>>> I accept that we have some things to learn w.r.t. running this test suite. We will work
>>> to run the xfstests "as is" without any outside "intelligence". We do recognise that
>>> is a dynamic set of tests that people are adding to regularly.
>>>
>>> I am not attempting to get just a series of "pass" results. If that were my goal
>>> I could accomplish it much easier and would not have engaged with the community
>>> on the mailing list. We want to help where we can and will accept constructive
>>> criticism.
>>
>> Sorry, it sounds like I came across too strong there - it was just a little worrying to see failing or problematic tests disabled or otherwise artificially restricted.
>>
>> I'm actually very excited to see you setting up ongoing, public testing using xfstests, I think it'll be a great benefit, especially if there's a way to see a particular test's results across several kernel versions and/or filesystems and/or architectures, so that patterns of failure can emerge.
>>
>> If you find that xfstests is missing some feature or behavior which would facilitate testing in the automated environment, please do let us know what you need - or send patches.  :)
>>
>> Thanks,
>> -Eric
>>
>>> Brad
>>>
>>
> 
> No harm, no foul. We really don't mind constructive criticism. We are also
> eager to get this setup and running. We will try to contribute more than
> just running tests.

Just running them and publishing results is definitely useful.

> I do want to point out that we are using the xfstests which is a snapshot
> in autotest. We do also look at the latest xfstests in the official xfstests
> repo and add it in when we see a delta. We will also work with the autotest
> maintainers to stay more up-to-date with xfstests.

Ah, I didn't know that autotest had a snapshot.  I wonder if there's a way to
tease that back out, and pull down xfstests from git daily.

FWIW, it's a little confusing - we have 2 repos:

git://oss.sgi.com/xfs/cmds/xfstests.git
git://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git

the one on kernel.org is where most rapid development seems to happen, and changes get pulled over to sgi.  Sometimes, it goes the other way.

Did autotest make any changes to what's upstream?

-Eric

> Thanks,
> Brad
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Ubuntu Ext4 regression testing

[Brad Figg]

On 09/12/2012 07:09 PM, Eric Sandeen wrote:
> On 9/12/12 9:04 PM, Brad Figg wrote:
>> On 09/12/2012 06:51 PM, Eric Sandeen wrote:
>>> On 9/12/12 7:41 PM, Brad Figg wrote:
>>>
>>>> Eric,
>>>>
>>>> Thanks for taking the time to point this out. We will adjust our testing accordingly.
>>>> We initially tried to run xfstest against ext2, ext3, ext4, xfs and btrfs. We are also
>>>> trying to get these tests to run on several different kernel versions as you can
>>>> see from our test results. We were running into issues on different kernels and various
>>>> file-systems while getting our act together, we did this as a band-aid.
>>>
>>> I see.
>>>
>>>> I accept that we have some things to learn w.r.t. running this test suite. We will work
>>>> to run the xfstests "as is" without any outside "intelligence". We do recognise that
>>>> is a dynamic set of tests that people are adding to regularly.
>>>>
>>>> I am not attempting to get just a series of "pass" results. If that were my goal
>>>> I could accomplish it much easier and would not have engaged with the community
>>>> on the mailing list. We want to help where we can and will accept constructive
>>>> criticism.
>>>
>>> Sorry, it sounds like I came across too strong there - it was just a little worrying to see failing or problematic tests disabled or otherwise artificially restricted.
>>>
>>> I'm actually very excited to see you setting up ongoing, public testing using xfstests, I think it'll be a great benefit, especially if there's a way to see a particular test's results across several kernel versions and/or filesystems and/or architectures, so that patterns of failure can emerge.
>>>
>>> If you find that xfstests is missing some feature or behavior which would facilitate testing in the automated environment, please do let us know what you need - or send patches.  :)
>>>
>>> Thanks,
>>> -Eric
>>>
>>>> Brad
>>>>
>>>
>>
>> No harm, no foul. We really don't mind constructive criticism. We are also
>> eager to get this setup and running. We will try to contribute more than
>> just running tests.
> 
> Just running them and publishing results is definitely useful.
> 
>> I do want to point out that we are using the xfstests which is a snapshot
>> in autotest. We do also look at the latest xfstests in the official xfstests
>> repo and add it in when we see a delta. We will also work with the autotest
>> maintainers to stay more up-to-date with xfstests.
> 
> Ah, I didn't know that autotest had a snapshot.  I wonder if there's a way to
> tease that back out, and pull down xfstests from git daily.
> 
> FWIW, it's a little confusing - we have 2 repos:
> 
> git://oss.sgi.com/xfs/cmds/xfstests.git
> git://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git
> 
> the one on kernel.org is where most rapid development seems to happen, and changes get pulled over to sgi.  Sometimes, it goes the other way.
> 
> Did autotest make any changes to what's upstream?
> 
> -Eric
> 
>> Thanks,
>> Brad
>>
> 

No, it seems to be an unmodified snapshot. As long as it's kept up-to-date,
which is something we have a vested interest in seeing is done, I'm fine
with this. We then take this (autotest) and also roll it into our QA team
for even more testing. They will also be making their testing results public.

I grabbed from the oss.sgi repository. If one is more "authoritative" or
one you'd rather see us use, just let me/us know.

Brad
-- 
Brad Figg brad.figg@canonical.com http://www.canonical.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Ubuntu Ext4 regression testing

[Theodore Ts'o]

On Wed, Sep 12, 2012 at 09:09:37PM -0500, Eric Sandeen wrote:
> 
> Ah, I didn't know that autotest had a snapshot.  I wonder if there's
> a way to tease that back out, and pull down xfstests from git daily.
> 
> FWIW, it's a little confusing - we have 2 repos:
> 
> git://oss.sgi.com/xfs/cmds/xfstests.git
> git://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git

This may be helpful; I have a xfstests build environment here:

git://git.kernel.org/pub/scm/fs/ext2/xfstests-bld.git

It will automatically pull down the xfstests and xfsprogs from
git.kernel.org, and builds xfstests in a hermetic environment (i.e.,
it doesn't depend on version of xfsprogs, libacl, libaio,
etc. installed on the build system).  I set this up back when I was
doing most of my work using Ubuntu LTS 10.04, and the positively
ancient versions of libacl, libaio, xfsprogs-dev, etc., weren't
compatible with the bleeding edge of xfstests.

So I used this build environment so I wasn't dependent on the
vagrancies of whatever happened to be in Ubunutu LTS 10.04.

The makefiles in xfstests-bld will also generate a tar file containing
the necessary xfstests and xfsprogs binaries which I could then drop
into a debootstrap environment (I'm currently using a x86-32 Debian
unstable chroot) which I then use in my KVM image.  So it allows me to
get and build the very latest version of xfstests from git.kernel.org
in a highly automated fashion.

Regards,

						- Ted

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs