Re: [PATCH] xfstests-bld: Simplify determination of number of CPUs in build-all

Re: [PATCH] xfstests-bld: Simplify determination of number of CPUs in build-all

[tytso]

On Fri, Mar 28, 2014 at 10:03:19AM +0100, Sedat Dilek wrote:
> Use 'getconf _NPROCESSORS_ONLN' to determine the total number
> of available ("online") CPUs.
> 
> I have seen this 1st in some build-scripts of Grml project and
> had some discussions with Thorsten Glaser on this topic.
> 
> Signed-off-by: Sedat Dilek <sedat.dilek@gmail.com>

Thanks, applied.

In the future, it would be great at least cc patches to
linux-ext4@vger.kernel.org, since I rely on patchwork to make sure
patches don't end up falling through the cracks.

If we start getting a huge number of patches to xfstests-bld, and
people start getting confused/annoyed about how xfstests-bld issues
get discussed on linux-ext4@vger.kernel.org, while xfstests patches
and discussion happen on xfs@oss.sgi.com, we could consider creating a
new mailing list --- especially given that based on a challenge which
Greg K-H gave us at the kernel pannel at Collab Summit, we'll at least
be looking at cleaning up and then trying to get into the linux kernel
mainline sources some combination of xfstests plus some infrastructure
automation (perhaps strongly based on what I've been working here in
the xfstests-bld tree) to run xfstests.

Cheers,

						- Ted

Re: [PATCH] xfstests-bld: Simplify determination of number of CPUs in build-all

[Sedat Dilek]

On Fri, Mar 28, 2014 at 5:18 PM,  <tytso@mit.edu> wrote:
> On Fri, Mar 28, 2014 at 10:03:19AM +0100, Sedat Dilek wrote:
>> Use 'getconf _NPROCESSORS_ONLN' to determine the total number
>> of available ("online") CPUs.
>>
>> I have seen this 1st in some build-scripts of Grml project and
>> had some discussions with Thorsten Glaser on this topic.
>>
>> Signed-off-by: Sedat Dilek <sedat.dilek@gmail.com>
>
> Thanks, applied.
>

...and pushed?

> In the future, it would be great at least cc patches to
> linux-ext4@vger.kernel.org, since I rely on patchwork to make sure
> patches don't end up falling through the cracks.
>
> If we start getting a huge number of patches to xfstests-bld, and
> people start getting confused/annoyed about how xfstests-bld issues
> get discussed on linux-ext4@vger.kernel.org, while xfstests patches
> and discussion happen on xfs@oss.sgi.com, we could consider creating a
> new mailing list --- especially given that based on a challenge which
> Greg K-H gave us at the kernel pannel at Collab Summit, we'll at least
> be looking at cleaning up and then trying to get into the linux kernel
> mainline sources some combination of xfstests plus some infrastructure
> automation (perhaps strongly based on what I've been working here in
> the xfstests-bld tree) to run xfstests.
>

Please, document the handling of patches (and where someone can get informed).

- Sedat -

Re: [PATCH] xfstests-bld: Simplify determination of number of CPUs in build-all

[Theodore Ts'o]

On Sat, Mar 29, 2014 at 11:04:51AM +0100, Sedat Dilek wrote:
> >>
> >> Signed-off-by: Sedat Dilek <sedat.dilek@gmail.com>
> >
> > Thanks, applied.
> >
> 
> ...and pushed?

Sorry, the conference network in Napa was a bit of a disaster.  Pushed now.

> Please, document the handling of patches (and where someone can get informed).

That's a good suggestion.  I'll update the documentation to include
that.  (Probably on the airplane ride home.)

Cheers,

					- Ted

Re: [Lsf] [PATCH] xfstests-bld: Simplify determination of number of CPUs in build-all

[Dave Chinner]

[cc'ing the xfstests list: xfs@oss.sgi.com]

On Fri, Mar 28, 2014 at 12:18:06PM -0400, tytso@mit.edu wrote:
> If we start getting a huge number of patches to xfstests-bld, and
> people start getting confused/annoyed about how xfstests-bld issues
> get discussed on linux-ext4@vger.kernel.org, while xfstests patches
> and discussion happen on xfs@oss.sgi.com, we could consider creating a
> new mailing list ---

/me puts on his xfstests Maintainer Hat

That's a problem of your own making, Ted: please don't speak on
behalf of what the upstream xfstests developers might or might not
do just because of what you do with it as a user. The existance of
many different environments people have built up around it is one of
the strengths of xfstests. Hence the very act of considering
enforcing One True Way of running xfstests is, IMO, harmful
to the wider filesystem and xfstests community.

You're also being rather presumptive that the existence of
xfstests-bld requires us to change anything about the way xfstests
is run.  Your xfstests environment - while interesting and
potentially useful to others - is not unique and is not the only way
of doing in-place testing.

> especially given that based on a challenge which
> Greg K-H gave us at the kernel pannel at Collab Summit,
               ^^^^

Hmmmm. Not the way I remember it. Perhaps I should go look at the
video and check that Greg was addressing me directly as the xfstests
maintainer with those comments. After all, those lights on stage can
be blinding.....

> we'll at least
> be looking at cleaning up and then trying to get into the linux kernel
> mainline sources some combination of xfstests plus some infrastructure
> automation (perhaps strongly based on what I've been working here in
> the xfstests-bld tree) to run xfstests.

Now you're pre-empting the discussions we need to have about
xfstests and what best serves it's user community. xfstests is
consumed by many end-users that are not kernel developers (e.g.
distro QA departments, storage product vendors, etc), so anything we
decide needs to work not only for kernel developers but also benefit
the wider community of xfstests users.

There are many advantages even to filesystem developers to
staying outside the kernel tree. Think about this for a moment: to
update xfstests to pick up a new regression test to test a
regression, you need to update the kernel tree. That will also pull
in the fix for the regression. To revert the kernel tree to before
the fix came in so that you can run before/after fix comfirmation,
it also removes the new regression test from xfstests harness and so
you can't run the regression test in place.

IOWs, bisects based on regression tests become rather difficult
because of this - bisects require the test not to change from
bisection point to bisection point, and running xfstests directly out
of the kernel tree that you are building kernels from during the
bisect is going to have this exact problem.

Therefore, *if* we move xfstests to the kernel tree we will still
need to maintain that flexibility and configurability of the current
code so that developers and external users can continue to package
the tests up into their own QA environment. That implies we need to
work out packaging and distribution issues that we don't currently
care about, as well as some method of in-place test execution for
people like Greg doing -stable kernel testing.

So, rather than going off half-cocked about some random build
environment for xfstests defining the future of filesystem testing,
we need to step back and think about what Greg actually wants. That
is, "make tests" in the kernel tests/ tree to be able to run
xfstests. That's the problem Greg wanted solved, and that does not
necessarily mean wholesale changes to xfstests or it's development
model. And one possible solution to this is simply making the kernel
tests/ directory just another downstream consumer of the upstream
xfstests project....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [Lsf] [PATCH] xfstests-bld: Simplify determination of number of CPUs in build-all

[Theodore Ts'o]

On Mon, Mar 31, 2014 at 01:51:48PM +1100, Dave Chinner wrote:
> /me puts on his xfstests Maintainer Hat
> 
> That's a problem of your own making, Ted: please don't speak on
> behalf of what the upstream xfstests developers might or might not
> do just because of what you do with it as a user. The existance of
> many different environments people have built up around it is one of
> the strengths of xfstests. Hence the very act of considering
> enforcing One True Way of running xfstests is, IMO, harmful
> to the wider filesystem and xfstests community.

I wasn't presuming to do so.

I am merely pushing xfstests-bld because it's better that more people
run tests.  And if it makes it easier for people to run tests because
there is a turn-key way to run them, so much the better.  Having a
diversity of ways of running xfstests is good.  Having lots of people
run xfstests, even if many of them run it the same way, is better.

> You're also being rather presumptive that the existence of
> xfstests-bld requires us to change anything about the way xfstests
> is run.

I never said that any changes was needed to xfstests repository, or
how anyone else might choose to run xfstests.

> > especially given that based on a challenge which
> > Greg K-H gave us at the kernel pannel at Collab Summit,
>                ^^^^
> 
> Hmmmm. Not the way I remember it. Perhaps I should go look at the
> video and check that Greg was addressing me directly as the xfstests
> maintainer with those comments. After all, those lights on stage can
> be blinding.....

Sorry, I thought in the discussions we had afterwards you agreed with
me that *some* way of running xfststs easily was going to be a
requirement, as it is somewhat doubtful that most non-file system
developers will have the patience to deal with the "some assembly
required" in order to actually run xfstests, and actually putting
xfstests into the kernel sources wasn't going to be particularly
useful unless we have some way of making it possible to run the tests
in a semi-automated fashion.

Whether that method is based on what I have in xfstests-bld, or some
other way is I agree certainly up for discussion.  At the moment the
system I have is only set up for ext4, although I will happily accept
patches and work with other file system developers to enhance it so it
can work with other file systems.

And of course, whether changes in the mainline kernel tree are
manually propagated changes from the xfstests.git tree, or whether
primary development happens in the kernel tree, is ultimately going to
be up to you and the XFS developers who have stewardship of xfstests.
I'm not sure I would be that excited about manual propagation of
changes from one git tree to another, but that is of course, up to
you.

Cheers,

						- Ted

Re: [Lsf] [PATCH] xfstests-bld: Simplify determination of number of CPUs in build-all

[Dave Chinner]

On Mon, Mar 31, 2014 at 10:37:11PM -0400, Theodore Ts'o wrote:
> On Mon, Mar 31, 2014 at 01:51:48PM +1100, Dave Chinner wrote:
> > > especially given that based on a challenge which
> > > Greg K-H gave us at the kernel pannel at Collab Summit,
> >                ^^^^
> > 
> > Hmmmm. Not the way I remember it. Perhaps I should go look at the
> > video and check that Greg was addressing me directly as the xfstests
> > maintainer with those comments. After all, those lights on stage can
> > be blinding.....
> 
> Sorry, I thought in the discussions we had afterwards you agreed with
> me that *some* way of running xfststs easily was going to be a
> requirement, as it is somewhat doubtful that most non-file system
> developers will have the patience to deal with the "some assembly
> required" in order to actually run xfstests, and actually putting
> xfstests into the kernel sources wasn't going to be particularly
> useful unless we have some way of making it possible to run the tests
> in a semi-automated fashion.

Right, there was follup discussion about how we could potentially
acheive what Greg wanted, but it was more of a "we've got pieces
that we might be able to leverage" discussion than anything else.

> Whether that method is based on what I have in xfstests-bld, or some
> other way is I agree certainly up for discussion.  At the moment the
> system I have is only set up for ext4, although I will happily accept
> patches and work with other file system developers to enhance it so it
> can work with other file systems.

Right, that's pretty much where we finished the discussion at
the collab summit ;)

But that this is why I responded to your previous email - while you
likely had the best of intentions, you gave the impression that
xfstests-bld type of solution was fait accompli, when in fact we
haven't even really started defining the problem space and gathering
requirements yet...

> And of course, whether changes in the mainline kernel tree are
> manually propagated changes from the xfstests.git tree, or whether
> primary development happens in the kernel tree, is ultimately going to
> be up to you and the XFS developers who have stewardship of xfstests.
> I'm not sure I would be that excited about manual propagation of
> changes from one git tree to another, but that is of course, up to
> you.

And this is exactly my point, Ted. Again, you are presuming that the
implementation is going to require syncing commits across disparate
git trees and other such games will be needed to maintain separate
packages. Nothing could be further from the truth: we already have
this problem with the shared XFS kernel/userspace code and it's a
royal PITA keeping them in sync. Hence introducing the same
maintenance problem with new code and infrastructure is highly
undesirable and something we'll try to avoid at all costs.

Basically, we need to start with a proposal that outlines what we
need in the kernel tree for testing filesystems in place and go from
there. Once we've agreed on the big picture stuff and solved the
maintenance, packaging and "how do we run in place" architectural
issues, then we can start working on the implementation. I'm hoping
to write something up in the next week or two to being the process
and discussions, because I'm sure the -stable folk will have some
input on what they would like to be able to do.... :)

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: [Lsf] [PATCH] xfstests-bld: Simplify determination of number of CPUs in build-all

[Theodore Ts'o]

On Wed, Apr 02, 2014 at 09:28:23AM +1100, Dave Chinner wrote:
> > And of course, whether changes in the mainline kernel tree are
> > manually propagated changes from the xfstests.git tree, or whether
> > primary development happens in the kernel tree, is ultimately going to
> > be up to you and the XFS developers who have stewardship of xfstests.
> > I'm not sure I would be that excited about manual propagation of
> > changes from one git tree to another, but that is of course, up to
> > you.
> 
> And this is exactly my point, Ted. Again, you are presuming that the
> implementation is going to require syncing commits across disparate
> git trees and other such games will be needed to maintain separate
> packages. Nothing could be further from the truth: we already have
> this problem with the shared XFS kernel/userspace code and it's a
> royal PITA keeping them in sync. Hence introducing the same
> maintenance problem with new code and infrastructure is highly
> undesirable and something we'll try to avoid at all costs.

Actually, I was presuming that the thing that makes the most sense was
to move all or most of the tests in xfstests into the kernel tests
tree.  And then you complained that I was making a presumption that
this was the only sane thing to do.  That's why I said, "if you want
to do something insane, be my guest".

I have nothing against doing a formal requirements process, that's
fine, but I think there are certain things about what the final
solution of "test in the kernel git tree" will look like that are
pretty obvious.

Cheers,

						- Ted

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [Lsf] [PATCH] xfstests-bld: Simplify determination of number of CPUs in build-all

[Dave Chinner]

On Wed, Apr 02, 2014 at 10:26:20AM -0400, Theodore Ts'o wrote:
> On Wed, Apr 02, 2014 at 09:28:23AM +1100, Dave Chinner wrote:
> > > And of course, whether changes in the mainline kernel tree are
> > > manually propagated changes from the xfstests.git tree, or whether
> > > primary development happens in the kernel tree, is ultimately going to
> > > be up to you and the XFS developers who have stewardship of xfstests.
> > > I'm not sure I would be that excited about manual propagation of
> > > changes from one git tree to another, but that is of course, up to
> > > you.
> > 
> > And this is exactly my point, Ted. Again, you are presuming that the
> > implementation is going to require syncing commits across disparate
> > git trees and other such games will be needed to maintain separate
> > packages. Nothing could be further from the truth: we already have
> > this problem with the shared XFS kernel/userspace code and it's a
> > royal PITA keeping them in sync. Hence introducing the same
> > maintenance problem with new code and infrastructure is highly
> > undesirable and something we'll try to avoid at all costs.
> 
> Actually, I was presuming that the thing that makes the most sense was
> to move all or most of the tests in xfstests into the kernel tests
> tree.  And then you complained that I was making a presumption that
> this was the only sane thing to do.  That's why I said, "if you want
> to do something insane, be my guest".

So, you think there's only one "sane" solution. That's a massive
assumption, and that's what I'm pointing out.

It's perfectly sane to treat the kernel tree as just another
downstream consumer of xfstests, and simple add infrastructure to
the kernel tree to source and build xfstests from the upstream repo.
Indeed, this is the existing model that seems to work for just about
everyone, so I see no reason why we should treat Greg's request any
differently.

IOWs, you're assumming that the only thing that makes sense is that
"kernel tree == upstream". We don't push code into the kernel tree
just because it's the Kernel Tree. We push code there if it makes
sense to do so - either from a commercial or technical PO. We have
no commercial motive to move xfstests into the kernel tree, and I
see a lot technical downsides to moving xfstests in it's entirity to the
kernel tree.

So, really, putting some infrastructure into the kernel tree to use
xfstests (i.e. a downstream consumer) makes a lot more sense from a
technical point of view than making the kernel tree the xfstests
upstream.

It's a sane way to solve the problem that Greg wanted solved - the
kernel tree carries the infrastructure needed to solve the "test in
place" use case that -stable maintainers want, but it doesn't turn
test development nor filesystem developer work flows on their head
nor make xfstests less useful to existing users.

> I have nothing against doing a formal requirements process, that's

I'm not doing anything formal, just doing the right thing to make
sure we don't do all the work and then be told "not acceptable"....

> fine, but I think there are certain things about what the final
> solution of "test in the kernel git tree" will look like that are
> pretty obvious.

Certain things, yes. Like using kvm to run the tests in a container.
Just what form that takes is not clear right now...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: [Lsf] [PATCH] xfstests-bld: Simplify determination of number of CPUs in build-all

[Lukáš Czerner]

On Thu, 3 Apr 2014, Dave Chinner wrote:

> Date: Thu, 3 Apr 2014 12:14:11 +1100
> From: Dave Chinner <david@fromorbit.com>
> To: Theodore Ts'o <tytso@mit.edu>
> Cc: Sedat Dilek <sedat.dilek@gmail.com>, lsf@lists.linux-foundation.org,
>     linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, xfs@oss.sgi.com
> Subject: Re: [Lsf] [PATCH] xfstests-bld: Simplify determination of number of
>     CPUs in build-all
> 
> On Wed, Apr 02, 2014 at 10:26:20AM -0400, Theodore Ts'o wrote:
> > On Wed, Apr 02, 2014 at 09:28:23AM +1100, Dave Chinner wrote:
> > > > And of course, whether changes in the mainline kernel tree are
> > > > manually propagated changes from the xfstests.git tree, or whether
> > > > primary development happens in the kernel tree, is ultimately going to
> > > > be up to you and the XFS developers who have stewardship of xfstests.
> > > > I'm not sure I would be that excited about manual propagation of
> > > > changes from one git tree to another, but that is of course, up to
> > > > you.
> > > 
> > > And this is exactly my point, Ted. Again, you are presuming that the
> > > implementation is going to require syncing commits across disparate
> > > git trees and other such games will be needed to maintain separate
> > > packages. Nothing could be further from the truth: we already have
> > > this problem with the shared XFS kernel/userspace code and it's a
> > > royal PITA keeping them in sync. Hence introducing the same
> > > maintenance problem with new code and infrastructure is highly
> > > undesirable and something we'll try to avoid at all costs.
> > 
> > Actually, I was presuming that the thing that makes the most sense was
> > to move all or most of the tests in xfstests into the kernel tests
> > tree.  And then you complained that I was making a presumption that
> > this was the only sane thing to do.  That's why I said, "if you want
> > to do something insane, be my guest".
> 
> So, you think there's only one "sane" solution. That's a massive
> assumption, and that's what I'm pointing out.
> 
> It's perfectly sane to treat the kernel tree as just another
> downstream consumer of xfstests, and simple add infrastructure to
> the kernel tree to source and build xfstests from the upstream repo.

That's what I think is the best approach as well. We'll still have
separate tree for the xfstests which works for everyone and
infrastructure in kernel tree which will simply use it.

Looking at rcutorture tests in kernel tree, they are indeed using
qemu and it looks like that they are running those tests directly
from init script - we really need an image to boot from.

We could use virtme

https://git.kernel.org/cgit/utils/kernel/virtme/virtme.git/

which seems like a best option. Alternatively we could allow users
passing their own image as well.

Then we would need init script which would download and build
xfstests, xfsprogs and possible other dependencies (such as fio)
and finally run xfstests itself. Colnsole output from the qemu can
be redirected to the file in the host, so that's ok, but we still
need a way to retrieve 'retults' directory from the test.

We'd also need a devices to test on (at least TEST_DEV and
SCRATCH_DEV). Those could be files a of set size exported to qemu as
block devices.

I think that looking at rcutorture test infrastructure is a good
start and we can possibly reuse parts of it for actually running the
qemu and then reuse parts of Ted's solution to run xfstests in the
virtual guest.

Thanks!
-Lukas

> Indeed, this is the existing model that seems to work for just about
> everyone, so I see no reason why we should treat Greg's request any
> differently.
> 
> IOWs, you're assumming that the only thing that makes sense is that
> "kernel tree == upstream". We don't push code into the kernel tree
> just because it's the Kernel Tree. We push code there if it makes
> sense to do so - either from a commercial or technical PO. We have
> no commercial motive to move xfstests into the kernel tree, and I
> see a lot technical downsides to moving xfstests in it's entirity to the
> kernel tree.
> 
> So, really, putting some infrastructure into the kernel tree to use
> xfstests (i.e. a downstream consumer) makes a lot more sense from a
> technical point of view than making the kernel tree the xfstests
> upstream.
> 
> It's a sane way to solve the problem that Greg wanted solved - the
> kernel tree carries the infrastructure needed to solve the "test in
> place" use case that -stable maintainers want, but it doesn't turn
> test development nor filesystem developer work flows on their head
> nor make xfstests less useful to existing users.
> 
> > I have nothing against doing a formal requirements process, that's
> 
> I'm not doing anything formal, just doing the right thing to make
> sure we don't do all the work and then be told "not acceptable"....
> 
> > fine, but I think there are certain things about what the final
> > solution of "test in the kernel git tree" will look like that are
> > pretty obvious.
> 
> Certain things, yes. Like using kvm to run the tests in a container.
> Just what form that takes is not clear right now...
> 
> Cheers,
> 
> Dave.
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [Lsf] [PATCH] xfstests-bld: Simplify determination of number of CPUs in build-all

[Andy Lutomirski]

On Thu, Apr 3, 2014 at 3:26 AM, Lukáš Czerner <lczerner@redhat.com> wrote:
> On Thu, 3 Apr 2014, Dave Chinner wrote:
>
>> Date: Thu, 3 Apr 2014 12:14:11 +1100
>> From: Dave Chinner <david@fromorbit.com>
>> To: Theodore Ts'o <tytso@mit.edu>
>> Cc: Sedat Dilek <sedat.dilek@gmail.com>, lsf@lists.linux-foundation.org,
>>     linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, xfs@oss.sgi.com
>> Subject: Re: [Lsf] [PATCH] xfstests-bld: Simplify determination of number of
>>     CPUs in build-all
>>
>> On Wed, Apr 02, 2014 at 10:26:20AM -0400, Theodore Ts'o wrote:
>> > On Wed, Apr 02, 2014 at 09:28:23AM +1100, Dave Chinner wrote:
>> > > > And of course, whether changes in the mainline kernel tree are
>> > > > manually propagated changes from the xfstests.git tree, or whether
>> > > > primary development happens in the kernel tree, is ultimately going to
>> > > > be up to you and the XFS developers who have stewardship of xfstests.
>> > > > I'm not sure I would be that excited about manual propagation of
>> > > > changes from one git tree to another, but that is of course, up to
>> > > > you.
>> > >
>> > > And this is exactly my point, Ted. Again, you are presuming that the
>> > > implementation is going to require syncing commits across disparate
>> > > git trees and other such games will be needed to maintain separate
>> > > packages. Nothing could be further from the truth: we already have
>> > > this problem with the shared XFS kernel/userspace code and it's a
>> > > royal PITA keeping them in sync. Hence introducing the same
>> > > maintenance problem with new code and infrastructure is highly
>> > > undesirable and something we'll try to avoid at all costs.
>> >
>> > Actually, I was presuming that the thing that makes the most sense was
>> > to move all or most of the tests in xfstests into the kernel tests
>> > tree.  And then you complained that I was making a presumption that
>> > this was the only sane thing to do.  That's why I said, "if you want
>> > to do something insane, be my guest".
>>
>> So, you think there's only one "sane" solution. That's a massive
>> assumption, and that's what I'm pointing out.
>>
>> It's perfectly sane to treat the kernel tree as just another
>> downstream consumer of xfstests, and simple add infrastructure to
>> the kernel tree to source and build xfstests from the upstream repo.
>
> That's what I think is the best approach as well. We'll still have
> separate tree for the xfstests which works for everyone and
> infrastructure in kernel tree which will simply use it.
>
> Looking at rcutorture tests in kernel tree, they are indeed using
> qemu and it looks like that they are running those tests directly
> from init script - we really need an image to boot from.
>
> We could use virtme
>
> https://git.kernel.org/cgit/utils/kernel/virtme/virtme.git/

virtme will eventually be able to use a separate OS image, probably in
the form of a directory with appropriate xattrs set.  I could support
images on a block device too, but that's boring :)

>
> which seems like a best option. Alternatively we could allow users
> passing their own image as well.
>
> Then we would need init script which would download and build
> xfstests, xfsprogs and possible other dependencies (such as fio)
> and finally run xfstests itself. Colnsole output from the qemu can
> be redirected to the file in the host, so that's ok, but we still
> need a way to retrieve 'retults' directory from the test.
>
> We'd also need a devices to test on (at least TEST_DEV and
> SCRATCH_DEV). Those could be files a of set size exported to qemu as
> block devices.

I just pushed a pair of sample scripts that invokes xfstests inside
virtme.  They're a total of 42 lines, including comments and
instructions.  You can now do:

PATH=.:$PATH samples/xfstests test.img scratch.img ~/apps/xfstests/

from inside a virtme checkout.  You'll have to compile xfstests first, though.

They will be considerably more useful once I add read-write host
windows to virtme.

If anyone wants to use virtme for something other than
experimentation, let me know and we can talk about interface
stability.

Some xfstests thoughts:

 - There's an undocumented way to write results outside the source
tree called RESULT_BASE.  It would be great if it were documented and
spelled consistently.

 - xfstests does not appreciate using symlinks to device nodes.

 - SCRATCH_MNT needs to be in /etc/fstab.  I think that this should be
changed or documented.  If the latter, then SCRATCH_DEV seems
redundant.

Some virtme thoughts:

- --script-sh is awkward.  Suggestions for a better interface are welcome.

 -  It would be great if the virtconsole driver were less buggy.  Even
the virtualized ttyS0 is slow enough to account for a significant
fraction of boot time.  It would also be great if the connection
between virtio ports and hvc devices were exposed in sysfs and through
/dev symlinks.

 - earlyprintk=virtconsole would be awesome.  This might need help
from qemu, though.  Or maybe there's a way to get the debug port to
work for that.

 - I should probably add simple way to use forcibly cached disks.

--Andy

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [Lsf] [PATCH] xfstests-bld: Simplify determination of number of CPUs in build-all

[Theodore Ts'o]

On Thu, Apr 03, 2014 at 10:05:26AM -0700, Andy Lutomirski wrote:
> 
> virtme will eventually be able to use a separate OS image, probably in
> the form of a directory with appropriate xattrs set.  I could support
> images on a block device too, but that's boring :)

When you say OS image, you mean "root file system, right"?  One of the
reasons why I'm actually build an actual root-file system image, and
didn't try the virtme approach was that I wanted to boot 32-bit
kernels on my development machine, which is 64-bit.

Having a 32-bit chroot environment would certainly work, though, and
would save the effort of creating the root file system image....  (and
of course having a 32-bit userspace also is a great way of exercising
the ioctl compatibility code paths :-).

> from inside a virtme checkout.  You'll have to compile xfstests first, though.

Fortunately, xfstests-bld will handle do this for you, since it grabs
and builds all of the depedencies automatically.  More importantly, it
allows the dependencies to be saved as part of the test output since
that's important when trying to have other people understand how to
reproduce a particular test result (since sometimes the latest
xfstests requires the latest xfs_io from xfsprogs, so it's a bad idea
to depend on the version of xfsprogs shipped by your distribution).
For example, for this merge window, I've been using the following to
do my tests:

fio		fio-2.1-19-g0b14f0a (Thu, 23 May 2013 21:27:54 +0200)
quota		0d0a674 (Tue, 26 Mar 2013 17:13:33 +0100)
xfsprogs	v3.2.0-alpha2-60-gaa210c4 (Thu, 13 Mar 2014 21:23:50 +1100)
xfstests-bld	1efde7a (Tue, 1 Apr 2014 14:42:07 -0400)
xfstests	linux-v3.8-336-g3948694 (Thu, 13 Mar 2014 15:20:54 +1100)

> They will be considerably more useful once I add read-write host
> windows to virtme.

Yes, you definitely want that for the results directories.

>  - There's an undocumented way to write results outside the source
> tree called RESULT_BASE.  It would be great if it were documented and
> spelled consistently.

There are a bunch of inconsistencies, which I've chalked up to
historical accidents and a desire to not break compatibility with
developers' test runners.  You mount the $SCRATCH_DIR on SCRATCH_MNT
but you mount $TEST_DEV on $TEST_DIR, for example.  I've just learned
to live with it....

>  - SCRATCH_MNT needs to be in /etc/fstab.  I think that this should be
> changed or documented.  If the latter, then SCRATCH_DEV seems
> redundant.

The various test scripts do need to be able to find the device where
the file system lives, and parsing /etc/fstab would be awkward.  So if
your comment is that either the /etc/fstab entry shouldn't be
required, or the xfstests runtime environment should be able to derive
$SCRATCH_DEV automatically from $SCRATCH_MNT, or vice versa, instead
of having the user specify both, I'd agree that would be nice, but
that's why I put together scripts like the ones I have in xfstests-bld
--- to make life easier.  :-)

Cheers,

						- Ted

Re: [Lsf] [PATCH] xfstests-bld: Simplify determination of number of CPUs in build-all

[Andy Lutomirski]

On Thu, Apr 3, 2014 at 10:35 AM, Theodore Ts'o <tytso@mit.edu> wrote:
> On Thu, Apr 03, 2014 at 10:05:26AM -0700, Andy Lutomirski wrote:
>>
>> virtme will eventually be able to use a separate OS image, probably in
>> the form of a directory with appropriate xattrs set.  I could support
>> images on a block device too, but that's boring :)
>
> When you say OS image, you mean "root file system, right"?  One of the
> reasons why I'm actually build an actual root-file system image, and
> didn't try the virtme approach was that I wanted to boot 32-bit
> kernels on my development machine, which is 64-bit.
>

Yes.  I've actually done this to test some vdso stuff.

> Having a 32-bit chroot environment would certainly work, though, and
> would save the effort of creating the root file system image....  (and
> of course having a 32-bit userspace also is a great way of exercising
> the ioctl compatibility code paths :-).

I agree, as long as there are giant quote marks around chroot.  No
actual chroot would be involved.

I'd like to support non-x86 architectures, too.  Last time I tried to
convince a modern kernel to boot on a released QEMU on ARM was
painful, though.  Maybe -M virt is the way to go here.

The tricky part here is that virtme currently relies on finding a
statically-linked busybox binary in $PATH.  It'll need to learn how to
find one that will run on the guest, or it will need to learn how to
live without busybox.

>
>> from inside a virtme checkout.  You'll have to compile xfstests first, though.
>
> Fortunately, xfstests-bld will handle do this for you, since it grabs
> and builds all of the depedencies automatically.  More importantly, it
> allows the dependencies to be saved as part of the test output since
> that's important when trying to have other people understand how to
> reproduce a particular test result (since sometimes the latest
> xfstests requires the latest xfs_io from xfsprogs, so it's a bad idea
> to depend on the version of xfsprogs shipped by your distribution).
> For example, for this merge window, I've been using the following to
> do my tests:
>
> fio             fio-2.1-19-g0b14f0a (Thu, 23 May 2013 21:27:54 +0200)
> quota           0d0a674 (Tue, 26 Mar 2013 17:13:33 +0100)
> xfsprogs        v3.2.0-alpha2-60-gaa210c4 (Thu, 13 Mar 2014 21:23:50 +1100)
> xfstests-bld    1efde7a (Tue, 1 Apr 2014 14:42:07 -0400)
> xfstests        linux-v3.8-336-g3948694 (Thu, 13 Mar 2014 15:20:54 +1100)

I haven't actually looked at xfstests-bld yet.  I suspect it could be
made to work with virtme fairly easily.  My current hack uses assumes
you'll use distro packages for all the dependencies.

>
>> They will be considerably more useful once I add read-write host
>> windows to virtme.
>
> Yes, you definitely want that for the results directories.
>
>>  - There's an undocumented way to write results outside the source
>> tree called RESULT_BASE.  It would be great if it were documented and
>> spelled consistently.
>
> There are a bunch of inconsistencies, which I've chalked up to
> historical accidents and a desire to not break compatibility with
> developers' test runners.  You mount the $SCRATCH_DIR on SCRATCH_MNT
> but you mount $TEST_DEV on $TEST_DIR, for example.  I've just learned
> to live with it....

Given that RESULTS_BASE only occurs in an error message, I think it
could be fixed without breaking compatibility.

>
>>  - SCRATCH_MNT needs to be in /etc/fstab.  I think that this should be
>> changed or documented.  If the latter, then SCRATCH_DEV seems
>> redundant.
>
> The various test scripts do need to be able to find the device where
> the file system lives, and parsing /etc/fstab would be awkward.  So if
> your comment is that either the /etc/fstab entry shouldn't be
> required, or the xfstests runtime environment should be able to derive
> $SCRATCH_DEV automatically from $SCRATCH_MNT, or vice versa, instead
> of having the user specify both, I'd agree that would be nice, but
> that's why I put together scripts like the ones I have in xfstests-bld
> --- to make life easier.  :-)
>
> Cheers,
>
>                                                 - Ted



-- 
Andy Lutomirski
AMA Capital Management, LLC

Re: [Lsf] [PATCH] xfstests-bld: Simplify determination of number of CPUs in build-all

[Eric Sandeen]

On 4/3/14, 11:35 AM, Theodore Ts'o wrote:

>>  - There's an undocumented way to write results outside the source
>> tree called RESULT_BASE.  It would be great if it were documented and
>> spelled consistently.

I'm not actually certain that it was intended to be used this way.
See 1686f9ab "xfstests: Introduce a results directory" 
which explains just where this variable came from and what it's
for...

So that's probably why it's undocumented; I don't think it was
envisioned as a configurable.  As for consistency... patch
sent for the typo.

If the functionality is needed, just make sure it works right if
you set it manually, update the user docs, & send a patch.

> There are a bunch of inconsistencies, which I've chalked up to
> historical accidents and a desire to not break compatibility with
> developers' test runners.  You mount the $SCRATCH_DIR on SCRATCH_MNT

$SCRATCH_DEV you mean. ;)  I don't think there's any real resistance
to fixing things that really need to be fixed, but this one
doesn't seem too critical.  OTOH, adding an alias from SCRATCH_MNT
to SCRATCH_DIR for consistency could surely be done if anyone cared
enough to send the patch.

> but you mount $TEST_DEV on $TEST_DIR, for example.  I've just learned
> to live with it....
> 
>>  - SCRATCH_MNT needs to be in /etc/fstab.  I think that this should be
>> changed or documented.  If the latter, then SCRATCH_DEV seems
>> redundant.

Hm, I've never needed SCRATCH_MNT in /etc/fstab... 

> The various test scripts do need to be able to find the device where
> the file system lives, and parsing /etc/fstab would be awkward.  So if
> your comment is that either the /etc/fstab entry shouldn't be
> required, or the xfstests runtime environment should be able to derive
> $SCRATCH_DEV automatically from $SCRATCH_MNT, or vice versa, instead

I guess I don't know why you'd expect to derive one from the other...

> of having the user specify both, I'd agree that would be nice, but
> that's why I put together scripts like the ones I have in xfstests-bld
> --- to make life easier.  :-)

All I've ever had to do is set up the 4 variables in local.config.example
(by copying to local.config & editing appropriately) and it all just works
AFAIK.

(No doubt docs could be improved, but we can do that by sending patches.)  :)

-Eric

Re: [Lsf] [PATCH] xfstests-bld: Simplify determination of number of CPUs in build-all

[Andy Lutomirski]

On Thu, Apr 3, 2014 at 12:06 PM, Eric Sandeen <sandeen@redhat.com> wrote:
> On 4/3/14, 11:35 AM, Theodore Ts'o wrote:
>
>>>  - There's an undocumented way to write results outside the source
>>> tree called RESULT_BASE.  It would be great if it were documented and
>>> spelled consistently.
>
> I'm not actually certain that it was intended to be used this way.
> See 1686f9ab "xfstests: Introduce a results directory"
> which explains just where this variable came from and what it's
> for...

Yeah.  A real solution should just create and overmount results.

Unfortunately, AFAICT it's currently impossible to create a mountpoint
with no underlying dentry or to overmount a symlink, so this can be a
bit awkward.

>>>  - SCRATCH_MNT needs to be in /etc/fstab.  I think that this should be
>>> changed or documented.  If the latter, then SCRATCH_DEV seems
>>> redundant.
>
> Hm, I've never needed SCRATCH_MNT in /etc/fstab...
>
>> The various test scripts do need to be able to find the device where
>> the file system lives, and parsing /etc/fstab would be awkward.  So if
>> your comment is that either the /etc/fstab entry shouldn't be
>> required, or the xfstests runtime environment should be able to derive
>> $SCRATCH_DEV automatically from $SCRATCH_MNT, or vice versa, instead
>
> I guess I don't know why you'd expect to derive one from the other...

Sigh.

If $SCRATCH_MNT is specified, then the line in /etc/fstab is
unnecessary.  If $SCRATCH_MNT is not specified, then /etc/fstab will
do the trick.

What does not work is specifying $SCRATCH_DIR [sic] but not adding an
fstab entry.  Oops.


--Andy

Re: [Lsf] [PATCH] xfstests-bld: Simplify determination of number of CPUs in build-all

[Eric Sandeen]

On 4/3/14, 1:21 PM, Andy Lutomirski wrote:
> On Thu, Apr 3, 2014 at 12:06 PM, Eric Sandeen <sandeen@redhat.com> wrote:

...

>>> The various test scripts do need to be able to find the device where
>>> the file system lives, and parsing /etc/fstab would be awkward.  So if
>>> your comment is that either the /etc/fstab entry shouldn't be
>>> required, or the xfstests runtime environment should be able to derive
>>> $SCRATCH_DEV automatically from $SCRATCH_MNT, or vice versa, instead
>>
>> I guess I don't know why you'd expect to derive one from the other...
> 
> Sigh.
> 
> If $SCRATCH_MNT is specified, then the line in /etc/fstab is
> unnecessary.  If $SCRATCH_MNT is not specified, then /etc/fstab will
> do the trick.
> 
> What does not work is specifying $SCRATCH_DIR [sic] but not adding an
> fstab entry.  Oops.

Oh, I see.

I'd never really thought about xfstests devices as "part of the system" -
since they're constantly scribbled on, re-made, etc, and managed wholly
by xfstests (at least the scratch device) - so the notion of using fstab
was just outside my realm of expected behaviors.  :)

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [Lsf] [PATCH] xfstests-bld: Simplify determination of number of CPUs in build-all

[Theodore Ts'o]

On Thu, Apr 03, 2014 at 01:06:40PM -0600, Eric Sandeen wrote:
> On 4/3/14, 11:35 AM, Theodore Ts'o wrote:
> 
> >>  - There's an undocumented way to write results outside the source
> >> tree called RESULT_BASE.  It would be great if it were documented and
> >> spelled consistently.
> 
> I'm not actually certain that it was intended to be used this way.
> See 1686f9ab "xfstests: Introduce a results directory" 
> which explains just where this variable came from and what it's
> for...

The last paragraph states of the git description states:

    This is the first (small) step in being able to store test results
    in an external location for archival/data mining purposes.

The question I suppose is whether storing the results in an external
location is something that should be xfstests' responsibility, or of
the test harnesses that call xfstests.  I suspect that since there are
so many different ways people might want to archive the results, it
does make sense for the test harnesses to handle this job, but I
wouldn't want to "presume" on anything that the xfstests developers
might choose to do....

						- Ted

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [Lsf] [PATCH] xfstests-bld: Simplify determination of number of CPUs in build-all

[Dave Chinner]

On Thu, Apr 03, 2014 at 03:30:40PM -0400, Theodore Ts'o wrote:
> On Thu, Apr 03, 2014 at 01:06:40PM -0600, Eric Sandeen wrote:
> > On 4/3/14, 11:35 AM, Theodore Ts'o wrote:
> > 
> > >>  - There's an undocumented way to write results outside the source
> > >> tree called RESULT_BASE.  It would be great if it were documented and
> > >> spelled consistently.
> > 
> > I'm not actually certain that it was intended to be used this way.
> > See 1686f9ab "xfstests: Introduce a results directory" 
> > which explains just where this variable came from and what it's
> > for...
> 
> The last paragraph states of the git description states:
> 
>     This is the first (small) step in being able to store test results
>     in an external location for archival/data mining purposes.
> 
> The question I suppose is whether storing the results in an external
> location is something that should be xfstests' responsibility, or of
> the test harnesses that call xfstests.

The idea is that the external test harnesses can set up a
destination directory for the results (i.e. RESULT_BASE) and
xfstests will write all results into that directory.  i.e. setting
up a results store and managing it is something that is currently
done external to xfstests.

Managing a results store is complex and nobody has sent patches to
do this in a generic way, so all we currently have is a mechanism to
enable such functionality. Patches welcome!

> I suspect that since there are
> so many different ways people might want to archive the results, it
> does make sense for the test harnesses to handle this job, but I
> wouldn't want to "presume" on anything that the xfstests developers
> might choose to do....

ROWR! :)

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [Lsf] [PATCH] xfstests-bld: Simplify determination of number of CPUs in build-all

[Mel Gorman]

On Wed, Apr 02, 2014 at 10:26:20AM -0400, Theodore Ts'o wrote:
> On Wed, Apr 02, 2014 at 09:28:23AM +1100, Dave Chinner wrote:
> > > And of course, whether changes in the mainline kernel tree are
> > > manually propagated changes from the xfstests.git tree, or whether
> > > primary development happens in the kernel tree, is ultimately going to
> > > be up to you and the XFS developers who have stewardship of xfstests.
> > > I'm not sure I would be that excited about manual propagation of
> > > changes from one git tree to another, but that is of course, up to
> > > you.
> > 
> > And this is exactly my point, Ted. Again, you are presuming that the
> > implementation is going to require syncing commits across disparate
> > git trees and other such games will be needed to maintain separate
> > packages. Nothing could be further from the truth: we already have
> > this problem with the shared XFS kernel/userspace code and it's a
> > royal PITA keeping them in sync. Hence introducing the same
> > maintenance problem with new code and infrastructure is highly
> > undesirable and something we'll try to avoid at all costs.
> 
> Actually, I was presuming that the thing that makes the most sense was
> to move all or most of the tests in xfstests into the kernel tests
> tree.  And then you complained that I was making a presumption that
> this was the only sane thing to do.  That's why I said, "if you want
> to do something insane, be my guest".
> 

FWIW, I also don't think that xfstests or a framework should go into the
kernel tree for the following reasons

1. Tests can be created during the release cycle and it does not make
   sense to tie the availabilty of tests to the availability of a kernel.
   If someone is running tests against historical tests, they must first
   checkout the most recent kernel tree, build xfstests and then checkout
   the old tree. Yes, users can be given a patch against the kernel tree
   to add a new test but they still have this awkward switching back and
   forth between the branch that has the test and the kernel they are
   running the test against.

   Recent example -- trinity 1.3 does not trigger MM bugs. Until recently,
   the latest git version of trinity caused massive damage.  If trinity had
   been in the tree someone might have accidentally concluded that 3.13 was
   "fine"

2. It would be relatively easy for someone to mistakenly bisect xfstests
   instead of bisecting the kernel although my understanding is that tests
   do not change per-se, only new ones are added. There may be
   exceptions to this rule

3. With testing automation, it's not that hard to download the
   requirements for it. mmtests prefers to use a local mirror but falls
   back to downloading tarballs, checking out git trees or mercurcial
   trees as appropriate. It's just a few lines of code (most of which is
   auto-generated from a template).

4. Testing monoculture. Sometimes the value of a benchmark is because
   different people run it differently for whatever reason. My IO tests for
   example all tune the benchmark to at least 2*RAM but that misses the cases
   where there are regressions in IO to a file that fits in memory. xfstests
   might have similar problems but minimally it would be a shame if the
   framework always used the same mount options for example. FWIW I've
   had bugs reported that were real bugs and only revealed because the
   benchmark parameters made no sense but still found a bug by accident.

Overall I have little motivation to having all the tests in the kernel
tree. That said, I already have an automated framework I can use so I
would not feel as strongly about the problem.

The point is that putting tests in tree or out of tree is neither sane nor
insane. It's simply different approaches so people on the thread should
be wary of trying to label the other person as being mental just because
of where they want to stick a test.

-- 
Mel Gorman
SUSE Labs

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs