[PATCH 0/6] xfstests-bld: Fixes and improvements v1

[PATCH 0/6] xfstests-bld: Fixes and improvements v1

[Dmitry Monakhov]

Hi,

Please review and pull my patch-queue for xfstests-bld
# Fix build (currently xfstests-bld is broken)
xfstests-bld: build update xfstests repo
fix build typos

# Minor improvements
kvm-xfstests: update command line help
xfstest-bld: use parallel gzip if possible
kvm-xfstests: add initrd support

# Dockerize xfstests-bld build enviroment
# github has integration with hub.docker so each commit update
# trigger auto build so we have CI for free.
build: Add dockerfile


https://github.com/dmonakhov/xfstests-bld to_pull-v1

Re: [PATCH 6/6] Add dockerfile

[Theodore Ts'o]

On Fri, Aug 19, 2016 at 12:54:11AM +0400, Dmitry Monakhov wrote:
> Dockerfile is good way to create build environments
> Let's keep it as a reference build script.
> 
> XXX: Currently Ted does not have hub.docker account
>      so this docker file points to my hub. Should be updated.

I guess I'm a little confused about how this Dockerfile works.  First
of all, says "From Debian", so I guess the idea is to be based on an
unspecified version of Debian (why not debian:jessie?).  But then it
has apt-get commands?  How does that work?

And if the idea is to fetch a prebuilt docker image, and we're running
it with docker run --privilege, I'm not sure what value using docker
run is really providing.  We're not using the docker container
features.   And typing

docker run -i -t --privileged ... \
	"kvm-xfstests.sh --kernel /tmp/bzImage --update-files --update-xfstests-tar  smoke"

Is just *awkward*.  The whole point is to let kvm-xfstests read the
kernel from your ~/.config/gce-xfstests, so your work flow can be
something like this:

git am -s ~/mbox/patch.mbox
../make-ext4      # shortcut for something like "make O=/build/ext4 -j16"
kvm-xfstests smoke   # ~/.config/gce-xfststs sets GCE_KERNEL=/build/ext4

needing to type a long docker run command doesn't seem to add any
value.

Confused,

					- Ted

Re: [PATCH 6/6] Add dockerfile

[Dmitry Monakhov]

Theodore Ts'o <tytso@mit.edu> writes:

> On Fri, Aug 19, 2016 at 12:54:11AM +0400, Dmitry Monakhov wrote:
>> Dockerfile is good way to create build environments
>> Let's keep it as a reference build script.
>> 
>> XXX: Currently Ted does not have hub.docker account
>>      so this docker file points to my hub. Should be updated.
>
> I guess I'm a little confused about how this Dockerfile works.  First
> of all, says "From Debian", so I guess the idea is to be based on an
> unspecified version of Debian (why not debian:jessie?).  But then it
> has apt-get commands?  How does that work?
Yes, this is an official debian image https://hub.docker.com/_/debian .
plain 'debian' means 'debian:jessie', but you can change it to 'debian:sid'
This image already has apt-get  (otherwise it will be not very useful)
>
> And if the idea is to fetch a prebuilt docker image, and we're running
> it with docker run --privilege, I'm not sure what value using docker
> run is really providing.  We're not using the docker container
> features.   And typing
>
> docker run -i -t --privileged ... \
> 	"kvm-xfstests.sh --kernel /tmp/bzImage --update-files --update-xfstests-tar  smoke"
>
> Is just *awkward*.  The whole point is to let kvm-xfstests read the
> kernel from your ~/.config/gce-xfstests, so your work flow can be
> something like this:
>
Actually main idea is to have build environment for xfstests-bld development
plus automatic builds gives us CI. This is important because
xfstests-blk build is silently broken since May 30 due to 82c3a179
(fix proposed: https://github.com/dmonakhov/xfsprogs/commit/5dc7a0805d6b7a265863413a8e1b7acb191d5280)
But this was uncovered because most people do not build xfstests-bld
from scratch very often, we all too busy. Let's machine do that :)
In fact other good candidate for CI is https://travis-ci.com/

Once we have Dockerfile which encapsulates all dependencies
one can create xfstets-blk from scratch simply typing
$ docker build -t my-xfstests-bld-image --file Dockerfile
Or use precreated environment
$ docker pull XXX/xfstests-bld

Once image is created 
Log in to container and start hacking xfstest or e2fsprogs etc.
$ docker run -i -t XXX/xfstests-bld
$ emacs xfstest/tests/ext4/023 ....
$ kvm-xfstests --update-xfstest ext4/023
...

Or run push your kernel patches to container via bindmount
and then submit it to gce.
$ docker run -v my-stuff:/my-stuff XXX/xfstests-bld \
                git clone kernel.git && \
                cd kernel && \
                git am -s /my-stuff/patch.mbox && \
                make O=/build/ext4 -j16 && \
                cp /build/ext4/arch/x86/boot/bzImage /my-stuff && \
                cp my-stuff/gce-config ~/.config/gce-config && \
                gce-xfstests.sh --kernel /my-stuff/bzImage smoke

Other useful feature is ability to speed xfstests-bld by splitting
full test-set to groups and run kvm-xfstests instances in parallel.
For example:
docker run -v my-kernel:/my-kernel XXX/xfstests-bld  \
       kvm-xfstests.sh --kernel /my-kernel/bzImage -c 1k > 1k.log &
docker run -v my-kernel:/my-kernel XXX/xfstests-bld  \
       kvm-xfstests.sh --kernel /my-kernel/bzImage -c 4k > 4k.log &
docker run -v my-kernel:/my-kernel XXX/xfstests-bld  \
       kvm-xfstests.sh --kernel /my-kernel/bzImage -c dax > dax.log &
.....
w/o sand box they will conflict due to net ports (localhost:7500 etc)
In my case speed up is about x10
> git am -s ~/mbox/patch.mbox
> ../make-ext4      # shortcut for something like "make O=/build/ext4 -j16"
> kvm-xfstests smoke   # ~/.config/gce-xfststs sets GCE_KERNEL=/build/ext4
>
> needing to type a long docker run command doesn't seem to add any
> value.
>
> Confused,
>
> 					- Ted
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 6/6] Add dockerfile

[Theodore Ts'o]

On Fri, Aug 19, 2016 at 12:27:43PM +0300, Dmitry Monakhov wrote:
> Actually main idea is to have build environment for xfstests-bld development
> plus automatic builds gives us CI. This is important because
> xfstests-blk build is silently broken since May 30 due to 82c3a179
> (fix proposed: https://github.com/dmonakhov/xfsprogs/commit/5dc7a0805d6b7a265863413a8e1b7acb191d5280)
> But this was uncovered because most people do not build xfstests-bld
> from scratch very often, we all too busy. Let's machine do that :)
> In fact other good candidate for CI is https://travis-ci.com/

Actually, I'm rebuilding all the time, but I'm not upgrading to the
latest bleeding edge version of xfsprogs all the time.  In fact, I'm
currently still using xfsprogs v4.5.0, since that's plenty for the
testing I'm doing.  I have historically pinned the commit Id's for
certain projects where newer versions were either (a) known to be
flaky, or (b) known to be broken in some way -- that's why we
currently have explicit commit ID's for fio and quota.  Perhaps I
should add one for xfsprogs as well, so that we only upgrade to newer
versions when it known to be needed and where it's helpful.

> Once we have Dockerfile which encapsulates all dependencies
> one can create xfstets-blk from scratch simply typing
> $ docker build -t my-xfstests-bld-image --file Dockerfile
> Or use precreated environment
> $ docker pull XXX/xfstests-bld

If you use the precreated environment that will have a fixed commit
ID.  So the advantage of doing continuous integration of the bleeding
edge version of xfsprogs only works if you are using the "docker
build" line, right.

> Once image is created 
> Log in to container and start hacking xfstest or e2fsprogs etc.
> $ docker run -i -t XXX/xfstests-bld
> $ emacs xfstest/tests/ext4/023 ....
> $ kvm-xfstests --update-xfstest ext4/023
> ...

I'm not sure I see the advantage of doing this in a container, I
guess.  I just do in my standard laptop environment today.

> Other useful feature is ability to speed xfstests-bld by splitting
> full test-set to groups and run kvm-xfstests instances in parallel.
> For example:
> docker run -v my-kernel:/my-kernel XXX/xfstests-bld  \
>        kvm-xfstests.sh --kernel /my-kernel/bzImage -c 1k > 1k.log &
> docker run -v my-kernel:/my-kernel XXX/xfstests-bld  \
>        kvm-xfstests.sh --kernel /my-kernel/bzImage -c 4k > 4k.log &
> docker run -v my-kernel:/my-kernel XXX/xfstests-bld  \
>        kvm-xfstests.sh --kernel /my-kernel/bzImage -c dax > dax.log &
> .....
> w/o sand box they will conflict due to net ports (localhost:7500 etc)

... and because you will need separate scratch disks for each of the
docker runs.

OK, I can see how that's certainly an advantage..  This is what I use
gce-xfstests for, myself.  :-)

The main thing I don't really like about the Dockerfile is that with
the exception of the last use case, everyone would really need to have
their own customized version of the Docker file, right?  And for
someone who isn't Docker aware, I'm concerned that it might be very
confusing to use, especially compared to all of the documentation I
currently have for kvm-xfstests and gce-xfstests.

Hmm, I'll have to think about this and play with it a bit more...

     	       	  	      	       	    - Ted

Re: [PATCH 6/6] Add dockerfile

[Dmitry Monakhov]

[-- Attachment #1: Type: text/plain, Size: 4976 bytes --]

Theodore Ts'o <tytso@mit.edu> writes:

> On Fri, Aug 19, 2016 at 12:27:43PM +0300, Dmitry Monakhov wrote:
>> Actually main idea is to have build environment for xfstests-bld development
>> plus automatic builds gives us CI. This is important because
>> xfstests-blk build is silently broken since May 30 due to 82c3a179
>> (fix proposed: https://github.com/dmonakhov/xfsprogs/commit/5dc7a0805d6b7a265863413a8e1b7acb191d5280)
>> But this was uncovered because most people do not build xfstests-bld
>> from scratch very often, we all too busy. Let's machine do that :)
>> In fact other good candidate for CI is https://travis-ci.com/
>
> Actually, I'm rebuilding all the time, but I'm not upgrading to the
> latest bleeding edge version of xfsprogs all the time.  In fact, I'm
> currently still using xfsprogs v4.5.0, since that's plenty for the
> testing I'm doing.  I have historically pinned the commit Id's for
> certain projects where newer versions were either (a) known to be
> flaky, or (b) known to be broken in some way -- that's why we
> currently have explicit commit ID's for fio and quota.  Perhaps I
> should add one for xfsprogs as well, so that we only upgrade to newer
> versions when it known to be needed and where it's helpful.
Yes. That is reasonable point.
My original idea was to check that any commit of xfstests-bld will
healthy with default config. And if github/docker build farm may do it
for us for free is it not bad.
>
>> Once we have Dockerfile which encapsulates all dependencies
>> one can create xfstets-blk from scratch simply typing
>> $ docker build -t my-xfstests-bld-image --file Dockerfile
>> Or use precreated environment
>> $ docker pull XXX/xfstests-bld
>
> If you use the precreated environment that will have a fixed commit
> ID.  So the advantage of doing continuous integration of the bleeding
> edge version of xfsprogs only works if you are using the "docker
> build" line, right.
>
>> Once image is created 
>> Log in to container and start hacking xfstest or e2fsprogs etc.
>> $ docker run -i -t XXX/xfstests-bld
>> $ emacs xfstest/tests/ext4/023 ....
>> $ kvm-xfstests --update-xfstest ext4/023
>> ...
>
> I'm not sure I see the advantage of doing this in a container, I
> guess.  I just do in my standard laptop environment today.
I can not because I laptop from famous thinkpad t430 series with
flaky SSD which starts to return EIO after intensive load.
https://forums.lenovo.com/t5/ThinkPad-T400-T500-and-newer-T/T430s-Intel-SSD-520-180GB-issue/td-p/888083

Also. What about RH/SUSE/Fedora or even Gentoo people? They are exit :)
And if we give them just chance to try debian w/o complexities
maybe they will become debian adepts.
>
>> Other useful feature is ability to speed xfstests-bld by splitting
>> full test-set to groups and run kvm-xfstests instances in parallel.
>> For example:
>> docker run -v my-kernel:/my-kernel XXX/xfstests-bld  \
>>        kvm-xfstests.sh --kernel /my-kernel/bzImage -c 1k > 1k.log &
>> docker run -v my-kernel:/my-kernel XXX/xfstests-bld  \
>>        kvm-xfstests.sh --kernel /my-kernel/bzImage -c 4k > 4k.log &
>> docker run -v my-kernel:/my-kernel XXX/xfstests-bld  \
>>        kvm-xfstests.sh --kernel /my-kernel/bzImage -c dax > dax.log &
>> .....
>> w/o sand box they will conflict due to net ports (localhost:7500 etc)
>
> ... and because you will need separate scratch disks for each of the
> docker runs.
Space is not a problem for most people. Average disk size is more that 500Gb
>
> OK, I can see how that's certainly an advantage..  This is what I use
> gce-xfstests for, myself.  :-)
BTW: I would be good thing to have some sort of job preparation mode.
where test does not executed, but only listed. So one can get lists
of tests and then divide it to groups and run in parallel:
$ kvm-xfstests.sh -c 4k -g auto --list > all_tests.txt
# Next line divides all_tests.txt in 8 parts and run them parallel
$ parallel --joblog job.log -j 8 -m -a all_tests.txt "kvm-xfstests.sh -c 4k"
I'll send you a patch.
BTW: Sometimes many parallel jobs provoke BUGON on host kernel due to
some bug in KVM  :)
>
> The main thing I don't really like about the Dockerfile is that with
> the exception of the last use case, everyone would really need to have
> their own customized version of the Docker file, right?  And for
> someone who isn't Docker aware, I'm concerned that it might be very
> confusing to use, especially compared to all of the documentation I
> currently have for kvm-xfstests and gce-xfstests.
Absolutely no. Dockerfile is just script which used for container
creation(similar gen-image). As normal script it should be as generic
as possible. If I've embedded some ad-hoc references to my-config this
is just a common error than every scripts writers do :(
>
> Hmm, I'll have to think about this and play with it a bit more...
>
>      	       	  	      	       	    - Ted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 472 bytes --]

Re: [PATCH 6/6] Add dockerfile

[Theodore Ts'o]

On Sat, Aug 20, 2016 at 02:31:26PM +0300, Dmitry Monakhov wrote:
> > I'm not sure I see the advantage of doing this in a container, I
> > guess.  I just do in my standard laptop environment today.
> I can not because I laptop from famous thinkpad t430 series with
> flaky SSD which starts to return EIO after intensive load.
> https://forums.lenovo.com/t5/ThinkPad-T400-T500-and-newer-T/T430s-Intel-SSD-520-180GB-issue/td-p/888083

Hmm, I never buy a Lenovo SSD; they're overpriced.  So I always get my
Thinkpads with a 500mb HDD, and then replace it with a Samsumg 840/850
EVO/PRO SSD.

> Also. What about RH/SUSE/Fedora or even Gentoo people? They are exit :)
> And if we give them just chance to try debian w/o complexities
> maybe they will become debian adepts.

You only need Debain to build the image.  If you are just using the
pre-built root_fs.img, you can just download it and go.

Speaking of which, that's one thing which I don't understand about
your Dockerfile.  In the build step you just download the prebuilt
root_fs.img from kernel.org.  There are comments in your Dockerfile which say:

# Fetch and build xfstests-bld
# In order to build root_image from scratch privileged operation are required,
# so it is impossible during build stage.
# One can make it like this:
# docker run -i -t --privileged --rm dmonakhov/xfstests-bld "cd kvm-xfstests/test-appliance && ./gen-image"
# During build stage we simply fetch image precreated by tytso@

... but while this will create a new root_fs.img file, as soon as
you're you're done the container will exit, and then the root_fs.img
file will be blown away, right?

The other thing I'll note here is that the Dockerfile is serving two
use cases.  One is to user Docker to do automated build testing.  The
other is for people who want to run multiple instances of kvm-xfstests
using the container services.

For the second, we don't really want to include all of the build
artifacts and build and source trees in Docker image that the users
will have to pull down over the network.  And using kvm-xfstests
--update-xfstests means a lot of extra wasted I/O, since we're
packaging up the tar file, then copying it into file, and then
unpacking it inside the guest VM, etc.

So for the second case, you'd really want to create a Docker image
which had the minimum packages needed to actually *run* the tests,
with the pre-installed root_fs image.

For the first case, all you need to do is to have the Dockerfile
create the xfstests.tar.gz file.  I suppose this could be used as a
poor man's build engine if you don't want to run the build on your
local machine.  What I do is my top-level config.custom has:

BUILD_ENV="schroot -c jessie64 --"
SUDO_ENV="schroot -c jessie64 -u root --"

That's my pristine build chroot, and so when I run ./do-all, what gets
executed is:

schroot -c jessie64 -- make clean
schroot -c jessie64 -- make
schroot -c jessie64 -- make tarball
cd kvm-xfstests/test-appliance
schroot -c jessie64 -u root -- ./gen-image

I'm pretty sure this is going to be faster than waiting for Docker to
build the image, and then for me to download the image and extract the
xfststs.tar.gz file.  (But then again, I have a fully working laptop
with functional SSD's not sourced by Lenovo :-).

     		      	  	     	    - Ted

Re: [PATCH 6/6] Add dockerfile

[Theodore Ts'o]

Ok, I've checked in a Dockerfile into the xfstests-bld repository, and
played with it some, and I have a couple of observations:

First of all, despite some work cleaning up the Dockerfile, the
resulting image is somewhere between 150% and 200% larger than what it
would be if we build root_fs.img outside of the Docker.  A bunch of
the wasted space is simply because we have to include a 47 MB
xfstests.tar.gz file which then has to get re-inserted into the
root_fs image via the --update-xfstests command line.

Secondly, as an automated build procedure, it's rather lacking in two
ways.  First, it doesn't do the gen-image step, and secondly it only
builds the 64-bit x86_64 binaries --- and I normally like to use the
32-bit i386 image, since that can be used to test 32-bit and 64-bit
binaries, and it also forces us to test the 32/64-bit ioctl compat
code.  So it's really only useful as a CI mechanism --- and in order
to use it I have to give Docker read/write access to my github
repositories.  I might consider creating a throwaway github repository
on bitbucket just for the CI effort, but it's not high on my priority
list, since it doesn't test the gen-image part of the build process.

Third, it's a bit more inconvenient to use than the comments in your
Dockerfile would imply.  The command:

docker run -i -t --privileged --rm dmonakhov/xfstests-bld \
   kvm-xfstests.sh --kernel /tmp/bzImage --update-files --update-xfstests-tar smoke

... won't work because the docker because the image won't have
/tmp/bzImage.  So you would need to add "-v /tmp/bzImage:/tmp/bzImage"
to the command line, making it even more unweildy.

BTW, with changes I've just commited, we can drop the --kernel since
we now default to ~/linux, and so the command would look like this:

docker run -i -t -rm -v /build/ext4:/root/linux --privileged \
    tytso/xfstests-bld kvm-xfstests --update-files --update-xfstests smoke

So if I am going to publish something to the Docker Hub, it would an
addition to my current release process, where I would build the
root_fs using my existing Debian build chroots, and just create a
minimal tytso/kvm-xfstests image which would just have the needed
files, and would probably end up weighing in around 200-225 MB.  The
user wouldn't need to specify the --update-files and --update-xfstests
flags, and it would start faster.

On the other hand, I'm not at all convinced this is actually a great
way to run kvm-xfstests; for one thing, the log file is trapped inside
the Docker container, and so you would need to manually extract it in
order to keep a history of past test runs.  (This is also the
challenge of just sharding the test runs; collating the test results
becomes a big pain.)  And the whole concept of running a VM inside a
docker container reminds me a bit of the "Hitler uses Docker" rant at:
https://youtu.be/PivpCKEiQOQ?t=2m27s

I'm also not wild about encouraging people to run random Docker images
they download over the network with docker run --privileged.  That's
right up there with running with scissors, encouraging people to give
Docker read/write access to their github accounts, using the same
password across hundreds of web sites, etc.....

If the real goal is to allow people to shard the tests so it can be
run across multiple VM's, it might be better to give kvm-xfstests some
options so a different set of disks are used and either a different
set of ports, or just network ports altogether.

Cheers,

					- Ted

Re: [PATCH 6/6] Add dockerfile

[Dmitry Monakhov]

[-- Attachment #1: Type: text/plain, Size: 4760 bytes --]

Theodore Ts'o <tytso@mit.edu> writes:

> On Sat, Aug 20, 2016 at 02:31:26PM +0300, Dmitry Monakhov wrote:
>> > I'm not sure I see the advantage of doing this in a container, I
>> > guess.  I just do in my standard laptop environment today.
>> I can not because I laptop from famous thinkpad t430 series with
>> flaky SSD which starts to return EIO after intensive load.
>> https://forums.lenovo.com/t5/ThinkPad-T400-T500-and-newer-T/T430s-Intel-SSD-520-180GB-issue/td-p/888083
>
> Hmm, I never buy a Lenovo SSD; they're overpriced.  So I always get my
> Thinkpads with a 500mb HDD, and then replace it with a Samsumg 840/850
> EVO/PRO SSD.
>
>> Also. What about RH/SUSE/Fedora or even Gentoo people? They are exit :)
>> And if we give them just chance to try debian w/o complexities
>> maybe they will become debian adepts.
>
> You only need Debain to build the image.  If you are just using the
> pre-built root_fs.img, you can just download it and go.
>
> Speaking of which, that's one thing which I don't understand about
> your Dockerfile.  In the build step you just download the prebuilt
> root_fs.img from kernel.org.  There are comments in your Dockerfile which say:
>
> # Fetch and build xfstests-bld
> # In order to build root_image from scratch privileged operation are required,
> # so it is impossible during build stage.
> # One can make it like this:
> # docker run -i -t --privileged --rm dmonakhov/xfstests-bld "cd kvm-xfstests/test-appliance && ./gen-image"
> # During build stage we simply fetch image precreated by tytso@
>
> ... but while this will create a new root_fs.img file, as soon as
> you're you're done the container will exit, and then the root_fs.img
> file will be blown away, right?
Yes. This is my typo. Generated image should be saved somewhere
before container stopped. So command should be
docker run -i -t --privileged -v ~/my-data:/shared --rm dmonakhov/xfstests-bld \
    "cd kvm-xfstests/test-appliance \
    && ./gen-image \
    && cp root_fs.img /shared/"
Probably it should be moved to dedicated script, see later.
>
> The other thing I'll note here is that the Dockerfile is serving two
> use cases.  One is to user Docker to do automated build testing.  The
> other is for people who want to run multiple instances of kvm-xfstests
> using the container services.
>
> For the second, we don't really want to include all of the build
> artifacts and build and source trees in Docker image that the users
> will have to pull down over the network.  And using kvm-xfstests
> --update-xfstests means a lot of extra wasted I/O, since we're
> packaging up the tar file, then copying it into file, and then
> unpacking it inside the guest VM, etc.
>
> So for the second case, you'd really want to create a Docker image
> which had the minimum packages needed to actually *run* the tests,
> with the pre-installed root_fs image.
Yes. That is reasonable idea. Actually I already have it in my queue.
Probably it it reasonable to create dedicated directory ./docker
and place all docker related crap there:
# build scripts
docker/Dockerfile.build-env
docker/Dockerfile.run-env
# And some run scripts
# Build all environment from scratch
docker/docker-build.sh
# Move config, kernel, root_fs.img inside container and run {gce,kvm}-xfstests.sh
docker/docker-kvm-xfstests.sh 
docker/docker-gce-xfstests.sh
> For the first case, all you need to do is to have the Dockerfile
> create the xfstests.tar.gz file.  I suppose this could be used as a
> poor man's build engine if you don't want to run the build on your
> local machine.  What I do is my top-level config.custom has:
>
> BUILD_ENV="schroot -c jessie64 --"
> SUDO_ENV="schroot -c jessie64 -u root --"
>
> That's my pristine build chroot, and so when I run ./do-all, what gets
> executed is:
>
> schroot -c jessie64 -- make clean
> schroot -c jessie64 -- make
> schroot -c jessie64 -- make tarball
> cd kvm-xfstests/test-appliance
> schroot -c jessie64 -u root -- ./gen-image
>
> I'm pretty sure this is going to be faster than waiting for Docker to
> build the image, and then for me to download the image and extract the
> xfststs.tar.gz file.  (But then again, I have a fully working laptop
> with functional SSD's not sourced by Lenovo :-).
Definitely. But in your case we still need jessie64's chroot.
It should be prepared somehow. My point is that if we give an automated build
script this may help people to start with xfstests with less pain.
In my experience time spend on investigation of installation process
always longer than time to download precreated VM/container :)
Let's call this approach "Lazy developers are welcome"
>      		      	  	     	    - Ted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 472 bytes --]

Re: [PATCH 6/6] Add dockerfile

[Theodore Ts'o]

On Sun, Aug 21, 2016 at 03:02:38PM +0300, Dmitry Monakhov wrote:
> Definitely. But in your case we still need jessie64's chroot.
> It should be prepared somehow. My point is that if we give an automated build
> script this may help people to start with xfstests with less pain.
> In my experience time spend on investigation of installation process
> always longer than time to download precreated VM/container :)
> Let's call this approach "Lazy developers are welcome"

Well, for the really lazy developers, all they need to do now is:

git clone git://git.kernel.org/pub/scm/fs/ext2/xfstests-bld.git
cd xfstests-bld/kvm-xfstests
make install

:-)

This will download the root_fs.img from www.kernel.org, and install
kvm-xfstests into their home directory.

(Or they can substite the last command with: "sudo make
prefix=/usr/local install")

This should work for both Fedora and Debian systems.  There will be
some packages you will need to install (e.g., kvm) but I suspect many
people will have all of the prequisites installed already.

Cheers,

       	    	     	    			  - Ted

Re: [PATCH 6/6] Add dockerfile

[Theodore Ts'o]

On Sun, Aug 21, 2016 at 03:02:38PM +0300, Dmitry Monakhov wrote:
> Probably it it reasonable to create dedicated directory ./docker
> and place all docker related crap there:
> # build scripts
> docker/Dockerfile.build-env
> docker/Dockerfile.run-env
> # And some run scripts
> # Build all environment from scratch
> docker/docker-build.sh
> # Move config, kernel, root_fs.img inside container and run {gce,kvm}-xfstests.sh
> docker/docker-kvm-xfstests.sh 
> docker/docker-gce-xfstests.sh

I thought about that, but my current thinking is to have one
Dockerfile which is in the top-level directory, and one which is in
the kvm-xfstests directory.

I've already created a tytso/xfststs-bld Docker image which was
generated using:

docker build -t tytso/xfstests-bld https://github.com/tytso/xfstests-bld.git

By putting the Dockerfile in the top-level directory, I can use the
command "COPY . /devel/xfstests-bld" in the Dockerfile, and so we can
get a clean build build that way.

By putting a Dockerfile in the kvm-xfstests directory, then all I have
to do is to put a "COPY . /usr/local/lib/kvm-xfstests" which will copy
all of the kvm-xfstests directory directly where we want the files in
the kvm-xfstests Docker image.  I'm assuming that my release process
will first build root_fs.i386 and root_fs.x86_64 using debian chroots,
and then upload them to www.kernel.org.  I'll then run:

docker build -t tytso/kvm-xfstests https://github.com/tytso/xfstests-bld.git:kvm-xfstests

... and then the Dockerfile will pull the root_fs.i386 from
ftp.kernel.org.

What do you think?

						- Ted

Re: [PATCH 6/6] Add dockerfile

[Dmitry Monakhov]

[-- Attachment #1: Type: text/plain, Size: 2082 bytes --]

Theodore Ts'o <tytso@mit.edu> writes:

> On Sun, Aug 21, 2016 at 03:02:38PM +0300, Dmitry Monakhov wrote:
>> Probably it it reasonable to create dedicated directory ./docker
>> and place all docker related crap there:
>> # build scripts
>> docker/Dockerfile.build-env
>> docker/Dockerfile.run-env
>> # And some run scripts
>> # Build all environment from scratch
>> docker/docker-build.sh
>> # Move config, kernel, root_fs.img inside container and run {gce,kvm}-xfstests.sh
>> docker/docker-kvm-xfstests.sh 
>> docker/docker-gce-xfstests.sh
>
> I thought about that, but my current thinking is to have one
> Dockerfile which is in the top-level directory, and one which is in
> the kvm-xfstests directory.
>
> I've already created a tytso/xfststs-bld Docker image which was
> generated using:
>
> docker build -t tytso/xfstests-bld https://github.com/tytso/xfstests-bld.git
Nice. Now image is no longer generated by Russian hacker. Hitler
finally can sleep well :)
>
> By putting the Dockerfile in the top-level directory, I can use the
> command "COPY . /devel/xfstests-bld" in the Dockerfile, and so we can
> get a clean build build that way.
Yes. That is reasonable. I do not know that build context can emerge from URL.
>
> By putting a Dockerfile in the kvm-xfstests directory, then all I have
> to do is to put a "COPY . /usr/local/lib/kvm-xfstests" which will copy
> all of the kvm-xfstests directory directly where we want the files in
> the kvm-xfstests Docker image.  I'm assuming that my release process
> will first build root_fs.i386 and root_fs.x86_64 using debian chroots,
> and then upload them to www.kernel.org.  I'll then run:
>
> docker build -t tytso/kvm-xfstests https://github.com/tytso/xfstests-bld.git:kvm-xfstests
Cool. The only minor comment is that command should be:
docker build -t tytso/kvm-xfstests -f kvm-xfstests/Dockerfile https://github.com/tytso/xfstests-bld.git:kvm-xfstests
>
> ... and then the Dockerfile will pull the root_fs.i386 from
> ftp.kernel.org.
>
> What do you think?
>
> 						- Ted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 472 bytes --]

Re: [PATCH 1/6] xfstests-bld: build update xfstests repo

[Theodore Ts'o]

On Fri, Aug 19, 2016 at 12:54:06AM +0400, Dmitry Monakhov wrote:
> The official location for xfstests source is now:
> git://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git
> 
> Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>

Applied, thanks.

					- Ted

Re: [PATCH 2/6] fix build typos

[Theodore Ts'o]

On Fri, Aug 19, 2016 at 12:54:07AM +0400, Dmitry Monakhov wrote:
> build-all: --xfsprogs-only => SKIP_XFSPROGS, not SKIP_XFSTESTS
> get-all: fix typo in case of explicit xfsprogs_commit
> Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>

Applied, thanks.

					- Ted

Re: [PATCH 3/6] kvm-xfstests: update command line help

[Theodore Ts'o]

On Fri, Aug 19, 2016 at 12:54:08AM +0400, Dmitry Monakhov wrote:
> diff --git a/kvm-xfstests/util/parse_cli b/kvm-xfstests/util/parse_cli
> index 83400ea..edbbafa 100644
> --- a/kvm-xfstests/util/parse_cli
> +++ b/kvm-xfstests/util/parse_cli
> @@ -26,10 +26,47 @@ print_help ()
>      echo "	-o opts		- Extra kernel command line options"
>      echo "	-r ram		- Specify memory to be used in megabytes"
>      echo "	-X test		- Exclude test from running"
> +    echo "	--no-punch	- Do not run tests w/ patch hole"
> +    echo "	--no-collapse	- Do not run tests w/ collapse range"
> +    echo "	--no-insert	- Do not run tests w/ insert range"
> +    echo "	--no-zero	- Do not run tests w/ zero range"

The reason why I don't document these options is they depend on an
out-of-tree patch for xfstests.  It's in my githup repo, but it's not
something which xfstests upstream has seen fit to take.

>      if test "$GCE_XFSTESTS" != "yes" ; then
>  	echo "	--kernel file	- Boot the specified kernel"
>  
> +    else
> +	echo "GCE speciffic commands:"
> +	echo "	ls-instances"
> +        echo "	ls|ls-instances"
> +        echo "	rm|rm-instances|abort|abort-instances"
> +        echo "	start|start-instance|start-instances"
> +        echo "	stop|stop-instance|stop-instances"
> +        echo "	ls-disks|ls-disk"
> +        echo "	rm-disks|rm-disk"
> +        echo "	ls-results|ls-gcs"
> +        echo "	rm-results|rm-gcs"
> +        echo "	get-results"
> +        echo "	setup"
> +        echo "	ssh"
> +        echo "	console"
> +        echo "	serial"
> +        echo "	get-results"
> +        echo "	create-image"
> +        echo "	export-image"
> +        echo "	import-image"
> +        echo "	--numa"
> +        echo "	--log"
> +        echo "	--local-ssd"
> +        echo "	--pmem-device"
> +        echo "	--machtype"
> +        echo "	--image-project"
> +        echo "	--instance-name"
> +        echo "	--no-virtio-rng"
>      fi
> +    echo "Base image update commands:"
> +    echo "	--update-xfstests-tar"
> +    echo "	--update-xfstests"
> +    echo "	--update-files"
> +    echo
>      echo "	--no-log	- Don't save the log file for this run"
>      echo "	--no-action	- Print the command to start the VM"
>      echo ""

The options don't belong next to the GCE specific commands.  And
gce-import, as well as the base image update options, are really
intended for the maintainer (e.g., for me to update the xfstests-cloud
public image project).

The main issue is I'm a bit concerned about making the help message
too long.  My long term vision is to go in the direction of git and
gcloud's style of help messages / man pages.  So gce-xfstests --help
and kvm-xfstests --help will display a list of the most commonly used
/ useful commands and options.  And where "gce-xfstests get-results
--help" will display a full man page for the get-results subcommand,
and "gce-xfstests help commands" will display a full list of all
commands, and "gce-xfstests help topics" will allow you to see the
documentation currently available as markdown files in Documentation.

Cheers,

						- Ted

Re: [PATCH 3/6] kvm-xfstests: update command line help

[Dmitry Monakhov]

[-- Attachment #1: Type: text/plain, Size: 3497 bytes --]

Theodore Ts'o <tytso@mit.edu> writes:

> On Fri, Aug 19, 2016 at 12:54:08AM +0400, Dmitry Monakhov wrote:
>> diff --git a/kvm-xfstests/util/parse_cli b/kvm-xfstests/util/parse_cli
>> index 83400ea..edbbafa 100644
>> --- a/kvm-xfstests/util/parse_cli
>> +++ b/kvm-xfstests/util/parse_cli
>> @@ -26,10 +26,47 @@ print_help ()
>>      echo "	-o opts		- Extra kernel command line options"
>>      echo "	-r ram		- Specify memory to be used in megabytes"
>>      echo "	-X test		- Exclude test from running"
>> +    echo "	--no-punch	- Do not run tests w/ patch hole"
>> +    echo "	--no-collapse	- Do not run tests w/ collapse range"
>> +    echo "	--no-insert	- Do not run tests w/ insert range"
>> +    echo "	--no-zero	- Do not run tests w/ zero range"
>
> The reason why I don't document these options is they depend on an
> out-of-tree patch for xfstests.  It's in my githup repo, but it's not
> something which xfstests upstream has seen fit to take.
>
>>      if test "$GCE_XFSTESTS" != "yes" ; then
>>  	echo "	--kernel file	- Boot the specified kernel"
>>  
>> +    else
>> +	echo "GCE speciffic commands:"
>> +	echo "	ls-instances"
>> +        echo "	ls|ls-instances"
>> +        echo "	rm|rm-instances|abort|abort-instances"
>> +        echo "	start|start-instance|start-instances"
>> +        echo "	stop|stop-instance|stop-instances"
>> +        echo "	ls-disks|ls-disk"
>> +        echo "	rm-disks|rm-disk"
>> +        echo "	ls-results|ls-gcs"
>> +        echo "	rm-results|rm-gcs"
>> +        echo "	get-results"
>> +        echo "	setup"
>> +        echo "	ssh"
>> +        echo "	console"
>> +        echo "	serial"
>> +        echo "	get-results"
>> +        echo "	create-image"
>> +        echo "	export-image"
>> +        echo "	import-image"
>> +        echo "	--numa"
>> +        echo "	--log"
>> +        echo "	--local-ssd"
>> +        echo "	--pmem-device"
>> +        echo "	--machtype"
>> +        echo "	--image-project"
>> +        echo "	--instance-name"
>> +        echo "	--no-virtio-rng"
>>      fi
>> +    echo "Base image update commands:"
>> +    echo "	--update-xfstests-tar"
>> +    echo "	--update-xfstests"
>> +    echo "	--update-files"
>> +    echo
>>      echo "	--no-log	- Don't save the log file for this run"
>>      echo "	--no-action	- Print the command to start the VM"
>>      echo ""
>
> The options don't belong next to the GCE specific commands.  And
> gce-import, as well as the base image update options, are really
> intended for the maintainer (e.g., for me to update the xfstests-cloud
> public image project).
>
> The main issue is I'm a bit concerned about making the help message
> too long.  My long term vision is to go in the direction of git and
> gcloud's style of help messages / man pages.  So gce-xfstests --help
> and kvm-xfstests --help will display a list of the most commonly used
> / useful commands and options.  And where "gce-xfstests get-results
> --help" will display a full man page for the get-results subcommand,
> and "gce-xfstests help commands" will display a full list of all
> commands, and "gce-xfstests help topics" will allow you to see the
> documentation currently available as markdown files in Documentation.
Agree. My point was that it is easier to:
0 place info about all commands here fist: DONE
2 visualize it,
3 make a decision about readability
4 cut less interesting options.
5 goto 2
>
> Cheers,
>
> 						- Ted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 472 bytes --]

Re: [PATCH 3/6] kvm-xfstests: update command line help

[Theodore Ts'o]

I've improved the usage commands for gen-image and gce-xfstests
somewhat.  It's roughly based on your patch, but as discussed, there
are certain options and commands which I don't want to include in the
usage messages either because they rely on out-of-tree xfstests
patches, or they are too specialized to be of interest to most users.

Cheers,

					- Ted

Re: [PATCH 5/6] kvm-xfstests: add initrd support

[Theodore Ts'o]

On Fri, Aug 19, 2016 at 12:54:10AM +0400, Dmitry Monakhov wrote:
> +	--initrd) shift
> +		  INITRD="$1"
> +		  if test ! -f "$INITRD" ; then
> +		      print_help
> +		  fi
> +		  ;;

We should only allow --initrd for kvm-xfstests, or add support for
uploading the initrd to gce-xfstests, and then adding support to it to
the gce-kexec script.

We can just allow it for kvm-xfstests first, and then only later add
support to gce-xfstests, if you don't have time to work to get
gce-xfstests support for --initrd working.

Cheers,

						- Ted

Re: [PATCH 5/6] kvm-xfstests: add initrd support

[Dmitry Monakhov]

[-- Attachment #1: Type: text/plain, Size: 998 bytes --]

Theodore Ts'o <tytso@mit.edu> writes:

> On Fri, Aug 19, 2016 at 12:54:10AM +0400, Dmitry Monakhov wrote:
>> +	--initrd) shift
>> +		  INITRD="$1"
>> +		  if test ! -f "$INITRD" ; then
>> +		      print_help
>> +		  fi
>> +		  ;;
>
> We should only allow --initrd for kvm-xfstests, or add support for
> uploading the initrd to gce-xfstests, and then adding support to it to
> the gce-kexec script.
>
> We can just allow it for kvm-xfstests first, and then only later add
> support to gce-xfstests, if you don't have time to work to get
> gce-xfstests support for --initrd working.
No problem, but it looks like my knowledge about GCE is too low at the
moment. BTW are there are any way to make a bullet prof method to stop
gce instance after predefined timout? Your systemctl timeout script
does not always work. In my case it stuck somewhere inside FS and
timeout.service can not do it's job. Probably we can do it via
kernel watchdog or external watcher ala Jenkins.

>
> Cheers,
>
> 						- Ted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 472 bytes --]

Re: [PATCH 5/6] kvm-xfstests: add initrd support

[Theodore Ts'o]

On Fri, Aug 19, 2016 at 04:59:22PM +0300, Dmitry Monakhov wrote:
> No problem, but it looks like my knowledge about GCE is too low at the
> moment. BTW are there are any way to make a bullet prof method to stop
> gce instance after predefined timout? Your systemctl timeout script
> does not always work. In my case it stuck somewhere inside FS and
> timeout.service can not do it's job. Probably we can do it via
> kernel watchdog or external watcher ala Jenkins.

My long term vision was to use an external watcher that would run in
Google App Engine.  The idea would be that this would also take care
of launching separate VM's for each of the different test cases, and
then collate the reports into a single test report.  Long term I'd
also like to have the results stored into Google Cloud Datastore, and
do automatic flaky test detection.

For now, I just simply manually keep an eye on things using
"gce-xfstests ls -l", and if I see something running for too long,
I'll connect to it using "gce-xfstests console xfstests-XXXX" to grab
the results.  In the app-engine test runner vision it would use the
equivalent of "gce-xfstests serial xfstests-XXX" and store the
complete serial console output someplace safe.  What happens today
tends to be:

1) gce-xfstests -c overlay -g auto

2) periodically I'll run gce-xfstests ls -l, and notice when the VM
apparently is no longer making foreward progress.

3) Hmm, looks like overlayfs is blowing up.  And gce-xfstests console
doesn't give me enough history since it only stores the last N lines".

4) gce-xfstests abort xfstests-XXXXXX

5) rerun "gce-xfstests -c overaly -g auto", but now after it starts,
   also run: "script -c "gce-xfstests serial xfstests-XXXXX" console-XXXXX.out"

In practice this doesn't happen often enough that I've automated this,
and it's also why I haven't made it a high priority to create some
kind of external test running / monitoring service.

     		      	      		   - Ted

P.S. I recently added overlayfs support, and it looks like overlayfs
has a bug which ends up screwing up an inode link's count, and causing
the ext4 orphan list to get corrupted, and causing subsequent ext4
warnings and BUG's to get triggered.  So this isn't a hypothetical
example; it's just one that I haven't had time to track down yet.  :-)

Re: [PATCH 5/6] kvm-xfstests: add initrd support

[Theodore Ts'o]

On Fri, Aug 19, 2016 at 12:54:10AM +0400, Dmitry Monakhov wrote:
> Sometimes it is important to run kernel which was build with
> modules, so the only way to do this it to pass initrd parameter.
> 
> Example:
> # Run test on official FC24 kernel
> #wget https://mirror.yandex.ru/fedora/linux/releases/24/Server/x86_64/os/images/pxeboot/initrd.img
> #wget https://mirror.yandex.ru/fedora/linux/releases/24/Server/x86_64/os/images/pxeboot/vmlinuz
> #kvm-xfstests.sh --kernel vmlinuz --initrd initrd.img generic/013
> 
> Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>

OK, applied with some slight modifications so that --initrd is only
accepted for kvm-xfstests.

BTW, I tried using the Fedora 24 kernel, and there's something wonky
with the serial console.  It looks like every other character which is
typed is getting eaten.  I suppose it's good to be able to test with
random distribution kernels with initrds, but there may be weird
compatibility issues with different distro's initrds, so I am a bit
worried that this might be a bit of a support headache in the long
term.

						- Ted

Re: [PATCH 4/6] xfstest-bld: use parallel gzip if possible

[Theodore Ts'o]

On Fri, Aug 19, 2016 at 12:54:09AM +0400, Dmitry Monakhov wrote:
> diff --git a/kvm-xfstests/kvm-xfstests b/kvm-xfstests/kvm-xfstests
> index d608317..10782d0 100755
> --- a/kvm-xfstests/kvm-xfstests
> +++ b/kvm-xfstests/kvm-xfstests
> @@ -9,6 +9,7 @@ fi
>  . "$DIR/util/get-config"
>  
>  QUIET="quiet loglevel=0"
> +which pigz &>/dev/null && GZIP=pigz || GZIP=gzip
>  
>  . $DIR/util/parse_cli
>  
> @@ -71,7 +72,7 @@ then
>      fi
>      (cd "$DIR/test-appliance"; \
>       tar -X kvm-exclude-files -C files -cf - . | \
> -	 gzip -9n > "$TDIR/files.tar.gz")
> +	 $GZIP -9n > "$TDIR/files.tar.gz")
>      tar -r -f $VDH -C "$TDIR" files.tar.gz
>      rm -rf "$TDIR"
>  fi

There's no real point in using pigz for files.tar.gz, since the file
is so small.  In fact, it's slightly slower on my system:

% time gzip -9 < /tmp/files.tar  > /tmp/files.tar.gz

real   0m0.010s
user   0m0.000s
sys    0m0.000s

% time pigz -9 < /tmp/files.tar  > /tmp/files.tar.gz

real   0m0.012s
user   0m0.010s
sys    0m0.010s

For the xfstests.tar.gz file, it does make sense; it cuts down the
compression time form 3 seconds to half a second.  That's fair,
although at least in my workflow I'm not recreating the
xfstests.tar.gz file all that often.

Cheers,

						- Ted

Re: [PATCH 4/6] xfstest-bld: use parallel gzip if possible

[Theodore Ts'o]

On Fri, Aug 19, 2016 at 12:54:09AM +0400, Dmitry Monakhov wrote:
> Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
> ---
>  gen-tarball                                  | 4 +++-
>  kvm-xfstests/gce-xfstests                    | 5 +++--
>  kvm-xfstests/kvm-xfstests                    | 3 ++-
>  kvm-xfstests/test-appliance/gce-create-image | 3 ++-
>  4 files changed, 10 insertions(+), 5 deletions(-)

I've applied just the gen-tarball portion of the patch, because that's
where using the parallelized gzip makes a difference.  For small
files, pigz can actually take a tiny bit _more_ time.

       	    		      	     	 - Ted