xfstests and ext4

xfstests and ext4

[Steve French]

To get xfstests to build on Ubuntu 14.10 and run (after adding various
packages I could get all but 11 of the tests to build and run) I had
to install the following packages:

sudo apt-get install xfslibs-dev uuid-dev libtool e2fsprogs automake
libuuid1 libuuidm-ocaml-dev attr libattr1-dev libacl1-dev libaio-dev
xfsprogs libgdbm-dev gawk fio dbench

The only xfstest I saw failing on ext4  (other than the defrag
problems mentioned earlier which are fixed in e2fsprogs but not picked
up by the most recent Ubuntu yet) on 3.18-rc3 kernel was generic/315
(Ubuntu 14.10).

generic/315  [failed, exit status 1] - output mismatch (see
/home/sfrench/xfstests/results//generic/315.out.bad)
    --- tests/generic/315.out 2014-11-13 11:20:05.405440282 -0800
    +++ /home/sfrench/xfstests/results//generic/315.out.bad 2014-11-13
13:34:57.346960909 -0800
    @@ -1,2 +1,6 @@
     QA output created by 315
     Slience is golden
    +ls: cannot access /mnt/test/testfile.315: No such file or directory
    +./tests/generic/315: line 69: [: : integer expression expected
    +File size is changed to ( Bytes)
    +(see /home/sfrench/xfstests/results//generic/315.full for details)


The other 157 worked on ext4 and only 11 were not run.

Ran: ext4/001 ext4/002 ext4/003 ext4/271 ext4/301 ext4/302 ext4/303
ext4/304 ext4/305 ext4/306 ext4/307 ext4/308 generic/001 generic/002
generic/003 generic/004 generic/005 generic/006 generic/007
generic/008 generic/009 generic/010 generic/011 generic/012
generic/013 generic/014 generic/015 generic/016 generic/017
generic/018 generic/020 generic/021 generic/022 generic/023
generic/024 generic/025 generic/027 generic/028 generic/029
generic/031 generic/032 generic/033 generic/034 generic/035
generic/036 generic/037 generic/053 generic/062 generic/068
generic/069 generic/070 generic/074 generic/075 generic/076
generic/077 generic/079 generic/083 generic/088 generic/089
generic/091 generic/100 generic/105 generic/112 generic/113
generic/117 generic/120 generic/123 generic/124 generic/125
generic/126 generic/127 generic/128 generic/129 generic/130
generic/131 generic/132 generic/133 generic/135 generic/141
generic/169 generic/184 generic/192 generic/193 generic/198
generic/204 generic/207 generic/208 generic/209 generic/210
generic/211 generic/212 generic/213 generic/214 generic/215
generic/219 generic/221 generic/223 generic/224 generic/225
generic/226 generic/228 generic/230 generic/231 generic/232
generic/233 generic/234 generic/235 generic/236 generic/237
generic/239 generic/240 generic/241 generic/245 generic/246
generic/247 generic/248 generic/249 generic/255 generic/256
generic/257 generic/258 generic/263 generic/269 generic/270
generic/273 generic/274 generic/299 generic/300 generic/275
generic/277 generic/280 generic/285 generic/286 generic/294
generic/306 generic/307 generic/308 generic/309 generic/310
generic/311 generic/312 generic/313 generic/314 generic/315
generic/316 generic/317 generic/318 generic/319 generic/320
generic/321 generic/322 generic/323 generic/324 generic/325 shared/006
shared/272 shared/289 shared/298
Not run: generic/019 generic/026 generic/030 generic/093 generic/097
generic/099 generic/251 generic/260 generic/288 shared/032 shared/051
Failures: generic/315 (and defrag tests ext4/302, ext4/303, ext4/304
and generic/018 which are presumably fixed upstream)

Of the 11 tests that don't run on ext4 on Ubuntu, best case seems to
be 4 more could be run if you had the right hardware and recompiled
the kernel:
3 (251, 260, 288) require TRIM support in hardware (which doesn't work in my VM)
019 requires a kernel config option (CONFIG_FAIL_MAKE_REQUEST)
026 requires maximum acl count to be specified
030 requires xfs_io mremap support
3 are IRIX specific (093, 097, 099)
and two are for different file system types (shared/032 and shared/051)


-- 
Thanks,

Steve

Re: xfstests and ext4

[Dave Chinner]

On Thu, Nov 13, 2014 at 05:38:53PM -0600, Steve French wrote:
> To get xfstests to build on Ubuntu 14.10 and run (after adding various
> packages I could get all but 11 of the tests to build and run) I had
> to install the following packages:
> 
> sudo apt-get install xfslibs-dev uuid-dev libtool e2fsprogs automake
> libuuid1 libuuidm-ocaml-dev attr libattr1-dev libacl1-dev libaio-dev
> xfsprogs libgdbm-dev gawk fio dbench

Documentation patch for the README file?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: xfstests and ext4

[Steve French]

[-- Attachment #1: Type: text/plain, Size: 2185 bytes --]

On Thu, Nov 13, 2014 at 6:02 PM, Dave Chinner <david@fromorbit.com> wrote:
> On Thu, Nov 13, 2014 at 05:38:53PM -0600, Steve French wrote:
>> To get xfstests to build on Ubuntu 14.10 and run (after adding various
>> packages I could get all but 11 of the tests to build and run) I had
>> to install the following packages:
>>
>> sudo apt-get install xfslibs-dev uuid-dev libtool e2fsprogs automake
>> libuuid1 libuuidm-ocaml-dev attr libattr1-dev libacl1-dev libaio-dev
>> xfsprogs libgdbm-dev gawk fio dbench
>
> Documentation patch for the README file?
>
> Cheers,
>
> Dave.

Patch attached



[PATCH] common: update README to list build dependencies

Add sample list of packages (dependencies) required to build XFSTESTS
and add a step to the instructions to mention creating the test
user needed for some of the ACL tests.

Signed-off-by: Steve French <smfrench@gmail.com>
---
 README            | 6 ++++++


diff --git a/README b/README
index 8a362bd..8db7efe 100644
--- a/README
+++ b/README
@@ -4,8 +4,14 @@ _______________________

 Building Linux:
     - cd into the xfstests directory
+    - install prerequisite packages
+      For example, for Ubuntu:
+        "sudo apt-get install xfslibs-dev uuid-dev libtool e2fsprogs
+         automake gcc libuuid1 libuuidm-ocaml-dev attr libattr1-dev
+         libacl1-dev libaio-dev xfsprogs libgdbm-dev gawk fio dbench"
     - run make
     - run make install
+    - create fsgqa test user ("sudo useradd fsgqa")

 Building IRIX:
     - cd into the xfstests directory


-- 
Thanks,

Steve

On Thu, Nov 13, 2014 at 4:02 PM, Dave Chinner <david@fromorbit.com> wrote:
> On Thu, Nov 13, 2014 at 05:38:53PM -0600, Steve French wrote:
>> To get xfstests to build on Ubuntu 14.10 and run (after adding various
>> packages I could get all but 11 of the tests to build and run) I had
>> to install the following packages:
>>
>> sudo apt-get install xfslibs-dev uuid-dev libtool e2fsprogs automake
>> libuuid1 libuuidm-ocaml-dev attr libattr1-dev libacl1-dev libaio-dev
>> xfsprogs libgdbm-dev gawk fio dbench
>
> Documentation patch for the README file?
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com



-- 
Thanks,

Steve

[-- Attachment #2: 0001-common-update-README-to-list-build-dependencies.patch --]
[-- Type: text/x-patch, Size: 1081 bytes --]

From 1822e733684f031b13392782e2a7eef1ab102d17 Mon Sep 17 00:00:00 2001
From: Steve French <smfrench@gmail.com>
Date: Thu, 13 Nov 2014 17:07:27 -0800
Subject: [PATCH] common: update README to list build dependencies

Add sample list of packages (dependencies) required to build XFSTESTS
and add a step to the instructions to mention creating the test
user needed for some of the ACL tests.

Signed-off-by: Steve French <smfrench@gmail.com>
---
 README | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/README b/README
index 8a362bd..8db7efe 100644
--- a/README
+++ b/README
@@ -4,8 +4,14 @@ _______________________
 
 Building Linux:
 	- cd into the xfstests directory
+	- install prerequisite packages
+	  For example, for Ubuntu:
+		"sudo apt-get install xfslibs-dev uuid-dev libtool e2fsprogs
+		 automake gcc libuuid1 libuuidm-ocaml-dev attr libattr1-dev
+		 libacl1-dev libaio-dev xfsprogs libgdbm-dev gawk fio dbench"
 	- run make
 	- run make install
+	- create fsgqa test user ("sudo useradd fsgqa")
 	
 Building IRIX:
 	- cd into the xfstests directory 
-- 
2.1.0

Re: xfstests and ext4

[Dave Chinner]

On Thu, Nov 13, 2014 at 05:20:34PM -0800, Steve French wrote:
> On Thu, Nov 13, 2014 at 6:02 PM, Dave Chinner <david@fromorbit.com> wrote:
> > On Thu, Nov 13, 2014 at 05:38:53PM -0600, Steve French wrote:
> >> To get xfstests to build on Ubuntu 14.10 and run (after adding various
> >> packages I could get all but 11 of the tests to build and run) I had
> >> to install the following packages:
> >>
> >> sudo apt-get install xfslibs-dev uuid-dev libtool e2fsprogs automake
> >> libuuid1 libuuidm-ocaml-dev attr libattr1-dev libacl1-dev libaio-dev
> >> xfsprogs libgdbm-dev gawk fio dbench
> >
> > Documentation patch for the README file?
> >
> > Cheers,
> >
> > Dave.
> 
> Patch attached

There are two copies of it in this email - one pasted in the middle
of the rest of the quoted email text and so is whitespace damaged,
the other as a base64 encoded attachment so can't be quoted and
replied to easily. In future, can you send patches as separate
emails according to:

https://www.kernel.org/doc/Documentation/email-clients.txt

> [PATCH] common: update README to list build dependencies
> 
> Add sample list of packages (dependencies) required to build XFSTESTS
> and add a step to the instructions to mention creating the test
> user needed for some of the ACL tests.
> 
> Signed-off-by: Steve French <smfrench@gmail.com>
> ---
>  README            | 6 ++++++
> 
> 
> diff --git a/README b/README
> index 8a362bd..8db7efe 100644
> --- a/README
> +++ b/README
> @@ -4,8 +4,14 @@ _______________________
> 
>  Building Linux:
>      - cd into the xfstests directory
> +    - install prerequisite packages
> +      For example, for Ubuntu:
> +        "sudo apt-get install xfslibs-dev uuid-dev libtool e2fsprogs
> +         automake gcc libuuid1 libuuidm-ocaml-dev attr libattr1-dev
                                  ^^^^^^^^^^^^^^^^^^

What in xfstests is dependent on ocaml?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: xfstests and ext4

[Steve French]

On Mon, Nov 17, 2014 at 2:38 PM, Dave Chinner <david@fromorbit.com> wrote:

> the other as a base64 encoded attachment so can't be quoted and
> replied to easily. In future, can you send patches as separate
> emails according to:

Yes.  WIll do.  It is a pain in test  VMs to setup git with gmail two
phase authentication
but doable if I set it up to send mail from a different host.

>>  Building Linux:
>>      - cd into the xfstests directory
>> +    - install prerequisite packages
>> +      For example, for Ubuntu:
>> +        "sudo apt-get install xfslibs-dev uuid-dev libtool e2fsprogs
>> +         automake gcc libuuid1 libuuidm-ocaml-dev attr libattr1-dev
>                                   ^^^^^^^^^^^^^^^^^^
>
> What in xfstests is dependent on ocaml?

That libuuidm-ocaml-dev looks unneeded now (I ended up pulling in one
of the tools a different
way so whatever that pulled in looks superfluous now) - in an earlier
try at this I built a few
of the tools. But there is one missing. I missed in the cut-and-pastes
an obvious very
important dependency when I was transcribing the dependency list -
"quota"   I will add
that to the list and remove the libuuidm-ocaml-dev off now that I have
confirmed the dependencies
list again today on a fresh Ubuntu install.



-- 
Thanks,

Steve

Re: xfstests and ext4

[Dave Chinner]

On Mon, Nov 17, 2014 at 08:35:39PM -0600, Steve French wrote:
> On Mon, Nov 17, 2014 at 2:38 PM, Dave Chinner <david@fromorbit.com> wrote:
> 
> > the other as a base64 encoded attachment so can't be quoted and
> > replied to easily. In future, can you send patches as separate
> > emails according to:
> 
> Yes.  WIll do.  It is a pain in test  VMs to setup git with gmail two
> phase authentication
> but doable if I set it up to send mail from a different host.

No need for that. Keep the xfstests source tree on your
laptop/workstation and do all your edits there. When you've done
that, simply rsync the tree to the test VM(s) and run "make; ./check
..." from the shell you are running in the test VM.

That way you can manage the code multiple test VMs run from a single
source tree, all in one location, and you don't have to worry about
setting up for mail, backups, losing changes because a test machine
crash ate your recent changes, accidentally sending changes are
root, etc...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: xfstests and ext4

[Christoph Hellwig]

On Wed, Nov 19, 2014 at 07:28:18AM +1100, Dave Chinner wrote:
> No need for that. Keep the xfstests source tree on your
> laptop/workstation and do all your edits there. When you've done
> that, simply rsync the tree to the test VM(s) and run "make; ./check
> ..." from the shell you are running in the test VM.
> 
> That way you can manage the code multiple test VMs run from a single
> source tree, all in one location, and you don't have to worry about
> setting up for mail, backups, losing changes because a test machine
> crash ate your recent changes, accidentally sending changes are
> root, etc...

Or commit them to a local git branch that you can pull from the VMs
instead of the rsync.  The effect is the same, though.

Re: xfstests and ext4

[Theodore Ts'o]

On Mon, Nov 17, 2014 at 08:35:39PM -0600, Steve French wrote:
> 
> Yes.  WIll do.  It is a pain in test VMs to setup git with gmail two
> phase authentication
> but doable if I set it up to send mail from a different host.

There are lots of ways to skin a cat, but I just simply build xfstests
in my standard development environment, and then either create a
completely new test VM image from scratch, which doesn't take _that_ long:

<tytso@closure> {/u1/xfstests-bld/build-32/kvm-xfstests/test-appliance}
521% /usr/bin/time schroot -c sid -u root ./gen-image
     ...
/tmp/root_fs.raw.7430: 9867/65536 files (0.0% non-contiguous), 66071/262144 blocks
65.23user 15.65system 1:25.65elapsed 94%CPU (0avgtext+0avgdata 11076maxresident)k
42inputs+970048outputs (0major+1528344minor)pagefaults 0swaps

or I'll update the VM image from my build directory, which keeps the
OS files the same and only updates the files in /root/xfstests:

<tytso@closure> {/u1/xfstests-bld/build-32/kvm-xfstests/test-appliance}
523% /usr/bin/time schroot -c sid -u root -- ./gen-image --update
	...
9.98user 1.65system 0:12.81elapsed 90%CPU (0avgtext+0avgdata 11644maxresident)k
4674inputs+178512outputs (0major+204070minor)pagefaults 0swaps

The advantage of doing it that way is that I get replicable builds for
my xfstest VM's, and it also helps keep them small (only 66 megs),
which makes them easier to move around to different test servers.

(I also like not being dependent on the version of xfslibs-dev that
Ubuntu might have decided to ship, since a while back I was stuck on
pretty ancient LTS versions due to corporate policy reasons, and there
were a few tests that required the latest versions of xfsprogs/xfslibs
in order to build and run.  So that's why I use a completely hermetic
build in my infrastructure, which can be found in my xfstests-bld git
tree if you want to give it a try.)

Cheers,

						- Ted