WANTED: xfstests results in different architectures

WANTED: xfstests results in different architectures

[Chandra Seetharaman]

Hello All,

Couple of weeks backs we had a discussion in xfs meeting to collect
xfstests results. I volunteered to collect xfstests results from
different architectures and upload to XFS.org.

I can run and get the results for x86_64 and ppc64. If anyone has other
architectures that they can run the tests on and provide me the results,
I will filter them an upload to XFS.org.

Here is what I think would be of value to provide along with the results
(others, please feel free to add more to the list for the results to be
more useful)
    - Architecture of the system
    - Configuration - memory size and number of procs
    - Filesystem sizes
    - Commit ID of the kernel
    - which git tree (XFS git tree or Linus's)
    - xfsprogs version (or commit ID if from the git tree)

Thanks for your help in advance.

Regards,

Chandra


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: WANTED: xfstests results in different architectures

[Dave Chinner]

On Tue, Jun 18, 2013 at 05:59:59PM -0500, Chandra Seetharaman wrote:
> Hello All,
> 
> Couple of weeks backs we had a discussion in xfs meeting to collect
> xfstests results. I volunteered to collect xfstests results from
> different architectures and upload to XFS.org.
> 
> I can run and get the results for x86_64 and ppc64. If anyone has other
> architectures that they can run the tests on and provide me the results,
> I will filter them an upload to XFS.org.

How are you going to filter and display them on xfs.org? Should the
scripts to do this be part of xfstests?

FWIW, without a database of results that users can use to filter the
test results themselves, it will become unmanageable very quickly...

BTW, from my notes from the 2012 LSFMM XFs get-together, there are
these line items related to exactly this:

----
       - Public repository of test results so we can better track failures
                - Look into resurrecting old ASG xfstests results
                  repository and web iquery interface (Ben)
                - host on oss.sgi.com.
                - script to run xfstests and produce publishable output (Ben)
----

Ben, did you ever start to look into this?

> Here is what I think would be of value to provide along with the results
> (others, please feel free to add more to the list for the results to be
> more useful)
>     - Architecture of the system

	- base distro (e.g. /etc/release).

>     - Configuration - memory size and number of procs

I think that all the info that we ask people to report in bug
reports would be a good start....

>     - Filesystem sizes

More useful is the MKFS_OPTIONS and MOUNT_OPTIONS used to run the
tests, as that tells us how much non-default test coverage we are
getting. i.e. default testing or something different.

>     - Commit ID of the kernel

Not useful for kernels built with local, non-public changes, which
is generally 100% of the kernels and userspace packages I test
with.

>     - which git tree (XFS git tree or Linus's)
>     - xfsprogs version (or commit ID if from the git tree)

Same as for the kernel - base version is probably all that is useful
here.

You'd probably also want to capture the console output indicating
test runtimes and why certain tests weren't run.

If you create a pristine $RESULTS_DIR and output all the information
you want to gather into it, then it will be trivial for users to
send information onwards. Providing a command line parameter that
generates a unique results directory and then packages the results
up into a tarball would be a great start. We'd then have a single
file that can be sent up a central point with all the test results
available. We could even do all the filtering/processing before
upload.

IOWs, the idea behind $RESULTS_DIR is to make this sort of scripted
test result gathering simple to do....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: WANTED: xfstests results in different architectures

[Phil White]

On Wed, Jun 19, 2013 at 09:34:26AM +1000, Dave Chinner wrote:
> How are you going to filter and display them on xfs.org? Should the
> scripts to do this be part of xfstests?
> 
> FWIW, without a database of results that users can use to filter the
> test results themselves, it will become unmanageable very quickly...
> 
> BTW, from my notes from the 2012 LSFMM XFs get-together, there are
> these line items related to exactly this:
> 
> ----
>        - Public repository of test results so we can better track failures
>                 - Look into resurrecting old ASG xfstests results
>                   repository and web iquery interface (Ben)
>                 - host on oss.sgi.com.
>                 - script to run xfstests and produce publishable output (Ben)
> ----
> 
> Ben, did you ever start to look into this?

That became my work, not Ben's.

The current proposal is to gather architecture, distro, memory size, number of
procs, characteristics of the storage device (size, performance, type, xfs_info
output), xfsprogs version numbers, the time it takes for each test to run, the
date & time it was run, and the hostname of the system.

The plan was to output that information into a database which we could add
benchmark information into as well with an eye to detecting performance
regressions.

(I felt, for example, that the recent discussion of CRC work and CPU usage
could have been helped by that.)

The last stage was to write a script to upload test run data which oss would
be able to publish.

Some of that work is done.  I need to get back around to it.

-Phil

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: WANTED: xfstests results in different architectures

[Ben Myers]

Dave, Chandra,

On Wed, Jun 19, 2013 at 09:34:26AM +1000, Dave Chinner wrote:
> On Tue, Jun 18, 2013 at 05:59:59PM -0500, Chandra Seetharaman wrote:
> > Couple of weeks backs we had a discussion in xfs meeting to collect
> > xfstests results. I volunteered to collect xfstests results from
> > different architectures and upload to XFS.org.
> > 
> > I can run and get the results for x86_64 and ppc64. If anyone has other
> > architectures that they can run the tests on and provide me the results,
> > I will filter them an upload to XFS.org.
> 
> How are you going to filter and display them on xfs.org? Should the
> scripts to do this be part of xfstests?
> 
> FWIW, without a database of results that users can use to filter the
> test results themselves, it will become unmanageable very quickly...
> 
> BTW, from my notes from the 2012 LSFMM XFs get-together, there are
> these line items related to exactly this:
> 
> ----
>        - Public repository of test results so we can better track failures
>                 - Look into resurrecting old ASG xfstests results
>                   repository and web iquery interface (Ben)
>                 - host on oss.sgi.com.
>                 - script to run xfstests and produce publishable output (Ben)
> ----
> 
> Ben, did you ever start to look into this?

I did.  It is still pretty rough.  I'll post it in reply here.

My design criteria as best as I can remember...

1) not filesystem specific

2) capture the actual results

3) simple, lightweight, very little setup

4) compatible with something like hangcheck so you can figure out when a
machine has hung or crashed 

5) useable for an individual developer

6) useable for a maintainer

7) not web centric

8) uses tools that are likely to be installed with the base distro

9) includes what you're testing, along with the results.

10) upload target can be easily changed.

11) (not implemented) perl scripts to parse the output.

...and so on.  I have no objection to adding databases and web stuff on the
backend.

> > Here is what I think would be of value to provide along with the results
> > (others, please feel free to add more to the list for the results to be
> > more useful)
> >     - Architecture of the system
> 
> 	- base distro (e.g. /etc/release).
> 
> >     - Configuration - memory size and number of procs
> 
> I think that all the info that we ask people to report in bug
> reports would be a good start....
> 
> >     - Filesystem sizes
> 
> More useful is the MKFS_OPTIONS and MOUNT_OPTIONS used to run the
> tests, as that tells us how much non-default test coverage we are
> getting. i.e. default testing or something different.
> 
> >     - Commit ID of the kernel
> 
> Not useful for kernels built with local, non-public changes, which
> is generally 100% of the kernels and userspace packages I test
> with.

You're using a guilt workflow, right?  I think it could be useful if you grab
the commit id, and then a list of all the changes atop that.  I find this
useful with my quilt workflow.

> >     - which git tree (XFS git tree or Linus's)
> >     - xfsprogs version (or commit ID if from the git tree)
> 
> Same as for the kernel - base version is probably all that is useful
> here.
> 
> You'd probably also want to capture the console output indicating
> test runtimes and why certain tests weren't run.

Agreed, gathering runtimes and .notrun is worth doing.

> If you create a pristine $RESULTS_DIR and output all the information
> you want to gather into it, then it will be trivial for users to
> send information onwards. Providing a command line parameter that
> generates a unique results directory and then packages the results
> up into a tarball would be a great start. We'd then have a single
> file that can be sent up a central point with all the test results
> available. We could even do all the filtering/processing before
> upload.
> 
> IOWs, the idea behind $RESULTS_DIR is to make this sort of scripted
> test result gathering simple to do....

Posting results as you go has certain benefits, like hangcheck.

-Ben

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: WANTED: xfstests results in different architectures

[Chandra Seetharaman]

On Wed, 2013-06-19 at 09:34 +1000, Dave Chinner wrote:


Hi Dave,
> On Tue, Jun 18, 2013 at 05:59:59PM -0500, Chandra Seetharaman wrote:
> > Hello All,
> > 
> > Couple of weeks backs we had a discussion in xfs meeting to collect
> > xfstests results. I volunteered to collect xfstests results from
> > different architectures and upload to XFS.org.
> > 
> > I can run and get the results for x86_64 and ppc64. If anyone has other
> > architectures that they can run the tests on and provide me the results,
> > I will filter them an upload to XFS.org.
> 
> How are you going to filter and display them on xfs.org? Should the
> scripts to do this be part of xfstests?

I wasn't thinking of very elaborate filtering of all the results
submitted by all variants of kernel/xfsprogs. 

My thinking was very simple:
 - get test results based on publicly available git tree commit IDs
 - show the commit ID, and the failures seen in a specific arch.
 
This would help when anybody runs xfstests and sees a failure with any
newer code, they would know if it is a regression or already seen.

But, looks like there is more elaborate work in progress. I will sync up
with Ben and Phil to see how to help.
> 
> FWIW, without a database of results that users can use to filter the
> test results themselves, it will become unmanageable very quickly...
> 
> BTW, from my notes from the 2012 LSFMM XFs get-together, there are
> these line items related to exactly this:
> 
> ----
>        - Public repository of test results so we can better track failures
>                 - Look into resurrecting old ASG xfstests results
>                   repository and web iquery interface (Ben)
>                 - host on oss.sgi.com.
>                 - script to run xfstests and produce publishable output (Ben)
> ----
> 
> Ben, did you ever start to look into this?
> 
> > Here is what I think would be of value to provide along with the results
> > (others, please feel free to add more to the list for the results to be
> > more useful)
> >     - Architecture of the system
> 
> 	- base distro (e.g. /etc/release).
> 
> >     - Configuration - memory size and number of procs
> 
> I think that all the info that we ask people to report in bug
> reports would be a good start....
> 
> >     - Filesystem sizes
> 
> More useful is the MKFS_OPTIONS and MOUNT_OPTIONS used to run the
> tests, as that tells us how much non-default test coverage we are
> getting. i.e. default testing or something different.
> 
> >     - Commit ID of the kernel
> 
> Not useful for kernels built with local, non-public changes, which
> is generally 100% of the kernels and userspace packages I test
> with.
> 
> >     - which git tree (XFS git tree or Linus's)
> >     - xfsprogs version (or commit ID if from the git tree)
> 
> Same as for the kernel - base version is probably all that is useful
> here.
> 
> You'd probably also want to capture the console output indicating
> test runtimes and why certain tests weren't run.
> 
> If you create a pristine $RESULTS_DIR and output all the information
> you want to gather into it, then it will be trivial for users to
> send information onwards. Providing a command line parameter that
> generates a unique results directory and then packages the results
> up into a tarball would be a great start. We'd then have a single
> file that can be sent up a central point with all the test results
> available. We could even do all the filtering/processing before
> upload.
> 
> IOWs, the idea behind $RESULTS_DIR is to make this sort of scripted
> test result gathering simple to do....
> 
> Cheers,
> 
> Dave.


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: WANTED: xfstests results in different architectures

[Ben Myers]

Hi Chandra,

On Wed, Jun 19, 2013 at 04:00:39PM -0500, Chandra Seetharaman wrote:
> On Wed, 2013-06-19 at 09:34 +1000, Dave Chinner wrote:
> 
> 
> Hi Dave,
> > On Tue, Jun 18, 2013 at 05:59:59PM -0500, Chandra Seetharaman wrote:
> > > Hello All,
> > > 
> > > Couple of weeks backs we had a discussion in xfs meeting to collect
> > > xfstests results. I volunteered to collect xfstests results from
> > > different architectures and upload to XFS.org.
> > > 
> > > I can run and get the results for x86_64 and ppc64. If anyone has other
> > > architectures that they can run the tests on and provide me the results,
> > > I will filter them an upload to XFS.org.
> > 
> > How are you going to filter and display them on xfs.org? Should the
> > scripts to do this be part of xfstests?
> 
> I wasn't thinking of very elaborate filtering of all the results
> submitted by all variants of kernel/xfsprogs. 
> 
> My thinking was very simple:
>  - get test results based on publicly available git tree commit IDs
>  - show the commit ID, and the failures seen in a specific arch.
>  
> This would help when anybody runs xfstests and sees a failure with any
> newer code, they would know if it is a regression or already seen.
> 
> But, looks like there is more elaborate work in progress. I will sync up
> with Ben and Phil to see how to help.

Here is what little I have so far.  One of my goals is to be able to track all
of the testing in a centralized location for every patch that has been posted
to the list.  Tracking only the test results is useful, but I've found that it
is not valuable as I'd like unless I have a pretty good idea what code was
being tested.  Message-ID seems like the best way to accomplish this mapping to
mailing list archive.

Patch 1:

Grabs some information about the source trees in question.  We don't want this
to be xfs specific so it goes after an environment variable to figure out where
to look for sources.  Set SRCDIRS like a path:

export SRCDIRS="/path/to/kernel:/path/to/xfsprogs:/path/to/xfsdump:/path/to/xfstests"

It will print the git tree/branch/desc of each directory and any quilt patches
you have applied.  It prints md5sum/messageid/patchworkid for each patch that
is applied.  This doesn't address other workflows that use packaging, so I'm
toying with the idea of adding the ability to specify a package name in here,
which would allow those who prefer to use a packaging-based approach a way to
include sources with their test results.

Here is some sample output, which I admit isn't very pretty:

# ./check -g auto
fatal: ref HEAD is not a symbolic ref
SRCDIRS       -- /root/xfs:/root/xfsprogs:/root/xfsdump:/root/xfstests
    /root/xfs:
        URL           -- git://oss.sgi.com/xfs/xfs.git
        BRANCH        -- refs/heads/master
        DESC          -- v3.10-rc1-39-g2fb8b50
        PATCHES:
                /root/xfs/patches/xfs-update-mount-options-documentation.patch
                md5sum:         09a25d03e65fd969c058d6dd39fc1315
                X-Patchwork-Id: 5621
                Message-Id:     <1371617468-32559-2-git-send-email-david@fromorbit.com>
                /root/xfs/patches/xfs-add-pluging-for-bulkstat-readahead.patch
                md5sum:         0c3772ba0792d42332139268e86036e2
                X-Patchwork-Id: 5620
                Message-Id:     <1371617468-32559-3-git-send-email-david@fromorbit.com>
...
                Message-Id:     <1371617468-32559-12-git-send-email-david@fromorbit.com>
                /root/xfs/patches/xfs-use-inode-create-transaction.patch
                md5sum:         fb7259b0b3eb7c29a3dfc3a487c1584e
                X-Patchwork-Id: 5630
                Message-Id:     <1371617468-32559-13-git-send-email-david@fromorbit.com>
    /root/xfsprogs:
        URL           -- git://oss.sgi.com/xfs/cmds/xfsprogs.git
        BRANCH        --
        DESC          -- v3.1.9
        PATCHES:
    /root/xfsdump:
        URL           -- git://oss.sgi.com/xfs/cmds/xfsdump.git
        BRANCH        -- refs/heads/3.1.2
        DESC          -- v3.1.2
    /root/xfstests:
        URL           -- git://oss.sgi.com/xfs/cmds/xfstests.git
        BRANCH        -- refs/heads/master
        DESC          -- linux-v3.8-131-ge2549c6
        PATCHES:
                /root/xfstests/patches/keep-track-10
                md5sum:         64f67d028b133bb540a29d2a1223492c
                X-Patchwork-Id:
                Message-Id:
                /root/xfstests/patches/xfstests-check-cleanup-check.log-update-4
                md5sum:         eb41327c478bf99adb62cd8db3258ad5
                X-Patchwork-Id:
                Message-Id:
                /root/xfstests/patches/putresults-4
                md5sum:         f41fb9bfe8b88e5bab2de440de91e47a
                X-Patchwork-Id:
                Message-Id:
FSTYP         -- xfs (debug)
PLATFORM      -- Linux/x86_64 xfsqa-sum1 3.10.0-rc1+
MKFS_OPTIONS  -- -f -bsize=4096 /dev/sdb1
MOUNT_OPTIONS -- /dev/sdb1 /mnt/scratch

generic/001 5s ... 5s

You can see I'm testing a chunk of the 3.11 queue on this machine...  This
gives us a way to map back from a set of test results to the exact message on
the list.

Patch 2:

This guy just switches us to use a temporary check.log file so that it can be
uploaded later.  At the end it appends this to check.log in the results
directory so that the behavior looks the same as before.  Not much to see here.

Patch 3:

This one archives the description of what is being tested as well as those
sources which are not in the git tree that are being tested.  This should give
you all the info you need to duplicate the test run at a later date.  It also
archives test results during the run, including timestamps of when each test
starts and stops.  This is done so that on the upload end a cron job will be
able to tell whether a given system has hung in the middle of a test run, and
take action if a test is taking longer than expected (future work).  Currently
it is using curl to upload via ftp, which you control by setting an environment
variable:

export ARCHIVE_URL=ftp://anonymous:joe_at_xyz.com@ftp.xyz.com/xfstests/results

I think it might be helpful to add a few more url types.  Ideas that come to
mind are dir://path/to/my/local/archive, and tar:/path/to/tarball.tar

Here's what the current directory structure looks like:

/ftp/xfstests/results/xfsqa-sum1.americas.sgi.com/3.10.0-rc1+/1371679906
	hostname      ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
	kernel version                            ^^^^^^^^^^^
	start date                                            ^^^^^^^^^^

# ls
5015.start  5015.stop  check.desc  check.log  check.time  root/

# find root
root
root/xfs
root/xfs/patches
root/xfs/patches/xfs-update-mount-options-documentation.patch
...
root/xfs/patches/xfs-use-inode-create-transaction.patch
root/xfstests
root/xfstests/patches
root/xfstests/patches/keep-track-10
root/xfstests/patches/xfstests-check-cleanup-check.log-update-4
root/xfstests/patches/putresults-4

All of the patches are in there...

# cat *.st*
tests/generic/001 61919
tests/generic/001 61924

Those are the start and stop times for test 'generic/001'.

check.desc is the output you saw above, check.log is the standard output you'd
expect but only for this run, check.time is the amount of time each test took,
and you also get $seq.notrun, as well as test failure output uploaded.

-----

My future ideas are to write a perl script to parse through the test results
and print some reports.  Based upon that script, we could throw the output into
a website or database, for people who are into that kind of thing.

So, in conclusion, there isn't a whole lot here!  ;)

I would like to stress that it is important to keep track of what you're
testing along with the results, that making the test archival function part of
the test scripts themselves has some advantages, and hopefully this could be
extended to address everyone's preferred workflow.  It is important to have
something that is useful for individual developers as well as commercial
interests, without the fuss of a great deal of web/sql setup.  I've made
provisions for those who prefer to keep their testing private, and also for
people who wish to mine their test results on a larger scale.  Just set
ARCHIVE_URL to something that suits your needs.

I'm hoping to explore some of those ideas again soon.

Thanks,
	Ben

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

[RFC PATCH 1/3] xfstests: get some basic source tree info

[Ben Myers]

Grab some basic information about the souce trees being tested and save
it off for later.  This includes information about the git commit and
any patches which are applied.

Set SRCDIRS environment variable with paths to the sources you're
testing, colon delimited like PATH. 

e.g.

export SRCDIRS="/path/to/kernel:/path/to/xfsprogs:/path/to/xfsdump:/path/to/xfstests"

Signed-off-by: Ben Myers <bpm@sgi.com>

---
 check     |    4 ++++
 common.rc |   36 ++++++++++++++++++++++++++++++++++++
 2 files changed, 40 insertions(+)

Index: xfstests/check
===================================================================
--- xfstests.orig/check
+++ xfstests/check
@@ -316,6 +316,7 @@ END	{ if (NR > 0) {
 
 	echo "" >>$check.log
 	date >>$check.log
+	cat /tmp/check.desc >>$check.log
 	echo $list | fmt | sed -e 's/^/    /' -e "s;$SRC_DIR/;;g" >>$check.log
 	$interrupt && echo "Interrupted!" >>$check.log
         
@@ -363,6 +364,9 @@ rm -f $check.full
 
 [ -f $check.time ] || touch $check.time
 
+_full_source_details > /tmp/check.desc
+cat /tmp/check.desc
+
 # print out our test configuration
 echo "FSTYP         -- `_full_fstyp_details`"
 echo "PLATFORM      -- `_full_platform_details`"
Index: xfstests/common/rc
===================================================================
--- xfstests.orig/common/rc
+++ xfstests/common/rc
@@ -1738,6 +1738,48 @@ _full_platform_details()
      echo "$os/$platform $host $kernel"
 }
 
+_full_source_details()
+{
+    if [ -z $SRCDIRS ]; then
+        return
+    fi
+
+    echo "SRCDIRS       -- $SRCDIRS"
+
+    dirs=$(echo $SRCDIRS | tr ":" "\n")
+    for dir in $dirs
+    do
+        echo "    $dir:"
+        if [ -d $dir/.git ]; then
+            # git url, branch, and description
+            url=$(cd $dir; git remote show origin | grep 'Fetch URL' | awk '{print $3}')
+            echo -e "\tURL           -- $url"
+            branch=$(cd $dir; git symbolic-ref HEAD)
+            echo -e "\tBRANCH        -- $branch"
+            desc=$(cd $dir; git describe)
+            echo -e "\tDESC          -- $desc"
+        fi
+
+        if [ -d $dir/patches ]; then
+            # quilt patches which are applied
+            echo -e "\tPATCHES:"
+            for p in $(cd $dir; quilt applied 2> /dev/null)
+            do
+                p=${p##patches/}
+                md5=$(md5sum $dir/patches/$p)
+                md5_checksum=${md5%% *}
+                patch_fn=${md5##* }
+                patchworkid=$(egrep -i '^X-Patchwork-Id:' $dir/patches/$p)
+                messageid=$(egrep -i '^Message-Id:' $dir/patches/$p)
+                echo -e "\t\t$patch_fn"
+                echo -e "\t\tmd5sum:\t\t$md5_checksum"
+                echo -e "\t\tX-Patchwork-Id:\t${patchworkid##* }"
+                echo -e "\t\tMessage-Id:\t${messageid##* }"
+            done
+        fi
+    done
+}
+
 _setup_udf_scratchdir()
 {
     [ "$FSTYP" != "udf" ] \

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [RFC PATCH 1/3] xfstests: get some basic source tree info

[Eric Sandeen]

On 6/19/13 5:44 PM, Ben Myers wrote:
> Grab some basic information about the souce trees being tested and save
> it off for later.  

save it where?  Ah, results/check.log.

README updates maybe?

> This includes information about the git commit and
> any patches which are applied.
> 
> Set SRCDIRS environment variable with paths to the sources you're
> testing, colon delimited like PATH. 
> 
> e.g.
> 
> export SRCDIRS="/path/to/kernel:/path/to/xfsprogs:/path/to/xfsdump:/path/to/xfstests"

ok so after looking at the patch more carefully, this can be any
collection of paths to git trees in any order, right, and outputs i.e.:

SRCDIRS       -- /mnt/test2/git/linux-2.6:/mnt/test2/git/xfsdump:/mnt/test2/git/xfsprogs
    /mnt/test2/git/linux-2.6:
	URL           -- git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
	BRANCH        -- refs/heads/master
	DESC          -- v3.10-rc2-48-g4284097
    /mnt/test2/git/xfsdump:
	URL           -- git://oss.sgi.com/xfs/cmds/xfsdump.git
	BRANCH        -- refs/heads/master
	DESC          -- v3.1.2-4-g71f940f
...

But what if we're testing packaged bits, not from a git tree,
should that be collected?

> Signed-off-by: Ben Myers <bpm@sgi.com>
> 
> ---
>  check     |    4 ++++
>  common.rc |   36 ++++++++++++++++++++++++++++++++++++
>  2 files changed, 40 insertions(+)
> 
> Index: xfstests/check
> ===================================================================
> --- xfstests.orig/check
> +++ xfstests/check
> @@ -316,6 +316,7 @@ END	{ if (NR > 0) {
>  
>  	echo "" >>$check.log
>  	date >>$check.log
> +	cat /tmp/check.desc >>$check.log
>  	echo $list | fmt | sed -e 's/^/    /' -e "s;$SRC_DIR/;;g" >>$check.log
>  	$interrupt && echo "Interrupted!" >>$check.log
>          
> @@ -363,6 +364,9 @@ rm -f $check.full
>  
>  [ -f $check.time ] || touch $check.time
>  
> +_full_source_details > /tmp/check.desc
> +cat /tmp/check.desc
> +
>  # print out our test configuration
>  echo "FSTYP         -- `_full_fstyp_details`"
>  echo "PLATFORM      -- `_full_platform_details`"
> Index: xfstests/common/rc
> ===================================================================
> --- xfstests.orig/common/rc
> +++ xfstests/common/rc
> @@ -1738,6 +1738,48 @@ _full_platform_details()
>       echo "$os/$platform $host $kernel"
>  }
>  
> +_full_source_details()
> +{
> +    if [ -z $SRCDIRS ]; then
> +        return
> +    fi
> +
> +    echo "SRCDIRS       -- $SRCDIRS"
> +
> +    dirs=$(echo $SRCDIRS | tr ":" "\n")
> +    for dir in $dirs
> +    do
> +        echo "    $dir:"
> +        if [ -d $dir/.git ]; then
> +            # git url, branch, and description
> +            url=$(cd $dir; git remote show origin | grep 'Fetch URL' | awk '{print $3}')
> +            echo -e "\tURL           -- $url"
> +            branch=$(cd $dir; git symbolic-ref HEAD)
> +            echo -e "\tBRANCH        -- $branch"
> +            desc=$(cd $dir; git describe)
> +            echo -e "\tDESC          -- $desc"
> +        fi
> +
> +        if [ -d $dir/patches ]; then
> +            # quilt patches which are applied

what if it's guilt not quilt?

> +            echo -e "\tPATCHES:"
> +            for p in $(cd $dir; quilt applied 2> /dev/null)
> +            do
> +                p=${p##patches/}
> +                md5=$(md5sum $dir/patches/$p)
> +                md5_checksum=${md5%% *}
> +                patch_fn=${md5##* }
> +                patchworkid=$(egrep -i '^X-Patchwork-Id:' $dir/patches/$p)
> +                messageid=$(egrep -i '^Message-Id:' $dir/patches/$p)
> +                echo -e "\t\t$patch_fn"
> +                echo -e "\t\tmd5sum:\t\t$md5_checksum"
> +                echo -e "\t\tX-Patchwork-Id:\t${patchworkid##* }"
> +                echo -e "\t\tMessage-Id:\t${messageid##* }"
> +            done
> +        fi
> +    done
> +}
> +
>  _setup_udf_scratchdir()
>  {
>      [ "$FSTYP" != "udf" ] \
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [RFC PATCH 1/3] xfstests: get some basic source tree info

[Dave Chinner]

On Wed, Jun 19, 2013 at 10:20:10PM -0500, Eric Sandeen wrote:
> On 6/19/13 5:44 PM, Ben Myers wrote:
> > Grab some basic information about the souce trees being tested and save
> > it off for later.  
> 
> save it where?  Ah, results/check.log.
> 
> README updates maybe?
> 
> > This includes information about the git commit and
> > any patches which are applied.
> > 
> > Set SRCDIRS environment variable with paths to the sources you're
> > testing, colon delimited like PATH. 
> > 
> > e.g.
> > 
> > export SRCDIRS="/path/to/kernel:/path/to/xfsprogs:/path/to/xfsdump:/path/to/xfstests"
> 
> ok so after looking at the patch more carefully, this can be any
> collection of paths to git trees in any order, right, and outputs i.e.:
> 
> SRCDIRS       -- /mnt/test2/git/linux-2.6:/mnt/test2/git/xfsdump:/mnt/test2/git/xfsprogs
>     /mnt/test2/git/linux-2.6:
> 	URL           -- git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
> 	BRANCH        -- refs/heads/master
> 	DESC          -- v3.10-rc2-48-g4284097
>     /mnt/test2/git/xfsdump:
> 	URL           -- git://oss.sgi.com/xfs/cmds/xfsdump.git
> 	BRANCH        -- refs/heads/master
> 	DESC          -- v3.1.2-4-g71f940f
> ...
> 
> But what if we're testing packaged bits, not from a git tree,
> should that be collected?

Ideally, yes. My CI system builds packages, then ships them over to
the VMs that run tests and installs them before the tests are run.
So there's never any source trees available for such probing. And
often I just use the versions install on the distro I'm doing my
testing on.

> > +_full_source_details()
> > +{
> > +    if [ -z $SRCDIRS ]; then
> > +        return
> > +    fi
> > +
> > +    echo "SRCDIRS       -- $SRCDIRS"
> > +
> > +    dirs=$(echo $SRCDIRS | tr ":" "\n")
> > +    for dir in $dirs
> > +    do
> > +        echo "    $dir:"
> > +        if [ -d $dir/.git ]; then
> > +            # git url, branch, and description
> > +            url=$(cd $dir; git remote show origin | grep 'Fetch URL' | awk '{print $3}')
> > +            echo -e "\tURL           -- $url"
> > +            branch=$(cd $dir; git symbolic-ref HEAD)
> > +            echo -e "\tBRANCH        -- $branch"
> > +            desc=$(cd $dir; git describe)
> > +            echo -e "\tDESC          -- $desc"
> > +        fi
> > +
> > +        if [ -d $dir/patches ]; then
> > +            # quilt patches which are applied
> 
> what if it's guilt not quilt?

Then it will give a git signature that doesn't match anything that
was ever published.

But from my perspective, that simply doesn't matter.  I'm not going
to be sending lists of patches I'm testing to a remote server -
there's way too much scope for inadvertant information leaks in
doing this. Especially considering patch 3/3 sends the *patches* to
the remote server via the _archive function.  There's no way I'd be
recommending such functionality be used in environments that contain
both upstream and internal development trees.

What you propose might be fine for individual developer usage, but
it's not appropriate for public distribution of test results.
Result logs for public distribution need to be as anonymous as
possible with only the bare minimum of identifying information in
them.

Hence for public consumption, I'd suggest that we'd do far better to
just target released kernels and packages. e.g. the kernel release
tag like 3.10-rc4, and the xfsprogs/dump/test version being used.
Having a repository of full of failure reports from random developer
patchsets and kernels is not useful to anyone....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [RFC PATCH 1/3] xfstests: get some basic source tree info

[Ben Myers]

Hey Eric & Dave,

On Thu, Jun 20, 2013 at 01:50:05PM +1000, Dave Chinner wrote:
> On Wed, Jun 19, 2013 at 10:20:10PM -0500, Eric Sandeen wrote:
> > On 6/19/13 5:44 PM, Ben Myers wrote:
> > > Grab some basic information about the souce trees being tested and save
> > > it off for later.  
> > 
> > save it where?  Ah, results/check.log.
> > 
> > README updates maybe?
> > 
> > > This includes information about the git commit and
> > > any patches which are applied.
> > > 
> > > Set SRCDIRS environment variable with paths to the sources you're
> > > testing, colon delimited like PATH. 
> > > 
> > > e.g.
> > > 
> > > export SRCDIRS="/path/to/kernel:/path/to/xfsprogs:/path/to/xfsdump:/path/to/xfstests"
> > 
> > ok so after looking at the patch more carefully, this can be any
> > collection of paths to git trees in any order, right, and outputs i.e.:
> > 
> > SRCDIRS       -- /mnt/test2/git/linux-2.6:/mnt/test2/git/xfsdump:/mnt/test2/git/xfsprogs
> >     /mnt/test2/git/linux-2.6:
> > 	URL           -- git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
> > 	BRANCH        -- refs/heads/master
> > 	DESC          -- v3.10-rc2-48-g4284097
> >     /mnt/test2/git/xfsdump:
> > 	URL           -- git://oss.sgi.com/xfs/cmds/xfsdump.git
> > 	BRANCH        -- refs/heads/master
> > 	DESC          -- v3.1.2-4-g71f940f
> > ...
> > 
> > But what if we're testing packaged bits, not from a git tree,
> > should that be collected?

IMO we should support both git tree and packaged workflows.

> Ideally, yes. My CI system builds packages, then ships them over to
> the VMs that run tests and installs them before the tests are run.
> So there's never any source trees available for such probing. And
> often I just use the versions install on the distro I'm doing my
> testing on.

Any suggestions on how to track the sources the packaged workflow?  My
first thought is that we could package sources in the .rpm or .deb or
srpm and ship that over to the test machine.  Then add the name of the
rpm/deb to the SRCDIRS envvar and teach this thing to grab the sources
from there.  But if you're talking about a kernel-source rpm this could
get fairly large.

Alternatively, if you have a way of tagging what you're testing and
linking that up with your SCM you don't have to go after the sources on
the test box, just make sure to make a note of the tag.  Maybe add the
git url/branch/desc the rpm info which you could query with -qi?

> > > +_full_source_details()
> > > +{
> > > +    if [ -z $SRCDIRS ]; then
> > > +        return
> > > +    fi
> > > +
> > > +    echo "SRCDIRS       -- $SRCDIRS"
> > > +
> > > +    dirs=$(echo $SRCDIRS | tr ":" "\n")
> > > +    for dir in $dirs
> > > +    do
> > > +        echo "    $dir:"
> > > +        if [ -d $dir/.git ]; then
> > > +            # git url, branch, and description
> > > +            url=$(cd $dir; git remote show origin | grep 'Fetch URL' | awk '{print $3}')
> > > +            echo -e "\tURL           -- $url"
> > > +            branch=$(cd $dir; git symbolic-ref HEAD)
> > > +            echo -e "\tBRANCH        -- $branch"
> > > +            desc=$(cd $dir; git describe)
> > > +            echo -e "\tDESC          -- $desc"
> > > +        fi
> > > +
> > > +        if [ -d $dir/patches ]; then
> > > +            # quilt patches which are applied
> > 
> > what if it's guilt not quilt?

Right now I'm a quilt guy... but I think it would be nice if we could
support both.
 
> Then it will give a git signature that doesn't match anything that was
> ever published.
>
> But from my perspective, that simply doesn't matter.
>
> I'm not going to be sending lists of patches I'm testing to a remote
> server - there's way too much scope for inadvertant information leaks
> in doing this.  Especially considering patch 3/3 sends the *patches*
> to the remote server via the _archive function.  There's no way I'd be
> recommending such functionality be used in environments that contain
> both upstream and internal development trees.

I want to do this in a way that doesn't force users to use it.  Maybe
add an --upload option to check so that it's clearer whether you're
going to upload?
 
> What you propose might be fine for individual developer usage, but
> it's not appropriate for public distribution of test results.

I'm looking for something that is flexable enough for both individual
developer usage as well as public distribution, in the same mechanism.

> Result logs for public distribution need to be as anonymous as
> possible with only the bare minimum of identifying information in
> them.

To what end?  We're already posting to a publicly available mailing list
which is archived for posterity.  There is no anonymity here.  I guess
we could hash the hostname into something unrecognisable?  It's worth
keeping an email address, I think.
 
> Hence for public consumption, I'd suggest that we'd do far better to
> just target released kernels and packages. e.g. the kernel release tag
> like 3.10-rc4, and the xfsprogs/dump/test version being used.

I tend to agree that for public consumption you'd want to upload only
tthe results of tagged versions of the kernels/xfsprogs/dump/test.  You
generally wouldn't want to upload results from day-to-day development
work to a public location, but this does allow you to upload those
results to an internal server if you want to.

However, if Jane Developer wants to post test results for the patches
she has published, I want to enable her to do so... maybe to a different
location than the tagged versions.

> Having a repository of full of failure reports from random developer
> patchsets and kernels is not useful to anyone....

Too much data from test reports is a problem I'd love to have.  ;)

-Ben

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

[RFC PATCH 2/3] xfstests: use an intermediate check.log file

[Ben Myers]

Use an intermediate check.log file for each test run so that it can be
uploaded to a central location by a subsequent patch in this series.
The intermediate file is appended to $RESULTS_DIR/check.log a the end of
each run.

Signed-off-by: Ben Myers <bpm@sgi.com>

---
 check |   21 ++++++++++++---------
 1 file changed, 12 insertions(+), 9 deletions(-)

Index: xfstests.new2/check
===================================================================
--- xfstests.new2.orig/check
+++ xfstests.new2/check
@@ -314,11 +314,11 @@ END	{ if (NR > 0) {
 	    mv $tmp.out $check.time
 	fi
 
-	echo "" >>$check.log
-	date >>$check.log
-	cat /tmp/check.desc >>$check.log
-	echo $list | fmt | sed -e 's/^/    /' -e "s;$SRC_DIR/;;g" >>$check.log
-	$interrupt && echo "Interrupted!" >>$check.log
+	echo "" >>/tmp/check.log
+	date >>/tmp/check.log
+	cat /tmp/check.desc >>/tmp/check.log
+	echo $list | fmt | sed -e 's/^/    /' -e "s;$SRC_DIR/;;g" >>/tmp/check.log
+	$interrupt && echo "Interrupted!" >>/tmp/check.log
         
         if [ ! -z "$n_try" -a $n_try != 0 ]
 	then
@@ -328,20 +328,22 @@ END	{ if (NR > 0) {
 	if [ ! -z "$notrun" ]
 	then
 	    echo "Not run:$notrun"
-	    echo "Not run:$notrun" >>$check.log
+	    echo "Not run:$notrun" >>/tmp/check.log
 	fi
 
         if [ ! -z "$n_bad" -a $n_bad != 0 ]
 	then
 	    echo "Failures:$bad"
 	    echo "Failed $n_bad of $n_try tests"
-	    echo "Failures:$bad" | fmt >>$check.log
-	    echo "Failed $n_bad of $n_try tests" >>$check.log
+	    echo "Failures:$bad" | fmt >>/tmp/check.log
+	    echo "Failed $n_bad of $n_try tests" >>/tmp/check.log
 	else
 	    echo "Passed all $n_try tests"
-	    echo "Passed all $n_try tests" >>$check.log
+	    echo "Passed all $n_try tests" >>/tmp/check.log
 	fi
 	needwrap=false
+        cat /tmp/check.log >> $check.log
+	rm -f /tmp/check.log
     fi
 
     rm -f /tmp/*.rawout /tmp/*.out /tmp/*.err /tmp/*.time

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

[RFC PATCH 3/3] xfstests: upload test results

[Ben Myers]

Set the ARCHIVE_URL environment variable to upload test results to a
central location:

export ARCHIVE_URL=ftp://anonymous:joe_at_xyz.com@ftp.xyz.com/xfstests/results

---
 check     |   14 ++++++++++++++
 common.rc |   24 ++++++++++++++++++++++++
 2 files changed, 38 insertions(+)

Index: xfstests/check
===================================================================
--- xfstests.orig/check
+++ xfstests/check
@@ -28,6 +28,7 @@ n_bad=0
 bad=""
 notrun=""
 interrupt=true
+start_time=`date "+%s"`
 diff="diff -u"
 showme=false
 have_test_arg=false
@@ -312,6 +313,7 @@ END	{ if (NR > 0) {
 	}' \
 	    | sort -n >$tmp.out
 	    mv $tmp.out $check.time
+	    _archive $check.time $start_time
 	fi
 
 	echo "" >>/tmp/check.log
@@ -343,6 +345,7 @@ END	{ if (NR > 0) {
 	fi
 	needwrap=false
         cat /tmp/check.log >> $check.log
+	_archive /tmp/check.log $start_time
 	rm -f /tmp/check.log
     fi
 
@@ -368,6 +371,7 @@ rm -f $check.full
 
 _full_source_details > /tmp/check.desc
 cat /tmp/check.desc
+_archive /tmp/check.desc $start_time
 
 # print out our test configuration
 echo "FSTYP         -- `_full_fstyp_details`"
@@ -453,6 +457,8 @@ do
 	rm -f core $seqres.notrun
 
 	start=`_wallclock`
+	echo "$seq $start" > $tmp.start
+	_archive $tmp.start $start_time
 	$timestamp && echo -n "	["`date "+%T"`"]"
 	[ ! -x $seq ] && chmod u+x $seq # ensure we can run it
 	$LOGGER_PROG "run xfstest $seqnum"
@@ -460,6 +466,8 @@ do
 	sts=$?
 	$timestamp && _timestamp
 	stop=`_wallclock`
+	echo "$seq $stop" > $tmp.stop
+	_archive $tmp.stop $start_time
 
 	_fix_malloc <$tmp.rawout >$tmp.out
 	rm -f $tmp.rawout
@@ -468,6 +476,7 @@ do
 	then
 	    echo -n " [dumped core]"
 	    mv core $RESULT_BASE/$seqnum.core
+	    _archive $RESULT_BASE/$seqnum.core $start_time
 	    err=true
 	fi
 
@@ -476,6 +485,7 @@ do
 	    $timestamp || echo -n " [not run] "
 	    $timestamp && echo " [not run]" && echo -n "	$seqnum -- "
 	    cat $seqres.notrun
+            _archive $seqres.notrun $start_time
 	    notrun="$notrun $seqnum"
 	else
 	    if [ $sts -ne 0 ]
@@ -501,6 +511,8 @@ do
 		else
 		    echo " - output mismatch (see $seqres.out.bad)"
 		    mv $tmp.out $seqres.out.bad
+		    _archive $seqres.out $start_time
+		    _archive $seqres.out.bad $start_time
 		    $diff $seq.out $seqres.out.bad | {
 		        if test "$DIFF_LENGTH" -le 0; then
 				cat
@@ -515,7 +527,7 @@ do
 		fi
 	    fi
 	fi
-
+        rm -f $tmp.$seq.start $tmp.seq.stop
     fi
 
     # come here for each test, except when $showme is true
Index: xfstests/common/rc
===================================================================
--- xfstests.orig/common/rc
+++ xfstests/common/rc
@@ -1738,6 +1738,27 @@ _full_platform_details()
      echo "$os/$platform $host $kernel"
 }
 
+_archive()
+{
+    if [ -z "$ARCHIVE_URL" ]; then
+        return;
+    fi
+
+    file=$1
+    subdir=$2
+    args=$3
+
+    if [ ! -e $file ]; then
+        _fail "archive: $file does not exist"
+    fi
+
+    desc=$(cat /proc/version | awk '{print $3}')
+
+    url=$ARCHIVE_URL/`hostname -f`/$desc/$subdir
+    curl $args --ftp-create-dirs --silent --upload-file \
+         $file $url/`basename $file`
+}
+
 _full_source_details()
 {
     if [ -z $SRCDIRS ]; then
@@ -1775,6 +1796,7 @@ _full_source_details()
                 echo -e "\t\tmd5sum:\t\t$md5_checksum"
                 echo -e "\t\tX-Patchwork-Id:\t${patchworkid##* }"
                 echo -e "\t\tMessage-Id:\t${messageid##* }"
+                _archive $dir/patches/$p $start_time/$dir/patches
             done
         fi
     done

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs