[PATCH] Create test groups for cifs in xfstests

[PATCH] Create test groups for cifs in xfstests

[Steve French]

[-- Attachment #1: Type: text/plain, Size: 631 bytes --]

Attached is a trivial patch to create two xfstest groups for cifs

Since some of the tests in xfstests are only appropriate for local filesystems
and some of the "quick" tests are only quick on local, not network
filesystems, create two test groups for testing cifs to make
use of xfstests easier. (I will add a similar follow on patch for
nfs after this). The first new test group "cifs-quick"
includes a set of tests that execute fairly quickly (under a few minutes
each), and second "cifs" which includes the tests which can take
much longer or are not as appropriate for quick verification of
a cifs build.

-- 
Thanks,

Steve

[-- Attachment #2: 0001-xfstests-create-a-group-for-cifs-tests.patch --]
[-- Type: text/x-patch, Size: 9089 bytes --]

From a88089cea07275309cc14e3d4841a7807bc03e07 Mon Sep 17 00:00:00 2001
From: Steve French <sfrench@xfstest.smfdom>
Date: Thu, 14 Apr 2016 22:27:58 -0500
Subject: [PATCH] xfstests: create a group for cifs tests

Since some of the tests are only appropriate for local filesystems
and some of the "quick" tests are only quick on local, not network
filesystems, create two test groups for testing cifs to make
use of xfstests easier. (I will add a similar follow on patch for
nfs after this). The first new test group "cifs-quick"
includes a set of tests that execute fairly quickly (under a few minutes
each), and second "cifs" which includes the tests which can take
much longer or are not as appropriate for quick verification of
a cifs build.

./check -cifs -g cifs-quick

produces terser output than ./check -cifs which is noisier and starts
many tests only to skip them due to prereqs that are not
appropriate for network file systems.

This group of tests has also pointed out a few bugs we
need to address in cifs so I am hoping these will make it easier
for people to run xfstests in testing network file systems.

Signed-off-by: Steve French <steve.french@primarydata.com>
---
 tests/generic/group | 152 ++++++++++++++++++++++++++--------------------------
 1 file changed, 76 insertions(+), 76 deletions(-)

diff --git a/tests/generic/group b/tests/generic/group
index ef1a423..e577ae9 100644
--- a/tests/generic/group
+++ b/tests/generic/group
@@ -3,20 +3,20 @@
 # - do not start group names with a digit
 # - comment line before each group is "new" description
 #
-001 rw dir udf auto quick
-002 metadata udf auto quick
+001 rw dir udf auto quick cifs
+002 metadata udf auto quick cifs-quick
 003 atime auto quick
 004 auto quick
-005 dir udf auto quick
-006 dir udf auto quick
-007 dir udf auto quick
+005 dir udf auto quick cifs-quick
+006 dir udf auto quick cifs
+007 dir udf auto quick cifs
 008 auto quick prealloc
 009 auto quick prealloc
-010 other udf auto
-011 dir udf auto quick
+010 other udf auto cifs-quick
+011 dir udf auto quick cifs
 012 auto quick prealloc
-013 other ioctl udf auto quick
-014 rw udf auto quick
+013 other ioctl udf auto quick cifs
+014 rw udf auto quick cifs
 015 other auto quick
 016 auto quick prealloc
 017 auto prealloc
@@ -25,20 +25,20 @@
 020 metadata attr udf auto quick
 021 auto quick prealloc
 022 auto quick prealloc
-023 auto quick
-024 auto quick
+023 auto quick cifs-quick
+024 auto quick cifs-quick
 025 auto quick
 026 acl quick auto
 027 auto enospc
-028 auto quick
-029 auto quick rw
-030 auto quick rw
+028 auto quick cifs-quick
+029 auto quick rw cifs-quick
+030 auto quick rw cifs-quick
 031 auto quick prealloc rw
 032 auto quick rw
 033 auto quick rw
 034 auto quick metadata log
-035 auto quick
-036 auto aio rw stress
+035 auto quick cifs-quick
+036 auto aio rw stress cifs
 037 metadata auto quick
 038 auto stress
 039 metadata auto quick
@@ -69,40 +69,40 @@
 064 auto quick prealloc
 065 metadata auto quick
 066 metadata auto quick
-067 auto quick mount
+067 auto quick mount cifs-quick
 068 other auto freeze dangerous stress
-069 rw udf auto quick
+069 rw udf auto quick cifs
 070 attr udf auto quick stress
 071 auto quick prealloc
 072 auto metadata stress
 073 metadata auto quick
-074 rw udf auto
-075 rw udf auto quick
+074 rw udf auto cifs
+075 rw udf auto quick cifs-quick
 076 metadata rw udf auto quick stress
 077 acl attr auto enospc
 078 auto quick metadata
 079 acl attr ioctl metadata auto quick
-080 auto quick
+080 auto quick cifs-quick
 081 auto quick
 082 auto quick quota
 083 rw auto enospc stress
-084 auto metadata quick
+084 auto metadata quick cifs-quick
 085 auto freeze mount
 086 auto prealloc preallocrw quick
-087 perms auto quick
-088 perms auto quick
-089 metadata auto
+087 perms auto quick cifs-quick
+088 perms auto quick cifs-quick
+089 metadata auto cifs
 090 metadata auto quick
-091 rw auto quick
+091 rw auto quick cifs-quick
 092 auto quick prealloc
 093 attr cap udf auto
 094 auto quick prealloc
-095 auto rw stress
+095 auto rw stress cifs-quick
 096 auto prealloc quick
 097 udf auto
-098 auto quick metadata
+098 auto quick metadata cifs-quick
 099 udf auto
-100 udf auto
+100 udf auto cifs
 101 auto quick metadata
 102 auto rw
 103 auto enospc quick
@@ -111,7 +111,7 @@
 106 auto quick metadata
 107 auto quick metadata
 108 auto quick rw
-109 auto metadata dir
+109 auto metadata dir cifs-quick
 110 auto quick clone
 111 auto quick clone
 112 rw aio auto quick
@@ -122,28 +122,28 @@
 117 attr auto quick
 118 auto quick clone
 119 auto quick clone
-120 other auto quick
+120 other auto quick cifs-quick
 121 auto quick clone dedupe
 122 auto quick clone dedupe
-123 perms auto quick
-124 pattern auto quick
-125 other pnfs
-126 perms auto quick
-127 rw auto
-128 perms auto quick
-129 rw auto quick
-130 pattern auto quick
+123 perms auto quick cifs-quick
+124 pattern auto quick cifs-quick
+125 other pnfs cifs-quick
+126 perms auto quick cifs-quick
+127 rw auto cifs
+128 perms auto quick cifs-quick
+129 rw auto quick cifs
+130 pattern auto quick cifs-quick
 131 perms auto quick
 132 pattern auto
-133 rw auto
+133 rw auto cifs
 134 auto quick clone
-135 metadata auto quick
+135 metadata auto quick cifs
 136 auto quick clone dedupe
 137 auto clone dedupe
 138 auto quick clone
 139 auto quick clone
 140 auto quick clone
-141 rw auto quick
+141 rw auto quick cifs
 142 auto quick clone
 143 auto quick clone
 144 auto quick clone
@@ -171,7 +171,7 @@
 166 auto clone
 167 auto clone
 168 auto clone
-169 rw metadata auto quick
+169 rw metadata auto quick cifs
 170 auto clone
 171 auto quick clone
 172 auto quick clone
@@ -186,7 +186,7 @@
 181 auto quick clone
 182 auto quick clone dedupe
 183 auto quick clone
-184 metadata auto quick
+184 metadata auto quick cifs-quick
 185 auto quick clone
 186 auto clone
 187 auto clone
@@ -194,13 +194,13 @@
 189 auto quick clone
 190 auto quick clone
 191 auto quick clone
-192 atime auto
-193 metadata auto quick
+192 atime auto cifs
+193 metadata auto quick cifs
 194 auto quick clone
 195 auto quick clone
 196 auto quick clone
 197 auto quick clone
-198 auto aio quick
+198 auto aio quick cifs-quick
 199 auto quick clone
 200 auto quick clone
 201 auto quick clone
@@ -209,21 +209,21 @@
 204 metadata rw auto
 205 auto quick clone
 206 auto quick clone
-207 auto aio quick
-208 auto aio
-209 auto aio
-210 auto aio quick
-211 auto aio quick
-212 auto aio quick
+207 auto aio quick cifs-quick
+208 auto aio cifs
+209 auto aio cifs
+210 auto aio quick cifs
+211 auto aio quick cifs-quick
+212 auto aio quick cifs-quick
 213 rw auto prealloc quick enospc
 214 rw auto prealloc quick
-215 auto metadata quick
+215 auto metadata quick cifs-quick
 216 auto quick clone
 217 auto quick clone
 218 auto quick clone
 219 auto quota quick
 220 auto quick clone
-221 auto metadata quick
+221 auto metadata quick cifs-quick
 222 auto quick clone
 223 auto quick
 224 auto
@@ -238,20 +238,20 @@
 233 auto quota stress
 234 auto quota
 235 auto quota quick
-236 auto quick metadata
+236 auto quick metadata cifs-quick
 237 auto quick acl
 238 auto quick clone
-239 auto aio rw
+239 auto aio rw cifs-quick
 240 auto aio quick rw
-241 auto
+241 auto cifs
 242 auto clone
 243 auto clone
 244 auto quick quota
-245 auto quick dir
-246 auto quick rw
-247 auto quick rw
-248 auto quick rw
-249 auto quick rw
+245 auto quick dir cifs
+246 auto quick rw cifs-quick
+247 auto quick rw cifs
+248 auto quick rw cifs-quick
+249 auto quick rw cifs-quick
 250 auto quick prealloc rw
 251 ioctl trim
 252 auto quick prealloc rw
@@ -259,13 +259,13 @@
 254 auto quick clone
 255 auto quick prealloc
 256 auto quick
-257 dir auto quick
-258 auto quick
+257 dir auto quick cifs-quick
+258 auto quick cifs-quick
 259 auto quick clone
 260 auto quick trim
 261 auto quick clone
 262 auto quick clone
-263 rw auto quick
+263 rw auto quick cifs
 264 auto quick clone
 265 auto quick clone
 266 auto quick clone
@@ -287,8 +287,8 @@
 282 auto quick clone
 283 auto quick clone
 284 auto quick clone
-285 auto rw
-286 auto quick other
+285 auto rw cifs-quick
+286 auto quick other cifs
 287 auto quick clone
 288 auto quick ioctl trim
 289 auto quick clone
@@ -296,7 +296,7 @@
 291 auto quick clone
 292 auto quick clone
 293 auto quick clone
-294 auto quick
+294 auto quick cifs
 295 auto quick clone
 296 auto quick clone
 297 auto clone
@@ -308,24 +308,24 @@
 303 auto quick clone
 304 auto quick clone
 305 auto quick clone
-306 auto quick rw
+306 auto quick rw cifs
 307 auto quick
-308 auto quick
-309 auto quick
-310 auto
+308 auto quick cifs-quick
+309 auto quick cifs-quick
+310 auto cifs
 311 auto metadata log
 312 auto quick prealloc enospc
-313 auto quick
+313 auto quick cifs
 314 auto quick acl
 315 auto quick rw prealloc
 316 auto quick
-317 auto metadata quick
+317 auto metadata quick cifs
 318 acl attr auto quick
 319 acl auto quick
 320 auto rw
 321 auto quick metadata log
 322 auto quick metadata log
-323 auto aio stress
+323 auto aio stress cifs
 324 auto fsr quick
 325 auto quick data log
 326 auto quick clone
-- 
2.5.0

Re: [PATCH] Create test groups for cifs in xfstests

[Eric Sandeen]

Hi Steve -

On 4/14/16 10:53 PM, Steve French wrote:
> Attached is a trivial patch to create two xfstest groups for cifs
> 
> Since some of the tests in xfstests are only appropriate for local filesystems

Are there tests that are wholly inappropriate for cifs (or nfs) which try
to run anyway and flame out, explode, or fail nonsensically?

In that case, an _unsupported_fs or a _require_local_fs might be a
better way to exclude those tests.  (those don't exist yet, but
could easily).

> and some of the "quick" tests are only quick on local, not network
> filesystems,

Dave has a good idea on this, I'll let him chime in ;)

> create two test groups for testing cifs to make
> use of xfstests easier. (I will add a similar follow on patch for
> nfs after this). The first new test group "cifs-quick"
> includes a set of tests that execute fairly quickly (under a few minutes
> each), and second "cifs" which includes the tests which can take
> much longer or are not as appropriate for quick verification of
> a cifs build.

fs type specifications are a little odd in groups files; udf is in
there, but that's for hysterical raisins.

My worry is that as new tests are added, you're going to have to keep
chasing them with these new tags, and many new tests won't get run
because you'll have too much "./check -g cifs" muscle memory.  :)

More descriptive tags and functions like _require_local_fs, 
_unsupported_fs, or group tags to exclude slow network tests
keeps things a lot more generic, I think.

The commit log sounds like terse output is desired, too?
Perhaps a -q (quiet) switch could suppress all the "not run"
tests, although I guess I usually want to see why things don't
run, myself.

-Eric

Re: [PATCH] Create test groups for cifs in xfstests

[Dave Chinner]

On Thu, Apr 14, 2016 at 10:53:15PM -0500, Steve French wrote:
> Attached is a trivial patch to create two xfstest groups for cifs
> 
> Since some of the tests in xfstests are only appropriate for local filesystems
> and some of the "quick" tests are only quick on local, not network
> filesystems, create two test groups for testing cifs to make
> use of xfstests easier. (I will add a similar follow on patch for
> nfs after this). The first new test group "cifs-quick"
> includes a set of tests that execute fairly quickly (under a few minutes
> each), and second "cifs" which includes the tests which can take
> much longer or are not as appropriate for quick verification of
> a cifs build.

I'm not sure this is the best way to use test grouping. Yes, it
serves your immediate purpose, but I'd prefer to find a way of
grouping tests that is less protocol specific and hence easier to
maintain.

A couple of things stand out about your classifications that will
catch people unaware:

	- quick is a subset of auto, so when you run the auto group,
	  you also get the quick tests run. "cifs" and "cifs-quick"
	  are exclusive groups, so when you run the "cifs" group, it
	  doesn't run all the regression tests it could.

	- new tests are not going to add cifs or nfs
	  classifications, so they are not going to be run if you
	  are using cifs/cifs-quick rather than auto/quick as the
	  test groups that you run.

	- "quick" means runs in under 15-20s on typical test
	  systems. auto is generally limited to tests that run in
	  under 5 minutes. You're definition of "cifs-quick" is
	  roughly the same as "auto", which means nobody is going to
	  be able to classify them correctly without a CIFS setup.

	- similarly, there is no consistency between quick and
	  cifs-quick, or auto and cifs. Some are auto + cifs-quick,
	  others are quick + cifs. Often tests that are not marked
	  as quick will run really fast on SSD/RAID w/ bbwc, but are
	  horrifically slow on a spinning SATA drive. Hence we don't
	  classify them as quick, and I can't see how adding CIFS on
	  a slow SATA drive is going to make them any faster....

What I think would be a better solution is to define groups for
exclusion of certain tests, rather than trying define a compelte new
set of tests specific to a certain filesystem.

That is, define a "remotefs-slow" group, so that you can run:

./check -g quick -x remotefs-slow

ad it runs all the tests in the quick group except for those that
are known to be slow on remote filesystems. This way you'll
automatically run new tests that are added to the quick/auto groups,
rather than periodically having to make passes to add all the new
tests into your cifs/nfs groups. I suspect that the "slow" tests is
a much smaller set than cifs/nfs sets currently are...

This will preserve the expected behaviour across different types of
physical storage, but also encodes the fact that a remotefs layer
makes this test much slower. This might even extend to, say, iscsi
as well as there are a couple of tests (generic/127) that run in
~50s on local storage but take >1200s on my iscsi based test VMs
because of the latency all the sync writes introduce across the
network. i.e:

./check -g auto -x iscsi-slow

would knock 15 minutes of runtime from a test cycle on a couple of
my test VMs...

Such classifications would also be useful documentation when someone
asks "how long should we expect this test to run". Just looking at
the group files will tell use roughly what we should expect.....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: [PATCH] Create test groups for cifs in xfstests

[Aurélien Aptel]

[-- Attachment #1: Type: text/plain, Size: 598 bytes --]

On Thu, 14 Apr 2016 22:53:15 -0500 Steve French <smfrench-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
wrote:
> Attached is a trivial patch to create two xfstest groups for cifs

This is great! I remember trying it unsucessfully on cifs last year
where I had test that would just run forever and never return. I
will try again soon with your patch.

-- 
Aurélien Aptel / SUSE Labs Samba Team
GPG: 1839 CB5F 9F5B FB9B AA97  8C99 03C8 A49B 521B D5D3
SUSE Linux GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG
Nürnberg)

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]