[PATCH 1/3] xfstests-bld: use a separate default config file for each test runner

[PATCH 1/3] xfstests-bld: use a separate default config file for each test runner

[Eric Biggers]

From: Eric Biggers <ebiggers@google.com>

Instead of mixing together the default configuration for kvm-xfstests,
gce-xfstests, and android-xfstests in a single file "config", split it
into separate files for each test runner, along with a config.common
(named "xfstests-common" if overridden in ~/.config/).  This will allow
setting defaults for variables such as ROOT_FS, NR_CPU, and MEM that may
be meaningful for more than one test runner but should have different
default values.  It also more closely matches how the configs in
~/.config/ work.

For now keep sourcing kvm-xfstests/config.custom if present, but suggest
that the user move their settings into ~/.config/ where they aren't all
mixed together for all test runners.  Also keep supporting per-account
configs for gce-xfstests, though really one could instead just make
their ~/.config/gce-xfstests enable different settings based on their
active account (it is a shell script after all).

Signed-off-by: Eric Biggers <ebiggers@google.com>
---
 Documentation/kvm-quickstart.md     |  8 ++++----
 Documentation/kvm-xfstests.md       |  9 ++++-----
 config                              |  4 +++-
 kvm-xfstests/config.android         |  6 ++++++
 kvm-xfstests/config.common          |  7 +++++++
 kvm-xfstests/config.gce             | 15 +++++++++++++++
 kvm-xfstests/{config => config.kvm} | 35 ++++-------------------------------
 kvm-xfstests/util/get-config        | 22 +++++++++++++++++-----
 8 files changed, 60 insertions(+), 46 deletions(-)
 create mode 100644 kvm-xfstests/config.android
 create mode 100644 kvm-xfstests/config.common
 create mode 100644 kvm-xfstests/config.gce
 rename kvm-xfstests/{config => config.kvm} (54%)

diff --git a/Documentation/kvm-quickstart.md b/Documentation/kvm-quickstart.md
index c18a39e..8dcdc90 100644
--- a/Documentation/kvm-quickstart.md
+++ b/Documentation/kvm-quickstart.md
@@ -22,10 +22,10 @@
     kernel-configs directory; pick one whose version number is close
     to the kernel version you wish to build.
 
-4.  In the fstests/kvm-xfstests directory, take a look at the "config"
-    file and either edit that file in place, or (this is preferred)
-    put override values in ~/.config/kvm-xfstests or config.custom.
-    The most common values you will likely need to override is the
+4.  In the fstests/kvm-xfstests/ directory, take a look at the
+    "config.kvm" file and either edit that file in place, or (this is
+    preferred) put override values in ~/.config/kvm-xfstests.  The
+    most common values you will likely need to override are the
     location of the compiled kernel and the preferred timezone if you
     wish the log files to display times in your local timezone.
 
diff --git a/Documentation/kvm-xfstests.md b/Documentation/kvm-xfstests.md
index c480e4a..72f3aae 100644
--- a/Documentation/kvm-xfstests.md
+++ b/Documentation/kvm-xfstests.md
@@ -39,11 +39,10 @@ Please see the relevant documentation files for more details.
 ## Setup and configuration
 
 The configuration file for kvm-xfstests is found in the kvm-xfstests
-directory and is named config.  You can edit this file directly, but
-the better thing to do is to place override values in
-~/.config/kvm-xfstests or in kvm-xfstests/config.custom.  Please look
-at the kvm-xfstests/config file to see the shell variables you can
-set.
+directory and is named config.kvm.  You can edit this file directly,
+but the better thing to do is to place override values in
+~/.config/kvm-xfstests.  Please look at the kvm-xfstests/config.kvm
+file to see the shell variables you can set.
 
 Perhaps the most important configuration variable to set is KERNEL.
 This should point at the default location for the kernel that qemu
diff --git a/config b/config
index af0f21e..f28bf9f 100644
--- a/config
+++ b/config
@@ -1,4 +1,6 @@
-# Configuration file for xfstests-bld
+#
+# Configuration file for building xfstests and root filesystem images
+#
 
 XFSTESTS_GIT=git://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git
 XFSPROGS_GIT=git://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git
diff --git a/kvm-xfstests/config.android b/kvm-xfstests/config.android
new file mode 100644
index 0000000..4b668f8
--- /dev/null
+++ b/kvm-xfstests/config.android
@@ -0,0 +1,6 @@
+#
+# Default configuration for android-xfstests
+#
+# Variables set here may be overridden in ~/.config/android-xfstests
+#
+
diff --git a/kvm-xfstests/config.common b/kvm-xfstests/config.common
new file mode 100644
index 0000000..12cae0a
--- /dev/null
+++ b/kvm-xfstests/config.common
@@ -0,0 +1,7 @@
+#
+# Default configuration for all test runners (kvm-xfstests, gce-xfstests, etc.)
+#
+# Variables set here may be overridden in ~/.config/xfstests-common
+#
+
+PRIMARY_FSTYPE="ext4"
diff --git a/kvm-xfstests/config.gce b/kvm-xfstests/config.gce
new file mode 100644
index 0000000..96b64f8
--- /dev/null
+++ b/kvm-xfstests/config.gce
@@ -0,0 +1,15 @@
+#
+# Default configuration for gce-xfstests
+#
+# Variables set here may be overridden in ~/.config/gce-xfstests
+#
+
+#
+# GS_BUCKET=gs_bucket.thunk.org
+# GCE_PROJECT=tytso-xfstests-project
+# GCE_ZONE=us-central1-c
+# GCE_KERNEL=/u1/ext4-64/arch/x86/boot/bzImage
+
+# List of firewall rules to create.  By default the gce-xfstests web interface
+# is made available to everyone over the public Internet.
+GCE_FIREWALL_RULES=("allow-http --allow tcp:80 --target-tags http-server")
diff --git a/kvm-xfstests/config b/kvm-xfstests/config.kvm
similarity index 54%
rename from kvm-xfstests/config
rename to kvm-xfstests/config.kvm
index 994dcd3..43df96a 100644
--- a/kvm-xfstests/config
+++ b/kvm-xfstests/config.kvm
@@ -1,27 +1,12 @@
 #
-# Customize these, or put new values in:
+# Default configuration for kvm-xfstests
 #
-#		~/.config/kvm-xfstests
-#	or
-#		~/.config/gce-xfstests
-#	or
-#		~/.config/android-xfstests
-#
-# ... to affect specific commands only;
-#
-# or in config.custom to affect all commands.
+# Variables set here may be overridden in ~/.config/kvm-xfstests
 #
 
-###############################################################################
-# Common configuration
-#
-PRIMARY_FSTYPE="ext4" # all commands
-NR_CPU=2 # kvm-xfstests and gce-xfstests
-MEM=2048 # kvm-xfstests and gce-xfstests
+NR_CPU=2
+MEM=2048
 
-###############################################################################
-# KVM configuration
-#
 #QEMU=/usr/local/bin/qemu-system-x86_64
 QEMU=/usr/bin/kvm
 KERNEL=$HOME/linux
@@ -55,15 +40,3 @@ MONITOR="-monitor telnet:localhost:7498,server,nowait"
 # interception entirely.
 CONSOLE=" -serial mon:stdio"
 #CONSOLE=" -chardev stdio,id=console,signal=off -serial chardev:console"
-
-###############################################################################
-# GCE configuration
-#
-# GS_BUCKET=gs_bucket.thunk.org
-# GCE_PROJECT=tytso-xfstests-project
-# GCE_ZONE=us-central1-c
-# GCE_KERNEL=/u1/ext4-64/arch/x86/boot/bzImage
-
-# List of firewall rules to create.  By default the gce-xfstests web interface
-# is made available to everyone over the public Internet.
-GCE_FIREWALL_RULES=("allow-http --allow tcp:80 --target-tags http-server")
diff --git a/kvm-xfstests/util/get-config b/kvm-xfstests/util/get-config
index b1ffc8f..f37d8c1 100644
--- a/kvm-xfstests/util/get-config
+++ b/kvm-xfstests/util/get-config
@@ -12,12 +12,16 @@
 #
 # . "$DIR/util/get-config"
 
-. $DIR/config
+# Source the default configs
+. $DIR/config.common
+. $DIR/config.${XFSTESTS_FLAVOR}
 
-# Source the user-provided config if present (e.g. ~/.config/kvm-xfstests)
+# Source custom configs in ~/.config/ if present
+[ -f "$HOME/.config/xfstests-common" ] && . "$HOME/.config/xfstests-common"
 [ -f "$HOME/.config/${XFSTESTS_FLAVOR}-xfstests" ] && \
 	. "$HOME/.config/${XFSTESTS_FLAVOR}-xfstests"
 
+# For gce-xfstests, source the config for the active account if present
 if test "$XFSTESTS_FLAVOR" = "gce" -a -z "$GCE_ACCOUNT" -a \
    -n "$(ls $DIR/config-* 2> /dev/null)"
 then
@@ -32,11 +36,19 @@ then
 fi
 
 if test -n "$GCE_ACCOUNT" -a -f "$DIR/config-$GCE_ACCOUNT" ; then
-   . "$DIR/config-$GCE_ACCOUNT"
+    . "$DIR/config-$GCE_ACCOUNT"
 elif test -f $DIR/config.custom ; then
-   . $DIR/config.custom
+# Else, source kvm-xfstests/config.custom if present (deprecated)
+    echo -e 1>&2 "Warning: use of kvm-xfstests/config.custom is deprecated." \
+	    "Move your settings\n         into ~/.config/kvm-xfstests," \
+	    "~/.config/gce-xfstests,\n         ~/.config/android-xfstests," \
+	    "and/or ~/.config/xfstests-common"
+    . $DIR/config.custom
 fi
 
+# Source $KVM_CONFIG if present (deprecated)
 if test -n "$KVM_CONFIG" -a -f "$KVM_CONFIG"; then
-   . $KVM_CONFIG
+    echo -e 1>&2 "Warning: use of \$KVM_CONFIG is deprecated." \
+	    "Move your settings into\n         ~/.config/kvm-xfstests"
+    . $KVM_CONFIG
 fi
-- 
2.12.2.762.g0e3151a226-goog

[PATCH 2/3] gce-xfstests: move NR_CPU, MEM, and ROOT_FS into config.gce

[Eric Biggers]

From: Eric Biggers <ebiggers@google.com>

This allows them to be overridden by ~/.config/gce-xfstests.

Signed-off-by: Eric Biggers <ebiggers@google.com>
---
 Documentation/gce-xfstests.md | 12 +++++++-----
 kvm-xfstests/config.gce       | 12 ++++++++++++
 kvm-xfstests/gce-xfstests     |  9 ---------
 3 files changed, 19 insertions(+), 14 deletions(-)

diff --git a/Documentation/gce-xfstests.md b/Documentation/gce-xfstests.md
index 6d053d1..5b1c9a7 100644
--- a/Documentation/gce-xfstests.md
+++ b/Documentation/gce-xfstests.md
@@ -345,8 +345,10 @@ variable is defined:
         GCE_IMAGE_PROJECT="$GCE_PROJECT"
 
 Normally, the most recently created image in the xfstests image family
-will be used by default.  You can however override this by using the
--I option to specify a specific image file.  (For example:
-"gce-xfstests -I xfstests-201608130052 smoke".)  You can also use the
---image-project command line option to override the GCE_IMAGE_PROJECT
-setting in your configuration file.
+will be used by default.  You can however override this and use a
+specific image by setting `ROOT_FS` in your `~/.config/gce-xfstests`
+configuration file, or by using the -I option to gce-xfstests.  (For
+example: `ROOT_FS=xfstests-201608130052`, or "gce-xfstests -I
+xfstests-201608130052 smoke".)  You can also use the --image-project
+command line option to override the GCE_IMAGE_PROJECT setting in your
+configuration file.
diff --git a/kvm-xfstests/config.gce b/kvm-xfstests/config.gce
index 96b64f8..3358ae3 100644
--- a/kvm-xfstests/config.gce
+++ b/kvm-xfstests/config.gce
@@ -13,3 +13,15 @@
 # List of firewall rules to create.  By default the gce-xfstests web interface
 # is made available to everyone over the public Internet.
 GCE_FIREWALL_RULES=("allow-http --allow tcp:80 --target-tags http-server")
+
+# Use different defaults for GCE because the background ruby monitoring
+# program uses extra memory --- the n1-standard-2 machine type has 7.5 GB,
+# so use that.
+NR_CPU=2
+MEM=7680
+
+# Name of the GCE image from which the test instances will be created.
+# This can also be set by the -I option.
+# If "xfstests" (the default), this is actually treated as an image family, and
+# the latest image from the xfstests family will be used.
+ROOT_FS="xfstests"
diff --git a/kvm-xfstests/gce-xfstests b/kvm-xfstests/gce-xfstests
index 7b678bc..240467e 100755
--- a/kvm-xfstests/gce-xfstests
+++ b/kvm-xfstests/gce-xfstests
@@ -9,15 +9,6 @@ fi
 
 . "$DIR/util/get-config"
 
-# Use different defaults for GCE because the background ruby monitoring
-# program uses extra memory --- the n1-standard-2 machine type has 7.5 GB,
-# so use that.
-NR_CPU=2
-MEM=7680
-
-# We use ROOT_FS for the GCE image; this can be set by the -I option
-ROOT_FS="xfstests"
-
 if test -r "$DIR/test-appliance/config.custom" ; then
    . "$DIR/test-appliance/config.custom"
 fi
-- 
2.12.2.762.g0e3151a226-goog

[PATCH 3/3] android-xfstests: move ROOT_FS and ROOT_FS_URL into config.android

[Eric Biggers]

From: Eric Biggers <ebiggers@google.com>

This allows them to be overridden by ~/.config/android-xfstests.  Also
start treating an empty ROOT_FS_URL as meaning that no URL is provided.

Signed-off-by: Eric Biggers <ebiggers@google.com>
---
 Documentation/android-xfstests.md | 3 ++-
 kvm-xfstests/android-xfstests     | 9 +++------
 kvm-xfstests/config.android       | 6 ++++++
 3 files changed, 11 insertions(+), 7 deletions(-)

diff --git a/Documentation/android-xfstests.md b/Documentation/android-xfstests.md
index b309c21..5b37f9e 100644
--- a/Documentation/android-xfstests.md
+++ b/Documentation/android-xfstests.md
@@ -39,7 +39,8 @@ devices.  If you encounter a problem, please submit a fix!
   [building-xfstests](building-xfstests.md).  Then, either put the
   chroot tarball in the default location of
   kvm-xfstests/test-appliance/armhf_root_fs.tar.gz, or specify it with
-  the -I option to android-xfstests.
+  ROOT_FS in your ~/.config/android-xfstests or the -I option to
+  android-xfstests.
 
 ## Procedure
 
diff --git a/kvm-xfstests/android-xfstests b/kvm-xfstests/android-xfstests
index 3489bc2..e33955e 100755
--- a/kvm-xfstests/android-xfstests
+++ b/kvm-xfstests/android-xfstests
@@ -10,12 +10,6 @@ fi
 
 . "$DIR/util/get-config"
 
-# Path to chroot tarball; can be overridden with -I
-ROOT_FS="$DIR/test-appliance/armhf_root_fs.tar.gz"
-
-# Where to download the tarball from (at user's request) if we don't have it
-ROOT_FS_URL="https://www.kernel.org/pub/linux/kernel/people/tytso/kvm-xfstests/armhf_root_fs.tar.gz"
-
 # Chroot directory on device.
 # Note: this will be wiped clean when deploying a new chroot tarball.
 CHROOT_DIR="/data/xfstests-chroot"
@@ -142,6 +136,9 @@ setup_chroot()
     if ! [ -f "$ROOT_FS" ]; then
 	echo "The xfstests chroot tarball does not exist:"
 	echo "        $ROOT_FS"
+	if [ -z "$ROOT_FS_URL" ]; then
+	    exit 1
+	fi
 	ask_yesno "Would you like to download the latest public tarball to that location?"
 	wget -O "$ROOT_FS" "$ROOT_FS_URL"
 	echo "Finished downloading chroot tarball."
diff --git a/kvm-xfstests/config.android b/kvm-xfstests/config.android
index 4b668f8..ed7098c 100644
--- a/kvm-xfstests/config.android
+++ b/kvm-xfstests/config.android
@@ -4,3 +4,9 @@
 # Variables set here may be overridden in ~/.config/android-xfstests
 #
 
+# Path to chroot tarball; this can also be set by the -I option
+ROOT_FS="$DIR/test-appliance/armhf_root_fs.tar.gz"
+
+# Where to download the tarball from (at user's request) if we don't have it.
+# If you want to disable this functionality, set this to an empty string.
+ROOT_FS_URL="https://www.kernel.org/pub/linux/kernel/people/tytso/kvm-xfstests/armhf_root_fs.tar.gz"
-- 
2.12.2.762.g0e3151a226-goog

Re: [PATCH 3/3] android-xfstests: move ROOT_FS and ROOT_FS_URL into config.android

[Theodore Ts'o]

On Mon, Apr 17, 2017 at 02:33:27PM -0700, Eric Biggers wrote:
> From: Eric Biggers <ebiggers@google.com>
> 
> This allows them to be overridden by ~/.config/android-xfstests.  Also
> start treating an empty ROOT_FS_URL as meaning that no URL is provided.
> 
> Signed-off-by: Eric Biggers <ebiggers@google.com>

Thanks for the cleanup!  I've applied all three patches to my tree.

       	       		      	      	  - Ted