[PATCH] xfstests-bld: document how to add debian packages to GCE image

[PATCH] xfstests-bld: document how to add debian packages to GCE image

[Eric Biggers]

From: Eric Biggers <ebiggers@google.com>

Signed-off-by: Eric Biggers <ebiggers@google.com>
---
 Documentation/gce-xfstests.md | 14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)

diff --git a/Documentation/gce-xfstests.md b/Documentation/gce-xfstests.md
index becc23d..1b7fa3e 100644
--- a/Documentation/gce-xfstests.md
+++ b/Documentation/gce-xfstests.md
@@ -306,12 +306,14 @@ to be unpacked into a directory in /tmp instead.
 
 By default gce-xfstests uses the prebuilt image which is made
 available via the xfstests-cloud project.  However, if you want to
-build your own image, you must first build the xfstests tarball as
-described in the [instructions for building
-xfstests](building-xfstests.md).  Then run the command "gce-xfstests
-create-image".  This will create a new GCE image with a name such as
-"xfstests-201608132226" where 201608132226 indicates when the image
-was created (in this case, August 13, 2016 at 22:26).
+build your own image, you must first build a 64-bit (amd64) xfstests
+tarball as described in the [instructions for building
+xfstests](building-xfstests.md).  Then copy any additional Debian
+packages, for the amd64 architecture, you want to include in the GCE
+image into the root directory of GS_BUCKET.  Then run the command
+"gce-xfstests create-image".  This will create a new GCE image with a
+name such as "xfstests-201608132226" where 201608132226 indicates when
+the image was created (in this case, August 13, 2016 at 22:26).
 
 This image will be created as part of an image family called xfstests.
 By default, when you start a test using gce-xfstests, the most
-- 
2.8.0.rc3.226.g39d4020

Re: [PATCH] xfstests-bld: document how to add debian packages to GCE image

[Theodore Ts'o]

On Wed, Dec 21, 2016 at 10:06:28AM -0800, Eric Biggers wrote:
> +build your own image, you must first build a 64-bit (amd64) xfstests
> +tarball as described in the [instructions for building
> +xfstests](building-xfstests.md).  Then copy any additional Debian
> +packages, for the amd64 architecture, you want to include in the GCE
> +image into the root directory of GS_BUCKET.

This is phrased a bit awkwardly.  The official way is to put the
debian packages in kvm-xfstests/test-appliances/debs.  This works for
both building gce-xfstests and kvm-xfstests images.

And in general, you would need to use whatever debian packages for
whatever architecture is compatible for the image or chroot tarball
you are building.  For gce-xfstests, it is always amd64, since GCE
currently only supports 64-bit x86.  (But for kvm-xfstests or chroot
tarballs, it might be i386 or armhf.)

So how about,

   If you want to include any additional Debian packages, or you want
   to override the default Jessie packages with newer versions from
   jessie-backports, place them in the directory
   kvm-xfstests/test-appliances/debs.  For gce-xfstests, the packages
   must be for the amd64 architecture.

We might want break up the paragraph into multiple smaller paragraphs
just to make it easier to read.

				- Ted

Re: [PATCH] xfstests-bld: document how to add debian packages to GCE image

[Eric Biggers]

On Thu, Dec 22, 2016 at 12:04:33PM -0500, Theodore Ts'o wrote:
> On Wed, Dec 21, 2016 at 10:06:28AM -0800, Eric Biggers wrote:
> > +build your own image, you must first build a 64-bit (amd64) xfstests
> > +tarball as described in the [instructions for building
> > +xfstests](building-xfstests.md).  Then copy any additional Debian
> > +packages, for the amd64 architecture, you want to include in the GCE
> > +image into the root directory of GS_BUCKET.
> 
> This is phrased a bit awkwardly.  The official way is to put the
> debian packages in kvm-xfstests/test-appliances/debs.  This works for
> both building gce-xfstests and kvm-xfstests images.
> 

That works with kvm-xfstests (gen-image), but it didn't work when I tried it
with gce-xfstests (gce-create-image) because nothing was actually copying the
debian packages into the GS bucket for the test appliance to install.  How about
we update gce-create-image to copy the debian packages from
test-appliances/debs/ into the GS bucket (maybe gsutil rsync -d to a directory
'debs'), so that it works in the way you're suggesting?

Also, Documentation/building-rootfs.md already describes adding packages for
kvm-xfstests; it's just gce-xfstests that was missing the documentation.  So
perhaps the gce-xfstests documentation should refer to the kvm-xfstests
documentation for full details on adding packages, if the procedure is going to
be the same.

Eric

Re: [PATCH] xfstests-bld: document how to add debian packages to GCE image

[Theodore Ts'o]

On Wed, Dec 28, 2016 at 03:21:32PM -0600, Eric Biggers wrote:
> That works with kvm-xfstests (gen-image), but it didn't work when I tried it
> with gce-xfstests (gce-create-image) because nothing was actually copying the
> debian packages into the GS bucket for the test appliance to install.  How about
> we update gce-create-image to copy the debian packages from
> test-appliances/debs/ into the GS bucket (maybe gsutil rsync -d to a directory
> 'debs'), so that it works in the way you're suggesting?

Oh, right; I had forgotten.  I was originally planning on copying the
debian packages into the GS bucket, but I gave up on that because it
was too slow.  I didn't try using gsutil rsync, but the multiple round
trips is very likely going to make things slower than we'd like.

Actually, what would probably work better for gce-create-image is
simply add jessie-backports to the apt sources list, and then use
"apt-get install -t jessie-backports <package> ..." to get the newer
version of e2fsprogs, f2fs-tools, etc.  If we also use
snapshots.debian.org URL instead of the standard debian download site
(as we do with kvm-xfstests), it also has the advantage making the
build of the image be perfectly reproducible.

Converting kvm-xfstests to using jessie-backports as well is going to
be a bit tricky, since debootstrap doesn't understand how to install
from backports; so this would probably require hacking up something
using python-debian and possibly python-apt.  So perhaps we'll just
change gce-create-image to use jessie-backports first, and we can
convert kvm-xfstests later.

						- Ted

P.S.  BTW, I checked in with the f2fs-tools maintainer and he has no
objections to porting the latest version of f2fs-tools from Debian
stretch to Debian jessie, so that should be happening in the next week
or so.

Re: [PATCH] xfstests-bld: document how to add debian packages to GCE image

[Theodore Ts'o]

It looks like "gsutil rsync -cd" is actually pretty efficient, so I'll
be adopting your suggestion of using that so we can use the same
mechanism for the kvm-xfstests create-image script.

I'm also going to be teaching gce-xfstests-bld.sh to use the
jessie-backports version for e2fsprogs and btrfs-progs, and f2fs-tools
is currently waiting to be installed in the debian backports queue.
Once that's done, we won't actually need any extra packages for the
gce-xfstests image which I am distributing, since we can just use the
jessie-backports version.

We still need a similar mechanism for the other build mechanisms, but
we can do that later.

					- Ted