* [Qemu-devel] [PULL v4 10/38] scripts: Add archive-source.sh
2017-09-15 9:02 [Qemu-devel] [PULL v4 00/38] Test and build patches Fam Zheng
@ 2017-09-15 9:02 ` Fam Zheng
2017-09-15 9:02 ` [Qemu-devel] [PULL v4 11/38] tests: Add a test key pair Fam Zheng
` (10 subsequent siblings)
11 siblings, 0 replies; 22+ messages in thread
From: Fam Zheng @ 2017-09-15 9:02 UTC (permalink / raw)
To: qemu-devel; +Cc: Peter Maydell
Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <20170913030119.3957-4-famz@redhat.com>
---
scripts/archive-source.sh | 33 +++++++++++++++++++++++++++++++++
1 file changed, 33 insertions(+)
create mode 100755 scripts/archive-source.sh
diff --git a/scripts/archive-source.sh b/scripts/archive-source.sh
new file mode 100755
index 0000000000..78201e4218
--- /dev/null
+++ b/scripts/archive-source.sh
@@ -0,0 +1,33 @@
+#!/bin/sh
+#
+# Author: Fam Zheng <famz@redhat.com>
+#
+# Archive source tree, including submodules. This is created for test code to
+# export the source files, in order to be built in a different environment,
+# such as in a docker instance or VM.
+#
+# This code is licensed under the GPL version 2 or later. See
+# the COPYING file in the top-level directory.
+
+set -e
+
+if test $# -lt 1; then
+ echo "Usage: $0 <output tarball>"
+ exit 1
+fi
+
+submodules=$(git submodule foreach --recursive --quiet 'echo $name')
+
+if test -n "$submodules"; then
+ {
+ git ls-files
+ for sm in $submodules; do
+ (cd $sm; git ls-files) | sed "s:^:$sm/:"
+ done
+ } | grep -x -v $(for sm in $submodules; do echo "-e $sm"; done) > "$1".list
+else
+ git ls-files > "$1".list
+fi
+
+tar -cf "$1" -T "$1".list
+rm "$1".list
--
2.13.5
^ permalink raw reply related [flat|nested] 22+ messages in thread* [Qemu-devel] [PULL v4 11/38] tests: Add a test key pair
2017-09-15 9:02 [Qemu-devel] [PULL v4 00/38] Test and build patches Fam Zheng
2017-09-15 9:02 ` [Qemu-devel] [PULL v4 10/38] scripts: Add archive-source.sh Fam Zheng
@ 2017-09-15 9:02 ` Fam Zheng
2017-09-15 9:02 ` [Qemu-devel] [PULL v4 12/38] tests: Add vm test lib Fam Zheng
` (9 subsequent siblings)
11 siblings, 0 replies; 22+ messages in thread
From: Fam Zheng @ 2017-09-15 9:02 UTC (permalink / raw)
To: qemu-devel; +Cc: Peter Maydell
This will be used by setup test user ssh.
Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <20170913030119.3957-5-famz@redhat.com>
---
tests/keys/id_rsa | 27 +++++++++++++++++++++++++++
tests/keys/id_rsa.pub | 1 +
2 files changed, 28 insertions(+)
create mode 100644 tests/keys/id_rsa
create mode 100644 tests/keys/id_rsa.pub
diff --git a/tests/keys/id_rsa b/tests/keys/id_rsa
new file mode 100644
index 0000000000..2933eac3db
--- /dev/null
+++ b/tests/keys/id_rsa
@@ -0,0 +1,27 @@
+-----BEGIN RSA PRIVATE KEY-----
+MIIEowIBAAKCAQEAopAuOlmLV6LVHdFBj8/eeOwI9CqguIJPp7eAQSZvOiB4Ag/R
+coEhl/RBbrV5Yc/SmSD4PTpJO/iM10RwliNjDb4a3I8q3sykRJu9c9PI/YsH8WN9
++NH2NjKPtJIcKTu287IM5JYxyB6nDoOzILbTyJ1TDR/xH6qYEfBAyiblggdjcvhA
+RTf93QIn39F/xLypXvT1K2O9BJEsnJ8lEUvB2UXhKo/JTfSeZF8wPBeowaP9EONk
+7b+nuJOWHGg68Ji6wVi62tjwl2Szch6lxIhZBpnV7QNRKMfYHP6eIyF4pusazzZq
+Telsq6xI2ghecWLzb/MF5A+rklsGx2FNuJSAJwIDAQABAoIBAHHi4o/8VZNivz0x
+cWXn8erzKV6tUoWQvW85Lj/2RiwJvSlsnYZDkx5af1CpEE2HA/pFT8PNRqsd+MWC
+7AEy710cVsM4BYerBFYQaYxwzblaoojo88LSjVPw3h5Z0iLM8+IMVd36nwuc9dpE
+R8TecMZ1+U4Tl6BgqkK+9xToZRdPKdjS8L5MoFhGN+xY0vRbbJbGaV9Q0IHxLBkB
+rEBV7T1mUynneCHRUQlJQEwJmKpT8MH3IjsUXlG5YvnuuvcQJSNTaW2iDLxuOKp8
+cxW8+qL88zpb1D5dppoIu6rlrugN0azSq70ruFJQPc/A8GQrDKoGgRQiagxNY3u+
+vHZzXlECgYEA0dKO3gfkSxsDBb94sQwskMScqLhcKhztEa8kPxTx6Yqh+x8/scx3
+XhJyOt669P8U1v8a/2Al+s81oZzzfQSzO1Q7gEwSrgBcRMSIoRBUw9uYcy02ngb/
+j/ng3DGivfJztjjiSJwb46FHkJ2JR8mF2UisC6UMXk3NgFY/3vWQx78CgYEAxlcG
+T3hfSWSmTgKRczMJuHQOX9ULfTBIqwP5VqkkkiavzigGRirzb5lgnmuTSPTpF0LB
+XVPjR2M4q+7gzP0Dca3pocrvLEoxjwIKnCbYKnyyvnUoE9qHv4Kr+vDbgWpa2LXG
+JbLmE7tgTCIp20jOPPT4xuDvlbzQZBJ5qCQSoZkCgYEAgrotSSihlCnAOFSTXbu4
+CHp3IKe8xIBBNENq0eK61kcJpOxTQvOha3sSsJsU4JAM6+cFaxb8kseHIqonCj1j
+bhOM/uJmwQJ4el/4wGDsbxriYOBKpyq1D38gGhDS1IW6kk3erl6VAb36WJ/OaGum
+eTpN9vNeQWM4Jj2WjdNx4QECgYAwTdd6mU1TmZCrJRL5ZG+0nYc2rbMrnQvFoqUi
+BvWiJovggHzur90zy73tNzPaq9Ls2FQxf5G1vCN8NCRJqEEjeYCR59OSDMu/EXc2
+CnvQ9SevHOdS1oEDEjcCWZCMFzPi3XpRih1gptzQDe31uuiHjf3cqcGPzTlPdfRt
+D8P92QKBgC4UaBvIRwREVJsdZzpIzm224Bpe8LOmA7DeTnjlT0b3lkGiBJ36/Q0p
+VhYh/6cjX4/iuIs7gJbGon7B+YPB8scmOi3fj0+nkJAONue1mMfBNkba6qQTc6Y2
+5mEKw2/O7/JpND7ucU3OK9plcw/qnrWDgHxl0Iz95+OzUIIagxne
+-----END RSA PRIVATE KEY-----
diff --git a/tests/keys/id_rsa.pub b/tests/keys/id_rsa.pub
new file mode 100644
index 0000000000..d9888e312f
--- /dev/null
+++ b/tests/keys/id_rsa.pub
@@ -0,0 +1 @@
+ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCikC46WYtXotUd0UGPz9547Aj0KqC4gk+nt4BBJm86IHgCD9FygSGX9EFutXlhz9KZIPg9Okk7+IzXRHCWI2MNvhrcjyrezKREm71z08j9iwfxY3340fY2Mo+0khwpO7bzsgzkljHIHqcOg7MgttPInVMNH/EfqpgR8EDKJuWCB2Ny+EBFN/3dAiff0X/EvKle9PUrY70EkSycnyURS8HZReEqj8lN9J5kXzA8F6jBo/0Q42Ttv6e4k5YcaDrwmLrBWLra2PCXZLNyHqXEiFkGmdXtA1Eox9gc/p4jIXim6xrPNmpN6WyrrEjaCF5xYvNv8wXkD6uSWwbHYU24lIAn qemu-test
--
2.13.5
^ permalink raw reply related [flat|nested] 22+ messages in thread* [Qemu-devel] [PULL v4 12/38] tests: Add vm test lib
2017-09-15 9:02 [Qemu-devel] [PULL v4 00/38] Test and build patches Fam Zheng
2017-09-15 9:02 ` [Qemu-devel] [PULL v4 10/38] scripts: Add archive-source.sh Fam Zheng
2017-09-15 9:02 ` [Qemu-devel] [PULL v4 11/38] tests: Add a test key pair Fam Zheng
@ 2017-09-15 9:02 ` Fam Zheng
2017-09-15 9:02 ` [Qemu-devel] [PULL v4 13/38] tests: Add ubuntu.i386 image Fam Zheng
` (8 subsequent siblings)
11 siblings, 0 replies; 22+ messages in thread
From: Fam Zheng @ 2017-09-15 9:02 UTC (permalink / raw)
To: qemu-devel; +Cc: Peter Maydell
This is the common code to implement a "VM test" to
1) Download and initialize a pre-defined VM that has necessary
dependencies to build QEMU and SSH access.
2) Archive $SRC_PATH to a .tar file.
3) Boot the VM, and pass the source tar file to the guest.
4) SSH into the VM, untar the source tarball, build from the source.
Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <20170913030119.3957-6-famz@redhat.com>
---
tests/vm/basevm.py | 256 +++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 256 insertions(+)
create mode 100755 tests/vm/basevm.py
diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
new file mode 100755
index 0000000000..ea970dad19
--- /dev/null
+++ b/tests/vm/basevm.py
@@ -0,0 +1,256 @@
+#!/usr/bin/env python
+#
+# VM testing base class
+#
+# Copyright 2017 Red Hat Inc.
+#
+# Authors:
+# Fam Zheng <famz@redhat.com>
+#
+# This code is licensed under the GPL version 2 or later. See
+# the COPYING file in the top-level directory.
+#
+
+import os
+import sys
+import logging
+import time
+import datetime
+sys.path.append(os.path.join(os.path.dirname(__file__), "..", "..", "scripts"))
+from qemu import QEMUMachine
+import subprocess
+import hashlib
+import optparse
+import atexit
+import tempfile
+import shutil
+import multiprocessing
+import traceback
+
+SSH_KEY = open(os.path.join(os.path.dirname(__file__),
+ "..", "keys", "id_rsa")).read()
+SSH_PUB_KEY = open(os.path.join(os.path.dirname(__file__),
+ "..", "keys", "id_rsa.pub")).read()
+
+class BaseVM(object):
+ GUEST_USER = "qemu"
+ GUEST_PASS = "qemupass"
+ ROOT_PASS = "qemupass"
+
+ # The script to run in the guest that builds QEMU
+ BUILD_SCRIPT = ""
+ # The guest name, to be overridden by subclasses
+ name = "#base"
+ def __init__(self, debug=False, vcpus=None):
+ self._guest = None
+ self._tmpdir = tempfile.mkdtemp(prefix="vm-test-", suffix=".tmp", dir=".")
+ atexit.register(shutil.rmtree, self._tmpdir)
+
+ self._ssh_key_file = os.path.join(self._tmpdir, "id_rsa")
+ open(self._ssh_key_file, "w").write(SSH_KEY)
+ subprocess.check_call(["chmod", "600", self._ssh_key_file])
+
+ self._ssh_pub_key_file = os.path.join(self._tmpdir, "id_rsa.pub")
+ open(self._ssh_pub_key_file, "w").write(SSH_PUB_KEY)
+
+ self.debug = debug
+ self._stderr = sys.stderr
+ self._devnull = open(os.devnull, "w")
+ if self.debug:
+ self._stdout = sys.stdout
+ else:
+ self._stdout = self._devnull
+ self._args = [ \
+ "-nodefaults", "-m", "2G",
+ "-cpu", "host",
+ "-netdev", "user,id=vnet,hostfwd=:0.0.0.0:0-:22",
+ "-device", "virtio-net-pci,netdev=vnet",
+ "-vnc", ":0,to=20",
+ "-serial", "file:%s" % os.path.join(self._tmpdir, "serial.out")]
+ if vcpus:
+ self._args += ["-smp", str(vcpus)]
+ if os.access("/dev/kvm", os.R_OK | os.W_OK):
+ self._args += ["-enable-kvm"]
+ else:
+ logging.info("KVM not available, not using -enable-kvm")
+ self._data_args = []
+
+ def _download_with_cache(self, url, sha256sum=None):
+ def check_sha256sum(fname):
+ if not sha256sum:
+ return True
+ checksum = subprocess.check_output(["sha256sum", fname]).split()[0]
+ return sha256sum == checksum
+
+ cache_dir = os.path.expanduser("~/.cache/qemu-vm/download")
+ if not os.path.exists(cache_dir):
+ os.makedirs(cache_dir)
+ fname = os.path.join(cache_dir, hashlib.sha1(url).hexdigest())
+ if os.path.exists(fname) and check_sha256sum(fname):
+ return fname
+ logging.debug("Downloading %s to %s...", url, fname)
+ subprocess.check_call(["wget", "-c", url, "-O", fname + ".download"],
+ stdout=self._stdout, stderr=self._stderr)
+ os.rename(fname + ".download", fname)
+ return fname
+
+ def _ssh_do(self, user, cmd, check, interactive=False):
+ ssh_cmd = ["ssh", "-q",
+ "-o", "StrictHostKeyChecking=no",
+ "-o", "UserKnownHostsFile=" + os.devnull,
+ "-o", "ConnectTimeout=1",
+ "-p", self.ssh_port, "-i", self._ssh_key_file]
+ if interactive:
+ ssh_cmd += ['-t']
+ assert not isinstance(cmd, str)
+ ssh_cmd += ["%s@127.0.0.1" % user] + list(cmd)
+ logging.debug("ssh_cmd: %s", " ".join(ssh_cmd))
+ r = subprocess.call(ssh_cmd,
+ stdin=sys.stdin if interactive else self._devnull,
+ stdout=sys.stdout if interactive else self._stdout,
+ stderr=sys.stderr if interactive else self._stderr)
+ if check and r != 0:
+ raise Exception("SSH command failed: %s" % cmd)
+ return r
+
+ def ssh(self, *cmd):
+ return self._ssh_do(self.GUEST_USER, cmd, False)
+
+ def ssh_interactive(self, *cmd):
+ return self._ssh_do(self.GUEST_USER, cmd, False, True)
+
+ def ssh_root(self, *cmd):
+ return self._ssh_do("root", cmd, False)
+
+ def ssh_check(self, *cmd):
+ self._ssh_do(self.GUEST_USER, cmd, True)
+
+ def ssh_root_check(self, *cmd):
+ self._ssh_do("root", cmd, True)
+
+ def build_image(self, img):
+ raise NotImplementedError
+
+ def add_source_dir(self, src_dir):
+ name = "data-" + hashlib.sha1(src_dir).hexdigest()[:5]
+ tarfile = os.path.join(self._tmpdir, name + ".tar")
+ logging.debug("Creating archive %s for src_dir dir: %s", tarfile, src_dir)
+ subprocess.check_call(["./scripts/archive-source.sh", tarfile],
+ cwd=src_dir, stdin=self._devnull,
+ stdout=self._stdout, stderr=self._stderr)
+ self._data_args += ["-drive",
+ "file=%s,if=none,id=%s,cache=writeback,format=raw" % \
+ (tarfile, name),
+ "-device",
+ "virtio-blk,drive=%s,serial=%s,bootindex=1" % (name, name)]
+
+ def boot(self, img, extra_args=[]):
+ args = self._args + [
+ "-device", "VGA",
+ "-drive", "file=%s,if=none,id=drive0,cache=writeback" % img,
+ "-device", "virtio-blk,drive=drive0,bootindex=0"]
+ args += self._data_args + extra_args
+ logging.debug("QEMU args: %s", " ".join(args))
+ qemu_bin = os.environ.get("QEMU", "qemu-system-x86_64")
+ guest = QEMUMachine(binary=qemu_bin, args=args)
+ try:
+ guest.launch()
+ except:
+ logging.error("Failed to launch QEMU, command line:")
+ logging.error(" ".join([qemu_bin] + args))
+ logging.error("Log:")
+ logging.error(guest.get_log())
+ logging.error("QEMU version >= 2.10 is required")
+ raise
+ atexit.register(self.shutdown)
+ self._guest = guest
+ usernet_info = guest.qmp("human-monitor-command",
+ command_line="info usernet")
+ self.ssh_port = None
+ for l in usernet_info["return"].splitlines():
+ fields = l.split()
+ if "TCP[HOST_FORWARD]" in fields and "22" in fields:
+ self.ssh_port = l.split()[3]
+ if not self.ssh_port:
+ raise Exception("Cannot find ssh port from 'info usernet':\n%s" % \
+ usernet_info)
+
+ def wait_ssh(self, seconds=120):
+ starttime = datetime.datetime.now()
+ guest_up = False
+ while (datetime.datetime.now() - starttime).total_seconds() < seconds:
+ if self.ssh("exit 0") == 0:
+ guest_up = True
+ break
+ time.sleep(1)
+ if not guest_up:
+ raise Exception("Timeout while waiting for guest ssh")
+
+ def shutdown(self):
+ self._guest.shutdown()
+
+ def wait(self):
+ self._guest.wait()
+
+ def qmp(self, *args, **kwargs):
+ return self._guest.qmp(*args, **kwargs)
+
+def parse_args(vm_name):
+ parser = optparse.OptionParser(description="""
+ VM test utility. Exit codes: 0 = success, 1 = command line error, 2 = environment initialization failed, 3 = test command failed""")
+ parser.add_option("--debug", "-D", action="store_true",
+ help="enable debug output")
+ parser.add_option("--image", "-i", default="%s.img" % vm_name,
+ help="image file name")
+ parser.add_option("--force", "-f", action="store_true",
+ help="force build image even if image exists")
+ parser.add_option("--jobs", type=int, default=multiprocessing.cpu_count() / 2,
+ help="number of virtual CPUs")
+ parser.add_option("--build-image", "-b", action="store_true",
+ help="build image")
+ parser.add_option("--build-qemu",
+ help="build QEMU from source in guest")
+ parser.add_option("--interactive", "-I", action="store_true",
+ help="Interactively run command")
+ parser.disable_interspersed_args()
+ return parser.parse_args()
+
+def main(vmcls):
+ try:
+ args, argv = parse_args(vmcls.name)
+ if not argv and not args.build_qemu and not args.build_image:
+ print "Nothing to do?"
+ return 1
+ if args.debug:
+ logging.getLogger().setLevel(logging.DEBUG)
+ vm = vmcls(debug=args.debug, vcpus=args.jobs)
+ if args.build_image:
+ if os.path.exists(args.image) and not args.force:
+ sys.stderr.writelines(["Image file exists: %s\n" % args.image,
+ "Use --force option to overwrite\n"])
+ return 1
+ return vm.build_image(args.image)
+ if args.build_qemu:
+ vm.add_source_dir(args.build_qemu)
+ cmd = [vm.BUILD_SCRIPT.format(
+ configure_opts = " ".join(argv),
+ jobs=args.jobs)]
+ else:
+ cmd = argv
+ vm.boot(args.image + ",snapshot=on")
+ vm.wait_ssh()
+ except Exception as e:
+ if isinstance(e, SystemExit) and e.code == 0:
+ return 0
+ sys.stderr.write("Failed to prepare guest environment\n")
+ traceback.print_exc()
+ return 2
+
+ if args.interactive:
+ if vm.ssh_interactive(*cmd) == 0:
+ return 0
+ vm.ssh_interactive()
+ return 3
+ else:
+ if vm.ssh(*cmd) != 0:
+ return 3
--
2.13.5
^ permalink raw reply related [flat|nested] 22+ messages in thread* [Qemu-devel] [PULL v4 13/38] tests: Add ubuntu.i386 image
2017-09-15 9:02 [Qemu-devel] [PULL v4 00/38] Test and build patches Fam Zheng
` (2 preceding siblings ...)
2017-09-15 9:02 ` [Qemu-devel] [PULL v4 12/38] tests: Add vm test lib Fam Zheng
@ 2017-09-15 9:02 ` Fam Zheng
2017-09-15 9:02 ` [Qemu-devel] [PULL v4 14/38] tests: Add FreeBSD image Fam Zheng
` (7 subsequent siblings)
11 siblings, 0 replies; 22+ messages in thread
From: Fam Zheng @ 2017-09-15 9:02 UTC (permalink / raw)
To: qemu-devel; +Cc: Peter Maydell
This adds a 32bit guest.
The official LTS cloud image is downloaded and initialized with
cloud-init.
Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <20170913030119.3957-7-famz@redhat.com>
---
tests/vm/ubuntu.i386 | 88 ++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 88 insertions(+)
create mode 100755 tests/vm/ubuntu.i386
diff --git a/tests/vm/ubuntu.i386 b/tests/vm/ubuntu.i386
new file mode 100755
index 0000000000..e70dcb89ce
--- /dev/null
+++ b/tests/vm/ubuntu.i386
@@ -0,0 +1,88 @@
+#!/usr/bin/env python
+#
+# Ubuntu i386 image
+#
+# Copyright 2017 Red Hat Inc.
+#
+# Authors:
+# Fam Zheng <famz@redhat.com>
+#
+# This code is licensed under the GPL version 2 or later. See
+# the COPYING file in the top-level directory.
+#
+
+import os
+import sys
+import subprocess
+import basevm
+import time
+
+class UbuntuX86VM(basevm.BaseVM):
+ name = "ubuntu.i386"
+ BUILD_SCRIPT = """
+ set -e;
+ cd $(mktemp -d);
+ sudo chmod a+r /dev/vdb;
+ tar -xf /dev/vdb;
+ ./configure {configure_opts};
+ make -j{jobs};
+ make check;
+ """
+
+ def _gen_cloud_init_iso(self):
+ cidir = self._tmpdir
+ mdata = open(os.path.join(cidir, "meta-data"), "w")
+ mdata.writelines(["instance-id: ubuntu-vm-0\n",
+ "local-hostname: ubuntu-guest\n"])
+ mdata.close()
+ udata = open(os.path.join(cidir, "user-data"), "w")
+ udata.writelines(["#cloud-config\n",
+ "chpasswd:\n",
+ " list: |\n",
+ " root:%s\n" % self.ROOT_PASS,
+ " %s:%s\n" % (self.GUEST_USER, self.GUEST_PASS),
+ " expire: False\n",
+ "users:\n",
+ " - name: %s\n" % self.GUEST_USER,
+ " sudo: ALL=(ALL) NOPASSWD:ALL\n",
+ " ssh-authorized-keys:\n",
+ " - %s\n" % basevm.SSH_PUB_KEY,
+ " - name: root\n",
+ " ssh-authorized-keys:\n",
+ " - %s\n" % basevm.SSH_PUB_KEY])
+ udata.close()
+ subprocess.check_call(["genisoimage", "-output", "cloud-init.iso",
+ "-volid", "cidata", "-joliet", "-rock",
+ "user-data", "meta-data"],
+ cwd=cidir,
+ stdin=self._devnull, stdout=self._stdout,
+ stderr=self._stdout)
+ return os.path.join(cidir, "cloud-init.iso")
+
+ def build_image(self, img):
+ cimg = self._download_with_cache("https://cloud-images.ubuntu.com/releases/16.04/release/ubuntu-16.04-server-cloudimg-i386-disk1.img")
+ img_tmp = img + ".tmp"
+ subprocess.check_call(["cp", "-f", cimg, img_tmp])
+ subprocess.check_call(["qemu-img", "resize", img_tmp, "50G"])
+ self.boot(img_tmp, extra_args = ["-cdrom", self._gen_cloud_init_iso()])
+ self.wait_ssh()
+ self.ssh_root_check("touch /etc/cloud/cloud-init.disabled")
+ self.ssh_root_check("apt-get update")
+ self.ssh_root_check("apt-get install -y cloud-initramfs-growroot")
+ # Don't check the status in case the guest hang up too quickly
+ self.ssh_root("sync && reboot")
+ time.sleep(5)
+ self.wait_ssh()
+ # The previous update sometimes doesn't survive a reboot, so do it again
+ self.ssh_root_check("apt-get update")
+ self.ssh_root_check("apt-get build-dep -y qemu")
+ self.ssh_root_check("apt-get install -y libfdt-dev")
+ self.ssh_root("poweroff")
+ self.wait()
+ if os.path.exists(img):
+ os.remove(img)
+ os.rename(img_tmp, img)
+ return 0
+
+if __name__ == "__main__":
+ sys.exit(basevm.main(UbuntuX86VM))
--
2.13.5
^ permalink raw reply related [flat|nested] 22+ messages in thread* [Qemu-devel] [PULL v4 14/38] tests: Add FreeBSD image
2017-09-15 9:02 [Qemu-devel] [PULL v4 00/38] Test and build patches Fam Zheng
` (3 preceding siblings ...)
2017-09-15 9:02 ` [Qemu-devel] [PULL v4 13/38] tests: Add ubuntu.i386 image Fam Zheng
@ 2017-09-15 9:02 ` Fam Zheng
2017-09-15 9:02 ` [Qemu-devel] [PULL v4 15/38] tests: Add NetBSD image Fam Zheng
` (6 subsequent siblings)
11 siblings, 0 replies; 22+ messages in thread
From: Fam Zheng @ 2017-09-15 9:02 UTC (permalink / raw)
To: qemu-devel; +Cc: Peter Maydell
The image is prepared following instructions as in:
https://wiki.qemu.org/Hosts/BSD
Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <20170913030119.3957-8-famz@redhat.com>
---
tests/vm/freebsd | 42 ++++++++++++++++++++++++++++++++++++++++++
1 file changed, 42 insertions(+)
create mode 100755 tests/vm/freebsd
diff --git a/tests/vm/freebsd b/tests/vm/freebsd
new file mode 100755
index 0000000000..039dad8f69
--- /dev/null
+++ b/tests/vm/freebsd
@@ -0,0 +1,42 @@
+#!/usr/bin/env python
+#
+# FreeBSD VM image
+#
+# Copyright 2017 Red Hat Inc.
+#
+# Authors:
+# Fam Zheng <famz@redhat.com>
+#
+# This code is licensed under the GPL version 2 or later. See
+# the COPYING file in the top-level directory.
+#
+
+import os
+import sys
+import subprocess
+import basevm
+
+class FreeBSDVM(basevm.BaseVM):
+ name = "freebsd"
+ BUILD_SCRIPT = """
+ set -e;
+ cd $(mktemp -d /var/tmp/qemu-test.XXXXXX);
+ tar -xf /dev/vtbd1;
+ ./configure {configure_opts};
+ gmake -j{jobs};
+ gmake check;
+ """
+
+ def build_image(self, img):
+ cimg = self._download_with_cache("http://download.patchew.org/freebsd-11.1-amd64.img.xz",
+ sha256sum='adcb771549b37bc63826c501f05121a206ed3d9f55f49145908f7e1432d65891')
+ img_tmp_xz = img + ".tmp.xz"
+ img_tmp = img + ".tmp"
+ subprocess.check_call(["cp", "-f", cimg, img_tmp_xz])
+ subprocess.check_call(["xz", "-df", img_tmp_xz])
+ if os.path.exists(img):
+ os.remove(img)
+ os.rename(img_tmp, img)
+
+if __name__ == "__main__":
+ sys.exit(basevm.main(FreeBSDVM))
--
2.13.5
^ permalink raw reply related [flat|nested] 22+ messages in thread* [Qemu-devel] [PULL v4 15/38] tests: Add NetBSD image
2017-09-15 9:02 [Qemu-devel] [PULL v4 00/38] Test and build patches Fam Zheng
` (4 preceding siblings ...)
2017-09-15 9:02 ` [Qemu-devel] [PULL v4 14/38] tests: Add FreeBSD image Fam Zheng
@ 2017-09-15 9:02 ` Fam Zheng
2017-09-15 9:02 ` [Qemu-devel] [PULL v4 16/38] tests: Add OpenBSD image Fam Zheng
` (5 subsequent siblings)
11 siblings, 0 replies; 22+ messages in thread
From: Fam Zheng @ 2017-09-15 9:02 UTC (permalink / raw)
To: qemu-devel; +Cc: Peter Maydell
The image is prepared following instructions as in:
https://wiki.qemu.org/Hosts/BSD
Signed-off-by: Fam Zheng <famz@redhat.com>
Reviewed-by: Kamil Rytarowski <n54@gmx.com>
Message-Id: <20170913030119.3957-9-famz@redhat.com>
---
tests/vm/netbsd | 42 ++++++++++++++++++++++++++++++++++++++++++
1 file changed, 42 insertions(+)
create mode 100755 tests/vm/netbsd
diff --git a/tests/vm/netbsd b/tests/vm/netbsd
new file mode 100755
index 0000000000..3972d8b45c
--- /dev/null
+++ b/tests/vm/netbsd
@@ -0,0 +1,42 @@
+#!/usr/bin/env python
+#
+# NetBSD VM image
+#
+# Copyright 2017 Red Hat Inc.
+#
+# Authors:
+# Fam Zheng <famz@redhat.com>
+#
+# This code is licensed under the GPL version 2 or later. See
+# the COPYING file in the top-level directory.
+#
+
+import os
+import sys
+import subprocess
+import basevm
+
+class NetBSDVM(basevm.BaseVM):
+ name = "netbsd"
+ BUILD_SCRIPT = """
+ set -e;
+ cd $(mktemp -d /var/tmp/qemu-test.XXXXXX);
+ tar -xf /dev/rld1a;
+ ./configure --python=python2.7 {configure_opts};
+ gmake -j{jobs};
+ gmake check;
+ """
+
+ def build_image(self, img):
+ cimg = self._download_with_cache("http://download.patchew.org/netbsd-7.1-amd64.img.xz",
+ sha256sum='b633d565b0eac3d02015cd0c81440bd8a7a8df8512615ac1ee05d318be015732')
+ img_tmp_xz = img + ".tmp.xz"
+ img_tmp = img + ".tmp"
+ subprocess.check_call(["cp", "-f", cimg, img_tmp_xz])
+ subprocess.check_call(["xz", "-df", img_tmp_xz])
+ if os.path.exists(img):
+ os.remove(img)
+ os.rename(img_tmp, img)
+
+if __name__ == "__main__":
+ sys.exit(basevm.main(NetBSDVM))
--
2.13.5
^ permalink raw reply related [flat|nested] 22+ messages in thread* [Qemu-devel] [PULL v4 16/38] tests: Add OpenBSD image
2017-09-15 9:02 [Qemu-devel] [PULL v4 00/38] Test and build patches Fam Zheng
` (5 preceding siblings ...)
2017-09-15 9:02 ` [Qemu-devel] [PULL v4 15/38] tests: Add NetBSD image Fam Zheng
@ 2017-09-15 9:02 ` Fam Zheng
2017-09-15 9:02 ` [Qemu-devel] [PULL v4 17/38] Makefile: Add rules to run vm tests Fam Zheng
` (4 subsequent siblings)
11 siblings, 0 replies; 22+ messages in thread
From: Fam Zheng @ 2017-09-15 9:02 UTC (permalink / raw)
To: qemu-devel; +Cc: Peter Maydell
The image is prepared following instructions as in:
https://wiki.qemu.org/Hosts/BSD
Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <20170913030119.3957-10-famz@redhat.com>
---
tests/vm/openbsd | 43 +++++++++++++++++++++++++++++++++++++++++++
1 file changed, 43 insertions(+)
create mode 100755 tests/vm/openbsd
diff --git a/tests/vm/openbsd b/tests/vm/openbsd
new file mode 100755
index 0000000000..6ae16d97fd
--- /dev/null
+++ b/tests/vm/openbsd
@@ -0,0 +1,43 @@
+#!/usr/bin/env python
+#
+# OpenBSD VM image
+#
+# Copyright 2017 Red Hat Inc.
+#
+# Authors:
+# Fam Zheng <famz@redhat.com>
+#
+# This code is licensed under the GPL version 2 or later. See
+# the COPYING file in the top-level directory.
+#
+
+import os
+import sys
+import subprocess
+import basevm
+
+class OpenBSDVM(basevm.BaseVM):
+ name = "openbsd"
+ BUILD_SCRIPT = """
+ set -e;
+ cd $(mktemp -d /var/tmp/qemu-test.XXXXXX);
+ tar -xf /dev/rsd1c;
+ ./configure --cc=x86_64-unknown-openbsd6.1-gcc-4.9.4 --python=python2.7 {configure_opts};
+ gmake -j{jobs};
+ # XXX: "gmake check" seems to always hang or fail
+ #gmake check;
+ """
+
+ def build_image(self, img):
+ cimg = self._download_with_cache("http://download.patchew.org/openbsd-6.1-amd64.img.xz",
+ sha256sum='8c6cedc483e602cfee5e04f0406c64eb99138495e8ca580bc0293bcf0640c1bf')
+ img_tmp_xz = img + ".tmp.xz"
+ img_tmp = img + ".tmp"
+ subprocess.check_call(["cp", "-f", cimg, img_tmp_xz])
+ subprocess.check_call(["xz", "-df", img_tmp_xz])
+ if os.path.exists(img):
+ os.remove(img)
+ os.rename(img_tmp, img)
+
+if __name__ == "__main__":
+ sys.exit(basevm.main(OpenBSDVM))
--
2.13.5
^ permalink raw reply related [flat|nested] 22+ messages in thread* [Qemu-devel] [PULL v4 17/38] Makefile: Add rules to run vm tests
2017-09-15 9:02 [Qemu-devel] [PULL v4 00/38] Test and build patches Fam Zheng
` (6 preceding siblings ...)
2017-09-15 9:02 ` [Qemu-devel] [PULL v4 16/38] tests: Add OpenBSD image Fam Zheng
@ 2017-09-15 9:02 ` Fam Zheng
2017-09-15 9:02 ` [Qemu-devel] [PULL v4 18/38] MAINTAINERS: Add tests/vm entry Fam Zheng
` (3 subsequent siblings)
11 siblings, 0 replies; 22+ messages in thread
From: Fam Zheng @ 2017-09-15 9:02 UTC (permalink / raw)
To: qemu-devel; +Cc: Peter Maydell
Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <20170913030119.3957-11-famz@redhat.com>
---
Makefile | 2 ++
configure | 2 +-
tests/vm/Makefile.include | 42 ++++++++++++++++++++++++++++++++++++++++++
3 files changed, 45 insertions(+), 1 deletion(-)
create mode 100644 tests/vm/Makefile.include
diff --git a/Makefile b/Makefile
index 337a1f6f9b..946eb2ce35 100644
--- a/Makefile
+++ b/Makefile
@@ -822,6 +822,7 @@ endif
-include $(wildcard *.d tests/*.d)
include $(SRC_PATH)/tests/docker/Makefile.include
+include $(SRC_PATH)/tests/vm/Makefile.include
.PHONY: help
help:
@@ -845,6 +846,7 @@ help:
@echo 'Test targets:'
@echo ' check - Run all tests (check-help for details)'
@echo ' docker - Help about targets running tests inside Docker containers'
+ @echo ' vm-test - Help about targets running tests inside VM'
@echo ''
@echo 'Documentation targets:'
@echo ' html info pdf txt'
diff --git a/configure b/configure
index fd7e3a5e81..3918c47cd8 100755
--- a/configure
+++ b/configure
@@ -6546,7 +6546,7 @@ if test "$ccache_cpp2" = "yes"; then
fi
# build tree in object directory in case the source is not in the current directory
-DIRS="tests tests/tcg tests/tcg/cris tests/tcg/lm32 tests/libqos tests/qapi-schema tests/tcg/xtensa tests/qemu-iotests"
+DIRS="tests tests/tcg tests/tcg/cris tests/tcg/lm32 tests/libqos tests/qapi-schema tests/tcg/xtensa tests/qemu-iotests tests/vm"
DIRS="$DIRS docs docs/interop fsdev"
DIRS="$DIRS pc-bios/optionrom pc-bios/spapr-rtas pc-bios/s390-ccw"
DIRS="$DIRS roms/seabios roms/vgabios"
diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include
new file mode 100644
index 0000000000..5daa2a3b73
--- /dev/null
+++ b/tests/vm/Makefile.include
@@ -0,0 +1,42 @@
+# Makefile for VM tests
+
+.PHONY: vm-build-all
+
+IMAGES := ubuntu.i386 freebsd netbsd openbsd
+IMAGE_FILES := $(patsubst %, tests/vm/%.img, $(IMAGES))
+
+.PRECIOUS: $(IMAGE_FILES)
+
+vm-test:
+ @echo "vm-test: Test QEMU in preconfigured virtual machines"
+ @echo
+ @echo " vm-build-ubuntu.i386 - Build QEMU in ubuntu i386 VM"
+ @echo " vm-build-freebsd - Build QEMU in FreeBSD VM"
+ @echo " vm-build-netbsd - Build QEMU in NetBSD VM"
+ @echo " vm-build-openbsd - Build QEMU in OpenBSD VM"
+
+vm-build-all: $(addprefix vm-build-, $(IMAGES))
+
+tests/vm/%.img: $(SRC_PATH)/tests/vm/% \
+ $(SRC_PATH)/tests/vm/basevm.py \
+ $(SRC_PATH)/tests/vm/Makefile.include
+ $(call quiet-command, \
+ $< \
+ $(if $(V)$(DEBUG), --debug) \
+ --image "$@" \
+ --force \
+ --build-image $@, \
+ " VM-IMAGE $*")
+
+
+# Build in VM $(IMAGE)
+vm-build-%: tests/vm/%.img
+ $(call quiet-command, \
+ $(SRC_PATH)/tests/vm/$* \
+ $(if $(V)$(DEBUG), --debug) \
+ $(if $(DEBUG), --interactive) \
+ $(if $(J),--jobs $(J)) \
+ --image "$<" \
+ --build-qemu $(SRC_PATH), \
+ " VM-BUILD $*")
+
--
2.13.5
^ permalink raw reply related [flat|nested] 22+ messages in thread* [Qemu-devel] [PULL v4 18/38] MAINTAINERS: Add tests/vm entry
2017-09-15 9:02 [Qemu-devel] [PULL v4 00/38] Test and build patches Fam Zheng
` (7 preceding siblings ...)
2017-09-15 9:02 ` [Qemu-devel] [PULL v4 17/38] Makefile: Add rules to run vm tests Fam Zheng
@ 2017-09-15 9:02 ` Fam Zheng
2017-09-15 9:02 ` [Qemu-devel] [PULL v4 19/38] tests: Add README for vm tests Fam Zheng
` (2 subsequent siblings)
11 siblings, 0 replies; 22+ messages in thread
From: Fam Zheng @ 2017-09-15 9:02 UTC (permalink / raw)
To: qemu-devel; +Cc: Peter Maydell
Signed-off-by: Fam Zheng <famz@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-Id: <20170913030119.3957-12-famz@redhat.com>
---
MAINTAINERS | 1 +
1 file changed, 1 insertion(+)
diff --git a/MAINTAINERS b/MAINTAINERS
index 36eeb42d19..42f5454311 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1891,6 +1891,7 @@ S: Maintained
F: .travis.yml
F: .shippable.yml
F: tests/docker/
+F: tests/vm/
W: https://travis-ci.org/qemu/qemu
W: https://app.shippable.com/github/qemu/qemu
W: http://patchew.org/QEMU/
--
2.13.5
^ permalink raw reply related [flat|nested] 22+ messages in thread* [Qemu-devel] [PULL v4 19/38] tests: Add README for vm tests
2017-09-15 9:02 [Qemu-devel] [PULL v4 00/38] Test and build patches Fam Zheng
` (8 preceding siblings ...)
2017-09-15 9:02 ` [Qemu-devel] [PULL v4 18/38] MAINTAINERS: Add tests/vm entry Fam Zheng
@ 2017-09-15 9:02 ` Fam Zheng
2017-09-15 9:02 ` [Qemu-devel] [PULL v4 20/38] docker: Use archive-source.py Fam Zheng
2017-09-15 10:55 ` [Qemu-devel] [PULL v4 00/38] Test and build patches Peter Maydell
11 siblings, 0 replies; 22+ messages in thread
From: Fam Zheng @ 2017-09-15 9:02 UTC (permalink / raw)
To: qemu-devel; +Cc: Peter Maydell
Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <20170913030119.3957-13-famz@redhat.com>
---
tests/vm/README | 63 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 63 insertions(+)
create mode 100644 tests/vm/README
diff --git a/tests/vm/README b/tests/vm/README
new file mode 100644
index 0000000000..7d2fe4ac8d
--- /dev/null
+++ b/tests/vm/README
@@ -0,0 +1,63 @@
+=== VM test suite to run build in guests ===
+
+== Intro ==
+
+This test suite contains scripts that bootstrap various guest images that have
+necessary packages to build QEMU. The basic usage is documented in Makefile
+help which is displayed with "make vm-test".
+
+== Quick start ==
+
+Run "make vm-test" to list available make targets.
+
+== Manual invocation ==
+
+Each guest script is an executable script with the same command line options.
+For example to work with the netbsd guest, use $QEMU_SRC/tests/vm/netbsd:
+
+ $ cd $QEMU_SRC/tests/vm
+
+ # To bootstrap the image
+ $ ./netbsd --build-image --image /var/tmp/netbsd.img
+ <...>
+
+ # To run an arbitrary command in guest (the output will not be echoed unless
+ # --debug is added)
+ $ ./netbsd --debug --image /var/tmp/netbsd.img uname -a
+
+ # To build QEMU in guest
+ $ ./netbsd --debug --image /var/tmp/netbsd.img --build-qemu $QEMU_SRC
+
+ # To get to an interactive shell
+ $ ./netbsd --interactive --image /var/tmp/netbsd.img sh
+
+== Adding new guests ==
+
+Please look at existing guest scripts for how to add new guests.
+
+Most importantly, create a subclass of BaseVM and implement build_image()
+method and define BUILD_SCRIPT, then finally call basevm.main() from the
+script's main().
+
+ - Usually in build_image(), a template image is downloaded from a predefined
+ URL. BaseVM._download_with_cache() takes care of the cache and the
+ checksum, so consider using it.
+
+ - Once the image is downloaded, users, SSH server and QEMU build deps should
+ be set up:
+
+ * Root password set to BaseVM.ROOT_PASS
+ * User BaseVM.GUEST_USER is created, and password set to BaseVM.GUEST_PASS
+ * SSH service is enabled and started on boot, BaseVM.SSH_PUB_KEY is added
+ to authorized_keys of both root and the normal user
+ * DHCP client service is enabled and started on boot, so that it can
+ automatically configure the virtio-net-pci NIC and communicate with QEMU
+ user net (10.0.2.2)
+ * Necessary packages are installed to untar the source tarball and build
+ QEMU
+
+ - Write a proper BUILD_SCRIPT template, which should be a shell script that
+ untars a raw virtio-blk block device, which is the tarball data blob of the
+ QEMU source tree, then configure/build it. Running "make check" is also
+ recommended.
+
--
2.13.5
^ permalink raw reply related [flat|nested] 22+ messages in thread* [Qemu-devel] [PULL v4 20/38] docker: Use archive-source.py
2017-09-15 9:02 [Qemu-devel] [PULL v4 00/38] Test and build patches Fam Zheng
` (9 preceding siblings ...)
2017-09-15 9:02 ` [Qemu-devel] [PULL v4 19/38] tests: Add README for vm tests Fam Zheng
@ 2017-09-15 9:02 ` Fam Zheng
2017-09-15 10:55 ` [Qemu-devel] [PULL v4 00/38] Test and build patches Peter Maydell
11 siblings, 0 replies; 22+ messages in thread
From: Fam Zheng @ 2017-09-15 9:02 UTC (permalink / raw)
To: qemu-devel; +Cc: Peter Maydell
Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <20170913030119.3957-14-famz@redhat.com>
---
tests/docker/Makefile.include | 15 ++-------------
tests/docker/run | 8 +-------
2 files changed, 3 insertions(+), 20 deletions(-)
diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index d7dafdbd27..4bb02b1bb5 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -17,24 +17,13 @@ DOCKER_TOOLS := travis
TESTS ?= %
IMAGES ?= %
-# Make archive from git repo $1 to tar.gz $2
-make-archive-maybe = $(if $(wildcard $1/*), \
- $(call quiet-command, \
- (cd $1; if git diff-index --quiet HEAD -- &>/dev/null; then \
- git archive -1 HEAD --format=tar.gz; \
- else \
- git archive -1 $$(git stash create) --format=tar.gz; \
- fi) > $2, \
- "ARCHIVE","$(notdir $2)"))
-
CUR_TIME := $(shell date +%Y-%m-%d-%H.%M.%S.$$$$)
DOCKER_SRC_COPY := docker-src.$(CUR_TIME)
$(DOCKER_SRC_COPY):
@mkdir $@
- $(call make-archive-maybe, $(SRC_PATH), $@/qemu.tgz)
- $(call make-archive-maybe, $(SRC_PATH)/dtc, $@/dtc.tgz)
- $(call make-archive-maybe, $(SRC_PATH)/pixman, $@/pixman.tgz)
+ $(call quiet-command, $(SRC_PATH)/scripts/archive-source.sh $@/qemu.tar, \
+ "GEN", "$@/qemu.tar")
$(call quiet-command, cp $(SRC_PATH)/tests/docker/run $@/run, \
"COPY","RUNNER")
diff --git a/tests/docker/run b/tests/docker/run
index ec2541cbd9..52b76e443d 100755
--- a/tests/docker/run
+++ b/tests/docker/run
@@ -32,13 +32,7 @@ export TEST_DIR=/tmp/qemu-test
mkdir -p $TEST_DIR/{src,build,install}
# Extract the source tarballs
-tar -C $TEST_DIR/src -xzf $BASE/qemu.tgz
-for p in dtc pixman; do
- if test -f $BASE/$p.tgz; then
- tar -C $TEST_DIR/src/$p -xzf $BASE/$p.tgz
- export FEATURES="$FEATURES $p"
- fi
-done
+tar -C $TEST_DIR/src -xf $BASE/qemu.tar
if test -n "$SHOW_ENV"; then
if test -f /packages.txt; then
--
2.13.5
^ permalink raw reply related [flat|nested] 22+ messages in thread* Re: [Qemu-devel] [PULL v4 00/38] Test and build patches
2017-09-15 9:02 [Qemu-devel] [PULL v4 00/38] Test and build patches Fam Zheng
` (10 preceding siblings ...)
2017-09-15 9:02 ` [Qemu-devel] [PULL v4 20/38] docker: Use archive-source.py Fam Zheng
@ 2017-09-15 10:55 ` Peter Maydell
2017-09-15 11:36 ` Fam Zheng
2017-09-15 11:40 ` Daniel P. Berrange
11 siblings, 2 replies; 22+ messages in thread
From: Peter Maydell @ 2017-09-15 10:55 UTC (permalink / raw)
To: Fam Zheng; +Cc: QEMU Developers
On 15 September 2017 at 10:02, Fam Zheng <famz@redhat.com> wrote:
> The following changes since commit 04ef33052c205170c92df21ca0b4be4f3b102188:
>
> tcg/tci: do not use ldst label (never implemented) (2017-09-11 19:24:05 +0100)
>
> are available in the git repository at:
>
> git://github.com/famz/qemu.git tags/test-and-build-pull-request
>
> for you to fetch changes up to be78fe670401af14e6d63fce5c5467f751207871:
>
> buildsys: Move rdma libs to per object (2017-09-15 15:05:24 +0800)
>
> ----------------------------------------------------------------
>
> ----------------------------------------------------------------
>
> Alex Bennée (4):
> docker: ensure NOUSER for travis images
> docker: docker.py make --no-cache skip checksum test
> docker: don't install device-tree-compiler build-deps in travis.docker
> docker: reduce noise when building travis.docker
>
> Fam Zheng (34):
> docker: Update ubuntu image
> docker: Enable features explicitly in test-full
> tests/docker: Clean up paths
> gitignore: Ignore vm test images
> qemu.py: Add "wait()" method
> scripts: Add archive-source.sh
> tests: Add a test key pair
So, before I commit an ssh private key to our git repo,
can you explain why it's ok that this is public? The
commit message for the relevant patch doesn't really say.
thanks
-- PMM
^ permalink raw reply [flat|nested] 22+ messages in thread* Re: [Qemu-devel] [PULL v4 00/38] Test and build patches
2017-09-15 10:55 ` [Qemu-devel] [PULL v4 00/38] Test and build patches Peter Maydell
@ 2017-09-15 11:36 ` Fam Zheng
2017-09-15 11:40 ` Daniel P. Berrange
1 sibling, 0 replies; 22+ messages in thread
From: Fam Zheng @ 2017-09-15 11:36 UTC (permalink / raw)
To: Peter Maydell; +Cc: QEMU Developers
On Fri, 09/15 11:55, Peter Maydell wrote:
> On 15 September 2017 at 10:02, Fam Zheng <famz@redhat.com> wrote:
> > The following changes since commit 04ef33052c205170c92df21ca0b4be4f3b102188:
> >
> > tcg/tci: do not use ldst label (never implemented) (2017-09-11 19:24:05 +0100)
> >
> > are available in the git repository at:
> >
> > git://github.com/famz/qemu.git tags/test-and-build-pull-request
> >
> > for you to fetch changes up to be78fe670401af14e6d63fce5c5467f751207871:
> >
> > buildsys: Move rdma libs to per object (2017-09-15 15:05:24 +0800)
> >
> > ----------------------------------------------------------------
> >
> > ----------------------------------------------------------------
> >
> > Alex Bennée (4):
> > docker: ensure NOUSER for travis images
> > docker: docker.py make --no-cache skip checksum test
> > docker: don't install device-tree-compiler build-deps in travis.docker
> > docker: reduce noise when building travis.docker
> >
> > Fam Zheng (34):
> > docker: Update ubuntu image
> > docker: Enable features explicitly in test-full
> > tests/docker: Clean up paths
> > gitignore: Ignore vm test images
> > qemu.py: Add "wait()" method
> > scripts: Add archive-source.sh
> > tests: Add a test key pair
>
> So, before I commit an ssh private key to our git repo,
> can you explain why it's ok that this is public? The
> commit message for the relevant patch doesn't really say.
It's under tests/, and the key is only used to access a temporarily spawned test
VM.
Fam
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [Qemu-devel] [PULL v4 00/38] Test and build patches
2017-09-15 10:55 ` [Qemu-devel] [PULL v4 00/38] Test and build patches Peter Maydell
2017-09-15 11:36 ` Fam Zheng
@ 2017-09-15 11:40 ` Daniel P. Berrange
2017-09-15 12:03 ` Peter Maydell
` (2 more replies)
1 sibling, 3 replies; 22+ messages in thread
From: Daniel P. Berrange @ 2017-09-15 11:40 UTC (permalink / raw)
To: Peter Maydell; +Cc: Fam Zheng, QEMU Developers
On Fri, Sep 15, 2017 at 11:55:44AM +0100, Peter Maydell wrote:
> On 15 September 2017 at 10:02, Fam Zheng <famz@redhat.com> wrote:
> > The following changes since commit 04ef33052c205170c92df21ca0b4be4f3b102188:
> >
> > tcg/tci: do not use ldst label (never implemented) (2017-09-11 19:24:05 +0100)
> >
> > are available in the git repository at:
> >
> > git://github.com/famz/qemu.git tags/test-and-build-pull-request
> >
> > for you to fetch changes up to be78fe670401af14e6d63fce5c5467f751207871:
> >
> > buildsys: Move rdma libs to per object (2017-09-15 15:05:24 +0800)
> >
> > ----------------------------------------------------------------
> >
> > ----------------------------------------------------------------
> >
> > Alex Bennée (4):
> > docker: ensure NOUSER for travis images
> > docker: docker.py make --no-cache skip checksum test
> > docker: don't install device-tree-compiler build-deps in travis.docker
> > docker: reduce noise when building travis.docker
> >
> > Fam Zheng (34):
> > docker: Update ubuntu image
> > docker: Enable features explicitly in test-full
> > tests/docker: Clean up paths
> > gitignore: Ignore vm test images
> > qemu.py: Add "wait()" method
> > scripts: Add archive-source.sh
> > tests: Add a test key pair
>
> So, before I commit an ssh private key to our git repo,
> can you explain why it's ok that this is public? The
> commit message for the relevant patch doesn't really say.
IIUC, the public part of the key gets exposed to the guest images via
cloud-init metadata. During boot the guest read this metadata and add
the public key to authorized_keys. The private key is used by the test
suite on the host so that it can now login to the guests.
So the risk here is that if these guests were exposed to the LAN in any
way, someone could grab our private key and login to these guests.
What saves us is that the VMs are run with user mode slirp networking
so AFAICT, aren't exposed to the LAN. So as long as we don't change
this to any kind of real networking, I think its acceptable to have
the private key in it and doesn't expose developer's workstations to
undue risk and avoids consuming system entropy to generate new keys
during build.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 22+ messages in thread* Re: [Qemu-devel] [PULL v4 00/38] Test and build patches
2017-09-15 11:40 ` Daniel P. Berrange
@ 2017-09-15 12:03 ` Peter Maydell
2017-09-15 12:09 ` Daniel P. Berrange
2017-09-15 12:21 ` Fam Zheng
2017-09-15 14:47 ` Philippe Mathieu-Daudé
2 siblings, 1 reply; 22+ messages in thread
From: Peter Maydell @ 2017-09-15 12:03 UTC (permalink / raw)
To: Daniel P. Berrange; +Cc: Fam Zheng, QEMU Developers
On 15 September 2017 at 12:40, Daniel P. Berrange <berrange@redhat.com> wrote:
> IIUC, the public part of the key gets exposed to the guest images via
> cloud-init metadata. During boot the guest read this metadata and add
> the public key to authorized_keys. The private key is used by the test
> suite on the host so that it can now login to the guests.
>
> So the risk here is that if these guests were exposed to the LAN in any
> way, someone could grab our private key and login to these guests.
>
> What saves us is that the VMs are run with user mode slirp networking
> so AFAICT, aren't exposed to the LAN.
If I'm reading the right bit of the script we run QEMU with a
hostfwd specification using 0.0.0.0 as the host part -- doesn't
that listen on all interfaces including the LAN ones?
thanks
-- PMM
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [Qemu-devel] [PULL v4 00/38] Test and build patches
2017-09-15 12:03 ` Peter Maydell
@ 2017-09-15 12:09 ` Daniel P. Berrange
0 siblings, 0 replies; 22+ messages in thread
From: Daniel P. Berrange @ 2017-09-15 12:09 UTC (permalink / raw)
To: Peter Maydell; +Cc: Fam Zheng, QEMU Developers
On Fri, Sep 15, 2017 at 01:03:54PM +0100, Peter Maydell wrote:
> On 15 September 2017 at 12:40, Daniel P. Berrange <berrange@redhat.com> wrote:
> > IIUC, the public part of the key gets exposed to the guest images via
> > cloud-init metadata. During boot the guest read this metadata and add
> > the public key to authorized_keys. The private key is used by the test
> > suite on the host so that it can now login to the guests.
> >
> > So the risk here is that if these guests were exposed to the LAN in any
> > way, someone could grab our private key and login to these guests.
> >
> > What saves us is that the VMs are run with user mode slirp networking
> > so AFAICT, aren't exposed to the LAN.
>
> If I'm reading the right bit of the script we run QEMU with a
> hostfwd specification using 0.0.0.0 as the host part -- doesn't
> that listen on all interfaces including the LAN ones?
Actually yes, you are right, my bad.
That needs to be fixed to use 127.0.0.1 for sure.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [Qemu-devel] [PULL v4 00/38] Test and build patches
2017-09-15 11:40 ` Daniel P. Berrange
2017-09-15 12:03 ` Peter Maydell
@ 2017-09-15 12:21 ` Fam Zheng
2017-09-15 12:31 ` Daniel P. Berrange
2017-09-15 14:47 ` Philippe Mathieu-Daudé
2 siblings, 1 reply; 22+ messages in thread
From: Fam Zheng @ 2017-09-15 12:21 UTC (permalink / raw)
To: Daniel P. Berrange; +Cc: Peter Maydell, QEMU Developers
On Fri, 09/15 12:40, Daniel P. Berrange wrote:
> On Fri, Sep 15, 2017 at 11:55:44AM +0100, Peter Maydell wrote:
> > On 15 September 2017 at 10:02, Fam Zheng <famz@redhat.com> wrote:
> > > The following changes since commit 04ef33052c205170c92df21ca0b4be4f3b102188:
> > >
> > > tcg/tci: do not use ldst label (never implemented) (2017-09-11 19:24:05 +0100)
> > >
> > > are available in the git repository at:
> > >
> > > git://github.com/famz/qemu.git tags/test-and-build-pull-request
> > >
> > > for you to fetch changes up to be78fe670401af14e6d63fce5c5467f751207871:
> > >
> > > buildsys: Move rdma libs to per object (2017-09-15 15:05:24 +0800)
> > >
> > > ----------------------------------------------------------------
> > >
> > > ----------------------------------------------------------------
> > >
> > > Alex Bennée (4):
> > > docker: ensure NOUSER for travis images
> > > docker: docker.py make --no-cache skip checksum test
> > > docker: don't install device-tree-compiler build-deps in travis.docker
> > > docker: reduce noise when building travis.docker
> > >
> > > Fam Zheng (34):
> > > docker: Update ubuntu image
> > > docker: Enable features explicitly in test-full
> > > tests/docker: Clean up paths
> > > gitignore: Ignore vm test images
> > > qemu.py: Add "wait()" method
> > > scripts: Add archive-source.sh
> > > tests: Add a test key pair
> >
> > So, before I commit an ssh private key to our git repo,
> > can you explain why it's ok that this is public? The
> > commit message for the relevant patch doesn't really say.
>
> IIUC, the public part of the key gets exposed to the guest images via
> cloud-init metadata. During boot the guest read this metadata and add
> the public key to authorized_keys. The private key is used by the test
> suite on the host so that it can now login to the guests.
>
> So the risk here is that if these guests were exposed to the LAN in any
> way, someone could grab our private key and login to these guests.
>
> What saves us is that the VMs are run with user mode slirp networking
> so AFAICT, aren't exposed to the LAN. So as long as we don't change
> this to any kind of real networking, I think its acceptable to have
> the private key in it and doesn't expose developer's workstations to
> undue risk and avoids consuming system entropy to generate new keys
> during build.
The hostfwd does listen on a dynamic port on 0.0.0.0, so does vnc. I didn't
really care since it's for temporary guests and for me convenience outweighed a
bit. The VM test is indeed less restricted than the docker ones such as in that
network is always available. Should it be a problem?
We can probably add restrict=on to slirp and listen on loopback.
Fam
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [Qemu-devel] [PULL v4 00/38] Test and build patches
2017-09-15 12:21 ` Fam Zheng
@ 2017-09-15 12:31 ` Daniel P. Berrange
2017-09-15 12:51 ` Fam Zheng
0 siblings, 1 reply; 22+ messages in thread
From: Daniel P. Berrange @ 2017-09-15 12:31 UTC (permalink / raw)
To: Fam Zheng; +Cc: Peter Maydell, QEMU Developers
On Fri, Sep 15, 2017 at 08:21:53PM +0800, Fam Zheng wrote:
> On Fri, 09/15 12:40, Daniel P. Berrange wrote:
> > On Fri, Sep 15, 2017 at 11:55:44AM +0100, Peter Maydell wrote:
> > > On 15 September 2017 at 10:02, Fam Zheng <famz@redhat.com> wrote:
> > > > The following changes since commit 04ef33052c205170c92df21ca0b4be4f3b102188:
> > > >
> > > > tcg/tci: do not use ldst label (never implemented) (2017-09-11 19:24:05 +0100)
> > > >
> > > > are available in the git repository at:
> > > >
> > > > git://github.com/famz/qemu.git tags/test-and-build-pull-request
> > > >
> > > > for you to fetch changes up to be78fe670401af14e6d63fce5c5467f751207871:
> > > >
> > > > buildsys: Move rdma libs to per object (2017-09-15 15:05:24 +0800)
> > > >
> > > > ----------------------------------------------------------------
> > > >
> > > > ----------------------------------------------------------------
> > > >
> > > > Alex Bennée (4):
> > > > docker: ensure NOUSER for travis images
> > > > docker: docker.py make --no-cache skip checksum test
> > > > docker: don't install device-tree-compiler build-deps in travis.docker
> > > > docker: reduce noise when building travis.docker
> > > >
> > > > Fam Zheng (34):
> > > > docker: Update ubuntu image
> > > > docker: Enable features explicitly in test-full
> > > > tests/docker: Clean up paths
> > > > gitignore: Ignore vm test images
> > > > qemu.py: Add "wait()" method
> > > > scripts: Add archive-source.sh
> > > > tests: Add a test key pair
> > >
> > > So, before I commit an ssh private key to our git repo,
> > > can you explain why it's ok that this is public? The
> > > commit message for the relevant patch doesn't really say.
> >
> > IIUC, the public part of the key gets exposed to the guest images via
> > cloud-init metadata. During boot the guest read this metadata and add
> > the public key to authorized_keys. The private key is used by the test
> > suite on the host so that it can now login to the guests.
> >
> > So the risk here is that if these guests were exposed to the LAN in any
> > way, someone could grab our private key and login to these guests.
> >
> > What saves us is that the VMs are run with user mode slirp networking
> > so AFAICT, aren't exposed to the LAN. So as long as we don't change
> > this to any kind of real networking, I think its acceptable to have
> > the private key in it and doesn't expose developer's workstations to
> > undue risk and avoids consuming system entropy to generate new keys
> > during build.
>
> The hostfwd does listen on a dynamic port on 0.0.0.0, so does vnc. I didn't
> really care since it's for temporary guests and for me convenience outweighed a
> bit. The VM test is indeed less restricted than the docker ones such as in that
> network is always available. Should it be a problem?
AFAICT there's no functional reason why it needs to listen on 0.0.0.0,
instead of 127.0.0.1, so general security best practice says it should
not expose this listening port on LAN interfaces for the developers
machine, even if we think the risk is low.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [Qemu-devel] [PULL v4 00/38] Test and build patches
2017-09-15 12:31 ` Daniel P. Berrange
@ 2017-09-15 12:51 ` Fam Zheng
0 siblings, 0 replies; 22+ messages in thread
From: Fam Zheng @ 2017-09-15 12:51 UTC (permalink / raw)
To: Daniel P. Berrange; +Cc: Peter Maydell, QEMU Developers
On Fri, 09/15 13:31, Daniel P. Berrange wrote:
> On Fri, Sep 15, 2017 at 08:21:53PM +0800, Fam Zheng wrote:
> > On Fri, 09/15 12:40, Daniel P. Berrange wrote:
> > > On Fri, Sep 15, 2017 at 11:55:44AM +0100, Peter Maydell wrote:
> > > > On 15 September 2017 at 10:02, Fam Zheng <famz@redhat.com> wrote:
> > > > > The following changes since commit 04ef33052c205170c92df21ca0b4be4f3b102188:
> > > > >
> > > > > tcg/tci: do not use ldst label (never implemented) (2017-09-11 19:24:05 +0100)
> > > > >
> > > > > are available in the git repository at:
> > > > >
> > > > > git://github.com/famz/qemu.git tags/test-and-build-pull-request
> > > > >
> > > > > for you to fetch changes up to be78fe670401af14e6d63fce5c5467f751207871:
> > > > >
> > > > > buildsys: Move rdma libs to per object (2017-09-15 15:05:24 +0800)
> > > > >
> > > > > ----------------------------------------------------------------
> > > > >
> > > > > ----------------------------------------------------------------
> > > > >
> > > > > Alex Bennée (4):
> > > > > docker: ensure NOUSER for travis images
> > > > > docker: docker.py make --no-cache skip checksum test
> > > > > docker: don't install device-tree-compiler build-deps in travis.docker
> > > > > docker: reduce noise when building travis.docker
> > > > >
> > > > > Fam Zheng (34):
> > > > > docker: Update ubuntu image
> > > > > docker: Enable features explicitly in test-full
> > > > > tests/docker: Clean up paths
> > > > > gitignore: Ignore vm test images
> > > > > qemu.py: Add "wait()" method
> > > > > scripts: Add archive-source.sh
> > > > > tests: Add a test key pair
> > > >
> > > > So, before I commit an ssh private key to our git repo,
> > > > can you explain why it's ok that this is public? The
> > > > commit message for the relevant patch doesn't really say.
> > >
> > > IIUC, the public part of the key gets exposed to the guest images via
> > > cloud-init metadata. During boot the guest read this metadata and add
> > > the public key to authorized_keys. The private key is used by the test
> > > suite on the host so that it can now login to the guests.
> > >
> > > So the risk here is that if these guests were exposed to the LAN in any
> > > way, someone could grab our private key and login to these guests.
> > >
> > > What saves us is that the VMs are run with user mode slirp networking
> > > so AFAICT, aren't exposed to the LAN. So as long as we don't change
> > > this to any kind of real networking, I think its acceptable to have
> > > the private key in it and doesn't expose developer's workstations to
> > > undue risk and avoids consuming system entropy to generate new keys
> > > during build.
> >
> > The hostfwd does listen on a dynamic port on 0.0.0.0, so does vnc. I didn't
> > really care since it's for temporary guests and for me convenience outweighed a
> > bit. The VM test is indeed less restricted than the docker ones such as in that
> > network is always available. Should it be a problem?
>
> AFAICT there's no functional reason why it needs to listen on 0.0.0.0,
> instead of 127.0.0.1, so general security best practice says it should
> not expose this listening port on LAN interfaces for the developers
> machine, even if we think the risk is low.
Yes, makes sense, let's change it. The only disadvantage of 127.0.0.1 is if the
test is run on a remote host, you don't have to ssh to the host and proxy from
there to login to the guest. The test is automated, so accessing guest may be a
rare need outside patchew (a few months ago I frequently need to diagnose
hanging tests on patchew, no idea how this vm test will do :).
Fam
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [Qemu-devel] [PULL v4 00/38] Test and build patches
2017-09-15 11:40 ` Daniel P. Berrange
2017-09-15 12:03 ` Peter Maydell
2017-09-15 12:21 ` Fam Zheng
@ 2017-09-15 14:47 ` Philippe Mathieu-Daudé
2017-09-15 20:52 ` Fam Zheng
2 siblings, 1 reply; 22+ messages in thread
From: Philippe Mathieu-Daudé @ 2017-09-15 14:47 UTC (permalink / raw)
To: Daniel P. Berrange, Peter Maydell, Fam Zheng; +Cc: QEMU Developers
Hi Daniel,
On 09/15/2017 08:40 AM, Daniel P. Berrange wrote:
> On Fri, Sep 15, 2017 at 11:55:44AM +0100, Peter Maydell wrote:
[...]
>>
>> So, before I commit an ssh private key to our git repo,
>> can you explain why it's ok that this is public? The
>> commit message for the relevant patch doesn't really say.
>
> IIUC, the public part of the key gets exposed to the guest images via
> cloud-init metadata. During boot the guest read this metadata and add
> the public key to authorized_keys. The private key is used by the test
> suite on the host so that it can now login to the guests.
>
> So the risk here is that if these guests were exposed to the LAN in any
> way, someone could grab our private key and login to these guests.
>
> What saves us is that the VMs are run with user mode slirp networking
> so AFAICT, aren't exposed to the LAN. So as long as we don't change
> this to any kind of real networking, I think its acceptable to have
> the private key in it and doesn't expose developer's workstations to
> undue risk and avoids consuming system entropy to generate new keys
> during build.
which systems are you worried about? build-farms or developer's stations?
why do you want to generate more than 1 key? why not generate 1 key in
tests/vm/ (or clever ~/.cache/qemu-vm/ already used by those scripts)
once when the make vm-test rule is called, that would be 1 key per
repository clone (or 1 per user using ~/.cache).
Distrib aren't using the test suite in binary packages.
Regards,
Phil.
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [Qemu-devel] [PULL v4 00/38] Test and build patches
2017-09-15 14:47 ` Philippe Mathieu-Daudé
@ 2017-09-15 20:52 ` Fam Zheng
0 siblings, 0 replies; 22+ messages in thread
From: Fam Zheng @ 2017-09-15 20:52 UTC (permalink / raw)
To: Philippe Mathieu-Daudé
Cc: Daniel P. Berrange, Peter Maydell, QEMU Developers
On Fri, 09/15 11:47, Philippe Mathieu-Daudé wrote:
> why not generate 1 key in
> tests/vm/ (or clever ~/.cache/qemu-vm/ already used by those scripts) once
> when the make vm-test rule is called, that would be 1 key per repository
> clone (or 1 per user using ~/.cache).
Like I explained elsewhere, the BSD images must be generated after the keys in
order for the pub key to be added to the guest authorized_keys. And there is no
automatic way to do so. That's why the keys are committed to the repo, not
generated (otherwise we'd just use ~/.ssh pub keys).
Fam
^ permalink raw reply [flat|nested] 22+ messages in thread