[Qemu-devel] [PATCH 00/13] tests/vm: serial console autoinstall, misc fixes.

[Qemu-devel] [PATCH 00/13] tests/vm: serial console autoinstall, misc fixes.

[Gerd Hoffmann]

This patch series changes the way virtual machines for test builds are
managed.  They are created locally on the developer machine now.  The
installer is booted on the serial console and the scripts walks through
the dialogs to install and configure the guest.

That takes the download.patchew.org server out of the loop and makes it
alot easier to tweak the guest images (adding build dependencies for
example).

The install scripts take care to apply host proxy settings (from *_proxy
environment variables) to the guest, so any package downloads will be
routed through the proxy and can be cached that way.  This also makes
them work behind strict firewalls.

There are also a bunch of smaller tweaks for tests/vm to fix issues I
was struggling with.  See commit messages of individual patches for
details.

Known issue:  NetBSD package install is not working for me right now.
It did work a while ago.  Not sure what is going on here.

Do we have accelerator support for the BSDs?  A "make check" for a full
build takes ages, and I suspect tcg being used is part of the problem.
I did my tests using "TARGET_LIST=x86_64-softmmu" because of that.

Gerd Hoffmann (13):
  scripts: use git archive in archive-source
  tests/vm: send proxy environment variables over ssh
  tests/vm: send locale environment variables over ssh
  tests/vm: use ssh with pty unconditionally
  tests/vm: run test builds on snapshot
  tests/vm: add vm-boot-{ssh,serial}-<guest> targets
  tests/vm: add DEBUG=1 to help text
  tests/vm: serial console support helpers
  tests/vm: openbsd autoinstall, using serial console
  tests/vm: freebsd autoinstall, using serial console
  tests/vm: netbsd autoinstall, using serial console
  tests/vm: fedora autoinstall, using serial console
  tests/vm: ubuntu.i386: apt proxy setup

 tests/vm/basevm.py        | 125 ++++++++++++++++++++++---
 scripts/archive-source.sh |  72 +++++++--------
 tests/vm/Makefile.include |  25 ++++-
 tests/vm/fedora           | 187 ++++++++++++++++++++++++++++++++++++++
 tests/vm/freebsd          | 172 +++++++++++++++++++++++++++++++++--
 tests/vm/netbsd           | 178 ++++++++++++++++++++++++++++++++++--
 tests/vm/openbsd          | 150 +++++++++++++++++++++++++++---
 tests/vm/ubuntu.i386      |   4 +
 8 files changed, 830 insertions(+), 83 deletions(-)
 create mode 100755 tests/vm/fedora

-- 
2.18.1

[Qemu-devel] [PATCH 01/13] scripts: use git archive in archive-source

[Gerd Hoffmann]

Use git archive to create tarballs of qemu and submodules instead of
cloning the repository and the submodules.  This is a order of magnitude
faster because it doesn't fetch the submodules from the internet each
time the script runs.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 scripts/archive-source.sh | 72 +++++++++++++++++----------------------
 1 file changed, 31 insertions(+), 41 deletions(-)

diff --git a/scripts/archive-source.sh b/scripts/archive-source.sh
index 8b8994826000..93a48d2063f9 100755
--- a/scripts/archive-source.sh
+++ b/scripts/archive-source.sh
@@ -19,16 +19,25 @@ if test $# -lt 1; then
 fi
 
 tar_file=$(realpath "$1")
-list_file="${tar_file}.list"
-vroot_dir="${tar_file}.vroot"
+sub_tdir=$(mktemp -d "${tar_file%.tar}.sub.XXXXXXXX")
+sub_file="${sub_tdir}/submodule.tar"
 
 # We want a predictable list of submodules for builds, that is
 # independent of what the developer currently has initialized
 # in their checkout, because the build environment is completely
 # different to the host OS.
 submodules="dtc slirp ui/keycodemapdb tests/fp/berkeley-softfloat-3 tests/fp/berkeley-testfloat-3"
+sub_deinit=""
 
-trap "status=$?; rm -rf \"$list_file\" \"$vroot_dir\"; exit \$status" 0 1 2 3 15
+function cleanup() {
+    local status=$?
+    rm -rf "$sub_tdir"
+    if test "$sub_deinit" != ""; then
+	git submodule deinit $sub_deinit
+    fi
+    exit $status
+}
+trap "cleanup" 0 1 2 3 15
 
 if git diff-index --quiet HEAD -- &>/dev/null
 then
@@ -36,45 +45,26 @@ then
 else
     HEAD=$(git stash create)
 fi
-git clone --shared . "$vroot_dir"
-test $? -ne 0 && error "failed to clone into '$vroot_dir'"
-for sm in $submodules; do
-    if test -d "$sm/.git"
-    then
-       git clone --shared "$sm" "$vroot_dir/$sm"
-       test $? -ne 0 && error "failed to clone submodule $sm"
-    fi
-done
-
-cd "$vroot_dir"
-test $? -ne 0 && error "failed to change into '$vroot_dir'"
-
-git checkout $HEAD
-test $? -ne 0 && error "failed to checkout $HEAD revision"
 
+git archive --format tar $HEAD > "$tar_file"
+test $? -ne 0 && error "failed to archive qemu"
 for sm in $submodules; do
-    git submodule update --init $sm
-    test $? -ne 0 && error "failed to init submodule $sm"
+    status="$(git submodule status "$sm")"
+    smhash="${status#[ +-]}"
+    smhash="${smhash%% *}"
+    case "$status" in
+	-*)
+	    sub_deinit="$sub_deinit $sm"
+	    git submodule update --init "$sm"
+	    test $? -ne 0 && error "failed to update submodule $sm"
+	    ;;
+	+*)
+	    echo "WARNING: submodule $sm is out of sync"
+	    ;;
+    esac
+    (cd $sm; git archive --format tar --prefix "$sm/" $smhash) > "$sub_file"
+    test $? -ne 0 && error "failed to archive submodule $sm ($smhash)"
+    tar --concatenate --file "$tar_file" "$sub_file"
+    test $? -ne 0 && error "failed append submodule $sm to $tar_file"
 done
-
-if test -n "$submodules"; then
-    {
-        git ls-files || error "git ls-files failed"
-        for sm in $submodules; do
-            (cd $sm; git ls-files) | sed "s:^:$sm/:"
-            if test "${PIPESTATUS[*]}" != "0 0"; then
-                error "git ls-files in submodule $sm failed"
-            fi
-        done
-    } | grep -x -v $(for sm in $submodules; do echo "-e $sm"; done) > "$list_file"
-else
-    git ls-files > "$list_file"
-fi
-
-if test $? -ne 0; then
-    error "failed to generate list file"
-fi
-
-tar -cf "$tar_file" -T "$list_file" || error "failed to create tar file"
-
 exit 0
-- 
2.18.1

[Qemu-devel] [PATCH 02/13] tests/vm: send proxy environment variables over ssh

[Gerd Hoffmann]

Packages are fetched via proxy that way, if configured on the host.
That might be required to pass firewalls, and it allows to route
package downloads through a caching proxy server.

Needs AcceptEnv setup in sshd_config on the guest side to work.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 tests/vm/basevm.py | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
index 0556bdcf9e9f..6b46674f4497 100755
--- a/tests/vm/basevm.py
+++ b/tests/vm/basevm.py
@@ -38,6 +38,13 @@ class BaseVM(object):
     GUEST_PASS = "qemupass"
     ROOT_PASS = "qemupass"
 
+    envvars = [
+        "https_proxy",
+        "http_proxy",
+        "ftp_proxy",
+        "no_proxy",
+    ]
+
     # The script to run in the guest that builds QEMU
     BUILD_SCRIPT = ""
     # The guest name, to be overridden by subclasses
@@ -105,6 +112,8 @@ class BaseVM(object):
                    "-o", "UserKnownHostsFile=" + os.devnull,
                    "-o", "ConnectTimeout=1",
                    "-p", self.ssh_port, "-i", self._ssh_key_file]
+        for var in self.envvars:
+            ssh_cmd += ['-o', "SendEnv=%s" % var ]
         if interactive:
             ssh_cmd += ['-t']
         assert not isinstance(cmd, str)
-- 
2.18.1

[Qemu-devel] [PATCH 03/13] tests/vm: send locale environment variables over ssh

[Gerd Hoffmann]

Needed for unicode tests.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 tests/vm/basevm.py | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
index 6b46674f4497..20eec5420dbf 100755
--- a/tests/vm/basevm.py
+++ b/tests/vm/basevm.py
@@ -43,6 +43,20 @@ class BaseVM(object):
         "http_proxy",
         "ftp_proxy",
         "no_proxy",
+        "LANG",
+        "LC_CTYPE",
+        "LC_NUMERIC",
+        "LC_TIME",
+        "LC_COLLATE",
+        "LC_MONETARY",
+        "LC_MESSAGES",
+        "LC_PAPER",
+        "LC_NAME",
+        "LC_ADDRESS",
+        "LC_TELEPHONE",
+        "LC_MEASUREMENT",
+        "LC_IDENTIFICATION",
+        "LC_ALL",
     ]
 
     # The script to run in the guest that builds QEMU
-- 
2.18.1

[Qemu-devel] [PATCH 04/13] tests/vm: use ssh with pty unconditionally

[Gerd Hoffmann]

Allways ask ssh to run with a pseudo terminal.
Not having a terminal causes problems now and then.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 tests/vm/basevm.py | 13 ++++---------
 1 file changed, 4 insertions(+), 9 deletions(-)

diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
index 20eec5420dbf..9c6bb317ac89 100755
--- a/tests/vm/basevm.py
+++ b/tests/vm/basevm.py
@@ -120,16 +120,14 @@ class BaseVM(object):
         os.rename(fname + ".download", fname)
         return fname
 
-    def _ssh_do(self, user, cmd, check, interactive=False):
-        ssh_cmd = ["ssh", "-q",
+    def _ssh_do(self, user, cmd, check):
+        ssh_cmd = ["ssh", "-q", "-t",
                    "-o", "StrictHostKeyChecking=no",
                    "-o", "UserKnownHostsFile=" + os.devnull,
                    "-o", "ConnectTimeout=1",
                    "-p", self.ssh_port, "-i", self._ssh_key_file]
         for var in self.envvars:
             ssh_cmd += ['-o', "SendEnv=%s" % var ]
-        if interactive:
-            ssh_cmd += ['-t']
         assert not isinstance(cmd, str)
         ssh_cmd += ["%s@127.0.0.1" % user] + list(cmd)
         logging.debug("ssh_cmd: %s", " ".join(ssh_cmd))
@@ -141,9 +139,6 @@ class BaseVM(object):
     def ssh(self, *cmd):
         return self._ssh_do(self.GUEST_USER, cmd, False)
 
-    def ssh_interactive(self, *cmd):
-        return self._ssh_do(self.GUEST_USER, cmd, False, True)
-
     def ssh_root(self, *cmd):
         return self._ssh_do("root", cmd, False)
 
@@ -297,9 +292,9 @@ def main(vmcls):
         return 2
 
     if args.interactive:
-        if vm.ssh_interactive(*cmd) == 0:
+        if vm.ssh(*cmd) == 0:
             return 0
-        vm.ssh_interactive()
+        vm.ssh()
         return 3
     else:
         if vm.ssh(*cmd) != 0:
-- 
2.18.1

[Qemu-devel] [PATCH 05/13] tests/vm: run test builds on snapshot

[Gerd Hoffmann]

The build script doesn't shutdown the guest VMs properly,
which results in filesystem corruption and guest boot
failures sooner or later.

Use the --snapshot to run builds on a snapshot,
That way killing the VM doesn't corrupt the base image.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 tests/vm/Makefile.include | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include
index 992d823f6bd2..d628e0a986f6 100644
--- a/tests/vm/Makefile.include
+++ b/tests/vm/Makefile.include
@@ -53,6 +53,7 @@ vm-build-%: $(IMAGES_DIR)/%.img
 		$(if $(V),--verbose) \
 		--image "$<" \
 		$(if $(BUILD_TARGET),--build-target $(BUILD_TARGET)) \
+		--snapshot \
 		--build-qemu $(SRC_PATH) -- \
 		$(if $(TARGET_LIST),--target-list=$(TARGET_LIST)) \
 		$(if $(EXTRA_CONFIGURE_OPTS),$(EXTRA_CONFIGURE_OPTS)), \
-- 
2.18.1

[Qemu-devel] [PATCH 06/13] tests/vm: add vm-boot-{ssh, serial}-<guest> targets

[Gerd Hoffmann]

For testing/troubleshooting convinience.

make vm-boot-serial-<guest>
  Boot guest, with the serial console on stdio.

make vm-boot-ssh-<guest>
  Boot guest, login via ssh.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 tests/vm/Makefile.include | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include
index d628e0a986f6..47084d5717c6 100644
--- a/tests/vm/Makefile.include
+++ b/tests/vm/Makefile.include
@@ -20,6 +20,10 @@ vm-test:
 	@echo "  vm-build-all                    - Build QEMU in all VMs"
 	@echo "  vm-clean-all                    - Clean up VM images"
 	@echo
+	@echo "For trouble-shooting:"
+	@echo "  vm-boot-serial-<guest>          - Boot guest, serial console on stdio"
+	@echo "  vm-boot-ssh-<guest>             - Boot guest and login via ssh"
+	@echo
 	@echo "Special variables:"
 	@echo "    BUILD_TARGET=foo		 - override the build target"
 	@echo "    TARGET_LIST=a,b,c    	 - Override target list in builds."
@@ -59,3 +63,18 @@ vm-build-%: $(IMAGES_DIR)/%.img
 		$(if $(EXTRA_CONFIGURE_OPTS),$(EXTRA_CONFIGURE_OPTS)), \
 		"  VM-BUILD $*")
 
+vm-boot-serial-%: $(IMAGES_DIR)/%.img
+	qemu-system-x86_64 -enable-kvm -m 4G -smp 2 -nographic \
+		-drive if=none,id=vblk,cache=writeback,file="$<" \
+		-netdev user,id=vnet \
+		-device virtio-blk-pci,drive=vblk \
+		-device virtio-net-pci,netdev=vnet \
+	|| true
+
+vm-boot-ssh-%: $(IMAGES_DIR)/%.img
+	$(call quiet-command, \
+		$(SRC_PATH)/tests/vm/$* \
+		--image "$<" \
+		--interactive \
+		false, \
+		"  VM-BOOT-SSH $*")
-- 
2.18.1

[Qemu-devel] [PATCH 07/13] tests/vm: add DEBUG=1 to help text

[Gerd Hoffmann]

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 tests/vm/Makefile.include | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include
index 47084d5717c6..8714b5947958 100644
--- a/tests/vm/Makefile.include
+++ b/tests/vm/Makefile.include
@@ -25,6 +25,8 @@ vm-test:
 	@echo "  vm-boot-ssh-<guest>             - Boot guest and login via ssh"
 	@echo
 	@echo "Special variables:"
+	@echo "    DEBUG=1			 - be verbose, also start interactive"
+	@echo "					   shell on build failures"
 	@echo "    BUILD_TARGET=foo		 - override the build target"
 	@echo "    TARGET_LIST=a,b,c    	 - Override target list in builds."
 	@echo '    EXTRA_CONFIGURE_OPTS="..."'
-- 
2.18.1

[Qemu-devel] [PATCH 08/13] tests/vm: serial console support helpers

[Gerd Hoffmann]

Add a bunch of helpers to talk to the guest using the
serial console.

Also drop the hard-coded -serial parameter for the vm
so QEMUMachine.set_console() actually works.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 tests/vm/basevm.py | 82 ++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 80 insertions(+), 2 deletions(-)

diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
index 9c6bb317ac89..a27d2c72f5f5 100755
--- a/tests/vm/basevm.py
+++ b/tests/vm/basevm.py
@@ -13,7 +13,9 @@
 
 from __future__ import print_function
 import os
+import re
 import sys
+import socket
 import logging
 import time
 import datetime
@@ -91,8 +93,7 @@ class BaseVM(object):
             "-cpu", "max",
             "-netdev", "user,id=vnet,hostfwd=:127.0.0.1:0-:22",
             "-device", "virtio-net-pci,netdev=vnet",
-            "-vnc", "127.0.0.1:0,to=20",
-            "-serial", "file:%s" % os.path.join(self._tmpdir, "serial.out")]
+            "-vnc", "127.0.0.1:0,to=20"]
         if vcpus and vcpus > 1:
             self._args += ["-smp", str(vcpus)]
         if kvm_available(self.arch):
@@ -173,6 +174,8 @@ class BaseVM(object):
         logging.debug("QEMU args: %s", " ".join(args))
         qemu_bin = os.environ.get("QEMU", "qemu-system-" + self.arch)
         guest = QEMUMachine(binary=qemu_bin, args=args)
+        guest.set_machine('pc')
+        guest.set_console()
         try:
             guest.launch()
         except:
@@ -195,6 +198,81 @@ class BaseVM(object):
             raise Exception("Cannot find ssh port from 'info usernet':\n%s" % \
                             usernet_info)
 
+    def console_init(self, timeout = 120):
+        vm = self._guest
+        vm.console_socket.settimeout(timeout)
+
+    def console_log(self, text):
+        for line in re.split("[\r\n]", text):
+            # filter out terminal escape sequences
+            line = re.sub("\x1b\[[0-9;?]*[a-zA-Z]", "", line)
+            line = re.sub("\x1b\([0-9;?]*[a-zA-Z]", "", line)
+            # replace unprintable chars
+            line = re.sub("\x1b", "<esc>", line)
+            line = re.sub("[\x00-\x1f]", ".", line)
+            if line == "":
+                continue
+            # log console line
+            sys.stderr.write("con recv: %s\n" % line)
+
+    def console_wait(self, expect):
+        vm = self._guest
+        output = ""
+        while True:
+            try:
+                chars = vm.console_socket.recv(1024)
+            except socket.timeout:
+                sys.stderr.write("console: *** read timeout ***\n")
+                sys.stderr.write("console: waiting for: '%s'\n" % expect)
+                sys.stderr.write("console: line buffer:\n")
+                sys.stderr.write("\n")
+                self.console_log(output.rstrip())
+                sys.stderr.write("\n")
+                raise
+            output += chars
+            if expect in output:
+                break
+            if "\r" in output or "\n" in output:
+                lines = re.split("[\r\n]", output)
+                output = lines.pop()
+                if self.debug:
+                    self.console_log("\n".join(lines))
+        if self.debug:
+            self.console_log(output)
+
+    def console_send(self, command):
+        vm = self._guest
+        if self.debug:
+            logline = re.sub("\n", "<enter>", command)
+            logline = re.sub("[\x00-\x1f]", ".", logline)
+            sys.stderr.write("con send: %s\n" % logline)
+        for char in list(command):
+            vm.console_socket.send(char)
+            time.sleep(0.01)
+
+    def console_wait_send(self, wait, command):
+        self.console_wait(wait)
+        self.console_send(command)
+
+    def console_ssh_init(self, prompt, user, pw):
+        sshkey_cmd = "echo '%s' > .ssh/authorized_keys\n" % SSH_PUB_KEY.rstrip()
+        self.console_wait_send("login:",    "%s\n" % user)
+        self.console_wait_send("Password:", "%s\n" % pw)
+        self.console_wait_send(prompt,      "mkdir .ssh\n")
+        self.console_wait_send(prompt,      sshkey_cmd)
+        self.console_wait_send(prompt,      "chmod 755 .ssh\n")
+        self.console_wait_send(prompt,      "chmod 644 .ssh/authorized_keys\n")
+
+    def console_sshd_config(self, prompt):
+        self.console_wait(prompt)
+        self.console_send("echo 'PermitRootLogin yes' >> /etc/ssh/sshd_config\n")
+        for var in self.envvars:
+            self.console_wait(prompt)
+            self.console_send("echo 'AcceptEnv %s' >> /etc/ssh/sshd_config\n" % var)
+
+    def print_step(self, text):
+        sys.stderr.write("### %s ...\n" % text)
+
     def wait_ssh(self, seconds=300):
         starttime = datetime.datetime.now()
         endtime = starttime + datetime.timedelta(seconds=seconds)
-- 
2.18.1

[Qemu-devel] [PATCH 09/13] tests/vm: openbsd autoinstall, using serial console

[Gerd Hoffmann]

Instead of fetching the prebuilt image from patchew download the install
iso and prepare the image locally.  Install to disk, using the serial
console.  Create qemu user, configure ssh login.  Install packages
needed for qemu builds.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 tests/vm/openbsd | 150 +++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 138 insertions(+), 12 deletions(-)

diff --git a/tests/vm/openbsd b/tests/vm/openbsd
index 2105c01a267a..02f07026debe 100755
--- a/tests/vm/openbsd
+++ b/tests/vm/openbsd
@@ -13,34 +13,160 @@
 
 import os
 import sys
+import socket
 import subprocess
 import basevm
 
 class OpenBSDVM(basevm.BaseVM):
     name = "openbsd"
     arch = "x86_64"
+
+    link = "https://cdn.openbsd.org/pub/OpenBSD/6.5/amd64/install65.iso"
+    csum = "38d1f8cadd502f1c27bf05c5abde6cc505dd28f3f34f8a941048ff9a54f9f608"
+    size = "20G"
+    pkgs = [
+        # tools
+        "git",
+        "pkgconf",
+        "bzip2", "xz",
+
+        # gnu tools
+        "bash",
+        "gmake",
+        "gsed",
+        "bison",
+
+        # libs: usb
+        "libusb1",
+
+        # libs: crypto
+        "gnutls",
+
+        # libs: images
+        "jpeg",
+        "png",
+
+	# libs: ui
+        "sdl2",
+        "gtk+3",
+        "libxkbcommon",
+    ]
+
     BUILD_SCRIPT = """
         set -e;
-        rm -rf /var/tmp/qemu-test.*
-        cd $(mktemp -d /var/tmp/qemu-test.XXXXXX);
+        rm -rf /home/qemu/qemu-test.*
+        cd $(mktemp -d /home/qemu/qemu-test.XXXXXX);
+        mkdir src build; cd src;
         tar -xf /dev/rsd1c;
-        ./configure --cc=x86_64-unknown-openbsd6.1-gcc-4.9.4 --python=python2.7 {configure_opts};
-        gmake --output-sync -j{jobs} {verbose};
-        # XXX: "gmake check" seems to always hang or fail
-        #gmake --output-sync -j{jobs} check {verbose};
+        cd ../build
+        ../src/configure --cc=cc --python=python3 {configure_opts};
+        gmake --output-sync -j{jobs} {target} {verbose};
     """
 
     def build_image(self, img):
-        cimg = self._download_with_cache("http://download.patchew.org/openbsd-6.1-amd64.img.xz",
-                sha256sum='8c6cedc483e602cfee5e04f0406c64eb99138495e8ca580bc0293bcf0640c1bf')
-        img_tmp_xz = img + ".tmp.xz"
+        self.print_step("Downloading install iso")
+        cimg = self._download_with_cache(self.link, sha256sum=self.csum)
         img_tmp = img + ".tmp"
-        sys.stderr.write("Extracting the image...\n")
-        subprocess.check_call(["cp", "-f", cimg, img_tmp_xz])
-        subprocess.check_call(["xz", "-dvf", img_tmp_xz])
+        iso = img + ".install.iso"
+
+        self.print_step("Preparing iso and disk image")
+        subprocess.check_call(["cp", "-f", cimg, iso])
+        subprocess.check_call(["qemu-img", "create", "-f", "qcow2",
+                               img_tmp, self.size])
+
+        self.print_step("Booting installer")
+        self.boot(img_tmp, extra_args = [
+            "-machine", "graphics=off",
+            "-cdrom", iso
+        ])
+        self.console_init()
+        self.console_wait_send("boot>", "set tty com0\n")
+        self.console_wait_send("boot>", "\n")
+
+        # pre-install configuration
+        self.console_wait_send("(I)nstall",               "i\n")
+        self.console_wait_send("Terminal type",           "xterm\n")
+        self.console_wait_send("System hostname",         "openbsd\n")
+        self.console_wait_send("Which network interface", "vio0\n")
+        self.console_wait_send("IPv4 address",            "dhcp\n")
+        self.console_wait_send("IPv6 address",            "none\n")
+        self.console_wait_send("Which network interface", "done\n")
+        self.console_wait_send("DNS domain name",         "localnet\n")
+        self.console_wait("Password for root account")
+        self.console_send("%s\n" % self.ROOT_PASS)
+        self.console_wait("Password for root account")
+        self.console_send("%s\n" % self.ROOT_PASS)
+        self.console_wait_send("Start sshd(8)",           "yes\n")
+        self.console_wait_send("X Window System",         "\n")
+        self.console_wait_send("xenodm",                  "\n")
+        self.console_wait_send("console to com0",         "\n")
+        self.console_wait_send("Which speed",             "\n")
+
+        self.console_wait("Setup a user")
+        self.console_send("%s\n" % self.GUEST_USER)
+        self.console_wait("Full name")
+        self.console_send("%s\n" % self.GUEST_USER)
+        self.console_wait("Password")
+        self.console_send("%s\n" % self.GUEST_PASS)
+        self.console_wait("Password")
+        self.console_send("%s\n" % self.GUEST_PASS)
+
+        self.console_wait_send("Allow root ssh login",    "yes\n")
+        self.console_wait_send("timezone",                "UTC\n")
+        self.console_wait_send("root disk",               "\n")
+        self.console_wait_send("(W)hole disk",            "\n")
+        self.console_wait_send("(A)uto layout",           "\n")
+        self.console_wait_send("Location of sets",        "cd0\n")
+        self.console_wait_send("Pathname to the sets",    "\n")
+        self.console_wait_send("Set name(s)",             "\n")
+        self.console_wait_send("without verification",    "yes\n")
+
+        self.print_step("Installation started now, this will take a while")
+        self.console_wait_send("Location of sets",        "done\n")
+
+        self.console_wait("successfully completed")
+        self.print_step("Installation finished, rebooting")
+        self.console_wait_send("(R)eboot",                "reboot\n")
+
+        # setup qemu user
+        prompt = "$"
+        self.console_ssh_init(prompt, self.GUEST_USER, self.GUEST_PASS)
+        self.console_wait_send(prompt, "exit\n")
+
+        # setup root user
+        prompt = "openbsd#"
+        self.console_ssh_init(prompt, "root", self.ROOT_PASS)
+        self.console_sshd_config(prompt)
+
+        # setup virtio-blk #1 (tarfile)
+        self.console_wait(prompt)
+        self.console_send("echo 'chmod 666 /dev/rsd1c' >> /etc/rc.local\n")
+
+        # enable w+x for /home
+        self.console_wait(prompt)
+        self.console_send("sed -i -e '/home/s/rw,/rw,wxallowed,/' /etc/fstab\n")
+
+        # use http (be proxy cache friendly)
+        self.console_wait(prompt)
+        self.console_send("sed -i -e 's/https/http/' /etc/installurl\n")
+
+        self.print_step("Configuration finished, rebooting")
+        self.console_wait_send(prompt, "reboot\n")
+        self.console_wait("login:")
+        self.wait_ssh()
+
+        self.print_step("Installing packages")
+        self.ssh_root_check("pkg_add %s\n" % " ".join(self.pkgs))
+
+        # shutdown
+        self.ssh_root("halt -p")
+        self.wait()
+
         if os.path.exists(img):
             os.remove(img)
         os.rename(img_tmp, img)
+        os.remove(iso)
+        self.print_step("All done")
 
 if __name__ == "__main__":
     sys.exit(basevm.main(OpenBSDVM))
-- 
2.18.1

[Qemu-devel] [PATCH 10/13] tests/vm: freebsd autoinstall, using serial console

[Gerd Hoffmann]

Instead of fetching the prebuilt image from patchew download the install
iso and prepare the image locally.  Install to disk, using the serial
console.  Create qemu user, configure ssh login.  Install packages
needed for qemu builds.

Note that freebsd package downloads are delivered as non-cachable
content, so I had to configure squid with "ignore-no-store
ignore-private ignore-reload" for pkgmir.geo.freebsd.org to make the
caching actually work.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 tests/vm/freebsd | 172 ++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 163 insertions(+), 9 deletions(-)

diff --git a/tests/vm/freebsd b/tests/vm/freebsd
index b0066017a617..78e6b673f6f8 100755
--- a/tests/vm/freebsd
+++ b/tests/vm/freebsd
@@ -12,33 +12,187 @@
 #
 
 import os
+import re
 import sys
+import time
+import socket
 import subprocess
 import basevm
 
 class FreeBSDVM(basevm.BaseVM):
     name = "freebsd"
     arch = "x86_64"
+
+    link = "https://download.freebsd.org/ftp/releases/ISO-IMAGES/12.0/FreeBSD-12.0-RELEASE-amd64-disc1.iso.xz"
+    csum = "1d40015bea89d05b8bd13e2ed80c40b522a9ec1abd8e7c8b80954fb485fb99db"
+    size = "20G"
+    pkgs = [
+        # build tools
+        "git",
+        "pkgconf",
+        "bzip2",
+
+        # gnu tools
+        "bash",
+        "gmake",
+        "gsed",
+        "flex", "bison",
+
+        # libs: crypto
+        "gnutls",
+
+        # libs: images
+        "jpeg-turbo",
+        "png",
+
+	# libs: ui
+        "sdl2",
+        "gtk3",
+        "libxkbcommon",
+
+        # libs: opengl
+        "libepoxy",
+        "mesa-libs",
+    ]
+
     BUILD_SCRIPT = """
         set -e;
-        rm -rf /var/tmp/qemu-test.*
-        cd $(mktemp -d /var/tmp/qemu-test.XXXXXX);
+        rm -rf /home/qemu/qemu-test.*
+        cd $(mktemp -d /home/qemu/qemu-test.XXXXXX);
+        mkdir src build; cd src;
         tar -xf /dev/vtbd1;
-        ./configure {configure_opts};
+        cd ../build
+        ../src/configure --python=python3.6 {configure_opts};
         gmake --output-sync -j{jobs} {target} {verbose};
     """
 
+    def console_boot_serial(self):
+        self.console_wait_send("Autoboot", "3")
+        self.console_wait_send("OK", "set console=comconsole\n")
+        self.console_wait_send("OK", "boot\n")
+
     def build_image(self, img):
-        cimg = self._download_with_cache("http://download.patchew.org/freebsd-11.1-amd64.img.xz",
-                sha256sum='adcb771549b37bc63826c501f05121a206ed3d9f55f49145908f7e1432d65891')
-        img_tmp_xz = img + ".tmp.xz"
+        self.print_step("Downloading install iso")
+        cimg = self._download_with_cache(self.link, sha256sum=self.csum)
         img_tmp = img + ".tmp"
-        sys.stderr.write("Extracting the image...\n")
-        subprocess.check_call(["cp", "-f", cimg, img_tmp_xz])
-        subprocess.check_call(["xz", "-dvf", img_tmp_xz])
+        iso = img + ".install.iso"
+        iso_xz = iso + ".xz"
+
+        self.print_step("Preparing iso and disk image")
+        subprocess.check_call(["cp", "-f", cimg, iso_xz])
+        subprocess.check_call(["xz", "-dvf", iso_xz])
+        subprocess.check_call(["qemu-img", "create", "-f", "qcow2",
+                               img_tmp, self.size])
+
+        self.print_step("Booting installer")
+        self.boot(img_tmp, extra_args = [
+            "-machine", "graphics=off",
+            "-cdrom", iso
+        ])
+        self.console_init()
+	self.console_boot_serial()
+        self.console_wait_send("Console type",          "xterm\n")
+
+        # pre-install configuration
+        self.console_wait_send("Welcome",               "\n")
+        self.console_wait_send("Keymap Selection",      "\n")
+        self.console_wait_send("Set Hostname",          "freebsd\n")
+        self.console_wait_send("Distribution Select",   "\n")
+        self.console_wait_send("Partitioning",          "\n")
+        self.console_wait_send("Partition",             "\n")
+        self.console_wait_send("Scheme",                "\n")
+        self.console_wait_send("Editor",                "f")
+        self.console_wait_send("Confirmation",          "c")
+
+        self.print_step("Installation started now, this will take a while")
+
+        # post-install configuration
+        self.console_wait("New Password:")
+        self.console_send("%s\n" % self.ROOT_PASS)
+        self.console_wait("Retype New Password:")
+        self.console_send("%s\n" % self.ROOT_PASS)
+
+        self.console_wait_send("Network Configuration", "\n")
+        self.console_wait_send("IPv4",                  "y")
+        self.console_wait_send("DHCP",                  "y")
+        self.console_wait_send("IPv6",                  "n")
+        self.console_wait_send("Resolver",              "\n")
+
+        self.console_wait_send("Time Zone Selector",    "a\n")
+        self.console_wait_send("Confirmation",          "y")
+        self.console_wait_send("Time & Date",           "\n")
+        self.console_wait_send("Time & Date",           "\n")
+
+        self.console_wait_send("System Configuration",  "\n")
+        self.console_wait_send("System Hardening",      "\n")
+
+        # qemu user
+        self.console_wait_send("Add User Accounts", "y")
+        self.console_wait("Username")
+        self.console_send("%s\n" % self.GUEST_USER)
+        self.console_wait("Full name")
+        self.console_send("%s\n" % self.GUEST_USER)
+        self.console_wait_send("Uid",                   "\n")
+        self.console_wait_send("Login group",           "\n")
+        self.console_wait_send("Login group",           "\n")
+        self.console_wait_send("Login class",           "\n")
+        self.console_wait_send("Shell",                 "\n")
+        self.console_wait_send("Home directory",        "\n")
+        self.console_wait_send("Home directory perm",   "\n")
+        self.console_wait_send("Use password",          "\n")
+        self.console_wait_send("Use an empty password", "\n")
+        self.console_wait_send("Use a random password", "\n")
+        self.console_wait("Enter password:")
+        self.console_send("%s\n" % self.GUEST_PASS)
+        self.console_wait("Enter password again:")
+        self.console_send("%s\n" % self.GUEST_PASS)
+        self.console_wait_send("Lock out",              "\n")
+        self.console_wait_send("OK",                    "yes\n")
+        self.console_wait_send("Add another user",      "no\n")
+
+        self.console_wait_send("Final Configuration",   "\n")
+        self.console_wait_send("Manual Configuration",  "\n")
+        self.console_wait_send("Complete",              "\n")
+
+        self.print_step("Installation finished, rebooting")
+        self.console_boot_serial()
+
+        # setup qemu user
+        prompt = "$"
+        self.console_ssh_init(prompt, self.GUEST_USER, self.GUEST_PASS)
+        self.console_wait_send(prompt, "exit\n")
+
+        # setup root user
+        prompt = "root@freebsd:~ #"
+        self.console_ssh_init(prompt, "root", self.ROOT_PASS)
+        self.console_sshd_config(prompt)
+
+        # setup serial console
+        self.console_wait(prompt)
+        self.console_send("echo 'console=comconsole' >> /boot/loader.conf\n")
+
+        # setup virtio-blk #1 (tarfile)
+        self.console_wait(prompt)
+        self.console_send("echo 'chmod 666 /dev/vtbd1' >> /etc/rc.local\n")
+
+        self.print_step("Configuration finished, rebooting")
+        self.console_wait_send(prompt, "reboot\n")
+        self.console_wait("login:")
+        self.wait_ssh()
+
+        self.print_step("Installing packages")
+        self.ssh_root_check("pkg install -y %s\n" % " ".join(self.pkgs))
+
+        # shutdown
+        self.ssh_root("poweroff")
+        self.console_wait("Uptime:")
+        self.wait()
+
         if os.path.exists(img):
             os.remove(img)
         os.rename(img_tmp, img)
+        os.remove(iso)
+        self.print_step("All done")
 
 if __name__ == "__main__":
     sys.exit(basevm.main(FreeBSDVM))
-- 
2.18.1

[Qemu-devel] [PATCH 11/13] tests/vm: netbsd autoinstall, using serial console

[Gerd Hoffmann]

Instead of fetching the prebuilt image from patchew download the install
iso and prepare the image locally.  Install to disk, using the serial
console.  Create qemu user, configure ssh login.  Install packages
needed for qemu builds.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 tests/vm/netbsd | 178 +++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 169 insertions(+), 9 deletions(-)

diff --git a/tests/vm/netbsd b/tests/vm/netbsd
index 4c6624ea5ed5..eaf0ae21db42 100755
--- a/tests/vm/netbsd
+++ b/tests/vm/netbsd
@@ -13,32 +13,192 @@
 
 import os
 import sys
+import time
 import subprocess
 import basevm
 
 class NetBSDVM(basevm.BaseVM):
     name = "netbsd"
     arch = "x86_64"
+
+    link = "https://cdn.netbsd.org/pub/NetBSD/NetBSD-8.0/images/NetBSD-8.0-amd64.iso"
+    size = "20G"
+    pkgs = [
+        # tools
+        "git",
+        "pkgconf",
+        "bzip2", "xz",
+
+        # gnu tools
+        "bash",
+        "gmake",
+        "gsed",
+        "flex", "bison",
+
+        # libs: crypto
+        "gnutls",
+
+        # libs: images
+        "jpeg",
+        "png",
+
+	# libs: ui
+        "SDL2",
+        "gtk3+",
+        "libxkbcommon",
+    ]
+
     BUILD_SCRIPT = """
         set -e;
-        rm -rf /var/tmp/qemu-test.*
-        cd $(mktemp -d /var/tmp/qemu-test.XXXXXX);
+        rm -rf /home/qemu/qemu-test.*
+        cd $(mktemp -d /home/qemu/qemu-test.XXXXXX);
+        mkdir src build; cd src;
         tar -xf /dev/rld1a;
-        ./configure --python=python2.7 {configure_opts};
+        cd ../build
+        ../src/configure --python=python2.7 --disable-opengl {configure_opts};
         gmake --output-sync -j{jobs} {target} {verbose};
     """
 
     def build_image(self, img):
-        cimg = self._download_with_cache("http://download.patchew.org/netbsd-7.1-amd64.img.xz",
-                                         sha256sum='b633d565b0eac3d02015cd0c81440bd8a7a8df8512615ac1ee05d318be015732')
-        img_tmp_xz = img + ".tmp.xz"
+        cimg = self._download_with_cache(self.link)
         img_tmp = img + ".tmp"
-        sys.stderr.write("Extracting the image...\n")
-        subprocess.check_call(["cp", "-f", cimg, img_tmp_xz])
-        subprocess.check_call(["xz", "-dvf", img_tmp_xz])
+        iso = img + ".install.iso"
+
+        self.print_step("Preparing iso and disk image")
+        subprocess.check_call(["cp", "-f", cimg, iso])
+        subprocess.check_call(["qemu-img", "create", "-f", "qcow2",
+                               img_tmp, self.size])
+
+        self.print_step("Booting installer")
+        self.boot(img_tmp, extra_args = [
+            "-device", "VGA",
+            "-machine", "graphics=off",
+            "-cdrom", iso
+        ])
+        self.console_init()
+        self.console_wait("Primary Bootstrap")
+
+        # serial console boot menu output doesn't work for some
+        # reason, so we have to fly blind ...
+        for char in list("5consdev com0\n"):
+            time.sleep(0.2)
+            self.console_send(char)
+            self.console_wait("")
+        self.console_wait_send("> ", "boot\n")
+
+        self.console_wait_send("Terminal type",            "xterm\n")
+        self.console_wait_send("a: Installation messages", "a\n")
+        self.console_wait_send("b: US-English",            "b\n")
+        self.console_wait_send("a: Install NetBSD",        "a\n")
+        self.console_wait("Shall we continue?")
+        self.console_wait_send("b: Yes",                   "b\n")
+
+        self.console_wait_send("a: ld0",                   "a\n")
+        self.console_wait_send("a: This is the correct",   "a\n")
+        self.console_wait_send("b: Use the entire disk",   "b\n")
+        self.console_wait("NetBSD bootcode")
+        self.console_wait_send("a: Yes",                   "a\n")
+        self.console_wait_send("b: Use existing part",     "b\n")
+        self.console_wait_send("x: Partition sizes ok",    "x\n")
+        self.console_wait_send("for your NetBSD disk",     "\n")
+        self.console_wait("Shall we continue?")
+        self.console_wait_send("b: Yes",                   "b\n")
+
+        self.console_wait_send("b: Use serial port com0",  "b\n")
+        self.console_wait_send("f: Set serial baud rate",  "f\n")
+        self.console_wait_send("a: 9600",                  "a\n")
+        self.console_wait_send("x: Exit",                  "x\n")
+
+        self.console_wait_send("a: Full installation",     "a\n")
+        self.console_wait_send("a: CD-ROM",                "a\n")
+
+        self.print_step("Installation started now, this will take a while")
+        self.console_wait_send("Hit enter to continue",    "\n")
+
+        self.console_wait_send("d: Change root password",  "d\n")
+        self.console_wait_send("a: Yes",                   "a\n")
+        self.console_wait("New password:")
+        self.console_send("%s\n" % self.ROOT_PASS)
+        self.console_wait("New password:")
+        self.console_send("%s\n" % self.ROOT_PASS)
+        self.console_wait("Retype new password:")
+        self.console_send("%s\n" % self.ROOT_PASS)
+
+        self.console_wait_send("o: Add a user",            "o\n")
+        self.console_wait("username")
+        self.console_send("%s\n" % self.GUEST_USER)
+        self.console_wait("to group wheel")
+        self.console_wait_send("a: Yes",                   "a\n")
+        self.console_wait_send("a: /bin/sh",               "a\n")
+        self.console_wait("New password:")
+        self.console_send("%s\n" % self.GUEST_PASS)
+        self.console_wait("New password:")
+        self.console_send("%s\n" % self.GUEST_PASS)
+        self.console_wait("Retype new password:")
+        self.console_send("%s\n" % self.GUEST_PASS)
+
+        self.console_wait_send("a: Configure network",     "a\n")
+        self.console_wait_send("a: vioif0",                "a\n")
+        self.console_wait_send("Network media type",       "\n")
+        self.console_wait("autoconfiguration")
+        self.console_wait_send("a: Yes",                   "a\n")
+        self.console_wait_send("DNS domain",               "localnet\n")
+        self.console_wait("Are they OK?")
+        self.console_wait_send("a: Yes",                   "a\n")
+        self.console_wait("installed in /etc")
+        self.console_wait_send("a: Yes",                   "a\n")
+
+        self.console_wait_send("e: Enable install",        "e\n")
+        proxy = os.environ.get("http_proxy")
+        if not proxy is None:
+            self.console_wait_send("f: Proxy",             "f\n")
+            self.console_wait("Proxy")
+            self.console_send("%s\n" % proxy)
+        self.console_wait_send("x: Install pkgin",         "x\n")
+        self.console_init(1200)
+        self.console_wait_send("Hit enter to continue", "\n")
+        self.console_init()
+
+        self.console_wait_send("g: Enable sshd",           "g\n")
+        self.console_wait_send("x: Finished conf",         "x\n")
+        self.console_wait_send("Hit enter to continue",    "\n")
+
+        self.print_step("Installation finished, rebooting")
+        self.console_wait_send("d: Reboot the computer",   "d\n")
+
+        # setup qemu user
+        prompt = "localhost$"
+        self.console_ssh_init(prompt, self.GUEST_USER, self.GUEST_PASS)
+        self.console_wait_send(prompt, "exit\n")
+
+        # setup root user
+        prompt = "localhost#"
+        self.console_ssh_init(prompt, "root", self.ROOT_PASS)
+        self.console_sshd_config(prompt)
+
+        # setup virtio-blk #1 (tarfile)
+        self.console_wait(prompt)
+        self.console_send("echo 'chmod 666 /dev/rld1a' >> /etc/rc.local\n")
+
+        self.print_step("Configuration finished, rebooting")
+        self.console_wait_send(prompt, "reboot\n")
+        self.console_wait("login:")
+        self.wait_ssh()
+
+        self.print_step("Installing packages")
+        self.ssh_root_check("pkgin update\n")
+        self.ssh_root_check("pkgin -y install %s\n" % " ".join(self.pkgs))
+
+        # shutdown
+        self.ssh_root("/sbin/poweroff")
+        self.console_wait("entering state S5")
+        self.wait()
+
         if os.path.exists(img):
             os.remove(img)
         os.rename(img_tmp, img)
+        os.remove(iso)
+        self.print_step("All done")
 
 if __name__ == "__main__":
     sys.exit(basevm.main(NetBSDVM))
-- 
2.18.1

[Qemu-devel] [PATCH 12/13] tests/vm: fedora autoinstall, using serial console

[Gerd Hoffmann]

Download the install iso and prepare the image locally.  Install to
disk, using the serial console.  Create qemu user, configure ssh login.
Install packages needed for qemu builds.

Yes, we have docker images for fedora.  But for trouble-shooting it
might be helpful to have a vm too.  When vm builds fail you can use
it to figure whenever the vm setup or the guest os is the problem.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 tests/vm/basevm.py        |   9 +-
 tests/vm/Makefile.include |   3 +-
 tests/vm/fedora           | 187 ++++++++++++++++++++++++++++++++++++++
 3 files changed, 197 insertions(+), 2 deletions(-)
 create mode 100755 tests/vm/fedora

diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
index a27d2c72f5f5..ffd687578b5c 100755
--- a/tests/vm/basevm.py
+++ b/tests/vm/basevm.py
@@ -215,7 +215,7 @@ class BaseVM(object):
             # log console line
             sys.stderr.write("con recv: %s\n" % line)
 
-    def console_wait(self, expect):
+    def console_wait(self, expect, expectalt = None):
         vm = self._guest
         output = ""
         while True:
@@ -224,6 +224,8 @@ class BaseVM(object):
             except socket.timeout:
                 sys.stderr.write("console: *** read timeout ***\n")
                 sys.stderr.write("console: waiting for: '%s'\n" % expect)
+                if not expectalt is None:
+                    sys.stderr.write("console: waiting for: '%s' (alt)\n" % expectalt)
                 sys.stderr.write("console: line buffer:\n")
                 sys.stderr.write("\n")
                 self.console_log(output.rstrip())
@@ -232,6 +234,8 @@ class BaseVM(object):
             output += chars
             if expect in output:
                 break
+            if not expectalt is None and expectalt in output:
+                break
             if "\r" in output or "\n" in output:
                 lines = re.split("[\r\n]", output)
                 output = lines.pop()
@@ -239,6 +243,9 @@ class BaseVM(object):
                     self.console_log("\n".join(lines))
         if self.debug:
             self.console_log(output)
+        if not expectalt is None and expectalt in output:
+            return False
+        return True
 
     def console_send(self, command):
         vm = self._guest
diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include
index 8714b5947958..6e24ee786910 100644
--- a/tests/vm/Makefile.include
+++ b/tests/vm/Makefile.include
@@ -2,7 +2,7 @@
 
 .PHONY: vm-build-all vm-clean-all
 
-IMAGES := ubuntu.i386 freebsd netbsd openbsd centos
+IMAGES := ubuntu.i386 freebsd netbsd openbsd centos fedora
 IMAGES_DIR := $(HOME)/.cache/qemu-vm/images
 IMAGE_FILES := $(patsubst %, $(IMAGES_DIR)/%.img, $(IMAGES))
 
@@ -16,6 +16,7 @@ vm-test:
 	@echo "  vm-build-netbsd                 - Build QEMU in NetBSD VM"
 	@echo "  vm-build-openbsd                - Build QEMU in OpenBSD VM"
 	@echo "  vm-build-centos                 - Build QEMU in CentOS VM, with Docker"
+	@echo "  vm-build-fedora                 - Build QEMU in Fedora VM"
 	@echo ""
 	@echo "  vm-build-all                    - Build QEMU in all VMs"
 	@echo "  vm-clean-all                    - Clean up VM images"
diff --git a/tests/vm/fedora b/tests/vm/fedora
new file mode 100755
index 000000000000..d8cb62c803a4
--- /dev/null
+++ b/tests/vm/fedora
@@ -0,0 +1,187 @@
+#!/usr/bin/env python
+#
+# FreeBSD VM image
+#
+# Copyright 2017 Red Hat Inc.
+#
+# Authors:
+#  Fam Zheng <famz@redhat.com>
+#
+# This code is licensed under the GPL version 2 or later.  See
+# the COPYING file in the top-level directory.
+#
+
+import os
+import re
+import sys
+import time
+import socket
+import subprocess
+import basevm
+
+class FedoraVM(basevm.BaseVM):
+    name = "fedora"
+    arch = "x86_64"
+
+    base = "http://dl.fedoraproject.org/pub/fedora/linux/releases/30/"
+    link = base + "Server/x86_64/iso/Fedora-Server-netinst-x86_64-30-1.2.iso"
+    repo = base + "Server/x86_64/os/"
+    full = base + "Everything/x86_64/os/"
+    csum = "5e4eac4566d8c572bfb3bcf54b7d6c82006ec3c6c882a2c9235c6d3494d7b100"
+    size = "20G"
+    pkgs = [
+        # tools
+        'git',
+        'flex', 'bison',
+        'gcc', 'binutils', 'make',
+
+        # perl
+        'perl-Test-Harness',
+
+        # libs: usb
+        '"pkgconfig(libusb-1.0)"',
+        '"pkgconfig(libusbredirparser-0.5)"',
+
+        # libs: crypto
+        '"pkgconfig(gnutls)"',
+
+        # libs: ui
+        '"pkgconfig(sdl2)"',
+        '"pkgconfig(gtk+-3.0)"',
+        '"pkgconfig(ncursesw)"',
+
+        # libs: audio
+        '"pkgconfig(libpulse)"',
+        '"pkgconfig(alsa)"',
+    ]
+
+    BUILD_SCRIPT = """
+        set -e;
+        rm -rf /home/qemu/qemu-test.*
+        cd $(mktemp -d /home/qemu/qemu-test.XXXXXX);
+        mkdir src build; cd src;
+        tar -xf /dev/vdb;
+        cd ../build
+        ../src/configure --python=python3 {configure_opts};
+        gmake --output-sync -j{jobs} {target} {verbose};
+    """
+
+    def build_image(self, img):
+        self.print_step("Downloading install iso")
+        cimg = self._download_with_cache(self.link, sha256sum=self.csum)
+        img_tmp = img + ".tmp"
+        iso = img + ".install.iso"
+
+        self.print_step("Preparing iso and disk image")
+        subprocess.check_call(["cp", "-f", cimg, iso])
+        subprocess.check_call(["qemu-img", "create", "-f", "qcow2",
+                               img_tmp, self.size])
+
+        self.print_step("Booting installer")
+        self.boot(img_tmp, extra_args = [
+            "-machine", "graphics=off",
+            "-cdrom", iso
+        ])
+        self.console_init(300)
+        self.console_wait("installation process.")
+        time.sleep(0.3)
+        self.console_send("\t")
+        time.sleep(0.3)
+        self.console_send(" console=ttyS0")
+        proxy = os.environ.get("http_proxy")
+        if not proxy is None:
+            self.console_send(" proxy=%s" % proxy)
+        self.console_send(" inst.repo=%s" % self.repo)
+        self.console_send("\n")
+
+        self.console_wait_send("2) Use text mode",         "2\n")
+
+        self.console_wait_send("5) [!] Installation Dest", "5\n")
+        self.console_wait_send("1) [x]",                   "c\n")
+        self.console_wait_send("2) [ ] Use All Space",     "2\n")
+        self.console_wait_send("2) [x] Use All Space",     "c\n")
+        self.console_wait_send("1) [ ] Standard Part",     "1\n")
+        self.console_wait_send("1) [x] Standard Part",     "c\n")
+
+        self.console_wait_send("7) [!] Root password",     "7\n")
+        self.console_wait("Password:")
+        self.console_send("%s\n" % self.ROOT_PASS)
+        self.console_wait("Password (confirm):")
+        self.console_send("%s\n" % self.ROOT_PASS)
+
+        self.console_wait_send("8) [ ] User creation",     "8\n")
+        self.console_wait_send("1) [ ] Create user",       "1\n")
+        self.console_wait_send("3) User name",             "3\n")
+        self.console_wait_send("ENTER:", "%s\n" % self.GUEST_USER)
+        self.console_wait_send("4) [ ] Use password",      "4\n")
+        self.console_wait_send("5) Password",              "5\n")
+        self.console_wait("Password:")
+        self.console_send("%s\n" % self.GUEST_PASS)
+        self.console_wait("Password (confirm):")
+        self.console_send("%s\n" % self.GUEST_PASS)
+        self.console_wait_send("7) Groups",                "c\n")
+
+        while True:
+            good = self.console_wait("3) [x] Installation",
+                                     "3) [!] Installation")
+            self.console_send("r\n")
+            if good:
+                break
+            time.sleep(10)
+
+        while True:
+            good = self.console_wait("4) [x] Software",
+                                     "4) [!] Software")
+            self.console_send("r\n")
+            if good:
+                break
+            time.sleep(10)
+            self.console_send("r\n" % self.GUEST_PASS)
+
+        self.console_wait_send("'b' to begin install",     "b\n")
+
+        self.print_step("Installation started now, this will take a while")
+
+        self.console_wait_send("Installation complete",    "\n")
+        self.print_step("Installation finished, rebooting")
+
+        # setup qemu user
+        prompt = " ~]$"
+        self.console_ssh_init(prompt, self.GUEST_USER, self.GUEST_PASS)
+        self.console_wait_send(prompt, "exit\n")
+
+        # setup root user
+        prompt = " ~]#"
+        self.console_ssh_init(prompt, "root", self.ROOT_PASS)
+        self.console_sshd_config(prompt)
+
+        # setup virtio-blk #1 (tarfile)
+        self.console_wait(prompt)
+        self.console_send("echo 'KERNEL==\"vdb\" MODE=\"666\"' >> %s\n" %
+                          "/etc/udev/rules.d/99-qemu.rules")
+
+        self.print_step("Configuration finished, rebooting")
+        self.console_wait_send(prompt, "reboot\n")
+        self.console_wait("login:")
+        self.wait_ssh()
+
+        self.print_step("Installing packages")
+        self.ssh_root_check("rm -vf /etc/yum.repos.d/fedora*.repo\n")
+        self.ssh_root_check("echo '[fedora]' >> /etc/yum.repos.d/qemu.repo\n")
+        self.ssh_root_check("echo 'baseurl=%s' >> /etc/yum.repos.d/qemu.repo\n" % self.full)
+        self.ssh_root_check("echo 'gpgcheck=0' >> /etc/yum.repos.d/qemu.repo\n")
+        self.ssh_root_check("dnf install -y %s\n" % " ".join(self.pkgs))
+
+        # shutdown
+        self.ssh_root("poweroff")
+        self.console_wait("sleep state S5")
+        self.wait()
+
+        if os.path.exists(img):
+            os.remove(img)
+        os.rename(img_tmp, img)
+        os.remove(iso)
+        self.print_step("All done")
+
+if __name__ == "__main__":
+    sys.exit(basevm.main(FedoraVM))
-- 
2.18.1

[Qemu-devel] [PATCH 13/13] tests/vm: ubuntu.i386: apt proxy setup

[Gerd Hoffmann]

Configure apt proxy so package downloads
can be cached and can pass firewalls.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 tests/vm/ubuntu.i386 | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/tests/vm/ubuntu.i386 b/tests/vm/ubuntu.i386
index a22d137e76df..b869afd212fa 100755
--- a/tests/vm/ubuntu.i386
+++ b/tests/vm/ubuntu.i386
@@ -51,6 +51,10 @@ class UbuntuX86VM(basevm.BaseVM):
                           "    ssh-authorized-keys:\n",
                           "    - %s\n" % basevm.SSH_PUB_KEY,
                           "locale: en_US.UTF-8\n"])
+        proxy = os.environ.get("http_proxy")
+        if not proxy is None:
+            udata.writelines(["apt:\n",
+                              "  proxy: %s" % proxy])
         udata.close()
         subprocess.check_call(["genisoimage", "-output", "cloud-init.iso",
                                "-volid", "cidata", "-joliet", "-rock",
-- 
2.18.1

Re: [Qemu-devel] [PATCH 11/13] tests/vm: netbsd autoinstall, using serial console

[Kamil Rytarowski]

[-- Attachment #1: Type: text/plain, Size: 9434 bytes --]

On 08.05.2019 10:56, Gerd Hoffmann wrote:
> Instead of fetching the prebuilt image from patchew download the install
> iso and prepare the image locally.  Install to disk, using the serial
> console.  Create qemu user, configure ssh login.  Install packages
> needed for qemu builds.
> 

I recommend to add one extra step into generated image:

echo security.pax.mprotect.enabled=0 >> /etc/sysctl.conf

Alternatively (and preferably) enhance qemu to handle RWX allocation for
JIT on NetBSD.

Example in libffi.

https://github.com/libffi/libffi/commit/2bfcd29955c02b67fa10a68cc4200f6838181e0f

> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
> ---
>  tests/vm/netbsd | 178 +++++++++++++++++++++++++++++++++++++++++++++---
>  1 file changed, 169 insertions(+), 9 deletions(-)
> 
> diff --git a/tests/vm/netbsd b/tests/vm/netbsd
> index 4c6624ea5ed5..eaf0ae21db42 100755
> --- a/tests/vm/netbsd
> +++ b/tests/vm/netbsd
> @@ -13,32 +13,192 @@
>  
>  import os
>  import sys
> +import time
>  import subprocess
>  import basevm
>  
>  class NetBSDVM(basevm.BaseVM):
>      name = "netbsd"
>      arch = "x86_64"
> +
> +    link = "https://cdn.netbsd.org/pub/NetBSD/NetBSD-8.0/images/NetBSD-8.0-amd64.iso"
> +    size = "20G"
> +    pkgs = [
> +        # tools
> +        "git",
> +        "pkgconf",
> +        "bzip2", "xz",
> +
> +        # gnu tools
> +        "bash",
> +        "gmake",
> +        "gsed",
> +        "flex", "bison",
> +
> +        # libs: crypto
> +        "gnutls",
> +
> +        # libs: images
> +        "jpeg",
> +        "png",
> +
> +	# libs: ui
> +        "SDL2",
> +        "gtk3+",
> +        "libxkbcommon",
> +    ]
> +
>      BUILD_SCRIPT = """
>          set -e;
> -        rm -rf /var/tmp/qemu-test.*
> -        cd $(mktemp -d /var/tmp/qemu-test.XXXXXX);
> +        rm -rf /home/qemu/qemu-test.*
> +        cd $(mktemp -d /home/qemu/qemu-test.XXXXXX);
> +        mkdir src build; cd src;
>          tar -xf /dev/rld1a;
> -        ./configure --python=python2.7 {configure_opts};
> +        cd ../build
> +        ../src/configure --python=python2.7 --disable-opengl {configure_opts};
>          gmake --output-sync -j{jobs} {target} {verbose};
>      """
>  
>      def build_image(self, img):
> -        cimg = self._download_with_cache("http://download.patchew.org/netbsd-7.1-amd64.img.xz",
> -                                         sha256sum='b633d565b0eac3d02015cd0c81440bd8a7a8df8512615ac1ee05d318be015732')
> -        img_tmp_xz = img + ".tmp.xz"
> +        cimg = self._download_with_cache(self.link)
>          img_tmp = img + ".tmp"
> -        sys.stderr.write("Extracting the image...\n")
> -        subprocess.check_call(["cp", "-f", cimg, img_tmp_xz])
> -        subprocess.check_call(["xz", "-dvf", img_tmp_xz])
> +        iso = img + ".install.iso"
> +
> +        self.print_step("Preparing iso and disk image")
> +        subprocess.check_call(["cp", "-f", cimg, iso])
> +        subprocess.check_call(["qemu-img", "create", "-f", "qcow2",
> +                               img_tmp, self.size])
> +
> +        self.print_step("Booting installer")
> +        self.boot(img_tmp, extra_args = [
> +            "-device", "VGA",
> +            "-machine", "graphics=off",
> +            "-cdrom", iso
> +        ])
> +        self.console_init()
> +        self.console_wait("Primary Bootstrap")
> +
> +        # serial console boot menu output doesn't work for some
> +        # reason, so we have to fly blind ...
> +        for char in list("5consdev com0\n"):
> +            time.sleep(0.2)
> +            self.console_send(char)
> +            self.console_wait("")
> +        self.console_wait_send("> ", "boot\n")
> +
> +        self.console_wait_send("Terminal type",            "xterm\n")
> +        self.console_wait_send("a: Installation messages", "a\n")
> +        self.console_wait_send("b: US-English",            "b\n")
> +        self.console_wait_send("a: Install NetBSD",        "a\n")
> +        self.console_wait("Shall we continue?")
> +        self.console_wait_send("b: Yes",                   "b\n")
> +
> +        self.console_wait_send("a: ld0",                   "a\n")
> +        self.console_wait_send("a: This is the correct",   "a\n")
> +        self.console_wait_send("b: Use the entire disk",   "b\n")
> +        self.console_wait("NetBSD bootcode")
> +        self.console_wait_send("a: Yes",                   "a\n")
> +        self.console_wait_send("b: Use existing part",     "b\n")
> +        self.console_wait_send("x: Partition sizes ok",    "x\n")
> +        self.console_wait_send("for your NetBSD disk",     "\n")
> +        self.console_wait("Shall we continue?")
> +        self.console_wait_send("b: Yes",                   "b\n")
> +
> +        self.console_wait_send("b: Use serial port com0",  "b\n")
> +        self.console_wait_send("f: Set serial baud rate",  "f\n")
> +        self.console_wait_send("a: 9600",                  "a\n")
> +        self.console_wait_send("x: Exit",                  "x\n")
> +
> +        self.console_wait_send("a: Full installation",     "a\n")
> +        self.console_wait_send("a: CD-ROM",                "a\n")
> +
> +        self.print_step("Installation started now, this will take a while")
> +        self.console_wait_send("Hit enter to continue",    "\n")
> +
> +        self.console_wait_send("d: Change root password",  "d\n")
> +        self.console_wait_send("a: Yes",                   "a\n")
> +        self.console_wait("New password:")
> +        self.console_send("%s\n" % self.ROOT_PASS)
> +        self.console_wait("New password:")
> +        self.console_send("%s\n" % self.ROOT_PASS)
> +        self.console_wait("Retype new password:")
> +        self.console_send("%s\n" % self.ROOT_PASS)
> +
> +        self.console_wait_send("o: Add a user",            "o\n")
> +        self.console_wait("username")
> +        self.console_send("%s\n" % self.GUEST_USER)
> +        self.console_wait("to group wheel")
> +        self.console_wait_send("a: Yes",                   "a\n")
> +        self.console_wait_send("a: /bin/sh",               "a\n")
> +        self.console_wait("New password:")
> +        self.console_send("%s\n" % self.GUEST_PASS)
> +        self.console_wait("New password:")
> +        self.console_send("%s\n" % self.GUEST_PASS)
> +        self.console_wait("Retype new password:")
> +        self.console_send("%s\n" % self.GUEST_PASS)
> +
> +        self.console_wait_send("a: Configure network",     "a\n")
> +        self.console_wait_send("a: vioif0",                "a\n")
> +        self.console_wait_send("Network media type",       "\n")
> +        self.console_wait("autoconfiguration")
> +        self.console_wait_send("a: Yes",                   "a\n")
> +        self.console_wait_send("DNS domain",               "localnet\n")
> +        self.console_wait("Are they OK?")
> +        self.console_wait_send("a: Yes",                   "a\n")
> +        self.console_wait("installed in /etc")
> +        self.console_wait_send("a: Yes",                   "a\n")
> +
> +        self.console_wait_send("e: Enable install",        "e\n")
> +        proxy = os.environ.get("http_proxy")
> +        if not proxy is None:
> +            self.console_wait_send("f: Proxy",             "f\n")
> +            self.console_wait("Proxy")
> +            self.console_send("%s\n" % proxy)
> +        self.console_wait_send("x: Install pkgin",         "x\n")
> +        self.console_init(1200)
> +        self.console_wait_send("Hit enter to continue", "\n")
> +        self.console_init()
> +
> +        self.console_wait_send("g: Enable sshd",           "g\n")
> +        self.console_wait_send("x: Finished conf",         "x\n")
> +        self.console_wait_send("Hit enter to continue",    "\n")
> +
> +        self.print_step("Installation finished, rebooting")
> +        self.console_wait_send("d: Reboot the computer",   "d\n")
> +
> +        # setup qemu user
> +        prompt = "localhost$"
> +        self.console_ssh_init(prompt, self.GUEST_USER, self.GUEST_PASS)
> +        self.console_wait_send(prompt, "exit\n")
> +
> +        # setup root user
> +        prompt = "localhost#"
> +        self.console_ssh_init(prompt, "root", self.ROOT_PASS)
> +        self.console_sshd_config(prompt)
> +
> +        # setup virtio-blk #1 (tarfile)
> +        self.console_wait(prompt)
> +        self.console_send("echo 'chmod 666 /dev/rld1a' >> /etc/rc.local\n")
> +
> +        self.print_step("Configuration finished, rebooting")
> +        self.console_wait_send(prompt, "reboot\n")
> +        self.console_wait("login:")
> +        self.wait_ssh()
> +
> +        self.print_step("Installing packages")
> +        self.ssh_root_check("pkgin update\n")
> +        self.ssh_root_check("pkgin -y install %s\n" % " ".join(self.pkgs))
> +
> +        # shutdown
> +        self.ssh_root("/sbin/poweroff")
> +        self.console_wait("entering state S5")
> +        self.wait()
> +
>          if os.path.exists(img):
>              os.remove(img)
>          os.rename(img_tmp, img)
> +        os.remove(iso)
> +        self.print_step("All done")
>  
>  if __name__ == "__main__":
>      sys.exit(basevm.main(NetBSDVM))
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 850 bytes --]

Re: [Qemu-devel] [PATCH 11/13] tests/vm: netbsd autoinstall, using serial console

[Gerd Hoffmann]

  Hi,

> I recommend to add one extra step into generated image:
> 
> echo security.pax.mprotect.enabled=0 >> /etc/sysctl.conf

Done.

> Alternatively (and preferably) enhance qemu to handle RWX allocation for
> JIT on NetBSD.
> 
> Example in libffi.
> 
> https://github.com/libffi/libffi/commit/2bfcd29955c02b67fa10a68cc4200f6838181e0f

So, from looking at the patch it seems you need two mappings of the same
page, one writable and one executable.

Or, maybe it is also possible with one mapping which is writable first
when you fill it with code, then gets flipped over to executable when
you are done with the initialization and want use it.

Is that correct?

I suspect supporting that in tcg isn't exactly trivial.
Does OpenBSD work the same way btw?

cheers,
  Gerd

Re: [Qemu-devel] [PATCH 03/13] tests/vm: send locale environment variables over ssh

[Thomas Huth]

On 08/05/2019 10.56, Gerd Hoffmann wrote:
> Needed for unicode tests.
> 
> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
> ---
>  tests/vm/basevm.py | 14 ++++++++++++++
>  1 file changed, 14 insertions(+)
> 
> diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
> index 6b46674f4497..20eec5420dbf 100755
> --- a/tests/vm/basevm.py
> +++ b/tests/vm/basevm.py
> @@ -43,6 +43,20 @@ class BaseVM(object):
>          "http_proxy",
>          "ftp_proxy",
>          "no_proxy",
> +        "LANG",
> +        "LC_CTYPE",
> +        "LC_NUMERIC",
> +        "LC_TIME",
> +        "LC_COLLATE",
> +        "LC_MONETARY",
> +        "LC_MESSAGES",
> +        "LC_PAPER",
> +        "LC_NAME",
> +        "LC_ADDRESS",
> +        "LC_TELEPHONE",
> +        "LC_MEASUREMENT",
> +        "LC_IDENTIFICATION",
> +        "LC_ALL",
>      ]

I gave your patch series a try, but now I see a lot of these messages in
the output:

perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LC_ALL = (unset),
	LC_TIME = "en_DK.UTF-8",
	LANG = "en_DK.UTF-8"
    are supported and installed on your system.

Is it really a good idea to use the host locale in the guest, too? ... I
don't think so... at least you should make sure that *all* possible
locales are installed in that case.

"gmake check" then also finally failed:

--- /home/qemu/qemu-test.AOvcgx/src/tests/qapi-schema/unicode-str.err
2019-05-09 05:56:17.000000000 +0000
+++ -	2019-05-09 06:12:54.451392000 +0000
@@ -1 +1 @@
-tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name 'é'
+tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name '\xe9'

Note that if you just wanted to fix the python3 utf-8 problem with this
patch here, there is a separate fix available for that problem already:

 https://lists.gnu.org/archive/html/qemu-devel/2019-05/msg01247.html

 HTH,
  Thomas

Re: [Qemu-devel] [PATCH 03/13] tests/vm: send locale environment variables over ssh

[Gerd Hoffmann]

  Hi,

> "gmake check" then also finally failed:
> 
> --- /home/qemu/qemu-test.AOvcgx/src/tests/qapi-schema/unicode-str.err
> 2019-05-09 05:56:17.000000000 +0000
> +++ -	2019-05-09 06:12:54.451392000 +0000
> @@ -1 +1 @@
> -tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name 'é'
> +tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name '\xe9'

Well, that failure disappeared in my tests.  But that'll probably only works
in case the guest actually has support for the given locale.

> Note that if you just wanted to fix the python3 utf-8 problem with this
> patch here, there is a separate fix available for that problem already:
> 
>  https://lists.gnu.org/archive/html/qemu-devel/2019-05/msg01247.html

Ah, ok.  Guess I can drop the patch then.

cheers,
  Gerd

Re: [Qemu-devel] [PATCH 00/13] tests/vm: serial console autoinstall, misc fixes.

[Thomas Huth]

On 08/05/2019 10.56, Gerd Hoffmann wrote:
> This patch series changes the way virtual machines for test builds are
> managed.  They are created locally on the developer machine now.  The
> installer is booted on the serial console and the scripts walks through
> the dialogs to install and configure the guest.
> 
> That takes the download.patchew.org server out of the loop and makes it
> alot easier to tweak the guest images (adding build dependencies for
> example).
> 
> The install scripts take care to apply host proxy settings (from *_proxy
> environment variables) to the guest, so any package downloads will be
> routed through the proxy and can be cached that way.  This also makes
> them work behind strict firewalls.
> 
> There are also a bunch of smaller tweaks for tests/vm to fix issues I
> was struggling with.  See commit messages of individual patches for
> details.
> 
> Known issue:  NetBSD package install is not working for me right now.
> It did work a while ago.  Not sure what is going on here.

I now gave your series another try and replaced patch 3 with the python3
fix from Eduardo locally here. FreeBSD works great. OpenBSD is fine too,
except for the known issue that the "gmake check" does not work - but
this issue has been there before already. NetBSD also does not work for
me, so I guess you should hold off that patch for now?

So for patches 1, 2 and 4 - 10 (I did not check the Linux images yet):

Tested-by: Thomas Huth <thuth@redhat.com>

> Do we have accelerator support for the BSDs?  A "make check" for a full
> build takes ages, and I suspect tcg being used is part of the problem.
> I did my tests using "TARGET_LIST=x86_64-softmmu" because of that.

I think they should be running with "--enable-kvm". Did you make sure
that you've enabled multiple CPUs with J=8 for example? ... but for me,
the compilation is also quite a bit slower, indeed. I think part of the
problem might be clang which is compiling a little bit slower than GCC
as far as I know...?

 Thomas

Re: [Qemu-devel] [PATCH 12/13] tests/vm: fedora autoinstall, using serial console

[Thomas Huth]

On 08/05/2019 10.56, Gerd Hoffmann wrote:
> Download the install iso and prepare the image locally.  Install to
> disk, using the serial console.  Create qemu user, configure ssh login.
> Install packages needed for qemu builds.
> 
> Yes, we have docker images for fedora.  But for trouble-shooting it
> might be helpful to have a vm too.  When vm builds fail you can use
> it to figure whenever the vm setup or the guest os is the problem.
> 
> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
> ---
>  tests/vm/basevm.py        |   9 +-
>  tests/vm/Makefile.include |   3 +-
>  tests/vm/fedora           | 187 ++++++++++++++++++++++++++++++++++++++
>  3 files changed, 197 insertions(+), 2 deletions(-)
>  create mode 100755 tests/vm/fedora
[...]
> diff --git a/tests/vm/fedora b/tests/vm/fedora
> new file mode 100755
> index 000000000000..d8cb62c803a4
> --- /dev/null
> +++ b/tests/vm/fedora
> @@ -0,0 +1,187 @@
> +#!/usr/bin/env python
> +#
> +# FreeBSD VM image
> +#
> +# Copyright 2017 Red Hat Inc.
> +#
> +# Authors:
> +#  Fam Zheng <famz@redhat.com>

All the above information needs some update, obviously.

> +class FedoraVM(basevm.BaseVM):
> +    name = "fedora"
> +    arch = "x86_64"
> +
> +    base = "http://dl.fedoraproject.org/pub/fedora/linux/releases/30/"
> +    link = base + "Server/x86_64/iso/Fedora-Server-netinst-x86_64-30-1.2.iso"
> +    repo = base + "Server/x86_64/os/"
> +    full = base + "Everything/x86_64/os/"
> +    csum = "5e4eac4566d8c572bfb3bcf54b7d6c82006ec3c6c882a2c9235c6d3494d7b100"
> +    size = "20G"
> +    pkgs = [
> +        # tools
> +        'git',
> +        'flex', 'bison',
> +        'gcc', 'binutils', 'make',
> +
> +        # perl
> +        'perl-Test-Harness',
> +
> +        # libs: usb
> +        '"pkgconfig(libusb-1.0)"',
> +        '"pkgconfig(libusbredirparser-0.5)"',
> +
> +        # libs: crypto
> +        '"pkgconfig(gnutls)"',
> +
> +        # libs: ui
> +        '"pkgconfig(sdl2)"',
> +        '"pkgconfig(gtk+-3.0)"',
> +        '"pkgconfig(ncursesw)"',
> +
> +        # libs: audio
> +        '"pkgconfig(libpulse)"',
> +        '"pkgconfig(alsa)"',
> +    ]
> +
> +    BUILD_SCRIPT = """
> +        set -e;
> +        rm -rf /home/qemu/qemu-test.*
> +        cd $(mktemp -d /home/qemu/qemu-test.XXXXXX);
> +        mkdir src build; cd src;
> +        tar -xf /dev/vdb;
> +        cd ../build
> +        ../src/configure --python=python3 {configure_opts};
> +        gmake --output-sync -j{jobs} {target} {verbose};
> +    """
> +
> +    def build_image(self, img):
> +        self.print_step("Downloading install iso")
> +        cimg = self._download_with_cache(self.link, sha256sum=self.csum)
> +        img_tmp = img + ".tmp"
> +        iso = img + ".install.iso"
> +
> +        self.print_step("Preparing iso and disk image")
> +        subprocess.check_call(["cp", "-f", cimg, iso])
> +        subprocess.check_call(["qemu-img", "create", "-f", "qcow2",
> +                               img_tmp, self.size])
> +
> +        self.print_step("Booting installer")
> +        self.boot(img_tmp, extra_args = [
> +            "-machine", "graphics=off",
> +            "-cdrom", iso
> +        ])
> +        self.console_init(300)
> +        self.console_wait("installation process.")
> +        time.sleep(0.3)
> +        self.console_send("\t")
> +        time.sleep(0.3)
> +        self.console_send(" console=ttyS0")
> +        proxy = os.environ.get("http_proxy")
> +        if not proxy is None:
> +            self.console_send(" proxy=%s" % proxy)
> +        self.console_send(" inst.repo=%s" % self.repo)
> +        self.console_send("\n")
> +
> +        self.console_wait_send("2) Use text mode",         "2\n")
> +
> +        self.console_wait_send("5) [!] Installation Dest", "5\n")
> +        self.console_wait_send("1) [x]",                   "c\n")
> +        self.console_wait_send("2) [ ] Use All Space",     "2\n")
> +        self.console_wait_send("2) [x] Use All Space",     "c\n")
> +        self.console_wait_send("1) [ ] Standard Part",     "1\n")
> +        self.console_wait_send("1) [x] Standard Part",     "c\n")
> +
> +        self.console_wait_send("7) [!] Root password",     "7\n")
> +        self.console_wait("Password:")
> +        self.console_send("%s\n" % self.ROOT_PASS)
> +        self.console_wait("Password (confirm):")
> +        self.console_send("%s\n" % self.ROOT_PASS)
> +
> +        self.console_wait_send("8) [ ] User creation",     "8\n")
> +        self.console_wait_send("1) [ ] Create user",       "1\n")
> +        self.console_wait_send("3) User name",             "3\n")
> +        self.console_wait_send("ENTER:", "%s\n" % self.GUEST_USER)
> +        self.console_wait_send("4) [ ] Use password",      "4\n")
> +        self.console_wait_send("5) Password",              "5\n")
> +        self.console_wait("Password:")
> +        self.console_send("%s\n" % self.GUEST_PASS)
> +        self.console_wait("Password (confirm):")
> +        self.console_send("%s\n" % self.GUEST_PASS)
> +        self.console_wait_send("7) Groups",                "c\n")
> +
> +        while True:
> +            good = self.console_wait("3) [x] Installation",
> +                                     "3) [!] Installation")
> +            self.console_send("r\n")
> +            if good:
> +                break
> +            time.sleep(10)
> +
> +        while True:
> +            good = self.console_wait("4) [x] Software",
> +                                     "4) [!] Software")
> +            self.console_send("r\n")
> +            if good:
> +                break
> +            time.sleep(10)
> +            self.console_send("r\n" % self.GUEST_PASS)
> +
> +        self.console_wait_send("'b' to begin install",     "b\n")
> +
> +        self.print_step("Installation started now, this will take a while")
> +
> +        self.console_wait_send("Installation complete",    "\n")
> +        self.print_step("Installation finished, rebooting")
> +
> +        # setup qemu user
> +        prompt = " ~]$"
> +        self.console_ssh_init(prompt, self.GUEST_USER, self.GUEST_PASS)
> +        self.console_wait_send(prompt, "exit\n")
> +
> +        # setup root user
> +        prompt = " ~]#"
> +        self.console_ssh_init(prompt, "root", self.ROOT_PASS)
> +        self.console_sshd_config(prompt)
> +
> +        # setup virtio-blk #1 (tarfile)
> +        self.console_wait(prompt)
> +        self.console_send("echo 'KERNEL==\"vdb\" MODE=\"666\"' >> %s\n" %
> +                          "/etc/udev/rules.d/99-qemu.rules")
> +
> +        self.print_step("Configuration finished, rebooting")
> +        self.console_wait_send(prompt, "reboot\n")
> +        self.console_wait("login:")
> +        self.wait_ssh()
> +
> +        self.print_step("Installing packages")
> +        self.ssh_root_check("rm -vf /etc/yum.repos.d/fedora*.repo\n")
> +        self.ssh_root_check("echo '[fedora]' >> /etc/yum.repos.d/qemu.repo\n")
> +        self.ssh_root_check("echo 'baseurl=%s' >> /etc/yum.repos.d/qemu.repo\n" % self.full)
> +        self.ssh_root_check("echo 'gpgcheck=0' >> /etc/yum.repos.d/qemu.repo\n")
> +        self.ssh_root_check("dnf install -y %s\n" % " ".join(self.pkgs))
> +
> +        # shutdown
> +        self.ssh_root("poweroff")
> +        self.console_wait("sleep state S5")
> +        self.wait()
> +
> +        if os.path.exists(img):
> +            os.remove(img)
> +        os.rename(img_tmp, img)
> +        os.remove(iso)
> +        self.print_step("All done")
> +
> +if __name__ == "__main__":
> +    sys.exit(basevm.main(FedoraVM))
> 

Maybe you could also do a kickstart installation instead?

 Thomas

Re: [Qemu-devel] [PATCH 00/13] tests/vm: serial console autoinstall, misc fixes.

[Philippe Mathieu-Daudé]

Hi Thomas,

On 5/9/19 1:53 PM, Thomas Huth wrote:
> On 08/05/2019 10.56, Gerd Hoffmann wrote:
>> This patch series changes the way virtual machines for test builds are
>> managed.  They are created locally on the developer machine now.  The
>> installer is booted on the serial console and the scripts walks through
>> the dialogs to install and configure the guest.
>>
>> That takes the download.patchew.org server out of the loop and makes it
>> alot easier to tweak the guest images (adding build dependencies for
>> example).
>>
>> The install scripts take care to apply host proxy settings (from *_proxy
>> environment variables) to the guest, so any package downloads will be
>> routed through the proxy and can be cached that way.  This also makes
>> them work behind strict firewalls.
>>
>> There are also a bunch of smaller tweaks for tests/vm to fix issues I
>> was struggling with.  See commit messages of individual patches for
>> details.
>>
>> Known issue:  NetBSD package install is not working for me right now.
>> It did work a while ago.  Not sure what is going on here.
> 
> I now gave your series another try and replaced patch 3 with the python3
> fix from Eduardo locally here. FreeBSD works great. OpenBSD is fine too,
> except for the known issue that the "gmake check" does not work - but
> this issue has been there before already. [...]

"gmake check" was working on OpenBSD with this series:
https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg07513.html
I think most of the patch proposed there have been merged, so are you
talking about a new issue?

Re: [Qemu-devel] [PATCH 13/13] tests/vm: ubuntu.i386: apt proxy setup

[Philippe Mathieu-Daudé]

On 5/8/19 10:56 AM, Gerd Hoffmann wrote:
> Configure apt proxy so package downloads
> can be cached and can pass firewalls.

Yay \o/

> 
> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
> ---
>  tests/vm/ubuntu.i386 | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/tests/vm/ubuntu.i386 b/tests/vm/ubuntu.i386
> index a22d137e76df..b869afd212fa 100755
> --- a/tests/vm/ubuntu.i386
> +++ b/tests/vm/ubuntu.i386
> @@ -51,6 +51,10 @@ class UbuntuX86VM(basevm.BaseVM):
>                            "    ssh-authorized-keys:\n",
>                            "    - %s\n" % basevm.SSH_PUB_KEY,
>                            "locale: en_US.UTF-8\n"])
> +        proxy = os.environ.get("http_proxy")

Some users only use 'https_proxy' (or FTP):
https://lists.gnu.org/archive/html/qemu-devel/2018-06/msg08413.html

> +        if not proxy is None:
> +            udata.writelines(["apt:\n",
> +                              "  proxy: %s" % proxy])
>          udata.close()
>          subprocess.check_call(["genisoimage", "-output", "cloud-init.iso",
>                                 "-volid", "cidata", "-joliet", "-rock",
> 

Re: [Qemu-devel] [PATCH 00/13] tests/vm: serial console autoinstall, misc fixes.

[Thomas Huth]

On 09/05/2019 14.04, Philippe Mathieu-Daudé wrote:
[...]
>> I now gave your series another try and replaced patch 3 with the python3
>> fix from Eduardo locally here. FreeBSD works great. OpenBSD is fine too,
>> except for the known issue that the "gmake check" does not work - but
>> this issue has been there before already. [...]
> 
> "gmake check" was working on OpenBSD with this series:
> https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg07513.html
> I think most of the patch proposed there have been merged, so are you
> talking about a new issue?

Oh, true, I remembered the patches, but was not aware that they've been
merged.

The issue that I've seen is this one:

  [...]
  TEST    check-qtest-arm: tests/pca9552-test
  TEST    check-qtest-arm: tests/ds1338-test
  TEST    check-qtest-arm: tests/microbit-test
  TEST    check-qtest-arm: tests/m25p80-test
  TEST    check-qtest-arm: tests/test-arm-mptimer
  TEST    check-qtest-arm: tests/boot-serial-test
qemu-system-arm: cannot set up guest memory 'ram': Cannot allocate memory
Broken pipe
/home/qemu/qemu-test.znJ6fy/src/tests/libqtest.c:135: kill_qemu() tried
to terminate QEMU process but encountered exit status 1
ERROR - too few tests run (expected 2, got 0)
Abort trap (core dumped)
gmake: *** [/home/qemu/qemu-test.znJ6fy/src/tests/Makefile.include:903:
check-qtest-arm] Error 1

 Thomas

Re: [Qemu-devel] [PATCH 12/13] tests/vm: fedora autoinstall, using serial console

[Gerd Hoffmann]

  Hi,

> > +        # shutdown
> > +        self.ssh_root("poweroff")
> > +        self.console_wait("sleep state S5")
> > +        self.wait()
> > +
> > +        if os.path.exists(img):
> > +            os.remove(img)
> > +        os.rename(img_tmp, img)
> > +        os.remove(iso)
> > +        self.print_step("All done")
> > +
> > +if __name__ == "__main__":
> > +    sys.exit(basevm.main(FedoraVM))

> Maybe you could also do a kickstart installation instead?

Well, the tricky part is how to get the kickstart/autoinstall/whatever
file passed to the guest then, in a way that works for every guest ...

cheers,
  Gerd

Re: [Qemu-devel] [PATCH 13/13] tests/vm: ubuntu.i386: apt proxy setup

[Gerd Hoffmann]

On Thu, May 09, 2019 at 02:12:11PM +0200, Philippe Mathieu-Daudé wrote:
> On 5/8/19 10:56 AM, Gerd Hoffmann wrote:
> > Configure apt proxy so package downloads
> > can be cached and can pass firewalls.
> 
> Yay \o/
> 
> > 
> > Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
> > ---
> >  tests/vm/ubuntu.i386 | 4 ++++
> >  1 file changed, 4 insertions(+)
> > 
> > diff --git a/tests/vm/ubuntu.i386 b/tests/vm/ubuntu.i386
> > index a22d137e76df..b869afd212fa 100755
> > --- a/tests/vm/ubuntu.i386
> > +++ b/tests/vm/ubuntu.i386
> > @@ -51,6 +51,10 @@ class UbuntuX86VM(basevm.BaseVM):
> >                            "    ssh-authorized-keys:\n",
> >                            "    - %s\n" % basevm.SSH_PUB_KEY,
> >                            "locale: en_US.UTF-8\n"])
> > +        proxy = os.environ.get("http_proxy")
> 
> Some users only use 'https_proxy' (or FTP):
> https://lists.gnu.org/archive/html/qemu-devel/2018-06/msg08413.html

Well, this configures apt only, and the apt repos are configured
to use http.  So for package installs we are all set.

We might additionally setup AcceptEnv in sshd_config (i.e. basically do
the same thing console_sshd_config() in basevm.py does for serial
console installs) so proxy evironment pass-through works for ubuntu ssh
logins too.  I didn't see a strong need for that with package downloads
being covered already, so I didn't try ...

cheers,
  Gerd

Re: [Qemu-devel] [PATCH 12/13] tests/vm: fedora autoinstall, using serial console

[Daniel P. Berrangé]

On Thu, May 09, 2019 at 03:10:03PM +0200, Gerd Hoffmann wrote:
>   Hi,
> 
> > > +        # shutdown
> > > +        self.ssh_root("poweroff")
> > > +        self.console_wait("sleep state S5")
> > > +        self.wait()
> > > +
> > > +        if os.path.exists(img):
> > > +            os.remove(img)
> > > +        os.rename(img_tmp, img)
> > > +        os.remove(iso)
> > > +        self.print_step("All done")
> > > +
> > > +if __name__ == "__main__":
> > > +    sys.exit(basevm.main(FedoraVM))
> 
> > Maybe you could also do a kickstart installation instead?
> 
> Well, the tricky part is how to get the kickstart/autoinstall/whatever
> file passed to the guest then, in a way that works for every guest ...

Libosinfo reports use of cdrom, disk, floppy or initrd injection for
providing the installer automation file. Distros support one of more
of these methods, but there's no single one that works everywhere
afaik.

FWIW, virt-install recently got ability to auto-install images using
the libosinfo installer files. We don't have coverage for any BSD's
though in libosinfo at this time though.

Regards,
Daniel

[1] https://gitlab.com/libosinfo/osinfo-db/tree/master/data/install-script
    The <injection-method> element in the XML files
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [PATCH 00/13] tests/vm: serial console autoinstall, misc fixes.

[Gerd Hoffmann]

  Hi,

> > Do we have accelerator support for the BSDs?  A "make check" for a full
> > build takes ages, and I suspect tcg being used is part of the problem.
> > I did my tests using "TARGET_LIST=x86_64-softmmu" because of that.
> 
> I think they should be running with "--enable-kvm".

The images themself yes, but the tests running *inside* (on make check) don't.

cheers,
  Gerd

Re: [Qemu-devel] [PATCH 00/13] tests/vm: serial console autoinstall, misc fixes.

[Thomas Huth]

On 09/05/2019 15.50, Gerd Hoffmann wrote:
>   Hi,
> 
>>> Do we have accelerator support for the BSDs?  A "make check" for a full
>>> build takes ages, and I suspect tcg being used is part of the problem.
>>> I did my tests using "TARGET_LIST=x86_64-softmmu" because of that.
>>
>> I think they should be running with "--enable-kvm".
> 
> The images themself yes, but the tests running *inside* (on make check) don't.

No, we don't have accelerator support for *BSD, as far as I know. But we
also do not run that much TCG tests during "make check" that you should
see such a big difference here. And for me, the compilation step is
already way slower than on the host, so I think the problem is likely
something else...

 Thomas

Re: [Qemu-devel] [PATCH 11/13] tests/vm: netbsd autoinstall, using serial console

[Richard Henderson]

On 5/8/19 11:47 PM, Gerd Hoffmann wrote:
> So, from looking at the patch it seems you need two mappings of the same
> page, one writable and one executable.
> 
> Or, maybe it is also possible with one mapping which is writable first
> when you fill it with code, then gets flipped over to executable when
> you are done with the initialization and want use it.
> 
> Is that correct?

That's certainly the way I read that patch.

> I suspect supporting that in tcg isn't exactly trivial.

It shouldn't be too hard, if required.  All of the writing to the buffer is
isolated to a couple of inline functions.

I do wonder if using paxctl -m as a part of the build process for affected *BSD
isn't just as easy?


r~

Re: [Qemu-devel] [PATCH 11/13] tests/vm: netbsd autoinstall, using serial console

[Kamil Rytarowski]

[-- Attachment #1: Type: text/plain, Size: 1648 bytes --]

On 09.05.2019 08:47, Gerd Hoffmann wrote:
>   Hi,
> 
>> I recommend to add one extra step into generated image:
>>
>> echo security.pax.mprotect.enabled=0 >> /etc/sysctl.conf
> 
> Done.
> 

Thanks! Once there will be PaX MPROTECT support in qemu, we can and
should drop it.

>> Alternatively (and preferably) enhance qemu to handle RWX allocation for
>> JIT on NetBSD.
>>
>> Example in libffi.
>>
>> https://github.com/libffi/libffi/commit/2bfcd29955c02b67fa10a68cc4200f6838181e0f
> 
> So, from looking at the patch it seems you need two mappings of the same
> page, one writable and one executable.
> > Or, maybe it is also possible with one mapping which is writable first
> when you fill it with code, then gets flipped over to executable when
> you are done with the initialization and want use it.
> 
> Is that correct?
> 

If we need RWX in the same time we need double mapping (RW and RX). This
has been done in libffi.

If we can switch between RW and RX it's sufficient to use a single
mapping, however there is need to allocate a memory region with mmap(2)
using the PROT_MPROTECT() macro.

Example from the LLVM code:

https://github.com/llvm-mirror/llvm/blob/90dd07f5c5946a3d9d6861effe3291620c88c06f/lib/Support/Unix/Memory.inc#L99

> I suspect supporting that in tcg isn't exactly trivial.
> Does OpenBSD work the same way btw?
> 

No. OpenBSD does not implement escape API.

There is need to pass "-z wxneeded" to the linker flags, but I'm not the
right person to test this or write a patch.

https://man.openbsd.org/ld

FreeBSD does not support W^X.

> cheers,
>   Gerd
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Qemu-devel] [PATCH 00/13] tests/vm: serial console autoinstall, misc fixes.

[Kamil Rytarowski]

[-- Attachment #1: Type: text/plain, Size: 2989 bytes --]

On 08.05.2019 10:56, Gerd Hoffmann wrote:
> This patch series changes the way virtual machines for test builds are
> managed.  They are created locally on the developer machine now.  The
> installer is booted on the serial console and the scripts walks through
> the dialogs to install and configure the guest.
> 
> That takes the download.patchew.org server out of the loop and makes it
> alot easier to tweak the guest images (adding build dependencies for
> example).
> 
> The install scripts take care to apply host proxy settings (from *_proxy
> environment variables) to the guest, so any package downloads will be
> routed through the proxy and can be cached that way.  This also makes
> them work behind strict firewalls.
> 
> There are also a bunch of smaller tweaks for tests/vm to fix issues I
> was struggling with.  See commit messages of individual patches for
> details.
> 
> Known issue:  NetBSD package install is not working for me right now.
> It did work a while ago.  Not sure what is going on here.
> 

Error log? What is the command? pkgin install?

> Do we have accelerator support for the BSDs?

KVM-style?

NetBSD does support HAXM (--accel hax) and in a downstream copy NVMM
(-accel nvmm).

http://blog.netbsd.org/tnf/entry/the_hardware_assisted_virtualization_challenge

http://blog.netbsd.org/tnf/entry/from_zero_to_nvmm

Once NVMM will stabilize we intend to submit it upstream.

There is no support for hardware assisted acceleration in qemu for any
other BSD.

>  A "make check" for a full
> build takes ages, and I suspect tcg being used is part of the problem.
> I did my tests using "TARGET_LIST=x86_64-softmmu" because of that.
> 
> Gerd Hoffmann (13):
>   scripts: use git archive in archive-source
>   tests/vm: send proxy environment variables over ssh
>   tests/vm: send locale environment variables over ssh
>   tests/vm: use ssh with pty unconditionally
>   tests/vm: run test builds on snapshot
>   tests/vm: add vm-boot-{ssh,serial}-<guest> targets
>   tests/vm: add DEBUG=1 to help text
>   tests/vm: serial console support helpers
>   tests/vm: openbsd autoinstall, using serial console
>   tests/vm: freebsd autoinstall, using serial console
>   tests/vm: netbsd autoinstall, using serial console
>   tests/vm: fedora autoinstall, using serial console
>   tests/vm: ubuntu.i386: apt proxy setup
> 
>  tests/vm/basevm.py        | 125 ++++++++++++++++++++++---
>  scripts/archive-source.sh |  72 +++++++--------
>  tests/vm/Makefile.include |  25 ++++-
>  tests/vm/fedora           | 187 ++++++++++++++++++++++++++++++++++++++
>  tests/vm/freebsd          | 172 +++++++++++++++++++++++++++++++++--
>  tests/vm/netbsd           | 178 ++++++++++++++++++++++++++++++++++--
>  tests/vm/openbsd          | 150 +++++++++++++++++++++++++++---
>  tests/vm/ubuntu.i386      |   4 +
>  8 files changed, 830 insertions(+), 83 deletions(-)
>  create mode 100755 tests/vm/fedora
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Qemu-devel] [PATCH 11/13] tests/vm: netbsd autoinstall, using serial console

[Kamil Rytarowski]

[-- Attachment #1: Type: text/plain, Size: 1101 bytes --]

On 09.05.2019 18:39, Richard Henderson wrote:
> On 5/8/19 11:47 PM, Gerd Hoffmann wrote:
>> So, from looking at the patch it seems you need two mappings of the same
>> page, one writable and one executable.
>>
>> Or, maybe it is also possible with one mapping which is writable first
>> when you fill it with code, then gets flipped over to executable when
>> you are done with the initialization and want use it.
>>
>> Is that correct?
> 
> That's certainly the way I read that patch.
> 
>> I suspect supporting that in tcg isn't exactly trivial.
> 
> It shouldn't be too hard, if required.  All of the writing to the buffer is
> isolated to a couple of inline functions.
> 
> I do wonder if using paxctl -m as a part of the build process for affected *BSD
> isn't just as easy?
> 

paxctl(8) is available only for NetBSD.

paxctl(8) is the last resort solution and shall be avoided due to a
security risk.

Only few exceptions shall be allowed to switch mapping protection or
request RWX mappings (mainly JIT and a process running under a debugger).

> 
> r~
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Qemu-devel] [PATCH 00/13] tests/vm: serial console autoinstall, misc fixes.

[Kamil Rytarowski]

On 09.05.2019 15:57, Thomas Huth wrote:
> On 09/05/2019 15.50, Gerd Hoffmann wrote:
>>   Hi,
>>
>>>> Do we have accelerator support for the BSDs?  A "make check" for a full
>>>> build takes ages, and I suspect tcg being used is part of the problem.
>>>> I did my tests using "TARGET_LIST=x86_64-softmmu" because of that.
>>>
>>> I think they should be running with "--enable-kvm".
>>
>> The images themself yes, but the tests running *inside* (on make check) don't.
>
> No, we don't have accelerator support for *BSD, as far as I know.

As mentioned in the other mail, KVM-style?

NetBSD does support HAXM (--accel hax) and in a downstream copy NVMM
(-accel nvmm).

http://blog.netbsd.org/tnf/entry/the_hardware_assisted_virtualization_challenge

http://blog.netbsd.org/tnf/entry/from_zero_to_nvmm

Once NVMM will stabilize we intend to submit it upstream.

There is no support for hardware assisted acceleration in qemu for any
other BSD.

> But we
> also do not run that much TCG tests during "make check" that you should
> see such a big difference here. And for me, the compilation step is
> already way slower than on the host, so I think the problem is likely
> something else...
>
>  Thomas
>

Re: [Qemu-devel] [PATCH 00/13] tests/vm: serial console autoinstall, misc fixes.

[Gerd Hoffmann]

On Thu, May 09, 2019 at 08:52:23PM +0200, Kamil Rytarowski wrote:
> On 08.05.2019 10:56, Gerd Hoffmann wrote:
> > This patch series changes the way virtual machines for test builds are
> > managed.  They are created locally on the developer machine now.  The
> > installer is booted on the serial console and the scripts walks through
> > the dialogs to install and configure the guest.
> > 
> > That takes the download.patchew.org server out of the loop and makes it
> > alot easier to tweak the guest images (adding build dependencies for
> > example).
> > 
> > The install scripts take care to apply host proxy settings (from *_proxy
> > environment variables) to the guest, so any package downloads will be
> > routed through the proxy and can be cached that way.  This also makes
> > them work behind strict firewalls.
> > 
> > There are also a bunch of smaller tweaks for tests/vm to fix issues I
> > was struggling with.  See commit messages of individual patches for
> > details.
> > 
> > Known issue:  NetBSD package install is not working for me right now.
> > It did work a while ago.  Not sure what is going on here.
> > 
> 
> Error log? What is the command? pkgin install?

Looked like a dependency problem, the error log complained that it
couldn't find a new enough tcl version for tk.

"fixed" that by installing git-base instead of git, which drop the tk
dependency of git.

cheers,
  Gerd