* [Qemu-devel] [PULL 1/3] block/nvme: optimize the performance of nvme driver based on vfio-pci
2019-01-06 5:34 [Qemu-devel] [PULL 0/3] Block and testing patches Fam Zheng
@ 2019-01-06 5:34 ` Fam Zheng
2019-01-06 5:34 ` [Qemu-devel] [PULL 2/3] docker: Use a stable snapshot for Debian Sid Fam Zheng
` (2 subsequent siblings)
3 siblings, 0 replies; 6+ messages in thread
From: Fam Zheng @ 2019-01-06 5:34 UTC (permalink / raw)
To: qemu-devel; +Cc: peter.maydell, Fam Zheng
From: Li Feng <lifeng1519@gmail.com>
When the IO size is larger than 2 pages, we move the the pointer one by
one in the pagelist, this is inefficient.
This is a simple benchmark result:
Before:
$ qemu-io -c 'write 0 1G' nvme://0000:00:04.0/1
wrote 1073741824/1073741824 bytes at offset 0
1 GiB, 1 ops; 0:00:02.41 (424.504 MiB/sec and 0.4146 ops/sec)
$ qemu-io -c 'read 0 1G' nvme://0000:00:04.0/1
read 1073741824/1073741824 bytes at offset 0
1 GiB, 1 ops; 0:00:02.03 (503.055 MiB/sec and 0.4913 ops/sec)
After:
$ qemu-io -c 'write 0 1G' nvme://0000:00:04.0/1
wrote 1073741824/1073741824 bytes at offset 0
1 GiB, 1 ops; 0:00:02.17 (471.517 MiB/sec and 0.4605 ops/sec)
$ qemu-io -c 'read 0 1G' nvme://0000:00:04.0/1
read 1073741824/1073741824 bytes at offset 0
1 GiB, 1 ops; 0:00:01.94 (526.770 MiB/sec and 0.5144 ops/sec)
Signed-off-by: Li Feng <lifeng1519@gmail.com>
Message-Id: <20181101103807.25862-1-lifeng1519@gmail.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
---
block/nvme.c | 16 ++++++----------
1 file changed, 6 insertions(+), 10 deletions(-)
diff --git a/block/nvme.c b/block/nvme.c
index 29294038fc..982097b5b1 100644
--- a/block/nvme.c
+++ b/block/nvme.c
@@ -837,7 +837,7 @@ try_map:
}
for (j = 0; j < qiov->iov[i].iov_len / s->page_size; j++) {
- pagelist[entries++] = iova + j * s->page_size;
+ pagelist[entries++] = cpu_to_le64(iova + j * s->page_size);
}
trace_nvme_cmd_map_qiov_iov(s, i, qiov->iov[i].iov_base,
qiov->iov[i].iov_len / s->page_size);
@@ -850,20 +850,16 @@ try_map:
case 0:
abort();
case 1:
- cmd->prp1 = cpu_to_le64(pagelist[0]);
+ cmd->prp1 = pagelist[0];
cmd->prp2 = 0;
break;
case 2:
- cmd->prp1 = cpu_to_le64(pagelist[0]);
- cmd->prp2 = cpu_to_le64(pagelist[1]);;
+ cmd->prp1 = pagelist[0];
+ cmd->prp2 = pagelist[1];
break;
default:
- cmd->prp1 = cpu_to_le64(pagelist[0]);
- cmd->prp2 = cpu_to_le64(req->prp_list_iova);
- for (i = 0; i < entries - 1; ++i) {
- pagelist[i] = cpu_to_le64(pagelist[i + 1]);
- }
- pagelist[entries - 1] = 0;
+ cmd->prp1 = pagelist[0];
+ cmd->prp2 = cpu_to_le64(req->prp_list_iova + sizeof(uint64_t));
break;
}
trace_nvme_cmd_map_qiov(s, cmd, req, qiov, entries);
--
2.11.0
^ permalink raw reply related [flat|nested] 6+ messages in thread* [Qemu-devel] [PULL 2/3] docker: Use a stable snapshot for Debian Sid
2019-01-06 5:34 [Qemu-devel] [PULL 0/3] Block and testing patches Fam Zheng
2019-01-06 5:34 ` [Qemu-devel] [PULL 1/3] block/nvme: optimize the performance of nvme driver based on vfio-pci Fam Zheng
@ 2019-01-06 5:34 ` Fam Zheng
2019-01-06 5:34 ` [Qemu-devel] [PULL 3/3] tests: vm: auto_install OpenBSD Fam Zheng
2019-01-07 15:31 ` [Qemu-devel] [PULL 0/3] Block and testing patches Peter Maydell
3 siblings, 0 replies; 6+ messages in thread
From: Fam Zheng @ 2019-01-06 5:34 UTC (permalink / raw)
To: qemu-devel; +Cc: peter.maydell, Fam Zheng
From: Philippe Mathieu-Daudé <philmd@redhat.com>
The Debian Sid repository is not garanteed to be stable, as his
'unstable' name suggest :)
To allow quick testing, Debian maintainers might push packages
various time a day. Sometime package dependencies might break,
which is annoying when using this repository for stable development
(which is not recommended, but Sid provides edge packages we use
for testing).
Debian provides repositories snapshots which are suitable for our
use. Pick a recent date that works. When required, update to newer
releases will be easy.
This fixes current issues with this image:
$ make docker-image-debian-sid
[...]
The following packages have unmet dependencies:
build-essential : Depends: dpkg-dev (>= 1.17.11) but it is not going to be installed
git : Depends: perl but it is not going to be installed
Depends: liberror-perl but it is not going to be installed
pkg-config : Depends: libdpkg-perl but it is not going to be installed
texinfo : Depends: perl (>= 5.26.2-6) but it is not going to be installed
Depends: libtext-unidecode-perl but it is not going to be installed
Depends: libxml-libxml-perl but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20181101183705.5422-1-philmd@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
---
tests/docker/dockerfiles/debian-sid.docker | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/tests/docker/dockerfiles/debian-sid.docker b/tests/docker/dockerfiles/debian-sid.docker
index 9a3d168705..4e4cda0ba5 100644
--- a/tests/docker/dockerfiles/debian-sid.docker
+++ b/tests/docker/dockerfiles/debian-sid.docker
@@ -13,6 +13,10 @@
FROM debian:sid-slim
+# Use a snapshot known to work (see http://snapshot.debian.org/#Usage)
+ENV DEBIAN_SNAPSHOT_DATE "20181030"
+RUN sed -i "s%^deb \(https\?://\)deb.debian.org/debian/\? \(.*\)%deb [check-valid-until=no] \1snapshot.debian.org/archive/debian/${DEBIAN_SNAPSHOT_DATE} \2%" /etc/apt/sources.list
+
# Duplicate deb line as deb-src
RUN cat /etc/apt/sources.list | sed "s/^deb\ /deb-src /" >> /etc/apt/sources.list
--
2.11.0
^ permalink raw reply related [flat|nested] 6+ messages in thread* [Qemu-devel] [PULL 3/3] tests: vm: auto_install OpenBSD
2019-01-06 5:34 [Qemu-devel] [PULL 0/3] Block and testing patches Fam Zheng
2019-01-06 5:34 ` [Qemu-devel] [PULL 1/3] block/nvme: optimize the performance of nvme driver based on vfio-pci Fam Zheng
2019-01-06 5:34 ` [Qemu-devel] [PULL 2/3] docker: Use a stable snapshot for Debian Sid Fam Zheng
@ 2019-01-06 5:34 ` Fam Zheng
2019-01-06 8:46 ` Carlo Arenas
2019-01-07 15:31 ` [Qemu-devel] [PULL 0/3] Block and testing patches Peter Maydell
3 siblings, 1 reply; 6+ messages in thread
From: Fam Zheng @ 2019-01-06 5:34 UTC (permalink / raw)
To: qemu-devel; +Cc: peter.maydell, Fam Zheng
From: Fam Zheng <famz@redhat.com>
Upgrade OpenBSD to 6.4 using auto_install. Especially, drop SDL1,
include SDL2.
Also do the build in $HOME since both /var/tmp and /tmp are tmpfs with
limited capacities.
Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <20181031025712.18602-1-famz@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
---
tests/vm/basevm.py | 6 ++--
tests/vm/openbsd | 85 ++++++++++++++++++++++++++++++++++++++++++++++--------
2 files changed, 76 insertions(+), 15 deletions(-)
diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
index 5caf77d6b8..6fb446d4c5 100755
--- a/tests/vm/basevm.py
+++ b/tests/vm/basevm.py
@@ -68,8 +68,6 @@ class BaseVM(object):
self._args = [ \
"-nodefaults", "-m", "4G",
"-cpu", "max",
- "-netdev", "user,id=vnet,hostfwd=:127.0.0.1:0-:22",
- "-device", "virtio-net-pci,netdev=vnet",
"-vnc", "127.0.0.1:0,to=20",
"-serial", "file:%s" % os.path.join(self._tmpdir, "serial.out")]
if vcpus and vcpus > 1:
@@ -146,8 +144,10 @@ class BaseVM(object):
"-device",
"virtio-blk,drive=%s,serial=%s,bootindex=1" % (name, name)]
- def boot(self, img, extra_args=[]):
+ def boot(self, img, extra_args=[], extra_usernet_args=""):
args = self._args + [
+ "-netdev", "user,id=vnet,hostfwd=:127.0.0.1:0-:22" + extra_usernet_args,
+ "-device", "virtio-net-pci,netdev=vnet",
"-device", "VGA",
"-drive", "file=%s,if=none,id=drive0,cache=writeback" % img,
"-device", "virtio-blk,drive=drive0,bootindex=0"]
diff --git a/tests/vm/openbsd b/tests/vm/openbsd
index cfe0572c59..99a7e98d80 100755
--- a/tests/vm/openbsd
+++ b/tests/vm/openbsd
@@ -14,6 +14,9 @@
import os
import sys
import subprocess
+import time
+import atexit
+import tempfile
import basevm
class OpenBSDVM(basevm.BaseVM):
@@ -21,25 +24,83 @@ class OpenBSDVM(basevm.BaseVM):
arch = "x86_64"
BUILD_SCRIPT = """
set -e;
- rm -rf /var/tmp/qemu-test.*
- cd $(mktemp -d /var/tmp/qemu-test.XXXXXX);
+ rm -rf $HOME/qemu-test.*
+ cd $(mktemp -d $HOME/qemu-test.XXXXXX);
tar -xf /dev/rsd1c;
- ./configure --cc=x86_64-unknown-openbsd6.1-gcc-4.9.4 --python=python2.7 {configure_opts};
+ ./configure {configure_opts};
gmake --output-sync -j{jobs} {verbose};
# XXX: "gmake check" seems to always hang or fail
#gmake --output-sync -j{jobs} check {verbose};
"""
+ def _install_os(self, img):
+ tmpdir = tempfile.mkdtemp()
+ pxeboot = self._download_with_cache("https://fastly.cdn.openbsd.org/pub/OpenBSD/6.4/amd64/pxeboot",
+ sha256sum="d87ab39d941ff926d693943a927585945456ccedb76ea504a251b4b93cd4c266")
+ bsd_rd = self._download_with_cache("https://fastly.cdn.openbsd.org/pub/OpenBSD/6.4/amd64/bsd.rd",
+ sha256sum="89505c683cbcd75582fe475e847ed53d89e2b8180c3e3d61f4eb4b76b5e11f5c")
+ install = self._download_with_cache("https://fastly.cdn.openbsd.org/pub/OpenBSD/6.4/amd64/install64.iso",
+ sha256sum='81833b79e23dc0f961ac5fb34484bca66386deb3181ddb8236870fa4f488cdd2')
+ subprocess.check_call(["qemu-img", "create", img, "32G"])
+ subprocess.check_call(["cp", pxeboot, os.path.join(tmpdir, "auto_install")])
+ subprocess.check_call(["cp", bsd_rd, os.path.join(tmpdir, "bsd")])
+
+ self._gen_install_conf(tmpdir)
+ # BOOTP filename being auto_install makes sure OpenBSD installer
+ # not prompt for "auto install mode"
+ usernet_args = ",tftp=%s,bootfile=/auto_install" % tmpdir
+ usernet_args += ",tftp-server-name=10.0.2.4"
+ usernet_args += ",guestfwd=tcp:10.0.2.4:80-cmd:cat %s" % \
+ os.path.join(tmpdir, "install.conf")
+ self.boot(img,
+ extra_args=["-boot", "once=n", "-no-reboot",
+ "-cdrom", install],
+ extra_usernet_args=usernet_args)
+ self.wait()
+
+ def _gen_install_conf(self, tmpdir):
+ contents = """\
+HTTP/1.0 200 OK
+
+System hostname = qemu-openbsd
+Password for root = qemupass
+Public ssh key for root = {pub_key}
+Allow root ssh login = yes
+Network interfaces = vio0
+IPv4 address for vio0 = dhcp
+Setup a user = qemu
+Password for user = qemupass
+Public ssh key for user = {pub_key}
+What timezone are you in = US/Eastern
+Server = fastly.cdn.openbsd.org
+Use http = yes
+Default IPv4 route = 10.0.2.2
+Location of sets = cd0
+Set name(s) = all
+Continue without verification = yes
+""".format(pub_key=basevm.SSH_PUB_KEY)
+ with open(os.path.join(tmpdir, "install.conf"), "w") as f:
+ f.write(contents)
+
def build_image(self, img):
- cimg = self._download_with_cache("http://download.patchew.org/openbsd-6.1-amd64.img.xz",
- sha256sum='8c6cedc483e602cfee5e04f0406c64eb99138495e8ca580bc0293bcf0640c1bf')
- img_tmp_xz = img + ".tmp.xz"
- img_tmp = img + ".tmp"
- subprocess.check_call(["cp", "-f", cimg, img_tmp_xz])
- subprocess.check_call(["xz", "-df", img_tmp_xz])
- if os.path.exists(img):
- os.remove(img)
- os.rename(img_tmp, img)
+
+ self._install_os(img + ".tmp")
+
+ self.boot(img + ".tmp")
+ self.wait_ssh()
+
+ self.ssh_root("usermod -G operator qemu")
+ self.ssh_root("echo https://fastly.cdn.openbsd.org/pub/OpenBSD > /etc/installurl")
+ for pkg in ["git", "gmake", "glib2", "bison", "sdl2"]:
+ self.ssh_root("pkg_add " + pkg)
+ self.ssh_root("ln -sf /usr/local/bin/python2.7 /usr/local/bin/python")
+ self.ssh_root("ln -sf /usr/local/bin/python2.7-2to3 /usr/local/bin/2to3")
+ self.ssh_root("ln -sf /usr/local/bin/python2.7-config /usr/local/bin/python-config")
+ self.ssh_root("ln -sf /usr/local/bin/pydoc2.7 /usr/local/bin/pydoc")
+ self.ssh_root("shutdown -p now")
+ self.wait()
+
+ subprocess.check_call(["mv", img + ".tmp", img])
if __name__ == "__main__":
sys.exit(basevm.main(OpenBSDVM))
--
2.11.0
^ permalink raw reply related [flat|nested] 6+ messages in thread* Re: [Qemu-devel] [PULL 0/3] Block and testing patches
2019-01-06 5:34 [Qemu-devel] [PULL 0/3] Block and testing patches Fam Zheng
` (2 preceding siblings ...)
2019-01-06 5:34 ` [Qemu-devel] [PULL 3/3] tests: vm: auto_install OpenBSD Fam Zheng
@ 2019-01-07 15:31 ` Peter Maydell
3 siblings, 0 replies; 6+ messages in thread
From: Peter Maydell @ 2019-01-07 15:31 UTC (permalink / raw)
To: Fam Zheng; +Cc: QEMU Developers
On Sun, 6 Jan 2019 at 05:35, Fam Zheng <fam@euphon.net> wrote:
>
> The following changes since commit 9b2e891ec5ccdb4a7d583b77988848282606fdea:
>
> Merge remote-tracking branch 'remotes/marcel/tags/rdma-pull-request' into staging (2018-12-22 11:25:31 +0000)
>
> are available in the git repository at:
>
> https://famz@github.com/famz/qemu tags/block-and-testing-pull-request
I think something in your workflow has put in the wrong URL
for your repo here: this looks like the one you use to log
in with (ie to push with), not the general one other people
pull from the repo with. I manually used the github URL I
have on record for you (git://github.com/famz/qemu).
> for you to fetch changes up to 3baa6942169cce0f58b88484ad010742c382480a:
>
> tests: vm: auto_install OpenBSD (2019-01-02 20:51:15 +0800)
This makes my OpenBSD build test fail:
DEBUG:root:QEMU args: -nodefaults -m 4G -cpu max -vnc
127.0.0.1:0,to=20 -serial file:/home/peter.maydell
/qemu-openbsd/build/vm-test-0cyRK6.tmp/serial.out -smp 18 -enable-kvm
-netdev user,id=vnet,hostfwd=:127.
0.0.1:0-:22,tftp=/tmp/tmpXrGR4K,bootfile=/auto_install,tftp-server-name=10.0.2.4,guestfwd=tcp:10.0.2.4:8
0-cmd:cat /tmp/tmpXrGR4K/install.conf -device
virtio-net-pci,netdev=vnet -device VGA -drive file=tests/v
m/openbsd.img.tmp,if=none,id=drive0,cache=writeback -device
virtio-blk,drive=drive0,bootindex=0 -boot on
ce=n -no-reboot -cdrom
/home/peter.maydell/.cache/qemu-vm/download/535472578c58eb79f8b3c168a22fa5bcd2c1e
6d4
DEBUG:QMP:>>> {'execute': 'quit'}
DEBUG:qemu:Error launching VM
DEBUG:qemu:Command: 'qemu-system-x86_64 -chardev
socket,id=mon,path=/var/tmp/tmp2kj6Xg/qemu-35162-monito
r.sock -mon chardev=mon,mode=control -display none -vga none
-nodefaults -m 4G -cpu max -vnc 127.0.0.1:0
,to=20 -serial file:/home/peter.maydell/qemu-openbsd/build/vm-test-0cyRK6.tmp/serial.out
-smp 18 -enable
-kvm -netdev user,id=vnet,hostfwd=:127.0.0.1:0-:22,tftp=/tmp/tmpXrGR4K,bootfile=/auto_install,tftp-serve
r-name=10.0.2.4,guestfwd=tcp:10.0.2.4:80-cmd:cat
/tmp/tmpXrGR4K/install.conf -device virtio-net-pci,netd
ev=vnet -device VGA -drive
file=tests/vm/openbsd.img.tmp,if=none,id=drive0,cache=writeback
-device virti
o-blk,drive=drive0,bootindex=0 -boot once=n -no-reboot -cdrom
/home/peter.maydell/.cache/qemu-vm/downloa
d/535472578c58eb79f8b3c168a22fa5bcd2c1e6d4'
DEBUG:qemu:Output: "qemu-system-x86_64: -netdev
user,id=vnet,hostfwd=:127.0.0.1:0-:22,tftp=/tmp/tmpXrGR4K,bootfile=/auto_install,tftp-server-name=10.0.2.4,guestfwd=tcp:10.0.2.4:80-cmd:cat
/tmp/tmpXrGR4K/install.conf: Invalid parameter 'tftp-server-name'\n"
ERROR:root:Failed to launch QEMU, command line:
ERROR:root:qemu-system-x86_64 -nodefaults -m 4G -cpu max -vnc
127.0.0.1:0,to=20 -serial
file:/home/peter.maydell/qemu-openbsd/build/vm-test-0cyRK6.tmp/serial.out
-smp 18 -enable-kvm -netdev
user,id=vnet,hostfwd=:127.0.0.1:0-:22,tftp=/tmp/tmpXrGR4K,bootfile=/auto_install,tftp-server-name=10.0.2.4,guestfwd=tcp:10.0.2.4:80-cmd:cat
/tmp/tmpXrGR4K/install.conf -device virtio-net-pci,netdev=vnet -device
VGA -drive file=tests/vm/openbsd.img.tmp,if=none,id=drive0,cache=writeback
-device virtio-blk,drive=drive0,bootindex=0 -boot once=n -no-reboot
-cdrom /home/peter.maydell/.cache/qemu-vm/download/535472578c58eb79f8b3c168a22fa5bcd2c1e6d4
ERROR:root:Log:
ERROR:root:qemu-system-x86_64: -netdev
user,id=vnet,hostfwd=:127.0.0.1:0-:22,tftp=/tmp/tmpXrGR4K,bootfile=/auto_install,tftp-server-name=10.0.2.4,guestfwd=tcp:10.0.2.4:80-cmd:cat
/tmp/tmpXrGR4K/install.conf: Invalid parameter 'tftp-server-name'
ERROR:root:QEMU version >= 2.10 is required
Failed to prepare guest environment
The host is an Ubuntu 18.04.1, with QEMU version
2.11.1(Debian 1:2.11+dfsg-1ubuntu7.8), so that should
be good enough for a ">= 2.10" requirement.
I notice that our qapi/net.json says tftp-server-name is
"Since 3.1", though, so maybe the error message is wrong.
I don't really want to have to build and maintain a non-distro
QEMU binary for running these VM tests if we can avoid it.
thanks
-- PMM
^ permalink raw reply [flat|nested] 6+ messages in thread