[Qemu-devel] [Bug 595117] Re: qemu-nbd slow and missing "writeback" cache option

[Qemu-devel] [Bug 595117] Re: qemu-nbd slow and missing "writeback" cache option

[Serge Hallyn]

** Also affects: qemu
   Importance: Undecided
       Status: New

** Changed in: qemu-kvm (Ubuntu)
       Status: New => Confirmed

-- 
qemu-nbd slow and missing "writeback" cache option
https://bugs.launchpad.net/bugs/595117
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.

Status in QEMU: New
Status in “qemu-kvm” package in Ubuntu: Confirmed

Bug description:
Binary package hint: qemu-kvm

dpkg -l | grep qemu
ii  kvm                                  1:84+dfsg-0ubuntu16+0.12.3+noroms+0ubuntu9            dummy transitional pacakge from kvm to qemu-
ii  qemu                                 0.12.3+noroms-0ubuntu9                                dummy transitional pacakge from qemu to qemu
ii  qemu-common                          0.12.3+noroms-0ubuntu9                                qemu common functionality (bios, documentati
ii  qemu-kvm                             0.12.3+noroms-0ubuntu9                                Full virtualization on i386 and amd64 hardwa
ii  qemu-kvm-extras                      0.12.3+noroms-0ubuntu9                                fast processor emulator binaries for non-x86
ii  qemu-launcher                        1.7.4-1ubuntu2                                        GTK+ front-end to QEMU computer emulator
ii  qemuctl                              0.2-2                                                 controlling GUI for qemu

lucid amd64.

qemu-nbd is a lot slower when writing to disk than say nbd-server.

It appears it is because by default the disk image it serves is open with O_SYNC. The --nocache option, unintuitively, makes matters a bit better because it causes the image to be open with O_DIRECT instead of O_SYNC.

The qemu code allows an image to be open without any of those flags, but unfortunately qemu-nbd doesn't have the option to do that (qemu doesn't allow the image to be open with both O_SYNC and O_DIRECT though).

The default of qemu-img (of using O_SYNC) is not very sensible because anyway, the client (the kernel) uses caches (write-back), (and "qemu-nbd -d" doesn't flush those by the way). So if for instance qemu-nbd is killed, regardless of whether qemu-nbd uses O_SYNC, O_DIRECT or not, the data in the image will not be consistent anyway, unless "syncs" are done by the client (like fsync on the nbd device or sync mount option), and with qemu-nbd's O_SYNC mode, those "sync"s will be extremely slow.

Attached is a patch that adds a --cache={off,none,writethrough,writeback} option to qemu-nbd.

--cache=off is the same as --nocache (that is use O_DIRECT), writethrough is using O_SYNC and is still the default so this patch doesn't change the functionality. writeback is none of those flags, so is the addition of this patch. The patch also does an fsync upon "qemu-nbd -d" to make sure data is flushed to the image before removing the nbd.

Consider this test scenario:

dd bs=1M count=100 of=a < /dev/null
qemu-nbd --cache=<x> -c /dev/nbd0 a
cp /dev/zero /dev/nbd0
time perl -MIO::Handle -e 'STDOUT->sync or die$!' 1<> /dev/nbd0

With cache=writethrough (the default), it takes over 10 minutes to write those 100MB worth of zeroes. Running a strace, we see the recvfrom and sentos delayed by each 1kb write(2)s to disk (10 to 30 ms per write).

With cache=off, it takes about 30 seconds.

With cache=writeback, it takes about 3 seconds, which is similar to the performance you get with nbd-server

Note that the cp command runs instantly as the data is buffered by the client (the kernel), and not sent to qemu-nbd until the fsync(2) is called.

[Qemu-devel] [Bug 595117] Re: qemu-nbd slow and missing "writeback" cache option

[Serge Hallyn]

Noone has confirmed, but have passed along to upstream.  If upstream
takes this patch then we will likely pull it into our patchset.

** Changed in: qemu-kvm (Ubuntu)
       Status: Confirmed => Incomplete

-- 
qemu-nbd slow and missing "writeback" cache option
https://bugs.launchpad.net/bugs/595117
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.

Status in QEMU: New
Status in “qemu-kvm” package in Ubuntu: Incomplete

Bug description:
Binary package hint: qemu-kvm

dpkg -l | grep qemu
ii  kvm                                  1:84+dfsg-0ubuntu16+0.12.3+noroms+0ubuntu9            dummy transitional pacakge from kvm to qemu-
ii  qemu                                 0.12.3+noroms-0ubuntu9                                dummy transitional pacakge from qemu to qemu
ii  qemu-common                          0.12.3+noroms-0ubuntu9                                qemu common functionality (bios, documentati
ii  qemu-kvm                             0.12.3+noroms-0ubuntu9                                Full virtualization on i386 and amd64 hardwa
ii  qemu-kvm-extras                      0.12.3+noroms-0ubuntu9                                fast processor emulator binaries for non-x86
ii  qemu-launcher                        1.7.4-1ubuntu2                                        GTK+ front-end to QEMU computer emulator
ii  qemuctl                              0.2-2                                                 controlling GUI for qemu

lucid amd64.

qemu-nbd is a lot slower when writing to disk than say nbd-server.

It appears it is because by default the disk image it serves is open with O_SYNC. The --nocache option, unintuitively, makes matters a bit better because it causes the image to be open with O_DIRECT instead of O_SYNC.

The qemu code allows an image to be open without any of those flags, but unfortunately qemu-nbd doesn't have the option to do that (qemu doesn't allow the image to be open with both O_SYNC and O_DIRECT though).

The default of qemu-img (of using O_SYNC) is not very sensible because anyway, the client (the kernel) uses caches (write-back), (and "qemu-nbd -d" doesn't flush those by the way). So if for instance qemu-nbd is killed, regardless of whether qemu-nbd uses O_SYNC, O_DIRECT or not, the data in the image will not be consistent anyway, unless "syncs" are done by the client (like fsync on the nbd device or sync mount option), and with qemu-nbd's O_SYNC mode, those "sync"s will be extremely slow.

Attached is a patch that adds a --cache={off,none,writethrough,writeback} option to qemu-nbd.

--cache=off is the same as --nocache (that is use O_DIRECT), writethrough is using O_SYNC and is still the default so this patch doesn't change the functionality. writeback is none of those flags, so is the addition of this patch. The patch also does an fsync upon "qemu-nbd -d" to make sure data is flushed to the image before removing the nbd.

Consider this test scenario:

dd bs=1M count=100 of=a < /dev/null
qemu-nbd --cache=<x> -c /dev/nbd0 a
cp /dev/zero /dev/nbd0
time perl -MIO::Handle -e 'STDOUT->sync or die$!' 1<> /dev/nbd0

With cache=writethrough (the default), it takes over 10 minutes to write those 100MB worth of zeroes. Running a strace, we see the recvfrom and sentos delayed by each 1kb write(2)s to disk (10 to 30 ms per write).

With cache=off, it takes about 30 seconds.

With cache=writeback, it takes about 3 seconds, which is similar to the performance you get with nbd-server

Note that the cp command runs instantly as the data is buffered by the client (the kernel), and not sent to qemu-nbd until the fsync(2) is called.

[Qemu-devel] [Bug 595117] Re: qemu-nbd slow and missing "writeback" cache option

[Anthony Liguori]

Patches should go to qemu-devel, not bug reports.

** Changed in: qemu
       Status: New => Invalid

-- 
qemu-nbd slow and missing "writeback" cache option
https://bugs.launchpad.net/bugs/595117
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.

Status in QEMU: Invalid
Status in “qemu-kvm” package in Ubuntu: Incomplete

Bug description:
Binary package hint: qemu-kvm

dpkg -l | grep qemu
ii  kvm                                  1:84+dfsg-0ubuntu16+0.12.3+noroms+0ubuntu9            dummy transitional pacakge from kvm to qemu-
ii  qemu                                 0.12.3+noroms-0ubuntu9                                dummy transitional pacakge from qemu to qemu
ii  qemu-common                          0.12.3+noroms-0ubuntu9                                qemu common functionality (bios, documentati
ii  qemu-kvm                             0.12.3+noroms-0ubuntu9                                Full virtualization on i386 and amd64 hardwa
ii  qemu-kvm-extras                      0.12.3+noroms-0ubuntu9                                fast processor emulator binaries for non-x86
ii  qemu-launcher                        1.7.4-1ubuntu2                                        GTK+ front-end to QEMU computer emulator
ii  qemuctl                              0.2-2                                                 controlling GUI for qemu

lucid amd64.

qemu-nbd is a lot slower when writing to disk than say nbd-server.

It appears it is because by default the disk image it serves is open with O_SYNC. The --nocache option, unintuitively, makes matters a bit better because it causes the image to be open with O_DIRECT instead of O_SYNC.

The qemu code allows an image to be open without any of those flags, but unfortunately qemu-nbd doesn't have the option to do that (qemu doesn't allow the image to be open with both O_SYNC and O_DIRECT though).

The default of qemu-img (of using O_SYNC) is not very sensible because anyway, the client (the kernel) uses caches (write-back), (and "qemu-nbd -d" doesn't flush those by the way). So if for instance qemu-nbd is killed, regardless of whether qemu-nbd uses O_SYNC, O_DIRECT or not, the data in the image will not be consistent anyway, unless "syncs" are done by the client (like fsync on the nbd device or sync mount option), and with qemu-nbd's O_SYNC mode, those "sync"s will be extremely slow.

Attached is a patch that adds a --cache={off,none,writethrough,writeback} option to qemu-nbd.

--cache=off is the same as --nocache (that is use O_DIRECT), writethrough is using O_SYNC and is still the default so this patch doesn't change the functionality. writeback is none of those flags, so is the addition of this patch. The patch also does an fsync upon "qemu-nbd -d" to make sure data is flushed to the image before removing the nbd.

Consider this test scenario:

dd bs=1M count=100 of=a < /dev/null
qemu-nbd --cache=<x> -c /dev/nbd0 a
cp /dev/zero /dev/nbd0
time perl -MIO::Handle -e 'STDOUT->sync or die$!' 1<> /dev/nbd0

With cache=writethrough (the default), it takes over 10 minutes to write those 100MB worth of zeroes. Running a strace, we see the recvfrom and sentos delayed by each 1kb write(2)s to disk (10 to 30 ms per write).

With cache=off, it takes about 30 seconds.

With cache=writeback, it takes about 3 seconds, which is similar to the performance you get with nbd-server

Note that the cp command runs instantly as the data is buffered by the client (the kernel), and not sent to qemu-nbd until the fsync(2) is called.

[Qemu-devel] [Bug 595117] Re: qemu-nbd slow and missing "writeback" cache option

[Dustin Kirkland]

Stephane-

Could you please send that patch to the qemu-devel@ mailing list?
Thanks!

-- 
qemu-nbd slow and missing "writeback" cache option
https://bugs.launchpad.net/bugs/595117
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.

Status in QEMU: Invalid
Status in “qemu-kvm” package in Ubuntu: Incomplete

Bug description:
Binary package hint: qemu-kvm

dpkg -l | grep qemu
ii  kvm                                  1:84+dfsg-0ubuntu16+0.12.3+noroms+0ubuntu9            dummy transitional pacakge from kvm to qemu-
ii  qemu                                 0.12.3+noroms-0ubuntu9                                dummy transitional pacakge from qemu to qemu
ii  qemu-common                          0.12.3+noroms-0ubuntu9                                qemu common functionality (bios, documentati
ii  qemu-kvm                             0.12.3+noroms-0ubuntu9                                Full virtualization on i386 and amd64 hardwa
ii  qemu-kvm-extras                      0.12.3+noroms-0ubuntu9                                fast processor emulator binaries for non-x86
ii  qemu-launcher                        1.7.4-1ubuntu2                                        GTK+ front-end to QEMU computer emulator
ii  qemuctl                              0.2-2                                                 controlling GUI for qemu

lucid amd64.

qemu-nbd is a lot slower when writing to disk than say nbd-server.

It appears it is because by default the disk image it serves is open with O_SYNC. The --nocache option, unintuitively, makes matters a bit better because it causes the image to be open with O_DIRECT instead of O_SYNC.

The qemu code allows an image to be open without any of those flags, but unfortunately qemu-nbd doesn't have the option to do that (qemu doesn't allow the image to be open with both O_SYNC and O_DIRECT though).

The default of qemu-img (of using O_SYNC) is not very sensible because anyway, the client (the kernel) uses caches (write-back), (and "qemu-nbd -d" doesn't flush those by the way). So if for instance qemu-nbd is killed, regardless of whether qemu-nbd uses O_SYNC, O_DIRECT or not, the data in the image will not be consistent anyway, unless "syncs" are done by the client (like fsync on the nbd device or sync mount option), and with qemu-nbd's O_SYNC mode, those "sync"s will be extremely slow.

Attached is a patch that adds a --cache={off,none,writethrough,writeback} option to qemu-nbd.

--cache=off is the same as --nocache (that is use O_DIRECT), writethrough is using O_SYNC and is still the default so this patch doesn't change the functionality. writeback is none of those flags, so is the addition of this patch. The patch also does an fsync upon "qemu-nbd -d" to make sure data is flushed to the image before removing the nbd.

Consider this test scenario:

dd bs=1M count=100 of=a < /dev/null
qemu-nbd --cache=<x> -c /dev/nbd0 a
cp /dev/zero /dev/nbd0
time perl -MIO::Handle -e 'STDOUT->sync or die$!' 1<> /dev/nbd0

With cache=writethrough (the default), it takes over 10 minutes to write those 100MB worth of zeroes. Running a strace, we see the recvfrom and sentos delayed by each 1kb write(2)s to disk (10 to 30 ms per write).

With cache=off, it takes about 30 seconds.

With cache=writeback, it takes about 3 seconds, which is similar to the performance you get with nbd-server

Note that the cp command runs instantly as the data is buffered by the client (the kernel), and not sent to qemu-nbd until the fsync(2) is called.

[Qemu-devel] Re: [Bug 595117] Re: qemu-nbd slow and missing "writeback" cache option

[Stephane Chazelas]

2010-06-16 20:36:00 -0000, Dustin Kirkland:
[...]
> Could you please send that patch to the qemu-devel@ mailing list?
> Thanks!
[...]

Hi Dustin, it looks like qemu-devel is subscribed to bugs in
there, so the bug report is on the list already.

Note that I still consider it as a bug because:
  - slow performance for no good reason
  - --nocache option is misleading
  - no fsync on "-d" which to my mind is a bug.

Cheers,
Stephane

-- 
qemu-nbd slow and missing "writeback" cache option
https://bugs.launchpad.net/bugs/595117
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.

Status in QEMU: Invalid
Status in “qemu-kvm” package in Ubuntu: Incomplete

Bug description:
Binary package hint: qemu-kvm

dpkg -l | grep qemu
ii  kvm                                  1:84+dfsg-0ubuntu16+0.12.3+noroms+0ubuntu9            dummy transitional pacakge from kvm to qemu-
ii  qemu                                 0.12.3+noroms-0ubuntu9                                dummy transitional pacakge from qemu to qemu
ii  qemu-common                          0.12.3+noroms-0ubuntu9                                qemu common functionality (bios, documentati
ii  qemu-kvm                             0.12.3+noroms-0ubuntu9                                Full virtualization on i386 and amd64 hardwa
ii  qemu-kvm-extras                      0.12.3+noroms-0ubuntu9                                fast processor emulator binaries for non-x86
ii  qemu-launcher                        1.7.4-1ubuntu2                                        GTK+ front-end to QEMU computer emulator
ii  qemuctl                              0.2-2                                                 controlling GUI for qemu

lucid amd64.

qemu-nbd is a lot slower when writing to disk than say nbd-server.

It appears it is because by default the disk image it serves is open with O_SYNC. The --nocache option, unintuitively, makes matters a bit better because it causes the image to be open with O_DIRECT instead of O_SYNC.

The qemu code allows an image to be open without any of those flags, but unfortunately qemu-nbd doesn't have the option to do that (qemu doesn't allow the image to be open with both O_SYNC and O_DIRECT though).

The default of qemu-img (of using O_SYNC) is not very sensible because anyway, the client (the kernel) uses caches (write-back), (and "qemu-nbd -d" doesn't flush those by the way). So if for instance qemu-nbd is killed, regardless of whether qemu-nbd uses O_SYNC, O_DIRECT or not, the data in the image will not be consistent anyway, unless "syncs" are done by the client (like fsync on the nbd device or sync mount option), and with qemu-nbd's O_SYNC mode, those "sync"s will be extremely slow.

Attached is a patch that adds a --cache={off,none,writethrough,writeback} option to qemu-nbd.

--cache=off is the same as --nocache (that is use O_DIRECT), writethrough is using O_SYNC and is still the default so this patch doesn't change the functionality. writeback is none of those flags, so is the addition of this patch. The patch also does an fsync upon "qemu-nbd -d" to make sure data is flushed to the image before removing the nbd.

Consider this test scenario:

dd bs=1M count=100 of=a < /dev/null
qemu-nbd --cache=<x> -c /dev/nbd0 a
cp /dev/zero /dev/nbd0
time perl -MIO::Handle -e 'STDOUT->sync or die$!' 1<> /dev/nbd0

With cache=writethrough (the default), it takes over 10 minutes to write those 100MB worth of zeroes. Running a strace, we see the recvfrom and sentos delayed by each 1kb write(2)s to disk (10 to 30 ms per write).

With cache=off, it takes about 30 seconds.

With cache=writeback, it takes about 3 seconds, which is similar to the performance you get with nbd-server

Note that the cp command runs instantly as the data is buffered by the client (the kernel), and not sent to qemu-nbd until the fsync(2) is called.

[Qemu-devel] Re: [Bug 595117] Re: qemu-nbd slow and missing "writeback" cache option

[Dustin Kirkland]

Stephane-

I understand your plight.  However, according to the rules and
policies of the QEMU project, you must submit the patch on the
qemu-devel@ mailing list, in addition to (or instead of) in the bug
tracker.  It's not my project, not my policy.  I'm just trying to make
sure you get your patch in front of the right audience such that it
can be discussed and accepted.

-- 
qemu-nbd slow and missing "writeback" cache option
https://bugs.launchpad.net/bugs/595117
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.

Status in QEMU: Invalid
Status in “qemu-kvm” package in Ubuntu: Incomplete

Bug description:
Binary package hint: qemu-kvm

dpkg -l | grep qemu
ii  kvm                                  1:84+dfsg-0ubuntu16+0.12.3+noroms+0ubuntu9            dummy transitional pacakge from kvm to qemu-
ii  qemu                                 0.12.3+noroms-0ubuntu9                                dummy transitional pacakge from qemu to qemu
ii  qemu-common                          0.12.3+noroms-0ubuntu9                                qemu common functionality (bios, documentati
ii  qemu-kvm                             0.12.3+noroms-0ubuntu9                                Full virtualization on i386 and amd64 hardwa
ii  qemu-kvm-extras                      0.12.3+noroms-0ubuntu9                                fast processor emulator binaries for non-x86
ii  qemu-launcher                        1.7.4-1ubuntu2                                        GTK+ front-end to QEMU computer emulator
ii  qemuctl                              0.2-2                                                 controlling GUI for qemu

lucid amd64.

qemu-nbd is a lot slower when writing to disk than say nbd-server.

It appears it is because by default the disk image it serves is open with O_SYNC. The --nocache option, unintuitively, makes matters a bit better because it causes the image to be open with O_DIRECT instead of O_SYNC.

The qemu code allows an image to be open without any of those flags, but unfortunately qemu-nbd doesn't have the option to do that (qemu doesn't allow the image to be open with both O_SYNC and O_DIRECT though).

The default of qemu-img (of using O_SYNC) is not very sensible because anyway, the client (the kernel) uses caches (write-back), (and "qemu-nbd -d" doesn't flush those by the way). So if for instance qemu-nbd is killed, regardless of whether qemu-nbd uses O_SYNC, O_DIRECT or not, the data in the image will not be consistent anyway, unless "syncs" are done by the client (like fsync on the nbd device or sync mount option), and with qemu-nbd's O_SYNC mode, those "sync"s will be extremely slow.

Attached is a patch that adds a --cache={off,none,writethrough,writeback} option to qemu-nbd.

--cache=off is the same as --nocache (that is use O_DIRECT), writethrough is using O_SYNC and is still the default so this patch doesn't change the functionality. writeback is none of those flags, so is the addition of this patch. The patch also does an fsync upon "qemu-nbd -d" to make sure data is flushed to the image before removing the nbd.

Consider this test scenario:

dd bs=1M count=100 of=a < /dev/null
qemu-nbd --cache=<x> -c /dev/nbd0 a
cp /dev/zero /dev/nbd0
time perl -MIO::Handle -e 'STDOUT->sync or die$!' 1<> /dev/nbd0

With cache=writethrough (the default), it takes over 10 minutes to write those 100MB worth of zeroes. Running a strace, we see the recvfrom and sentos delayed by each 1kb write(2)s to disk (10 to 30 ms per write).

With cache=off, it takes about 30 seconds.

With cache=writeback, it takes about 3 seconds, which is similar to the performance you get with nbd-server

Note that the cp command runs instantly as the data is buffered by the client (the kernel), and not sent to qemu-nbd until the fsync(2) is called.

[Qemu-devel] [Bug 595117] Re: qemu-nbd slow and missing "writeback" cache option

[Brian Murray]

** Tags added: patch

-- 
qemu-nbd slow and missing "writeback" cache option
https://bugs.launchpad.net/bugs/595117
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.

Status in QEMU: Invalid
Status in “qemu-kvm” package in Ubuntu: Incomplete

Bug description:
Binary package hint: qemu-kvm

dpkg -l | grep qemu
ii  kvm                                  1:84+dfsg-0ubuntu16+0.12.3+noroms+0ubuntu9            dummy transitional pacakge from kvm to qemu-
ii  qemu                                 0.12.3+noroms-0ubuntu9                                dummy transitional pacakge from qemu to qemu
ii  qemu-common                          0.12.3+noroms-0ubuntu9                                qemu common functionality (bios, documentati
ii  qemu-kvm                             0.12.3+noroms-0ubuntu9                                Full virtualization on i386 and amd64 hardwa
ii  qemu-kvm-extras                      0.12.3+noroms-0ubuntu9                                fast processor emulator binaries for non-x86
ii  qemu-launcher                        1.7.4-1ubuntu2                                        GTK+ front-end to QEMU computer emulator
ii  qemuctl                              0.2-2                                                 controlling GUI for qemu

lucid amd64.

qemu-nbd is a lot slower when writing to disk than say nbd-server.

It appears it is because by default the disk image it serves is open with O_SYNC. The --nocache option, unintuitively, makes matters a bit better because it causes the image to be open with O_DIRECT instead of O_SYNC.

The qemu code allows an image to be open without any of those flags, but unfortunately qemu-nbd doesn't have the option to do that (qemu doesn't allow the image to be open with both O_SYNC and O_DIRECT though).

The default of qemu-img (of using O_SYNC) is not very sensible because anyway, the client (the kernel) uses caches (write-back), (and "qemu-nbd -d" doesn't flush those by the way). So if for instance qemu-nbd is killed, regardless of whether qemu-nbd uses O_SYNC, O_DIRECT or not, the data in the image will not be consistent anyway, unless "syncs" are done by the client (like fsync on the nbd device or sync mount option), and with qemu-nbd's O_SYNC mode, those "sync"s will be extremely slow.

Attached is a patch that adds a --cache={off,none,writethrough,writeback} option to qemu-nbd.

--cache=off is the same as --nocache (that is use O_DIRECT), writethrough is using O_SYNC and is still the default so this patch doesn't change the functionality. writeback is none of those flags, so is the addition of this patch. The patch also does an fsync upon "qemu-nbd -d" to make sure data is flushed to the image before removing the nbd.

Consider this test scenario:

dd bs=1M count=100 of=a < /dev/null
qemu-nbd --cache=<x> -c /dev/nbd0 a
cp /dev/zero /dev/nbd0
time perl -MIO::Handle -e 'STDOUT->sync or die$!' 1<> /dev/nbd0

With cache=writethrough (the default), it takes over 10 minutes to write those 100MB worth of zeroes. Running a strace, we see the recvfrom and sentos delayed by each 1kb write(2)s to disk (10 to 30 ms per write).

With cache=off, it takes about 30 seconds.

With cache=writeback, it takes about 3 seconds, which is similar to the performance you get with nbd-server

Note that the cp command runs instantly as the data is buffered by the client (the kernel), and not sent to qemu-nbd until the fsync(2) is called.

[Qemu-devel] [Bug 595117] Re: qemu-nbd slow and missing "writeback" cache option

[Serge Hallyn]

** Changed in: qemu-kvm (Ubuntu)
       Status: Incomplete => Confirmed

-- 
qemu-nbd slow and missing "writeback" cache option
https://bugs.launchpad.net/bugs/595117
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.

Status in QEMU: Invalid
Status in “qemu-kvm” package in Ubuntu: Confirmed

Bug description:
Binary package hint: qemu-kvm

dpkg -l | grep qemu
ii  kvm                                  1:84+dfsg-0ubuntu16+0.12.3+noroms+0ubuntu9            dummy transitional pacakge from kvm to qemu-
ii  qemu                                 0.12.3+noroms-0ubuntu9                                dummy transitional pacakge from qemu to qemu
ii  qemu-common                          0.12.3+noroms-0ubuntu9                                qemu common functionality (bios, documentati
ii  qemu-kvm                             0.12.3+noroms-0ubuntu9                                Full virtualization on i386 and amd64 hardwa
ii  qemu-kvm-extras                      0.12.3+noroms-0ubuntu9                                fast processor emulator binaries for non-x86
ii  qemu-launcher                        1.7.4-1ubuntu2                                        GTK+ front-end to QEMU computer emulator
ii  qemuctl                              0.2-2                                                 controlling GUI for qemu

lucid amd64.

qemu-nbd is a lot slower when writing to disk than say nbd-server.

It appears it is because by default the disk image it serves is open with O_SYNC. The --nocache option, unintuitively, makes matters a bit better because it causes the image to be open with O_DIRECT instead of O_SYNC.

The qemu code allows an image to be open without any of those flags, but unfortunately qemu-nbd doesn't have the option to do that (qemu doesn't allow the image to be open with both O_SYNC and O_DIRECT though).

The default of qemu-img (of using O_SYNC) is not very sensible because anyway, the client (the kernel) uses caches (write-back), (and "qemu-nbd -d" doesn't flush those by the way). So if for instance qemu-nbd is killed, regardless of whether qemu-nbd uses O_SYNC, O_DIRECT or not, the data in the image will not be consistent anyway, unless "syncs" are done by the client (like fsync on the nbd device or sync mount option), and with qemu-nbd's O_SYNC mode, those "sync"s will be extremely slow.

Attached is a patch that adds a --cache={off,none,writethrough,writeback} option to qemu-nbd.

--cache=off is the same as --nocache (that is use O_DIRECT), writethrough is using O_SYNC and is still the default so this patch doesn't change the functionality. writeback is none of those flags, so is the addition of this patch. The patch also does an fsync upon "qemu-nbd -d" to make sure data is flushed to the image before removing the nbd.

Consider this test scenario:

dd bs=1M count=100 of=a < /dev/null
qemu-nbd --cache=<x> -c /dev/nbd0 a
cp /dev/zero /dev/nbd0
time perl -MIO::Handle -e 'STDOUT->sync or die$!' 1<> /dev/nbd0

With cache=writethrough (the default), it takes over 10 minutes to write those 100MB worth of zeroes. Running a strace, we see the recvfrom and sentos delayed by each 1kb write(2)s to disk (10 to 30 ms per write).

With cache=off, it takes about 30 seconds.

With cache=writeback, it takes about 3 seconds, which is similar to the performance you get with nbd-server

Note that the cp command runs instantly as the data is buffered by the client (the kernel), and not sent to qemu-nbd until the fsync(2) is called.

[Qemu-devel] [Bug 595117] Re: qemu-nbd slow and missing "writeback" cache option

[Serge Hallyn]

** Changed in: qemu-kvm (Ubuntu)
       Status: Confirmed => Incomplete

-- 
qemu-nbd slow and missing "writeback" cache option
https://bugs.launchpad.net/bugs/595117
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.

Status in QEMU: Invalid
Status in “qemu-kvm” package in Ubuntu: Incomplete

Bug description:
Binary package hint: qemu-kvm

dpkg -l | grep qemu
ii  kvm                                  1:84+dfsg-0ubuntu16+0.12.3+noroms+0ubuntu9            dummy transitional pacakge from kvm to qemu-
ii  qemu                                 0.12.3+noroms-0ubuntu9                                dummy transitional pacakge from qemu to qemu
ii  qemu-common                          0.12.3+noroms-0ubuntu9                                qemu common functionality (bios, documentati
ii  qemu-kvm                             0.12.3+noroms-0ubuntu9                                Full virtualization on i386 and amd64 hardwa
ii  qemu-kvm-extras                      0.12.3+noroms-0ubuntu9                                fast processor emulator binaries for non-x86
ii  qemu-launcher                        1.7.4-1ubuntu2                                        GTK+ front-end to QEMU computer emulator
ii  qemuctl                              0.2-2                                                 controlling GUI for qemu

lucid amd64.

qemu-nbd is a lot slower when writing to disk than say nbd-server.

It appears it is because by default the disk image it serves is open with O_SYNC. The --nocache option, unintuitively, makes matters a bit better because it causes the image to be open with O_DIRECT instead of O_SYNC.

The qemu code allows an image to be open without any of those flags, but unfortunately qemu-nbd doesn't have the option to do that (qemu doesn't allow the image to be open with both O_SYNC and O_DIRECT though).

The default of qemu-img (of using O_SYNC) is not very sensible because anyway, the client (the kernel) uses caches (write-back), (and "qemu-nbd -d" doesn't flush those by the way). So if for instance qemu-nbd is killed, regardless of whether qemu-nbd uses O_SYNC, O_DIRECT or not, the data in the image will not be consistent anyway, unless "syncs" are done by the client (like fsync on the nbd device or sync mount option), and with qemu-nbd's O_SYNC mode, those "sync"s will be extremely slow.

Attached is a patch that adds a --cache={off,none,writethrough,writeback} option to qemu-nbd.

--cache=off is the same as --nocache (that is use O_DIRECT), writethrough is using O_SYNC and is still the default so this patch doesn't change the functionality. writeback is none of those flags, so is the addition of this patch. The patch also does an fsync upon "qemu-nbd -d" to make sure data is flushed to the image before removing the nbd.

Consider this test scenario:

dd bs=1M count=100 of=a < /dev/null
qemu-nbd --cache=<x> -c /dev/nbd0 a
cp /dev/zero /dev/nbd0
time perl -MIO::Handle -e 'STDOUT->sync or die$!' 1<> /dev/nbd0

With cache=writethrough (the default), it takes over 10 minutes to write those 100MB worth of zeroes. Running a strace, we see the recvfrom and sentos delayed by each 1kb write(2)s to disk (10 to 30 ms per write).

With cache=off, it takes about 30 seconds.

With cache=writeback, it takes about 3 seconds, which is similar to the performance you get with nbd-server

Note that the cp command runs instantly as the data is buffered by the client (the kernel), and not sent to qemu-nbd until the fsync(2) is called.

Re: [Qemu-devel] [Bug 595117] Re: qemu-nbd slow and missing "writeback" cache option

[Jamie Lokier]

Serge Hallyn wrote:
> The default of qemu-img (of using O_SYNC) is not very sensible
> because anyway, the client (the kernel) uses caches (write-back),
> (and "qemu-nbd -d" doesn't flush those by the way). So if for
> instance qemu-nbd is killed, regardless of whether qemu-nbd uses
> O_SYNC, O_DIRECT or not, the data in the image will not be
> consistent anyway, unless "syncs" are done by the client (like fsync
> on the nbd device or sync mount option), and with qemu-nbd's O_SYNC
> mode, those "sync"s will be extremely slow.

Do the "client syncs" cause the nbd server to fsync or fdatasync the file?

> It appears it is because by default the disk image it serves is open
> with O_SYNC. The --nocache option, unintuitively, makes matters a
> bit better because it causes the image to be open with O_DIRECT
> instead of O_SYNC.
[...]
> --cache=off is the same as --nocache (that is use O_DIRECT),
> writethrough is using O_SYNC and is still the default so this patch
> doesn't change the functionality. writeback is none of those flags,
> so is the addition of this patch. The patch also does an fsync upon
> "qemu-nbd -d" to make sure data is flushed to the image before
> removing the nbd.

I really wish qemu's options didn't give the false impression
"nocache" does less caching than "writethrough".  O_DIRECT does
caching in the disk controller/hardware, while O_SYNC hopefully does
not, nowadays.

-- Jamie

Re: [Qemu-devel] [Bug 595117] Re: qemu-nbd slow and missing "writeback" cache option

[Christoph Hellwig]

On Thu, Jun 24, 2010 at 01:16:03AM +0100, Jamie Lokier wrote:
> Serge Hallyn wrote:
> > The default of qemu-img (of using O_SYNC) is not very sensible
> > because anyway, the client (the kernel) uses caches (write-back),
> > (and "qemu-nbd -d" doesn't flush those by the way). So if for
> > instance qemu-nbd is killed, regardless of whether qemu-nbd uses
> > O_SYNC, O_DIRECT or not, the data in the image will not be
> > consistent anyway, unless "syncs" are done by the client (like fsync
> > on the nbd device or sync mount option), and with qemu-nbd's O_SYNC
> > mode, those "sync"s will be extremely slow.
> 
> Do the "client syncs" cause the nbd server to fsync or fdatasync the file?

NBD does not have support for cache flushes.  Any nbd server needs to
use O_DSYNC-like semantics.

> I really wish qemu's options didn't give the false impression
> "nocache" does less caching than "writethrough".  O_DIRECT does
> caching in the disk controller/hardware, while O_SYNC hopefully does
> not, nowadays.

The current cache= options are misleading in many ways.  I'll post a
patchset soon to distangle the notion of using direct vs buffered I/O
from exposing and implementing a guest visible volatile write cache.

Exposing these improvements on the command linkes will have to wait for
the new -blockdev option.

Re: [Qemu-devel] [Bug 595117] Re: qemu-nbd slow and missing "writeback" cache option

[Stephane Chazelas]

2010-06-24 00:16:03 -0000, Jamie Lokier:
> Serge Hallyn wrote:
> > The default of qemu-img (of using O_SYNC) is not very sensible
> > because anyway, the client (the kernel) uses caches (write-back),
> > (and "qemu-nbd -d" doesn't flush those by the way). So if for
> > instance qemu-nbd is killed, regardless of whether qemu-nbd uses
> > O_SYNC, O_DIRECT or not, the data in the image will not be
> > consistent anyway, unless "syncs" are done by the client (like fsync
> > on the nbd device or sync mount option), and with qemu-nbd's O_SYNC
> > mode, those "sync"s will be extremely slow.
> 
> Do the "client syncs" cause the nbd server to fsync or fdatasync the
> file?

The clients syncs cause the data to be sent to the server. The
server then writes it to disk and each write blocks until the
data is written physically on disk with O_SYNC.

> > It appears it is because by default the disk image it serves is open
> > with O_SYNC. The --nocache option, unintuitively, makes matters a
> > bit better because it causes the image to be open with O_DIRECT
> > instead of O_SYNC.
> [...]
> > --cache=off is the same as --nocache (that is use O_DIRECT),
> > writethrough is using O_SYNC and is still the default so this patch
> > doesn't change the functionality. writeback is none of those flags,
> > so is the addition of this patch. The patch also does an fsync upon
> > "qemu-nbd -d" to make sure data is flushed to the image before
> > removing the nbd.
> 
> I really wish qemu's options didn't give the false impression
> "nocache" does less caching than "writethrough".  O_DIRECT does
> caching in the disk controller/hardware, while O_SYNC hopefully does
> not, nowadays.
[...]

Note that I use the same "none", "writethrough", "writeback" as
another utility shipped with qemu for consistency (see vl.c in
the source), I don't mind about the words as long as the
"writeback" functionality is available.

Cheers,
Stephane

-- 
qemu-nbd slow and missing "writeback" cache option
https://bugs.launchpad.net/bugs/595117
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.

Status in QEMU: Invalid
Status in “qemu-kvm” package in Ubuntu: Incomplete

Bug description:
Binary package hint: qemu-kvm

dpkg -l | grep qemu
ii  kvm                                  1:84+dfsg-0ubuntu16+0.12.3+noroms+0ubuntu9            dummy transitional pacakge from kvm to qemu-
ii  qemu                                 0.12.3+noroms-0ubuntu9                                dummy transitional pacakge from qemu to qemu
ii  qemu-common                          0.12.3+noroms-0ubuntu9                                qemu common functionality (bios, documentati
ii  qemu-kvm                             0.12.3+noroms-0ubuntu9                                Full virtualization on i386 and amd64 hardwa
ii  qemu-kvm-extras                      0.12.3+noroms-0ubuntu9                                fast processor emulator binaries for non-x86
ii  qemu-launcher                        1.7.4-1ubuntu2                                        GTK+ front-end to QEMU computer emulator
ii  qemuctl                              0.2-2                                                 controlling GUI for qemu

lucid amd64.

qemu-nbd is a lot slower when writing to disk than say nbd-server.

It appears it is because by default the disk image it serves is open with O_SYNC. The --nocache option, unintuitively, makes matters a bit better because it causes the image to be open with O_DIRECT instead of O_SYNC.

The qemu code allows an image to be open without any of those flags, but unfortunately qemu-nbd doesn't have the option to do that (qemu doesn't allow the image to be open with both O_SYNC and O_DIRECT though).

The default of qemu-img (of using O_SYNC) is not very sensible because anyway, the client (the kernel) uses caches (write-back), (and "qemu-nbd -d" doesn't flush those by the way). So if for instance qemu-nbd is killed, regardless of whether qemu-nbd uses O_SYNC, O_DIRECT or not, the data in the image will not be consistent anyway, unless "syncs" are done by the client (like fsync on the nbd device or sync mount option), and with qemu-nbd's O_SYNC mode, those "sync"s will be extremely slow.

Attached is a patch that adds a --cache={off,none,writethrough,writeback} option to qemu-nbd.

--cache=off is the same as --nocache (that is use O_DIRECT), writethrough is using O_SYNC and is still the default so this patch doesn't change the functionality. writeback is none of those flags, so is the addition of this patch. The patch also does an fsync upon "qemu-nbd -d" to make sure data is flushed to the image before removing the nbd.

Consider this test scenario:

dd bs=1M count=100 of=a < /dev/null
qemu-nbd --cache=<x> -c /dev/nbd0 a
cp /dev/zero /dev/nbd0
time perl -MIO::Handle -e 'STDOUT->sync or die$!' 1<> /dev/nbd0

With cache=writethrough (the default), it takes over 10 minutes to write those 100MB worth of zeroes. Running a strace, we see the recvfrom and sentos delayed by each 1kb write(2)s to disk (10 to 30 ms per write).

With cache=off, it takes about 30 seconds.

With cache=writeback, it takes about 3 seconds, which is similar to the performance you get with nbd-server

Note that the cp command runs instantly as the data is buffered by the client (the kernel), and not sent to qemu-nbd until the fsync(2) is called.

[Qemu-devel] [Bug 595117] Re: qemu-nbd slow and missing "writeback" cache option

[Launchpad Bug Tracker]

[Expired for qemu-kvm (Ubuntu) because there has been no activity for 60
days.]

** Changed in: qemu-kvm (Ubuntu)
       Status: Incomplete => Expired

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/595117

Title:
  qemu-nbd slow and missing "writeback" cache option

Status in QEMU:
  Invalid
Status in “qemu-kvm” package in Ubuntu:
  Expired

Bug description:
  Binary package hint: qemu-kvm

dpkg -l | grep qemu
ii  kvm                                  1:84+dfsg-0ubuntu16+0.12.3+noroms+0ubuntu9            dummy transitional pacakge from kvm to qemu-
ii  qemu                                 0.12.3+noroms-0ubuntu9                                dummy transitional pacakge from qemu to qemu
ii  qemu-common                          0.12.3+noroms-0ubuntu9                                qemu common functionality (bios, documentati
ii  qemu-kvm                             0.12.3+noroms-0ubuntu9                                Full virtualization on i386 and amd64 hardwa
ii  qemu-kvm-extras                      0.12.3+noroms-0ubuntu9                                fast processor emulator binaries for non-x86
ii  qemu-launcher                        1.7.4-1ubuntu2                                        GTK+ front-end to QEMU computer emulator
ii  qemuctl                              0.2-2                                                 controlling GUI for qemu

lucid amd64.

qemu-nbd is a lot slower when writing to disk than say nbd-server.

It appears it is because by default the disk image it serves is open with O_SYNC. The --nocache option, unintuitively, makes matters a bit better because it causes the image to be open with O_DIRECT instead of O_SYNC.

The qemu code allows an image to be open without any of those flags, but unfortunately qemu-nbd doesn't have the option to do that (qemu doesn't allow the image to be open with both O_SYNC and O_DIRECT though).

The default of qemu-img (of using O_SYNC) is not very sensible because anyway, the client (the kernel) uses caches (write-back), (and "qemu-nbd -d" doesn't flush those by the way). So if for instance qemu-nbd is killed, regardless of whether qemu-nbd uses O_SYNC, O_DIRECT or not, the data in the image will not be consistent anyway, unless "syncs" are done by the client (like fsync on the nbd device or sync mount option), and with qemu-nbd's O_SYNC mode, those "sync"s will be extremely slow.

Attached is a patch that adds a --cache={off,none,writethrough,writeback} option to qemu-nbd.

--cache=off is the same as --nocache (that is use O_DIRECT), writethrough is using O_SYNC and is still the default so this patch doesn't change the functionality. writeback is none of those flags, so is the addition of this patch. The patch also does an fsync upon "qemu-nbd -d" to make sure data is flushed to the image before removing the nbd.

Consider this test scenario:

dd bs=1M count=100 of=a < /dev/null
qemu-nbd --cache=<x> -c /dev/nbd0 a
cp /dev/zero /dev/nbd0
time perl -MIO::Handle -e 'STDOUT->sync or die$!' 1<> /dev/nbd0

With cache=writethrough (the default), it takes over 10 minutes to write those 100MB worth of zeroes. Running a strace, we see the recvfrom and sentos delayed by each 1kb write(2)s to disk (10 to 30 ms per write).

With cache=off, it takes about 30 seconds.

With cache=writeback, it takes about 3 seconds, which is similar to the performance you get with nbd-server

Note that the cp command runs instantly as the data is buffered by the client (the kernel), and not sent to qemu-nbd until the fsync(2) is called.

[Qemu-devel] [Bug 595117] Re: qemu-nbd slow and missing "writeback" cache option

[Stephane Chazelas]

For the record, there's more on that bug at
http://thread.gmane.org/gmane.linux.ubuntu.bugs.server/36923

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/595117

Title:
  qemu-nbd slow and missing "writeback" cache option

Status in QEMU:
  Invalid
Status in “qemu-kvm” package in Ubuntu:
  Expired

Bug description:
  Binary package hint: qemu-kvm

dpkg -l | grep qemu
ii  kvm                                  1:84+dfsg-0ubuntu16+0.12.3+noroms+0ubuntu9            dummy transitional pacakge from kvm to qemu-
ii  qemu                                 0.12.3+noroms-0ubuntu9                                dummy transitional pacakge from qemu to qemu
ii  qemu-common                          0.12.3+noroms-0ubuntu9                                qemu common functionality (bios, documentati
ii  qemu-kvm                             0.12.3+noroms-0ubuntu9                                Full virtualization on i386 and amd64 hardwa
ii  qemu-kvm-extras                      0.12.3+noroms-0ubuntu9                                fast processor emulator binaries for non-x86
ii  qemu-launcher                        1.7.4-1ubuntu2                                        GTK+ front-end to QEMU computer emulator
ii  qemuctl                              0.2-2                                                 controlling GUI for qemu

lucid amd64.

qemu-nbd is a lot slower when writing to disk than say nbd-server.

It appears it is because by default the disk image it serves is open with O_SYNC. The --nocache option, unintuitively, makes matters a bit better because it causes the image to be open with O_DIRECT instead of O_SYNC.

The qemu code allows an image to be open without any of those flags, but unfortunately qemu-nbd doesn't have the option to do that (qemu doesn't allow the image to be open with both O_SYNC and O_DIRECT though).

The default of qemu-img (of using O_SYNC) is not very sensible because anyway, the client (the kernel) uses caches (write-back), (and "qemu-nbd -d" doesn't flush those by the way). So if for instance qemu-nbd is killed, regardless of whether qemu-nbd uses O_SYNC, O_DIRECT or not, the data in the image will not be consistent anyway, unless "syncs" are done by the client (like fsync on the nbd device or sync mount option), and with qemu-nbd's O_SYNC mode, those "sync"s will be extremely slow.

Attached is a patch that adds a --cache={off,none,writethrough,writeback} option to qemu-nbd.

--cache=off is the same as --nocache (that is use O_DIRECT), writethrough is using O_SYNC and is still the default so this patch doesn't change the functionality. writeback is none of those flags, so is the addition of this patch. The patch also does an fsync upon "qemu-nbd -d" to make sure data is flushed to the image before removing the nbd.

Consider this test scenario:

dd bs=1M count=100 of=a < /dev/null
qemu-nbd --cache=<x> -c /dev/nbd0 a
cp /dev/zero /dev/nbd0
time perl -MIO::Handle -e 'STDOUT->sync or die$!' 1<> /dev/nbd0

With cache=writethrough (the default), it takes over 10 minutes to write those 100MB worth of zeroes. Running a strace, we see the recvfrom and sentos delayed by each 1kb write(2)s to disk (10 to 30 ms per write).

With cache=off, it takes about 30 seconds.

With cache=writeback, it takes about 3 seconds, which is similar to the performance you get with nbd-server

Note that the cp command runs instantly as the data is buffered by the client (the kernel), and not sent to qemu-nbd until the fsync(2) is called.

[Qemu-devel] [Bug 595117] Re: qemu-nbd slow and missing "writeback" cache option

[Serge Hallyn]

@Stephane,

did upstream ever accept your patch?

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/595117

Title:
  qemu-nbd slow and missing "writeback" cache option

Status in QEMU:
  Invalid
Status in “qemu-kvm” package in Ubuntu:
  Expired

Bug description:
  Binary package hint: qemu-kvm

dpkg -l | grep qemu
ii  kvm                                  1:84+dfsg-0ubuntu16+0.12.3+noroms+0ubuntu9            dummy transitional pacakge from kvm to qemu-
ii  qemu                                 0.12.3+noroms-0ubuntu9                                dummy transitional pacakge from qemu to qemu
ii  qemu-common                          0.12.3+noroms-0ubuntu9                                qemu common functionality (bios, documentati
ii  qemu-kvm                             0.12.3+noroms-0ubuntu9                                Full virtualization on i386 and amd64 hardwa
ii  qemu-kvm-extras                      0.12.3+noroms-0ubuntu9                                fast processor emulator binaries for non-x86
ii  qemu-launcher                        1.7.4-1ubuntu2                                        GTK+ front-end to QEMU computer emulator
ii  qemuctl                              0.2-2                                                 controlling GUI for qemu

lucid amd64.

qemu-nbd is a lot slower when writing to disk than say nbd-server.

It appears it is because by default the disk image it serves is open with O_SYNC. The --nocache option, unintuitively, makes matters a bit better because it causes the image to be open with O_DIRECT instead of O_SYNC.

The qemu code allows an image to be open without any of those flags, but unfortunately qemu-nbd doesn't have the option to do that (qemu doesn't allow the image to be open with both O_SYNC and O_DIRECT though).

The default of qemu-img (of using O_SYNC) is not very sensible because anyway, the client (the kernel) uses caches (write-back), (and "qemu-nbd -d" doesn't flush those by the way). So if for instance qemu-nbd is killed, regardless of whether qemu-nbd uses O_SYNC, O_DIRECT or not, the data in the image will not be consistent anyway, unless "syncs" are done by the client (like fsync on the nbd device or sync mount option), and with qemu-nbd's O_SYNC mode, those "sync"s will be extremely slow.

Attached is a patch that adds a --cache={off,none,writethrough,writeback} option to qemu-nbd.

--cache=off is the same as --nocache (that is use O_DIRECT), writethrough is using O_SYNC and is still the default so this patch doesn't change the functionality. writeback is none of those flags, so is the addition of this patch. The patch also does an fsync upon "qemu-nbd -d" to make sure data is flushed to the image before removing the nbd.

Consider this test scenario:

dd bs=1M count=100 of=a < /dev/null
qemu-nbd --cache=<x> -c /dev/nbd0 a
cp /dev/zero /dev/nbd0
time perl -MIO::Handle -e 'STDOUT->sync or die$!' 1<> /dev/nbd0

With cache=writethrough (the default), it takes over 10 minutes to write those 100MB worth of zeroes. Running a strace, we see the recvfrom and sentos delayed by each 1kb write(2)s to disk (10 to 30 ms per write).

With cache=off, it takes about 30 seconds.

With cache=writeback, it takes about 3 seconds, which is similar to the performance you get with nbd-server

Note that the cp command runs instantly as the data is buffered by the client (the kernel), and not sent to qemu-nbd until the fsync(2) is called.