[Qemu-devel] virtio-serial-pci very expensive during live migration

[Qemu-devel] virtio-serial-pci very expensive during live migration

[Chris Friesen]

Hi,

I recently made the unfortunate discovery that virtio-serial-pci is 
quite expensive to stop/start during live migration.

By default we support 32 ports, each of which uses 2 queues.  In my case 
it takes 2-3ms per queue to disconnect on the source host, and another 
2-3ms per queue to connect on the target host, for a total cost of >300ms.

In our case this roughly tripled the outage times of a libvirt-based 
live migration, from 150ms to almost 500ms.

It seems like the main problem is that we loop over all the queues, 
calling virtio_pci_set_host_notifier_internal() on each of them.  That 
in turn calls memory_region_add_eventfd(), which calls 
memory_region_transaction_commit(), which scans over all the address 
spaces, which seems to take the vast majority of the time.

Yes, setting the max_ports value to something smaller does help, but 
each port still adds 10-12ms to the overall live migration time, which 
is crazy.

Is there anything that could be done to make this code more efficient? 
Could we tweak the API so that we add all the eventfds and then do a 
single commit at the end?  Do we really need to scan the entire address 
space?  I don't know the code well enough to answer that sort of 
question, but I'm hoping that one of you does.

Thanks,
Chris

Re: [Qemu-devel] virtio-serial-pci very expensive during live migration

[Markus Armbruster]

Copying Amit.

Chris Friesen <chris.friesen@windriver.com> writes:

> Hi,
>
> I recently made the unfortunate discovery that virtio-serial-pci is
> quite expensive to stop/start during live migration.
>
> By default we support 32 ports, each of which uses 2 queues.  In my
> case it takes 2-3ms per queue to disconnect on the source host, and
> another 2-3ms per queue to connect on the target host, for a total
> cost of >300ms.
>
> In our case this roughly tripled the outage times of a libvirt-based
> live migration, from 150ms to almost 500ms.
>
> It seems like the main problem is that we loop over all the queues,
> calling virtio_pci_set_host_notifier_internal() on each of them.  That
> in turn calls memory_region_add_eventfd(), which calls
> memory_region_transaction_commit(), which scans over all the address
> spaces, which seems to take the vast majority of the time.
>
> Yes, setting the max_ports value to something smaller does help, but
> each port still adds 10-12ms to the overall live migration time, which
> is crazy.
>
> Is there anything that could be done to make this code more efficient?
> Could we tweak the API so that we add all the eventfds and then do a
> single commit at the end?  Do we really need to scan the entire
> address space?  I don't know the code well enough to answer that sort
> of question, but I'm hoping that one of you does.
>
> Thanks,
> Chris

Re: [Qemu-devel] virtio-serial-pci very expensive during live migration

[Paolo Bonzini]

Il 06/05/2014 22:01, Chris Friesen ha scritto:
>
> It seems like the main problem is that we loop over all the queues,
> calling virtio_pci_set_host_notifier_internal() on each of them.  That
> in turn calls memory_region_add_eventfd(), which calls
> memory_region_transaction_commit(), which scans over all the address
> spaces, which seems to take the vast majority of the time.

Yes, you can wrap the entire loop with memory_region_transaction_begin 
and memory_region_transaction_commit.  Can you try that?

Paolo

Re: [Qemu-devel] virtio-serial-pci very expensive during live migration

[Chris Friesen]

On 05/07/2014 12:39 AM, Paolo Bonzini wrote:
> Il 06/05/2014 22:01, Chris Friesen ha scritto:
>>
>> It seems like the main problem is that we loop over all the queues,
>> calling virtio_pci_set_host_notifier_internal() on each of them.  That
>> in turn calls memory_region_add_eventfd(), which calls
>> memory_region_transaction_commit(), which scans over all the address
>> spaces, which seems to take the vast majority of the time.
>
> Yes, you can wrap the entire loop with memory_region_transaction_begin
> and memory_region_transaction_commit.  Can you try that?

I tried the patch below.  Unfortunately it seems to cause qemu to crash.

That said, I'm on a patched version so it might be good if someone else 
tried it.  If it works for other people it might mean bugs in our other 
patches.

Chris





Index: qemu-1.4.2/hw/virtio-pci.c
===================================================================
--- qemu-1.4.2.orig/hw/virtio-pci.c
+++ qemu-1.4.2/hw/virtio-pci.c
@@ -32,6 +32,7 @@
  #include "virtio-pci.h"
  #include "qemu/range.h"
  #include "virtio-bus.h"
+#include "exec/memory.h"

  /* from Linux's linux/virtio_pci.h */

@@ -209,6 +210,7 @@ static void virtio_pci_start_ioeventfd(V
          return;
      }

+    memory_region_transaction_begin();
      for (n = 0; n < VIRTIO_PCI_QUEUE_MAX; n++) {
          if (!virtio_queue_get_num(proxy->vdev, n)) {
              continue;
@@ -219,10 +221,12 @@ static void virtio_pci_start_ioeventfd(V
              goto assign_error;
          }
      }
+    memory_region_transaction_commit();
      proxy->ioeventfd_started = true;
      return;

  assign_error:
+    memory_region_transaction_commit();
      while (--n >= 0) {
          if (!virtio_queue_get_num(proxy->vdev, n)) {
              continue;
@@ -244,6 +248,7 @@ static void virtio_pci_stop_ioeventfd(Vi
          return;
      }

+    memory_region_transaction_begin();
      for (n = 0; n < VIRTIO_PCI_QUEUE_MAX; n++) {
          if (!virtio_queue_get_num(proxy->vdev, n)) {
              continue;
@@ -252,6 +257,7 @@ static void virtio_pci_stop_ioeventfd(Vi
          r = virtio_pci_set_host_notifier_internal(proxy, n, false, false);
          assert(r >= 0);
      }
+    memory_region_transaction_commit();
      proxy->ioeventfd_started = false;
  }

Re: [Qemu-devel] virtio-serial-pci very expensive during live migration

[Amit Shah]

On (Wed) 07 May 2014 [16:25:50], Chris Friesen wrote:
> On 05/07/2014 12:39 AM, Paolo Bonzini wrote:
> >Il 06/05/2014 22:01, Chris Friesen ha scritto:
> >>
> >>It seems like the main problem is that we loop over all the queues,
> >>calling virtio_pci_set_host_notifier_internal() on each of them.  That
> >>in turn calls memory_region_add_eventfd(), which calls
> >>memory_region_transaction_commit(), which scans over all the address
> >>spaces, which seems to take the vast majority of the time.
> >
> >Yes, you can wrap the entire loop with memory_region_transaction_begin
> >and memory_region_transaction_commit.  Can you try that?
> 
> I tried the patch below.  Unfortunately it seems to cause qemu to crash.

This doesn't remove the memory_region_transaction_begin() and
_commit() from memory_region_add_eventfd(), which will still be
called.

Also, if you're not using all 32 ports, you can restrict the max
number of ports for a device using the
virtio-serial-pci.max_ports=uint32 property:

-device virtio-serial-pci,max_ports=2

		Amit

Re: [Qemu-devel] virtio-serial-pci very expensive during live migration

[Paolo Bonzini]

Il 08/05/2014 15:02, Amit Shah ha scritto:
>> > I tried the patch below.  Unfortunately it seems to cause qemu to crash.
> This doesn't remove the memory_region_transaction_begin() and
> _commit() from memory_region_add_eventfd(), which will still be
> called.

That's correct, transactions can be nested.

Paolo

Re: [Qemu-devel] virtio-serial-pci very expensive during live migration

[Amit Shah]

On (Thu) 08 May 2014 [15:14:26], Paolo Bonzini wrote:
> Il 08/05/2014 15:02, Amit Shah ha scritto:
> >>> I tried the patch below.  Unfortunately it seems to cause qemu to crash.
> >This doesn't remove the memory_region_transaction_begin() and
> >_commit() from memory_region_add_eventfd(), which will still be
> >called.
> 
> That's correct, transactions can be nested.

But each _commit() will add to the delay?

		Amit

Re: [Qemu-devel] virtio-serial-pci very expensive during live migration

[Paolo Bonzini]

Il 08/05/2014 15:30, Amit Shah ha scritto:
> On (Thu) 08 May 2014 [15:14:26], Paolo Bonzini wrote:
>> Il 08/05/2014 15:02, Amit Shah ha scritto:
>>>>> I tried the patch below.  Unfortunately it seems to cause qemu to crash.
>>> This doesn't remove the memory_region_transaction_begin() and
>>> _commit() from memory_region_add_eventfd(), which will still be
>>> called.
>>
>> That's correct, transactions can be nested.
>
> But each _commit() will add to the delay?

No, only the outermost one.

Paolo

Re: [Qemu-devel] virtio-serial-pci very expensive during live migration

[Amit Shah]

On (Thu) 08 May 2014 [15:36:13], Paolo Bonzini wrote:
> Il 08/05/2014 15:30, Amit Shah ha scritto:
> >On (Thu) 08 May 2014 [15:14:26], Paolo Bonzini wrote:
> >>Il 08/05/2014 15:02, Amit Shah ha scritto:
> >>>>>I tried the patch below.  Unfortunately it seems to cause qemu to crash.
> >>>This doesn't remove the memory_region_transaction_begin() and
> >>>_commit() from memory_region_add_eventfd(), which will still be
> >>>called.
> >>
> >>That's correct, transactions can be nested.
> >
> >But each _commit() will add to the delay?
> 
> No, only the outermost one.

Helps to read the code :-)

Chris, I just tried a simple test this way:

./x86_64-softmmu/qemu-system-x86_64 -device virtio-serial-pci -device virtserialport -S -monitor stdio -nographic

and it didn't crash for me.  This was with qemu.git.  Perhaps you can
try in a similar way.


		Amit

Re: [Qemu-devel] virtio-serial-pci very expensive during live migration

[Chris Friesen]

On 05/08/2014 07:47 AM, Amit Shah wrote:

> Chris, I just tried a simple test this way:
>
> ./x86_64-softmmu/qemu-system-x86_64 -device virtio-serial-pci -device virtserialport -S -monitor stdio -nographic
>
> and it didn't crash for me.  This was with qemu.git.  Perhaps you can
> try in a similar way.

I just tried it with the "stable-1.4" branch from upstream with my first 
patch added on.

Incidentally, to compile on Fedora 19 I had to delete 
"include/libfdt_env.h" in the qemu package since it was overriding the 
version from libfdt in the host, causing errors like:

/usr/include/fdt.h:58:2: error: unknown type name ‘fdt32_t’
   fdt32_t magic;    /* magic word FDT_MAGIC */




Anyway, it seems to boot up okay, which is better than what I was 
getting before.  I ran it as:


qemu-system-x86_64 -machine accel=kvm -m 1000 guest.img -device 
virtio-serial -chardev socket,path=/tmp/foo,server,nowait,id=foo -device 
virtserialport,chardev=foo,name=myfoo


My earlier version was using a yocto-modified version of qemu, further 
modified locally.  There's probably a bug in there somewhere.


Chris

Re: [Qemu-devel] virtio-serial-pci very expensive during live migration

[Chris Friesen]

On 05/08/2014 09:40 AM, Chris Friesen wrote:
> On 05/08/2014 07:47 AM, Amit Shah wrote:
>
>> Chris, I just tried a simple test this way:
>>
>> ./x86_64-softmmu/qemu-system-x86_64 -device virtio-serial-pci -device
>> virtserialport -S -monitor stdio -nographic
>>
>> and it didn't crash for me.  This was with qemu.git.  Perhaps you can
>> try in a similar way.
>
> I just tried it with the "stable-1.4" branch from upstream with my first
> patch added on.

> Anyway, it seems to boot up okay, which is better than what I was
> getting before.


Turns out I spoke too soon.  With the patch applied, it boots, but if I 
try to do a live migration both the source and destination crash.  This 
happens for both the master branch as well as the stable-1.4 branch.

If I back out the patch, it works fine.  If I leave the patch in and 
disable kvm acceleration it works fine.


I'm running the source as

/tmp/qemu-system-x86_64-upstream  -machine accel=kvm -m 1000 
test-cgcs-guest.img -device virtio-serial -chardev 
socket,path=/tmp/foo,server,nowait,id=foo -device 
virtserialport,chardev=foo,name=myfoo -monitor stdio


and the dest as

/tmp/qemu-system-x86_64-upstream  -machine accel=kvm -m 1000 
test-cgcs-guest.img -device virtio-serial -chardev 
socket,path=/tmp/foo,server,nowait,id=foo -device 
virtserialport,chardev=foo,name=myfoo -monitor stdio -incoming tcp:0:4444


and I'm triggering the migration with

migrate -d tcp:localhost:4444



On the dest side monitor I get:

(qemu) qemu: warning: error while loading state section id 3
load of migration failed



I managed to get gdb working, but it's not very helpful.  With gdb 
attached to the destination process I just get:

[Thread 0x7f760f6dc700 (LWP 15328) exited]
[Inferior 1 (process 15326) exited normally]

Re: [Qemu-devel] virtio-serial-pci very expensive during live migration

[Paolo Bonzini]

Il 09/05/2014 02:53, Chris Friesen ha scritto:
>
>
> Turns out I spoke too soon.  With the patch applied, it boots, but if I
> try to do a live migration both the source and destination crash.  This
> happens for both the master branch as well as the stable-1.4 branch.

The destination doesn't crash, it simply stops because it got an 
incomplete migration stream.  Thanks for the report, the patch seems 
correct so it's worthwhile looking at it more closely.

> If I back out the patch, it works fine.  If I leave the patch in and
> disable kvm acceleration it works fine.

Indeed, non-KVM doesn't use ioeventfd at all.

Paolo

Re: [Qemu-devel] virtio-serial-pci very expensive during live migration

[Chris Friesen]

On 05/09/2014 02:19 AM, Paolo Bonzini wrote:
> Il 09/05/2014 02:53, Chris Friesen ha scritto:
>>
>>
>> Turns out I spoke too soon.  With the patch applied, it boots, but if I
>> try to do a live migration both the source and destination crash.  This
>> happens for both the master branch as well as the stable-1.4 branch.
>
> The destination doesn't crash, it simply stops because it got an
> incomplete migration stream.  Thanks for the report, the patch seems
> correct so it's worthwhile looking at it more closely.
>
>> If I back out the patch, it works fine.  If I leave the patch in and
>> disable kvm acceleration it works fine.
>
> Indeed, non-KVM doesn't use ioeventfd at all.

Did anyone ever figure out why wrapping the whole thing in 
memory_region_transaction_begin()/memory_region_transaction_commit() 
caused problems?

Chris

Re: [Qemu-devel] virtio-serial-pci very expensive during live migration

[Paolo Bonzini]

Il 19/06/2014 17:19, Chris Friesen ha scritto:
> On 05/09/2014 02:19 AM, Paolo Bonzini wrote:
>> Il 09/05/2014 02:53, Chris Friesen ha scritto:
>>>
>>>
>>> Turns out I spoke too soon.  With the patch applied, it boots, but if I
>>> try to do a live migration both the source and destination crash.  This
>>> happens for both the master branch as well as the stable-1.4 branch.
>>
>> The destination doesn't crash, it simply stops because it got an
>> incomplete migration stream.  Thanks for the report, the patch seems
>> correct so it's worthwhile looking at it more closely.
>>
>>> If I back out the patch, it works fine.  If I leave the patch in and
>>> disable kvm acceleration it works fine.
>>
>> Indeed, non-KVM doesn't use ioeventfd at all.
>
> Did anyone ever figure out why wrapping the whole thing in
> memory_region_transaction_begin()/memory_region_transaction_commit()
> caused problems?

No, sorry, I didn't have time.  On the other hand patches went in that 
avoided unnecessary work on ioeventfds, so you may want to test again 
with the latest master branch.

Paolo

Re: [Qemu-devel] virtio-serial-pci very expensive during live migration

[Chris Friesen]

On 05/08/2014 07:30 AM, Amit Shah wrote:
> On (Thu) 08 May 2014 [15:14:26], Paolo Bonzini wrote:
>> Il 08/05/2014 15:02, Amit Shah ha scritto:
>>>>> I tried the patch below.  Unfortunately it seems to cause qemu to crash.
>>> This doesn't remove the memory_region_transaction_begin() and
>>> _commit() from memory_region_add_eventfd(), which will still be
>>> called.
>>
>> That's correct, transactions can be nested.
>
> But each _commit() will add to the delay?

No, the _commit() call checks "memory_region_transaction_depth" and only 
does the address space scan when it hits zero.  So with nested 
transactions only the outer commit will actually do the address space scan.

The fact remains that qemu crashes when I apply the patch.  I also tried 
patching it as below in virtio_pci_vmstate_change().  That would allow 
the VM to boot, but it would crash when I tried to do a live migration.

Chris



--- qemu-1.4.2.orig/hw/virtio-pci.c
+++ qemu-1.4.2/hw/virtio-pci.c
@@ -32,6 +32,7 @@
  #include "virtio-pci.h"
  #include "qemu/range.h"
  #include "virtio-bus.h"
+#include "exec/memory.h"

  /* from Linux's linux/virtio_pci.h */

@@ -854,7 +855,7 @@ static int virtio_pci_set_host_notifier(
  static void virtio_pci_vmstate_change(DeviceState *d, bool running)
  {
      VirtIOPCIProxy *proxy = to_virtio_pci_proxy(d);
-
+    memory_region_transaction_begin();
      if (running) {
          /* Try to find out if the guest has bus master disabled, but is
             in ready state. Then we have a buggy guest OS. */
@@ -866,6 +867,7 @@ static void virtio_pci_vmstate_change(De
      } else {
          virtio_pci_stop_ioeventfd(proxy);
      }
+    memory_region_transaction_commit();
  }

  static const VirtIOBindings virtio_pci_bindings = {

Re: [Qemu-devel] virtio-serial-pci very expensive during live migration

[Paolo Bonzini]

Il 08/05/2014 16:31, Chris Friesen ha scritto:
>
>
> The fact remains that qemu crashes when I apply the patch.  I also tried
> patching it as below in virtio_pci_vmstate_change().  That would allow
> the VM to boot, but it would crash when I tried to do a live migration.

Can you give us your command line and a backtrace?

Paolo

Re: [Qemu-devel] virtio-serial-pci very expensive during live migration

[Chris Friesen]

On 05/08/2014 08:34 AM, Paolo Bonzini wrote:
> Il 08/05/2014 16:31, Chris Friesen ha scritto:
>>
>>
>> The fact remains that qemu crashes when I apply the patch.  I also tried
>> patching it as below in virtio_pci_vmstate_change().  That would allow
>> the VM to boot, but it would crash when I tried to do a live migration.
>
> Can you give us your command line and a backtrace?

For the backtrace, you mean with a core file?  Or is there a better way?

Chris

Re: [Qemu-devel] virtio-serial-pci very expensive during live migration

[Paolo Bonzini]

Il 08/05/2014 17:57, Chris Friesen ha scritto:
>>>
>>> The fact remains that qemu crashes when I apply the patch.  I also tried
>>> patching it as below in virtio_pci_vmstate_change().  That would allow
>>> the VM to boot, but it would crash when I tried to do a live migration.
>>
>> Can you give us your command line and a backtrace?
>
> For the backtrace, you mean with a core file?  Or is there a better way?

You can attach gdb to the qemu process, and run "thread apply all bt" 
when QEMU crashes.

Paolo

Re: [Qemu-devel] virtio-serial-pci very expensive during live migration

[Chris Friesen]

On 05/08/2014 10:02 AM, Paolo Bonzini wrote:
> Il 08/05/2014 17:57, Chris Friesen ha scritto:
>>>>
>>>> The fact remains that qemu crashes when I apply the patch.  I also
>>>> tried
>>>> patching it as below in virtio_pci_vmstate_change().  That would allow
>>>> the VM to boot, but it would crash when I tried to do a live migration.
>>>
>>> Can you give us your command line and a backtrace?
>>
>> For the backtrace, you mean with a core file?  Or is there a better way?
>
> You can attach gdb to the qemu process, and run "thread apply all bt"
> when QEMU crashes.


For the commandline, it's coming from OpenStack via libvirt, so it's 
fairly complicated:

/usr/bin/kvm -c 0x00000000000000000000000000000001 -n 4 
--proc-type=secondary --file-prefix=vs -- -enable-dpdk -name 
instance-00000007 -S -machine pc-i440fx-1.4,accel=kvm,usb=off -m 512 
-mem-prealloc -mem-path /mnt/huge-2048kB/libvirt/qemu -smp 
2,sockets=2,cores=1,threads=1 -uuid b277d785-9c31-45be-91d3-15b7b1aca974 
-smbios type=1,manufacturer=OpenStack Foundation,product=OpenStack 
Nova,version=2013.2.2,serial=00000000-0000-3848-3342-4e5037313536,uuid=b277d785-9c31-45be-91d3-15b7b1aca974 
-no-user-config -nodefaults -chardev 
socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000007.monitor,server,nowait 
-mon chardev=charmonitor,id=monitor,mode=control -rtc 
base=utc,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device 
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device 
virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x3 -drive 
file=/etc/nova/instances/b277d785-9c31-45be-91d3-15b7b1aca974/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none 
-device 
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 
-chardev 
file,id=charserial0,path=/etc/nova/instances/b277d785-9c31-45be-91d3-15b7b1aca974/console.log 
-device isa-serial,chardev=charserial0,id=serial0 -chardev 
pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 
-chardev 
socket,id=charchannel0,path=/var/lib/libvirt/qemu/cgcs.heartbeat.instance-00000007.sock,server,nowait 
-device 
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=cgcs.heartbeat 
-device usb-tablet,id=input0 -vnc 0.0.0.0:0 -k en-us -vga cirrus -device 
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -device 
avp,port=4a05cedd-3868-449a-9728-899bbea10863,id=avp4a05cedd-38




As for the backtrace, I can attach gdb to the qemu process, but when I 
hit "c" to continue it gives an error:


Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols 
found)...done.
Loaded symbols for /lib64/libnss_files.so.2

warning: File "/lib64/libthread_db-1.0.so" auto-loading has been 
declined by your `auto-load safe-path' set to 
"$debugdir:$datadir/auto-load".

warning: Unable to find libthread_db matching inferior's thread library, 
thread debugging will not be available.
0x00007f56c6366d23 in select () from /lib64/libc.so.6
(gdb) c
Continuing.
Warning:
Cannot insert breakpoint -1.
Error accessing memory address 0x2f7c50: Input/output error.


Chris

Re: [Qemu-devel] virtio-serial-pci very expensive during live migration

[ChenLiang]

On 2014/5/8 22:34, Paolo Bonzini wrote:

> Il 08/05/2014 16:31, Chris Friesen ha scritto:
>>
>>
>> The fact remains that qemu crashes when I apply the patch.  I also tried
>> patching it as below in virtio_pci_vmstate_change().  That would allow
>> the VM to boot, but it would crash when I tried to do a live migration.
> 
> Can you give us your command line and a backtrace?
> 
> Paolo
> 
> 
> 

Hi,
I have test the patch at the qemu.git, qemu crashed when vm is booting.

the backtrace is:

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7f6bf67f9700 (LWP 9740)]
0x00007f6bfacb2b55 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00007f6bfacb2b55 in raise () from /lib64/libc.so.6
#1  0x00007f6bfacb4131 in abort () from /lib64/libc.so.6
#2  0x00007f6bfd51047c in kvm_io_ioeventfd_del (listener=
    0x7f6bfd9ffee0 <kvm_io_listener>, section=0x7f6bf67f87c0, match_data=true, data=
    0, e=0x7f6970000930) at /tmp/qemu/kvm-all.c:879
#3  0x00007f6bfd5163b5 in address_space_add_del_ioeventfds (as=
    0x7f6bfde3d6e0 <address_space_io>, fds_new=0x0, fds_new_nb=0, fds_old=
    0x7f6bfdfd8ce0, fds_old_nb=1) at /tmp/qemu/memory.c:628
#4  0x00007f6bfd51698e in address_space_update_ioeventfds (as=
    0x7f6bfde3d6e0 <address_space_io>) at /tmp/qemu/memory.c:687
#5  0x00007f6bfd517949 in address_space_update_topology (as=
    0x7f6bfde3d6e0 <address_space_io>) at /tmp/qemu/memory.c:780
#6  0x00007f6bfd517a68 in memory_region_transaction_commit ()
    at /tmp/qemu/memory.c:800
#7  0x00007f6bfd3e2942 in virtio_pci_stop_ioeventfd (proxy=0x7f6bfdfde080)
    at hw/virtio/virtio-pci.c:270
#8  0x00007f6bfd3e2aaa in virtio_ioport_write (opaque=0x7f6bfdfde080, addr=18, val=
    0) at hw/virtio/virtio-pci.c:309
#9  0x00007f6bfd3e2ff3 in virtio_pci_config_write (opaque=0x7f6bfdfde080, addr=18,
    val=0, size=1) at hw/virtio/virtio-pci.c:436
#10 0x00007f6bfd515368 in memory_region_write_accessor (mr=0x7f6bfdfde818, addr=18,
    value=0x7f6bf67f8b68, size=1, shift=0, mask=255) at /tmp/qemu/memory.c:441
#11 0x00007f6bfd5154a4 in access_with_adjusted_size (addr=18, value=0x7f6bf67f8b68,
    size=1, access_size_min=1, access_size_max=4, access=
    0x7f6bfd5152df <memory_region_write_accessor>, mr=0x7f6bfdfde818)
    at /tmp/qemu/memory.c:478
#12 0x00007f6bfd518305 in memory_region_dispatch_write (mr=0x7f6bfdfde818, addr=18,
    data=0, size=1) at /tmp/qemu/memory.c:985
#13 0x00007f6bfd51bb1b in io_mem_write (mr=0x7f6bfdfde818, addr=18, val=0, size=1)
    at /tmp/qemu/memory.c:1744
#14 0x00007f6bfd4a0c67 in address_space_rw (as=0x7f6bfde3d6e0 <address_space_io>,
    addr=49170, buf=0x7f6bfd189000 "", len=1, is_write=true)
    at /tmp/qemu/exec.c:2034
#15 0x00007f6bfd511e06 in kvm_handle_io (port=49170, data=0x7f6bfd189000, direction=
    1, size=1, count=1) at /tmp/qemu/kvm-all.c:1558
#16 0x00007f6bfd5123aa in kvm_cpu_exec (cpu=0x7f6bfdf54d50)
    at /tmp/qemu/kvm-all.c:1695
#17 0x00007f6bfd491db6 in qemu_kvm_cpu_thread_fn (arg=0x7f6bfdf54d50)
    at /tmp/qemu/cpus.c:873
#18 0x00007f6bfafff7f6 in start_thread () from /lib64/libpthread.so.0
#19 0x00007f6bfad5b09d in clone () from /lib64/libc.so.6
#20 0x0000000000000000 in ?? ()


the commandline is:
LC_ALL=C PATH=/bin:/sbin:/usr/bin:/usr/sbin HOME=/ QEMU_AUDIO_DRV=none /tmp/qemu/x86_
64-softmmu/qemu-system-x86_64 -name cl_suse -S -machine pc-i440fx-1.5,accel=kvm,usb=o
ff -m 10240 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid 5a09315c-d31
4-49a5-aa51-2168a71bf82d -no-user-config -nodefaults -chardev socket,id=charmonitor,p
ath=/var/lib/libvirt/qemu/cl_suse.monitor,server,nowait -mon chardev=charmonitor,id=m
onitor,mode=control -rtc base=utc -no-hpet -no-shutdown -device piix3-usb-uhci,id=usb
,bus=pci.0,addr=0x1.0x2 -drive file=/mnt/sdb/cl/cl_sles11sp3.img,if=none,id=drive-vir
tio-disk0,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,
addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -chardev pty,id=charser
ial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -
vnc 0.0.0.0:0 -device cirrus-vga,id=video0,vgamem_mb=9,bus=pci.0,addr=0x2 -device vir
tio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6

Re: [Qemu-devel] virtio-serial-pci very expensive during live migration

[Chris Friesen]

On 05/08/2014 07:44 PM, ChenLiang wrote:

> Hi,
> I have test the patch at the qemu.git, qemu crashed when vm is booting.
> 
> the backtrace is:
> 
> Program received signal SIGABRT, Aborted.
> [Switching to Thread 0x7f6bf67f9700 (LWP 9740)]
> 0x00007f6bfacb2b55 in raise () from /lib64/libc.so.6
> (gdb) bt
> #0  0x00007f6bfacb2b55 in raise () from /lib64/libc.so.6
> #1  0x00007f6bfacb4131 in abort () from /lib64/libc.so.6
> #2  0x00007f6bfd51047c in kvm_io_ioeventfd_del (listener=
>      0x7f6bfd9ffee0 <kvm_io_listener>, section=0x7f6bf67f87c0, match_data=true, data=
>      0, e=0x7f6970000930) at /tmp/qemu/kvm-all.c:879
> #3  0x00007f6bfd5163b5 in address_space_add_del_ioeventfds (as=
>      0x7f6bfde3d6e0 <address_space_io>, fds_new=0x0, fds_new_nb=0, fds_old=
>      0x7f6bfdfd8ce0, fds_old_nb=1) at /tmp/qemu/memory.c:628
> #4  0x00007f6bfd51698e in address_space_update_ioeventfds (as=
>      0x7f6bfde3d6e0 <address_space_io>) at /tmp/qemu/memory.c:687
> #5  0x00007f6bfd517949 in address_space_update_topology (as=
>      0x7f6bfde3d6e0 <address_space_io>) at /tmp/qemu/memory.c:780
> #6  0x00007f6bfd517a68 in memory_region_transaction_commit ()
>      at /tmp/qemu/memory.c:800
> #7  0x00007f6bfd3e2942 in virtio_pci_stop_ioeventfd (proxy=0x7f6bfdfde080)
>      at hw/virtio/virtio-pci.c:270


Hah...I attached gdb to the source qemu this time and got basically the same thing.
This is with the stable-1.4 branch plus the patch, using "-machine accel=kvm".

#0  0x00007f4e08e1e9e9 in raise () from /lib64/libc.so.6
#1  0x00007f4e08e200f8 in abort () from /lib64/libc.so.6
#2  0x00007f4e0b30e8a4 in kvm_io_ioeventfd_del (listener=<optimized out>, section=0x7f4e080f2c00, match_data=<optimized out>, data=0, e=<optimized out>) at /home/cfriesen/devel/qemu/kvm-all.c:804
#3  0x00007f4e0b311f60 in address_space_add_del_ioeventfds (fds_old_nb=64, fds_old=0x7f4df8004b40, fds_new_nb=0, fds_new=0x0, as=0x7f4e0bfb9fe0 <address_space_io>) at /home/cfriesen/devel/qemu/memory.c:603
#4  address_space_update_ioeventfds (as=0x7f4e0bfb9fe0 <address_space_io>) at /home/cfriesen/devel/qemu/memory.c:649
#5  address_space_update_topology (as=0x7f4e0bfb9fe0 <address_space_io>) at /home/cfriesen/devel/qemu/memory.c:730
#6  memory_region_transaction_commit () at /home/cfriesen/devel/qemu/memory.c:750
#7  0x00007f4e0b255cb5 in virtio_pci_stop_ioeventfd (proxy=0x7f4e0d3266a0) at hw/virtio-pci.c:259
#8  0x00007f4e0b3086d0 in virtio_vmstate_change (opaque=0x7f4e0d31c370, running=<optimized out>, state=<optimized out>) at /home/cfriesen/devel/qemu/hw/virtio.c:914
#9  0x00007f4e0b2bdf5b in vm_state_notify (running=running@entry=0, state=state@entry=RUN_STATE_FINISH_MIGRATE) at vl.c:1674
#10 0x00007f4e0b2c3f62 in do_vm_stop (state=RUN_STATE_FINISH_MIGRATE) at /home/cfriesen/devel/qemu/cpus.c:446
#11 vm_stop (state=state@entry=RUN_STATE_FINISH_MIGRATE) at /home/cfriesen/devel/qemu/cpus.c:1080
#12 0x00007f4e0b25e5c5 in buffered_file_thread (opaque=0x7f4e0b79a640 <current_migration.19392>) at migration.c:707
#13 0x00007f4e09cacc53 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f4e08ededbd in clone () from /lib64/libc.so.6

Chris

Re: [Qemu-devel] virtio-serial-pci very expensive during live migration

[Gonglei (Arei)]

Hi,


> Subject: Re: [Qemu-devel] virtio-serial-pci very expensive during live migration
> 
> Il 06/05/2014 22:01, Chris Friesen ha scritto:
> >
> > It seems like the main problem is that we loop over all the queues,
> > calling virtio_pci_set_host_notifier_internal() on each of them.  That
> > in turn calls memory_region_add_eventfd(), which calls
> > memory_region_transaction_commit(), which scans over all the address
> > spaces, which seems to take the vast majority of the time.
> 
> Yes, you can wrap the entire loop with memory_region_transaction_begin
> and memory_region_transaction_commit.  Can you try that?
> 
> Paolo
We have discussed the issue before:
https://lists.gnu.org/archive/html/qemu-devel/2014-03/msg00421.html

According Paolo's advice, I have finished the patch and tested it work well.

I will post the patch later.


Best regards,
-Gonglei