Re: [Qemu-devel] [BUG] living migrate vm pause forever

["Dr. David Alan Gilbert" <dgilbert@redhat.com>]

* Yuchen (yu.chen@h3c.com) wrote:
> Sometimes, living migrate vm pause forever, migrate job stop, but very small probability, I can’t reproduce.
> qemu wait semaphore from libvirt send migrate continue, however libvirt wait semaphore from qemu send vm pause.

Hi,
  I've copied in Jiri Denemark from libvirt.
Can you confirm exactly which qemu and libvirt versions you're using
please.

> follow stack:
> qemu:
> Thread 6 (Thread 0x7f50445f3700 (LWP 18120)):
> #0  0x00007f504b84d670 in sem_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x00005574eda1e164 in qemu_sem_wait (sem=sem@entry=0x5574ef6930e0) at qemu-2.12/util/qemu-thread-posix.c:322
> #2  0x00005574ed8dd72e in migration_maybe_pause (s=0x5574ef692f50, current_active_state=0x7f50445f2ae4, new_state=10)
>     at qemu-2.12/migration/migration.c:2106
> #3  0x00005574ed8df51a in migration_completion (s=0x5574ef692f50) at qemu-2.12/migration/migration.c:2137
> #4  migration_iteration_run (s=0x5574ef692f50) at qemu-2.12/migration/migration.c:2311
> #5  migration_thread (opaque=0x5574ef692f50) atqemu-2.12/migration/migration.c:2415
> #6  0x00007f504b847184 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
> #7  0x00007f504b574bed in clone () from /lib/x86_64-linux-gnu/libc.so.6

In migration_maybe_pause we have:

    migrate_set_state(&s->state, *current_active_state,
                      MIGRATION_STATUS_PRE_SWITCHOVER);
    qemu_sem_wait(&s->pause_sem);
    migrate_set_state(&s->state, MIGRATION_STATUS_PRE_SWITCHOVER,
                      new_state);

the line numbers don't match my 2.12.0 checkout; so I guess that it's
that qemu_sem_wait it's stuck at.

QEMU must have sent the switch to PRE_SWITCHOVER and that should have
sent an event to libvirt, and libvirt should notice that - I'm
not sure how to tell whether libvirt has seen that event yet or not?

Dave

> libvirt:
> Thread 95 (Thread 0x7fdb82ffd700 (LWP 28775)):
> #0  0x00007fdd177dc404 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x00007fdd198c3b07 in virCondWait (c=0x7fdbc4003000, m=0x7fdbc4002f30) at ../../../src/util/virthread.c:252
> #2  0x00007fdd198f36d2 in virDomainObjWait (vm=0x7fdbc4002f20) at ../../../src/conf/domain_conf.c:3303
> #3  0x00007fdd09ffaa44 in qemuMigrationRun (driver=0x7fdd000037b0, vm=0x7fdbc4002f20, persist_xml=0x0,
>     cookiein=0x7fdb780084e0 "<qemu-migration>\n  <name>mss-pl_652</name>\n  <uuid>1f2b2334-451e-424b-822a-ea10452abb38</uuid>\n  <hostname>mss </hostname>\n  <hostuuid>334e344a-4130-4336-5534-323544543642</hostuuid>\n</qemu-migra"..., cookieinlen=207, cookieout=0x7fdb82ffcad0, cookieoutlen=0x7fdb82ffcac8, flags=777,
>     resource=0, spec=0x7fdb82ffc670, dconn=0x0, graphicsuri=0x0, nmigrate_disks=0, migrate_disks=0x0, compression=0x7fdb78007990, migParams=0x7fdb82ffc900)
>     at ../../../src/qemu/qemu_migration.c:3937
> #4  0x00007fdd09ffb26a in doNativeMigrate (driver=0x7fdd000037b0, vm=0x7fdbc4002f20, persist_xml=0x0, uri=0x7fdb780073a0 "tcp://172.16.202.17:49152",
>     cookiein=0x7fdb780084e0 "<qemu-migration>\n  <name>mss-pl_652</name>\n  <uuid>1f2b2334-451e-424b-822a-ea10452abb38</uuid>\n  <hostname>mss</hostname>\n  <hos---Type <return> to continue, or q <return> to quit---
> tuuid>334e344a-4130-4336-5534-323544543642</hostuuid>\n</qemu-migra"..., cookieinlen=207, cookieout=0x7fdb82ffcad0, cookieoutlen=0x7fdb82ffcac8, flags=777,
>     resource=0, dconn=0x0, graphicsuri=0x0, nmigrate_disks=0, migrate_disks=0x0, compression=0x7fdb78007990, migParams=0x7fdb82ffc900)
>     at ../../../src/qemu/qemu_migration.c:4118
> #5  0x00007fdd09ffd808 in qemuMigrationPerformPhase (driver=0x7fdd000037b0, conn=0x7fdb500205d0, vm=0x7fdbc4002f20, persist_xml=0x0,
>     uri=0x7fdb780073a0 "tcp://172.16.202.17:49152", graphicsuri=0x0, nmigrate_disks=0, migrate_disks=0x0, compression=0x7fdb78007990, migParams=0x7fdb82ffc900,
>     cookiein=0x7fdb780084e0 "<qemu-migration>\n  <name>mss-pl_652</name>\n  <uuid>1f2b2334-451e-424b-822a-ea10452abb38</uuid>\n  <hostname>mss</hostname>\n  <hostuuid>334e344a-4130-4336-5534-323544543642</hostuuid>\n</qemu-migra"..., cookieinlen=207, cookieout=0x7fdb82ffcad0, cookieoutlen=0x7fdb82ffcac8, flags=777,
>     resource=0) at ../../../src/qemu/qemu_migration.c:5030
> #6  0x00007fdd09ffdbb5 in qemuMigrationPerform (driver=0x7fdd000037b0, conn=0x7fdb500205d0, vm=0x7fdbc4002f20, xmlin=0x0, persist_xml=0x0, dconnuri=0x0,
>     uri=0x7fdb780073a0 "tcp://172.16.202.17:49152", graphicsuri=0x0, listenAddress=0x0, nmigrate_disks=0, migrate_disks=0x0, nbdPort=0, compression=0x7fdb78007990,
>     migParams=0x7fdb82ffc900,
>     cookiein=0x7fdb780084e0 "<qemu-migration>\n  <name>mss-pl_652</name>\n  <uuid>1f2b2334-451e-424b-822a-ea10452abb38</uuid>\n  <hostname>mss</hostname>\n  <hostuuid>334e344a-4130-4336-5534-323544543642</hostuuid>\n</qemu-migra"..., cookieinlen=207, cookieout=0x7fdb82ffcad0, cookieoutlen=0x7fdb82ffcac8, flags=777,
>     dname=0x0, resource=0, v3proto=true) at ../../../src/qemu/qemu_migration.c:5124
> #7  0x00007fdd0a054725 in qemuDomainMigratePerform3 (dom=0x7fdb78007b00, xmlin=0x0,
>     cookiein=0x7fdb780084e0 "<qemu-migration>\n  <name>mss-pl_652</name>\n  <uuid>1f2b2334-451e-424b-822a-ea10452abb38</uuid>\n  <hostname>mss</hostname>\n  <hostuuid>334e344a-4130-4336-5534-323544543642</hostuuid>\n</qemu-migra"..., cookieinlen=207, cookieout=0x7fdb82ffcad0, cookieoutlen=0x7fdb82ffcac8, dconnuri=0x0,
>     uri=0x7fdb780073a0 "tcp://172.16.202.17:49152", flags=777, dname=0x0, resource=0) at ../../../src/qemu/qemu_driver.c:12996
> #8  0x00007fdd199ad0f0 in virDomainMigratePerform3 (domain=0x7fdb78007b00, xmlin=0x0,
>     cookiein=0x7fdb780084e0 "<qemu-migration>\n  <name>mss-pl_652</name>\n  <uuid>1f2b2334-451e-424b-822a-ea10452abb38</uuid>\n  <hostname>mss</hostname>\n  <hostuuid>334e344a-4130-4336-5534-323544543642</hostuuid>\n</qemu-migra"..., cookieinlen=207, cookieout=0x7fdb82ffcad0, cookieoutlen=0x7fdb82ffcac8, dconnuri=0x0,
>     uri=0x7fdb780073a0 "tcp://172.16.202.17:49152", flags=777, dname=0x0, bandwidth=0) at ../../../src/libvirt-domain.c:4698
> #9  0x000055d13923a939 in remoteDispatchDomainMigratePerform3 (server=0x55d13af90e60, client=0x55d13b0156f0, msg=0x55d13afbf620, rerr=0x7fdb82ffcbc0,
>     args=0x7fdb7800b220, ret=0x7fdb78021e90) at ../../../daemon/remote.c:4528
> #10 0x000055d13921a043 in remoteDispatchDomainMigratePerform3Helper (server=0x55d13af90e60, client=0x55d13b0156f0, msg=0x55d13afbf620, rerr=0x7fdb82ffcbc0,
>     args=0x7fdb7800b220, ret=0x7fdb78021e90) at ../../../daemon/remote_dispatch.h:7944
> #11 0x00007fdd19a260b4 in virNetServerProgramDispatchCall (prog=0x55d13af98b50, server=0x55d13af90e60, client=0x55d13b0156f0, msg=0x55d13afbf620)
>     at ../../../src/rpc/virnetserverprogram.c:436
> #12 0x00007fdd19a25c17 in virNetServerProgramDispatch (prog=0x55d13af98b50, server=0x55d13af90e60, client=0x55d13b0156f0, msg=0x55d13afbf620)
>     at ../../../src/rpc/virnetserverprogram.c:307
> #13 0x000055d13925933b in virNetServerProcessMsg (srv=0x55d13af90e60, client=0x55d13b0156f0, prog=0x55d13af98b50, msg=0x55d13afbf620)
>     at ../../../src/rpc/virnetserver.c:148
> -------------------------------------------------------------------------------------------------------------------------------------
> 本邮件及其附件含有新华三集团的保密信息，仅限于发送给上面地址中列出
> 的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、
> 或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本
> 邮件！
> This e-mail and its attachments contain confidential information from New H3C, which is
> intended only for the person or entity whose address is listed above. Any use of the
> information contained herein in any way (including, but not limited to, total or partial
> disclosure, reproduction, or dissemination) by persons other than the intended
> recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender
> by phone or email immediately and delete it!
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK