* [PATCH] migration/multifd: receive channel socket needs to be set to non-blocking @ 2024-09-20 10:05 Yuchen 2024-09-20 15:53 ` Peter Xu 0 siblings, 1 reply; 6+ messages in thread From: Yuchen @ 2024-09-20 10:05 UTC (permalink / raw) To: peterx@redhat.com, farosas@suse.de; +Cc: qemu-devel@nongnu.org When the migration network is disconnected, the source qemu can exit normally with an error, but the destination qemu is always blocked in recvmsg(), causes the destination qemu main thread to be blocked. The destination qemu block stack: Thread 13 (Thread 0x7f0178bfa640 (LWP 1895906) "multifdrecv_6"): #0 0x00007f041b5af56f in recvmsg () #1 0x000055573ebd0b42 in qio_channel_socket_readv #2 0x000055573ebce83f in qio_channel_readv #3 qio_channel_readv_all_eof #4 0x000055573ebce909 in qio_channel_readv_all #5 0x000055573eaa1b1f in multifd_recv_thread #6 0x000055573ec2f0b9 in qemu_thread_start #7 0x00007f041b52bf7a in start_thread #8 0x00007f041b5ae600 in clone3 Thread 1 (Thread 0x7f0410c62240 (LWP 1895156) "kvm"): #0 0x00007f041b528ae2 in __futex_abstimed_wait_common () #1 0x00007f041b5338b8 in __new_sem_wait_slow64.constprop.0 #2 0x000055573ec2fd34 in qemu_sem_wait (sem=0x555742b5a4e0) #3 0x000055573eaa2f09 in multifd_recv_sync_main () #4 0x000055573e7d590d in ram_load_precopy (f=f@entry=0x555742291c20) #5 0x000055573e7d5cbf in ram_load (opaque=<optimized out>, version_id=<optimized out>, f=0x555742291c20) #6 ram_load_entry (f=0x555742291c20, opaque=<optimized out>, version_id=<optimized out>) #7 0x000055573ea932e7 in qemu_loadvm_section_part_end (mis=0x555741136c00, f=0x555742291c20) #8 qemu_loadvm_state_main (f=f@entry=0x555742291c20, mis=mis@entry=0x555741136c00) #9 0x000055573ea94418 in qemu_loadvm_state (f=0x555742291c20, mode=mode@entry=VMS_MIGRATE) #10 0x000055573ea88be1 in process_incoming_migration_co (opaque=<optimized out>) #11 0x000055573ec43d13 in coroutine_trampoline (i0=<optimized out>, i1=<optimized out>) #12 0x00007f041b4f5d90 in ?? () from target:/usr/lib64/libc.so.6 #13 0x00007ffc11890270 in ?? () #14 0x0000000000000000 in ?? () Setting the receive channel to non-blocking can solve the problem. Signed-off-by: YuChen <Yu.Chen@h3c.com> --- migration/multifd.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/migration/multifd.c b/migration/multifd.c index 9b200f4ad9..7b2a768f05 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -1318,6 +1318,8 @@ void multifd_recv_new_channel(QIOChannel *ioc, Error **errp) id = qatomic_read(&multifd_recv_state->count); } + qio_channel_set_blocking(ioc, false, NULL); + p = &multifd_recv_state->params[id]; if (p->c != NULL) { error_setg(&local_err, "multifd: received id '%d' already setup'", -- 2.30.2 ------------------------------------------------------------------------------------------------------------------------------------- 本邮件及其附件含有新华三集团的保密信息，仅限于发送给上面地址中列出 的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、 或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本 邮件！ This e-mail and its attachments contain confidential information from New H3C, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it! ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH] migration/multifd: receive channel socket needs to be set to non-blocking 2024-09-20 10:05 [PATCH] migration/multifd: receive channel socket needs to be set to non-blocking Yuchen @ 2024-09-20 15:53 ` Peter Xu 2024-09-23 1:33 ` 回复: " Yuchen 0 siblings, 1 reply; 6+ messages in thread From: Peter Xu @ 2024-09-20 15:53 UTC (permalink / raw) To: Yuchen; +Cc: farosas@suse.de, qemu-devel@nongnu.org On Fri, Sep 20, 2024 at 10:05:42AM +0000, Yuchen wrote: > When the migration network is disconnected, the source > qemu can exit normally with an error, but the destination > qemu is always blocked in recvmsg(), causes the destination > qemu main thread to be blocked. > > The destination qemu block stack: > Thread 13 (Thread 0x7f0178bfa640 (LWP 1895906) "multifdrecv_6"): > #0 0x00007f041b5af56f in recvmsg () > #1 0x000055573ebd0b42 in qio_channel_socket_readv > #2 0x000055573ebce83f in qio_channel_readv > #3 qio_channel_readv_all_eof > #4 0x000055573ebce909 in qio_channel_readv_all > #5 0x000055573eaa1b1f in multifd_recv_thread > #6 0x000055573ec2f0b9 in qemu_thread_start > #7 0x00007f041b52bf7a in start_thread > #8 0x00007f041b5ae600 in clone3 > > Thread 1 (Thread 0x7f0410c62240 (LWP 1895156) "kvm"): > #0 0x00007f041b528ae2 in __futex_abstimed_wait_common () > #1 0x00007f041b5338b8 in __new_sem_wait_slow64.constprop.0 > #2 0x000055573ec2fd34 in qemu_sem_wait (sem=0x555742b5a4e0) > #3 0x000055573eaa2f09 in multifd_recv_sync_main () > #4 0x000055573e7d590d in ram_load_precopy (f=f@entry=0x555742291c20) > #5 0x000055573e7d5cbf in ram_load (opaque=<optimized out>, version_id=<optimized out>, f=0x555742291c20) > #6 ram_load_entry (f=0x555742291c20, opaque=<optimized out>, version_id=<optimized out>) > #7 0x000055573ea932e7 in qemu_loadvm_section_part_end (mis=0x555741136c00, f=0x555742291c20) > #8 qemu_loadvm_state_main (f=f@entry=0x555742291c20, mis=mis@entry=0x555741136c00) > #9 0x000055573ea94418 in qemu_loadvm_state (f=0x555742291c20, mode=mode@entry=VMS_MIGRATE) > #10 0x000055573ea88be1 in process_incoming_migration_co (opaque=<optimized out>) > #11 0x000055573ec43d13 in coroutine_trampoline (i0=<optimized out>, i1=<optimized out>) > #12 0x00007f041b4f5d90 in ?? () from target:/usr/lib64/libc.so.6 > #13 0x00007ffc11890270 in ?? () > #14 0x0000000000000000 in ?? () > > Setting the receive channel to non-blocking can solve the problem. Multifd threads are real threads and there's no coroutine, I'm slightly confused why it needs to use nonblock. Why recvmsg() didn't get kicked out when disconnect? Is it a generic Linux kernel are you using? I wonder whether that's the expected behavior for sockets. E.g., we do have multifd/cancel test (test_multifd_tcp_cancel) and I think that runs this path too with it always in block mode as of now.. > > Signed-off-by: YuChen <Yu.Chen@h3c.com> > --- > migration/multifd.c | 2 ++ > 1 file changed, 2 insertions(+) > > diff --git a/migration/multifd.c b/migration/multifd.c > index 9b200f4ad9..7b2a768f05 100644 > --- a/migration/multifd.c > +++ b/migration/multifd.c > @@ -1318,6 +1318,8 @@ void multifd_recv_new_channel(QIOChannel *ioc, Error **errp) > id = qatomic_read(&multifd_recv_state->count); > } > > + qio_channel_set_blocking(ioc, false, NULL); > + > p = &multifd_recv_state->params[id]; > if (p->c != NULL) { > error_setg(&local_err, "multifd: received id '%d' already setup'", > -- > 2.30.2 > ------------------------------------------------------------------------------------------------------------------------------------- > 本邮件及其附件含有新华三集团的保密信息，仅限于发送给上面地址中列出 > 的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、 > 或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本 > 邮件！ > This e-mail and its attachments contain confidential information from New H3C, which is > intended only for the person or entity whose address is listed above. Any use of the > information contained herein in any way (including, but not limited to, total or partial > disclosure, reproduction, or dissemination) by persons other than the intended > recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender > by phone or email immediately and delete it! -- Peter Xu ^ permalink raw reply [flat|nested] 6+ messages in thread
* 回复: [PATCH] migration/multifd: receive channel socket needs to be set to non-blocking 2024-09-20 15:53 ` Peter Xu @ 2024-09-23 1:33 ` Yuchen 2024-09-23 16:58 ` Daniel P. Berrangé 0 siblings, 1 reply; 6+ messages in thread From: Yuchen @ 2024-09-23 1:33 UTC (permalink / raw) To: Peter Xu; +Cc: farosas@suse.de, qemu-devel@nongnu.org > -----邮件原件----- > 发件人: Peter Xu <peterx@redhat.com> > 发送时间: 2024年9月20日 23:53 > 收件人: yuchen (CCSPL) <yu.chen@h3c.com> > 抄送: farosas@suse.de; qemu-devel@nongnu.org > 主题: Re: [PATCH] migration/multifd: receive channel socket needs to be set to > non-blocking > > On Fri, Sep 20, 2024 at 10:05:42AM +0000, Yuchen wrote: > > When the migration network is disconnected, the source qemu can exit > > normally with an error, but the destination qemu is always blocked in > > recvmsg(), causes the destination qemu main thread to be blocked. > > > > The destination qemu block stack: > > Thread 13 (Thread 0x7f0178bfa640 (LWP 1895906) "multifdrecv_6"): > > #0 0x00007f041b5af56f in recvmsg () > > #1 0x000055573ebd0b42 in qio_channel_socket_readv > > #2 0x000055573ebce83f in qio_channel_readv > > #3 qio_channel_readv_all_eof > > #4 0x000055573ebce909 in qio_channel_readv_all > > #5 0x000055573eaa1b1f in multifd_recv_thread > > #6 0x000055573ec2f0b9 in qemu_thread_start > > #7 0x00007f041b52bf7a in start_thread > > #8 0x00007f041b5ae600 in clone3 > > > > Thread 1 (Thread 0x7f0410c62240 (LWP 1895156) "kvm"): > > #0 0x00007f041b528ae2 in __futex_abstimed_wait_common () > > #1 0x00007f041b5338b8 in __new_sem_wait_slow64.constprop.0 > > #2 0x000055573ec2fd34 in qemu_sem_wait (sem=0x555742b5a4e0) > > #3 0x000055573eaa2f09 in multifd_recv_sync_main () > > #4 0x000055573e7d590d in ram_load_precopy > (f=f@entry=0x555742291c20) > > #5 0x000055573e7d5cbf in ram_load (opaque=<optimized out>, > > version_id=<optimized out>, f=0x555742291c20) > > #6 ram_load_entry (f=0x555742291c20, opaque=<optimized out>, > > version_id=<optimized out>) > > #7 0x000055573ea932e7 in qemu_loadvm_section_part_end > > (mis=0x555741136c00, f=0x555742291c20) > > #8 qemu_loadvm_state_main (f=f@entry=0x555742291c20, > > mis=mis@entry=0x555741136c00) > > #9 0x000055573ea94418 in qemu_loadvm_state (f=0x555742291c20, > > mode=mode@entry=VMS_MIGRATE) > > #10 0x000055573ea88be1 in process_incoming_migration_co > > (opaque=<optimized out>) > > #11 0x000055573ec43d13 in coroutine_trampoline (i0=<optimized out>, > > i1=<optimized out>) > > #12 0x00007f041b4f5d90 in ?? () from target:/usr/lib64/libc.so.6 > > #13 0x00007ffc11890270 in ?? () > > #14 0x0000000000000000 in ?? () > > > > Setting the receive channel to non-blocking can solve the problem. > > Multifd threads are real threads and there's no coroutine, I'm slightly confused > why it needs to use nonblock. > > Why recvmsg() didn't get kicked out when disconnect? Is it a generic Linux > kernel are you using? > My steps to reproduce: ifdown migration network, or disable migration network using iptables. The probability of recurrence of these two methods is very high. My test environment uses is linux-5.10.136. multifd thread block in kernel: # cat /proc/3416190/stack [<0>] wait_woken+0x43/0x80 [<0>] sk_wait_data+0x123/0x140 [<0>] tcp_recvmsg+0x4f8/0xa50 [<0>] inet6_recvmsg+0x5e/0x120 [<0>] ____sys_recvmsg+0x87/0x180 [<0>] ___sys_recvmsg+0x82/0x110 [<0>] __sys_recvmsg+0x56/0xa0 [<0>] do_syscall_64+0x3d/0x80 [<0>] entry_SYSCALL_64_after_hwframe+0x61/0xc6 > I wonder whether that's the expected behavior for sockets. E.g., we do have > multifd/cancel test (test_multifd_tcp_cancel) and I think that runs this path too > with it always in block mode as of now.. > My previous statement may not be accurate. The migration network socket is not disconnected. I use ifdown or iptables to simulate the network card failure. Because the TCP connection was not disconnected, so recvmsg() was blocked. Ordinary precopy migration, the destination also uses non-blocking, I think it's to avoid non-blocking. Qemu master lastest code: /** * migration_incoming_setup: Setup incoming migration * @f: file for main migration channel */ static void migration_incoming_setup(QEMUFile *f) { MigrationIncomingState *mis = migration_incoming_get_current(); if (!mis->from_src_file) { mis->from_src_file = f; } qemu_file_set_blocking(f, false); } > > > > Signed-off-by: YuChen <Yu.Chen@h3c.com> > > --- > > migration/multifd.c | 2 ++ > > 1 file changed, 2 insertions(+) > > > > diff --git a/migration/multifd.c b/migration/multifd.c index > > 9b200f4ad9..7b2a768f05 100644 > > --- a/migration/multifd.c > > +++ b/migration/multifd.c > > @@ -1318,6 +1318,8 @@ void multifd_recv_new_channel(QIOChannel *ioc, > Error **errp) > > id = qatomic_read(&multifd_recv_state->count); > > } > > > > + qio_channel_set_blocking(ioc, false, NULL); > > + > > p = &multifd_recv_state->params[id]; > > if (p->c != NULL) { > > error_setg(&local_err, "multifd: received id '%d' already > > setup'", > > -- > > 2.30.2 > > ---------------------------------------------------------------------- > > --------------------------------------------------------------- > > 本邮件及其附件含有新华三集团的保密信息，仅限于发送给上面地址中 > 列出 > > 的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或 > 部分地泄露、复制、 > > 或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件 > 通知发件人并删除本 > > 邮件！ > > This e-mail and its attachments contain confidential information from > > New H3C, which is intended only for the person or entity whose address > > is listed above. Any use of the information contained herein in any > > way (including, but not limited to, total or partial disclosure, > > reproduction, or dissemination) by persons other than the intended > > recipient(s) is prohibited. If you receive this e-mail in error, > > please notify the sender by phone or email immediately and delete it! > > -- > Peter Xu ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 回复: [PATCH] migration/multifd: receive channel socket needs to be set to non-blocking 2024-09-23 1:33 ` 回复: " Yuchen @ 2024-09-23 16:58 ` Daniel P. Berrangé 2024-09-24 8:25 ` 回复: " Yuchen 0 siblings, 1 reply; 6+ messages in thread From: Daniel P. Berrangé @ 2024-09-23 16:58 UTC (permalink / raw) To: Yuchen; +Cc: Peter Xu, farosas@suse.de, qemu-devel@nongnu.org On Mon, Sep 23, 2024 at 01:33:13AM +0000, Yuchen wrote: > > > > -----邮件原件----- > > 发件人: Peter Xu <peterx@redhat.com> > > 发送时间: 2024年9月20日 23:53 > > 收件人: yuchen (CCSPL) <yu.chen@h3c.com> > > 抄送: farosas@suse.de; qemu-devel@nongnu.org > > 主题: Re: [PATCH] migration/multifd: receive channel socket needs to be set to > > non-blocking > > > > On Fri, Sep 20, 2024 at 10:05:42AM +0000, Yuchen wrote: > > > When the migration network is disconnected, the source qemu can exit > > > normally with an error, but the destination qemu is always blocked in > > > recvmsg(), causes the destination qemu main thread to be blocked. > > > > > > The destination qemu block stack: > > > Thread 13 (Thread 0x7f0178bfa640 (LWP 1895906) "multifdrecv_6"): > > > #0 0x00007f041b5af56f in recvmsg () > > > #1 0x000055573ebd0b42 in qio_channel_socket_readv > > > #2 0x000055573ebce83f in qio_channel_readv > > > #3 qio_channel_readv_all_eof > > > #4 0x000055573ebce909 in qio_channel_readv_all > > > #5 0x000055573eaa1b1f in multifd_recv_thread > > > #6 0x000055573ec2f0b9 in qemu_thread_start > > > #7 0x00007f041b52bf7a in start_thread > > > #8 0x00007f041b5ae600 in clone3 > > > > > > Thread 1 (Thread 0x7f0410c62240 (LWP 1895156) "kvm"): > > > #0 0x00007f041b528ae2 in __futex_abstimed_wait_common () > > > #1 0x00007f041b5338b8 in __new_sem_wait_slow64.constprop.0 > > > #2 0x000055573ec2fd34 in qemu_sem_wait (sem=0x555742b5a4e0) > > > #3 0x000055573eaa2f09 in multifd_recv_sync_main () > > > #4 0x000055573e7d590d in ram_load_precopy > > (f=f@entry=0x555742291c20) > > > #5 0x000055573e7d5cbf in ram_load (opaque=<optimized out>, > > > version_id=<optimized out>, f=0x555742291c20) > > > #6 ram_load_entry (f=0x555742291c20, opaque=<optimized out>, > > > version_id=<optimized out>) > > > #7 0x000055573ea932e7 in qemu_loadvm_section_part_end > > > (mis=0x555741136c00, f=0x555742291c20) > > > #8 qemu_loadvm_state_main (f=f@entry=0x555742291c20, > > > mis=mis@entry=0x555741136c00) > > > #9 0x000055573ea94418 in qemu_loadvm_state (f=0x555742291c20, > > > mode=mode@entry=VMS_MIGRATE) > > > #10 0x000055573ea88be1 in process_incoming_migration_co > > > (opaque=<optimized out>) > > > #11 0x000055573ec43d13 in coroutine_trampoline (i0=<optimized out>, > > > i1=<optimized out>) > > > #12 0x00007f041b4f5d90 in ?? () from target:/usr/lib64/libc.so.6 > > > #13 0x00007ffc11890270 in ?? () > > > #14 0x0000000000000000 in ?? () > > > > > > Setting the receive channel to non-blocking can solve the problem. > > > > Multifd threads are real threads and there's no coroutine, I'm slightly confused > > why it needs to use nonblock. > > > > Why recvmsg() didn't get kicked out when disconnect? Is it a generic Linux > > kernel are you using? > > > My steps to reproduce: > ifdown migration network, or disable migration network using iptables. > The probability of recurrence of these two methods is very high. > > My test environment uses is linux-5.10.136. > > multifd thread block in kernel: > # cat /proc/3416190/stack > [<0>] wait_woken+0x43/0x80 > [<0>] sk_wait_data+0x123/0x140 > [<0>] tcp_recvmsg+0x4f8/0xa50 > [<0>] inet6_recvmsg+0x5e/0x120 > [<0>] ____sys_recvmsg+0x87/0x180 > [<0>] ___sys_recvmsg+0x82/0x110 > [<0>] __sys_recvmsg+0x56/0xa0 > [<0>] do_syscall_64+0x3d/0x80 > [<0>] entry_SYSCALL_64_after_hwframe+0x61/0xc6 > > > I wonder whether that's the expected behavior for sockets. E.g., we do have > > multifd/cancel test (test_multifd_tcp_cancel) and I think that runs this path too > > with it always in block mode as of now.. > > > My previous statement may not be accurate. The migration network socket is not disconnected. > I use ifdown or iptables to simulate the network card failure. > Because the TCP connection was not disconnected, so recvmsg() was blocked. How long did you wait after doing ifdown ? TCP is intended to wait if there is an interruption.... only eventually after relevant TCP timeouts are hit, it will terminate the connection. QEMU shouldn't proactively give up if the TCP conn is still in an active state as reported by the kernel, even if traffic isn't currently flowing. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 6+ messages in thread
* 回复: 回复: [PATCH] migration/multifd: receive channel socket needs to be set to non-blocking 2024-09-23 16:58 ` Daniel P. Berrangé @ 2024-09-24 8:25 ` Yuchen 2024-09-25 15:32 ` Peter Xu 0 siblings, 1 reply; 6+ messages in thread From: Yuchen @ 2024-09-24 8:25 UTC (permalink / raw) To: Daniel P. Berrangé; +Cc: Peter Xu, farosas@suse.de, qemu-devel@nongnu.org > -----邮件原件----- > 发件人: Daniel P. Berrangé <berrange@redhat.com> > 发送时间: 2024年9月24日 0:59 > 收件人: yuchen (CCSPL) <yu.chen@h3c.com> > 抄送: Peter Xu <peterx@redhat.com>; farosas@suse.de; > qemu-devel@nongnu.org > 主题: Re: 回复: [PATCH] migration/multifd: receive channel socket needs to > be set to non-blocking > > On Mon, Sep 23, 2024 at 01:33:13AM +0000, Yuchen wrote: > > > > > > > -----邮件原件----- > > > 发件人: Peter Xu <peterx@redhat.com> > > > 发送时间: 2024年9月20日 23:53 > > > 收件人: yuchen (CCSPL) <yu.chen@h3c.com> > > > 抄送: farosas@suse.de; qemu-devel@nongnu.org > > > 主题: Re: [PATCH] migration/multifd: receive channel socket needs to > > > be set to non-blocking > > > > > > On Fri, Sep 20, 2024 at 10:05:42AM +0000, Yuchen wrote: > > > > When the migration network is disconnected, the source qemu can > > > > exit normally with an error, but the destination qemu is always > > > > blocked in recvmsg(), causes the destination qemu main thread to be > blocked. > > > > > > > > The destination qemu block stack: > > > > Thread 13 (Thread 0x7f0178bfa640 (LWP 1895906) "multifdrecv_6"): > > > > #0 0x00007f041b5af56f in recvmsg () > > > > #1 0x000055573ebd0b42 in qio_channel_socket_readv > > > > #2 0x000055573ebce83f in qio_channel_readv > > > > #3 qio_channel_readv_all_eof > > > > #4 0x000055573ebce909 in qio_channel_readv_all > > > > #5 0x000055573eaa1b1f in multifd_recv_thread > > > > #6 0x000055573ec2f0b9 in qemu_thread_start > > > > #7 0x00007f041b52bf7a in start_thread > > > > #8 0x00007f041b5ae600 in clone3 > > > > > > > > Thread 1 (Thread 0x7f0410c62240 (LWP 1895156) "kvm"): > > > > #0 0x00007f041b528ae2 in __futex_abstimed_wait_common () > > > > #1 0x00007f041b5338b8 in __new_sem_wait_slow64.constprop.0 > > > > #2 0x000055573ec2fd34 in qemu_sem_wait (sem=0x555742b5a4e0) > > > > #3 0x000055573eaa2f09 in multifd_recv_sync_main () > > > > #4 0x000055573e7d590d in ram_load_precopy > > > (f=f@entry=0x555742291c20) > > > > #5 0x000055573e7d5cbf in ram_load (opaque=<optimized out>, > > > > version_id=<optimized out>, f=0x555742291c20) > > > > #6 ram_load_entry (f=0x555742291c20, opaque=<optimized out>, > > > > version_id=<optimized out>) > > > > #7 0x000055573ea932e7 in qemu_loadvm_section_part_end > > > > (mis=0x555741136c00, f=0x555742291c20) > > > > #8 qemu_loadvm_state_main (f=f@entry=0x555742291c20, > > > > mis=mis@entry=0x555741136c00) > > > > #9 0x000055573ea94418 in qemu_loadvm_state (f=0x555742291c20, > > > > mode=mode@entry=VMS_MIGRATE) > > > > #10 0x000055573ea88be1 in process_incoming_migration_co > > > > (opaque=<optimized out>) > > > > #11 0x000055573ec43d13 in coroutine_trampoline (i0=<optimized > > > > out>, i1=<optimized out>) > > > > #12 0x00007f041b4f5d90 in ?? () from target:/usr/lib64/libc.so.6 > > > > #13 0x00007ffc11890270 in ?? () > > > > #14 0x0000000000000000 in ?? () > > > > > > > > Setting the receive channel to non-blocking can solve the problem. > > > > > > Multifd threads are real threads and there's no coroutine, I'm > > > slightly confused why it needs to use nonblock. > > > > > > Why recvmsg() didn't get kicked out when disconnect? Is it a > > > generic Linux kernel are you using? > > > > > My steps to reproduce: > > ifdown migration network, or disable migration network using iptables. > > The probability of recurrence of these two methods is very high. > > > > My test environment uses is linux-5.10.136. > > > > multifd thread block in kernel: > > # cat /proc/3416190/stack > > [<0>] wait_woken+0x43/0x80 > > [<0>] sk_wait_data+0x123/0x140 > > [<0>] tcp_recvmsg+0x4f8/0xa50 > > [<0>] inet6_recvmsg+0x5e/0x120 > > [<0>] ____sys_recvmsg+0x87/0x180 > > [<0>] ___sys_recvmsg+0x82/0x110 > > [<0>] __sys_recvmsg+0x56/0xa0 > > [<0>] do_syscall_64+0x3d/0x80 > > [<0>] entry_SYSCALL_64_after_hwframe+0x61/0xc6 > > > > > I wonder whether that's the expected behavior for sockets. E.g., we > > > do have multifd/cancel test (test_multifd_tcp_cancel) and I think > > > that runs this path too with it always in block mode as of now.. > > > > > My previous statement may not be accurate. The migration network socket is > not disconnected. > > I use ifdown or iptables to simulate the network card failure. > > Because the TCP connection was not disconnected, so recvmsg() was > blocked. > > How long did you wait after doing ifdown ? TCP is intended to wait if I waited about 15 minutes, the source qemu migration threads quit, but the destination qemu migration threads is still there. > there is an interruption.... only eventually after relevant TCP timeouts are hit, it > will terminate the connection. QEMU shouldn't proactively give up if the TCP > conn is still in an active state as reported by the kernel, even if traffic isn't > currently flowing. > > Daniel, I agree with what you said, But in fact, the destination migration connection is not disconnected and is in the close wait state. The source qemu process lsof and top: # lsof -p 384509 ... kvm 384509 root 112u sock 0,8 0t0 157321811 protocol: TCP kvm 384509 root 113u sock 0,8 0t0 157321813 protocol: TCP kvm 384509 root 114u sock 0,8 0t0 157321815 protocol: TCP kvm 384509 root 115u sock 0,8 0t0 157321817 protocol: TCP kvm 384509 root 116u sock 0,8 0t0 157321819 protocol: TCP kvm 384509 root 117u sock 0,8 0t0 157321821 protocol: TCP kvm 384509 root 118u sock 0,8 0t0 157321823 protocol: TCP kvm 384509 root 119u sock 0,8 0t0 157321825 protocol: TCP # top -H -p 384509 top - 15:10:22 up 5 days, 18:54, 3 users, load average: 5.16, 4.61, 4.50 Threads: 8 total, 3 running, 5 sleeping, 0 stopped, 0 zombie %Cpu(s): 5.2 us, 5.2 sy, 0.0 ni, 89.3 id, 0.0 wa, 0.1 hi, 0.1 si, 0.0 st MiB Mem : 128298.7 total, 41490.2 free, 89470.2 used, 2168.0 buff/cache MiB Swap: 42922.0 total, 42910.4 free, 11.6 used. 38828.5 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 384596 root 20 0 11.9g 93516 40112 R 98.7 0.1 261:13.24 CPU 1/KVM 384595 root 20 0 11.9g 93516 40112 R 98.0 0.1 56:44.31 CPU 0/KVM 384509 root 20 0 11.9g 93516 40112 R 1.3 0.1 7:38.73 kvm 384563 root 20 0 11.9g 93516 40112 S 0.0 0.1 0:00.05 kvm 384598 root 20 0 11.9g 93516 40112 S 0.0 0.1 0:01.00 vnc_worker 1544593 root 20 0 11.9g 93516 40112 S 0.0 0.1 0:00.00 worker The destination qemu process lsof and top: # lsof -p 3236693 kvm 3236693 root 29u IPv6 159227758 0t0 TCP node18:49156->2.2.2.6:41880 (CLOSE_WAIT) kvm 3236693 root 30u IPv6 159227759 0t0 TCP node18:49156->2.2.2.6:41890 (ESTABLISHED) kvm 3236693 root 31u IPv6 159227760 0t0 TCP node18:49156->2.2.2.6:41902 (ESTABLISHED) kvm 3236693 root 32u IPv6 159227762 0t0 TCP node18:49156->2.2.2.6:41912 (ESTABLISHED) kvm 3236693 root 33u IPv6 159227761 0t0 TCP node18:49156->2.2.2.6:41904 (ESTABLISHED) kvm 3236693 root 34u IPv6 159227763 0t0 TCP node18:49156->2.2.2.6:41918 (ESTABLISHED) kvm 3236693 root 35u IPv6 159227764 0t0 TCP node18:49156->2.2.2.6:41924 (ESTABLISHED) kvm 3236693 root 36u IPv6 159227765 0t0 TCP node18:49156->2.2.2.6:41934 (ESTABLISHED) kvm 3236693 root 37u IPv6 159227766 0t0 TCP node18:49156->2.2.2.6:41942 (ESTABLISHED) # top -H -p 3236693 top - 15:09:25 up 5 days, 19:12, 2 users, load average: 0.63, 0.68, 0.89 Threads: 15 total, 0 running, 15 sleeping, 0 stopped, 0 zombie %Cpu(s): 1.3 us, 0.5 sy, 0.0 ni, 98.1 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st MiB Mem : 128452.1 total, 43515.4 free, 87291.7 used, 2527.4 buff/cache MiB Swap: 42973.0 total, 42968.4 free, 4.6 used. 41160.4 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3236693 root 20 0 11.3g 100192 38508 S 0.0 0.1 0:00.41 kvm 3236714 root 20 0 11.3g 100192 38508 S 0.0 0.1 0:00.00 kvm 3236745 root 20 0 11.3g 100192 38508 S 0.0 0.1 0:00.00 CPU 0/KVM 3236746 root 20 0 11.3g 100192 38508 S 0.0 0.1 0:00.00 CPU 1/KVM 3236748 root 20 0 11.3g 100192 38508 S 0.0 0.1 0:00.00 vnc_worker 3236750 root 20 0 11.3g 100192 38508 S 0.0 0.1 0:01.45 multifdrecv_4 3236751 root 20 0 11.3g 100192 38508 S 0.0 0.1 0:01.42 multifdrecv_5 3236752 root 20 0 11.3g 100192 38508 S 0.0 0.1 0:01.35 multifdrecv_6 3236753 root 20 0 11.3g 100192 38508 S 0.0 0.1 0:01.49 multifdrecv_7 3236754 root 20 0 11.3g 100192 38508 S 0.0 0.1 0:01.45 multifdrecv_1 3236755 root 20 0 11.3g 100192 38508 S 0.0 0.1 0:01.43 multifdrecv_2 3236756 root 20 0 11.3g 100192 38508 S 0.0 0.1 0:01.45 multifdrecv_3 3236757 root 20 0 11.3g 100192 38508 S 0.0 0.1 0:01.44 multifdrecv_0 So we should still set the multifd channel socket to non-blocking ? > With regards, > Daniel > -- > |: https://berrange.com -o- > https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- > https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- > https://www.instagram.com/dberrange :| ------------------------------------------------------------------------------------------------------------------------------------- 本邮件及其附件含有新华三集团的保密信息，仅限于发送给上面地址中列出 的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、 或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本 邮件！ This e-mail and its attachments contain confidential information from New H3C, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it! ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 回复: 回复: [PATCH] migration/multifd: receive channel socket needs to be set to non-blocking 2024-09-24 8:25 ` 回复: " Yuchen @ 2024-09-25 15:32 ` Peter Xu 0 siblings, 0 replies; 6+ messages in thread From: Peter Xu @ 2024-09-25 15:32 UTC (permalink / raw) To: Yuchen; +Cc: Daniel P. Berrangé, farosas@suse.de, qemu-devel@nongnu.org On Tue, Sep 24, 2024 at 08:25:22AM +0000, Yuchen wrote: > > > > -----邮件原件----- > > 发件人: Daniel P. Berrangé <berrange@redhat.com> > > 发送时间: 2024年9月24日 0:59 > > 收件人: yuchen (CCSPL) <yu.chen@h3c.com> > > 抄送: Peter Xu <peterx@redhat.com>; farosas@suse.de; > > qemu-devel@nongnu.org > > 主题: Re: 回复: [PATCH] migration/multifd: receive channel socket needs to > > be set to non-blocking > > > > On Mon, Sep 23, 2024 at 01:33:13AM +0000, Yuchen wrote: > > > > > > > > > > -----邮件原件----- > > > > 发件人: Peter Xu <peterx@redhat.com> > > > > 发送时间: 2024年9月20日 23:53 > > > > 收件人: yuchen (CCSPL) <yu.chen@h3c.com> > > > > 抄送: farosas@suse.de; qemu-devel@nongnu.org > > > > 主题: Re: [PATCH] migration/multifd: receive channel socket needs to > > > > be set to non-blocking > > > > > > > > On Fri, Sep 20, 2024 at 10:05:42AM +0000, Yuchen wrote: > > > > > When the migration network is disconnected, the source qemu can > > > > > exit normally with an error, but the destination qemu is always > > > > > blocked in recvmsg(), causes the destination qemu main thread to be > > blocked. > > > > > > > > > > The destination qemu block stack: > > > > > Thread 13 (Thread 0x7f0178bfa640 (LWP 1895906) "multifdrecv_6"): > > > > > #0 0x00007f041b5af56f in recvmsg () > > > > > #1 0x000055573ebd0b42 in qio_channel_socket_readv > > > > > #2 0x000055573ebce83f in qio_channel_readv > > > > > #3 qio_channel_readv_all_eof > > > > > #4 0x000055573ebce909 in qio_channel_readv_all > > > > > #5 0x000055573eaa1b1f in multifd_recv_thread > > > > > #6 0x000055573ec2f0b9 in qemu_thread_start > > > > > #7 0x00007f041b52bf7a in start_thread > > > > > #8 0x00007f041b5ae600 in clone3 > > > > > > > > > > Thread 1 (Thread 0x7f0410c62240 (LWP 1895156) "kvm"): > > > > > #0 0x00007f041b528ae2 in __futex_abstimed_wait_common () > > > > > #1 0x00007f041b5338b8 in __new_sem_wait_slow64.constprop.0 > > > > > #2 0x000055573ec2fd34 in qemu_sem_wait (sem=0x555742b5a4e0) > > > > > #3 0x000055573eaa2f09 in multifd_recv_sync_main () > > > > > #4 0x000055573e7d590d in ram_load_precopy > > > > (f=f@entry=0x555742291c20) > > > > > #5 0x000055573e7d5cbf in ram_load (opaque=<optimized out>, > > > > > version_id=<optimized out>, f=0x555742291c20) > > > > > #6 ram_load_entry (f=0x555742291c20, opaque=<optimized out>, > > > > > version_id=<optimized out>) > > > > > #7 0x000055573ea932e7 in qemu_loadvm_section_part_end > > > > > (mis=0x555741136c00, f=0x555742291c20) > > > > > #8 qemu_loadvm_state_main (f=f@entry=0x555742291c20, > > > > > mis=mis@entry=0x555741136c00) > > > > > #9 0x000055573ea94418 in qemu_loadvm_state (f=0x555742291c20, > > > > > mode=mode@entry=VMS_MIGRATE) > > > > > #10 0x000055573ea88be1 in process_incoming_migration_co > > > > > (opaque=<optimized out>) > > > > > #11 0x000055573ec43d13 in coroutine_trampoline (i0=<optimized > > > > > out>, i1=<optimized out>) > > > > > #12 0x00007f041b4f5d90 in ?? () from target:/usr/lib64/libc.so.6 > > > > > #13 0x00007ffc11890270 in ?? () > > > > > #14 0x0000000000000000 in ?? () > > > > > > > > > > Setting the receive channel to non-blocking can solve the problem. > > > > > > > > Multifd threads are real threads and there's no coroutine, I'm > > > > slightly confused why it needs to use nonblock. > > > > > > > > Why recvmsg() didn't get kicked out when disconnect? Is it a > > > > generic Linux kernel are you using? > > > > > > > My steps to reproduce: > > > ifdown migration network, or disable migration network using iptables. > > > The probability of recurrence of these two methods is very high. > > > > > > My test environment uses is linux-5.10.136. > > > > > > multifd thread block in kernel: > > > # cat /proc/3416190/stack > > > [<0>] wait_woken+0x43/0x80 > > > [<0>] sk_wait_data+0x123/0x140 > > > [<0>] tcp_recvmsg+0x4f8/0xa50 > > > [<0>] inet6_recvmsg+0x5e/0x120 > > > [<0>] ____sys_recvmsg+0x87/0x180 > > > [<0>] ___sys_recvmsg+0x82/0x110 > > > [<0>] __sys_recvmsg+0x56/0xa0 > > > [<0>] do_syscall_64+0x3d/0x80 > > > [<0>] entry_SYSCALL_64_after_hwframe+0x61/0xc6 > > > > > > > I wonder whether that's the expected behavior for sockets. E.g., we > > > > do have multifd/cancel test (test_multifd_tcp_cancel) and I think > > > > that runs this path too with it always in block mode as of now.. > > > > > > > My previous statement may not be accurate. The migration network socket is > > not disconnected. > > > I use ifdown or iptables to simulate the network card failure. > > > Because the TCP connection was not disconnected, so recvmsg() was > > blocked. > > > > How long did you wait after doing ifdown ? TCP is intended to wait if > > I waited about 15 minutes, the source qemu migration threads quit, but > the destination qemu migration threads is still there. > > > there is an interruption.... only eventually after relevant TCP timeouts are hit, it > > will terminate the connection. QEMU shouldn't proactively give up if the TCP > > conn is still in an active state as reported by the kernel, even if traffic isn't > > currently flowing. > > > > > > Daniel, I agree with what you said, But in fact, the destination migration connection is not disconnected > and is in the close wait state. > > The source qemu process lsof and top: > # lsof -p 384509 > ... > kvm 384509 root 112u sock 0,8 0t0 157321811 protocol: TCP > kvm 384509 root 113u sock 0,8 0t0 157321813 protocol: TCP > kvm 384509 root 114u sock 0,8 0t0 157321815 protocol: TCP > kvm 384509 root 115u sock 0,8 0t0 157321817 protocol: TCP > kvm 384509 root 116u sock 0,8 0t0 157321819 protocol: TCP > kvm 384509 root 117u sock 0,8 0t0 157321821 protocol: TCP > kvm 384509 root 118u sock 0,8 0t0 157321823 protocol: TCP > kvm 384509 root 119u sock 0,8 0t0 157321825 protocol: TCP > > # top -H -p 384509 > top - 15:10:22 up 5 days, 18:54, 3 users, load average: 5.16, 4.61, 4.50 > Threads: 8 total, 3 running, 5 sleeping, 0 stopped, 0 zombie > %Cpu(s): 5.2 us, 5.2 sy, 0.0 ni, 89.3 id, 0.0 wa, 0.1 hi, 0.1 si, 0.0 st > MiB Mem : 128298.7 total, 41490.2 free, 89470.2 used, 2168.0 buff/cache > MiB Swap: 42922.0 total, 42910.4 free, 11.6 used. 38828.5 avail Mem > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 384596 root 20 0 11.9g 93516 40112 R 98.7 0.1 261:13.24 CPU 1/KVM > 384595 root 20 0 11.9g 93516 40112 R 98.0 0.1 56:44.31 CPU 0/KVM > 384509 root 20 0 11.9g 93516 40112 R 1.3 0.1 7:38.73 kvm > 384563 root 20 0 11.9g 93516 40112 S 0.0 0.1 0:00.05 kvm > 384598 root 20 0 11.9g 93516 40112 S 0.0 0.1 0:01.00 vnc_worker > 1544593 root 20 0 11.9g 93516 40112 S 0.0 0.1 0:00.00 worker > > The destination qemu process lsof and top: > # lsof -p 3236693 > kvm 3236693 root 29u IPv6 159227758 0t0 TCP node18:49156->2.2.2.6:41880 (CLOSE_WAIT) > kvm 3236693 root 30u IPv6 159227759 0t0 TCP node18:49156->2.2.2.6:41890 (ESTABLISHED) > kvm 3236693 root 31u IPv6 159227760 0t0 TCP node18:49156->2.2.2.6:41902 (ESTABLISHED) > kvm 3236693 root 32u IPv6 159227762 0t0 TCP node18:49156->2.2.2.6:41912 (ESTABLISHED) > kvm 3236693 root 33u IPv6 159227761 0t0 TCP node18:49156->2.2.2.6:41904 (ESTABLISHED) > kvm 3236693 root 34u IPv6 159227763 0t0 TCP node18:49156->2.2.2.6:41918 (ESTABLISHED) > kvm 3236693 root 35u IPv6 159227764 0t0 TCP node18:49156->2.2.2.6:41924 (ESTABLISHED) > kvm 3236693 root 36u IPv6 159227765 0t0 TCP node18:49156->2.2.2.6:41934 (ESTABLISHED) > kvm 3236693 root 37u IPv6 159227766 0t0 TCP node18:49156->2.2.2.6:41942 (ESTABLISHED) > > # top -H -p 3236693 > top - 15:09:25 up 5 days, 19:12, 2 users, load average: 0.63, 0.68, 0.89 > Threads: 15 total, 0 running, 15 sleeping, 0 stopped, 0 zombie > %Cpu(s): 1.3 us, 0.5 sy, 0.0 ni, 98.1 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st > MiB Mem : 128452.1 total, 43515.4 free, 87291.7 used, 2527.4 buff/cache > MiB Swap: 42973.0 total, 42968.4 free, 4.6 used. 41160.4 avail Mem > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 3236693 root 20 0 11.3g 100192 38508 S 0.0 0.1 0:00.41 kvm > 3236714 root 20 0 11.3g 100192 38508 S 0.0 0.1 0:00.00 kvm > 3236745 root 20 0 11.3g 100192 38508 S 0.0 0.1 0:00.00 CPU 0/KVM > 3236746 root 20 0 11.3g 100192 38508 S 0.0 0.1 0:00.00 CPU 1/KVM > 3236748 root 20 0 11.3g 100192 38508 S 0.0 0.1 0:00.00 vnc_worker > 3236750 root 20 0 11.3g 100192 38508 S 0.0 0.1 0:01.45 multifdrecv_4 > 3236751 root 20 0 11.3g 100192 38508 S 0.0 0.1 0:01.42 multifdrecv_5 > 3236752 root 20 0 11.3g 100192 38508 S 0.0 0.1 0:01.35 multifdrecv_6 > 3236753 root 20 0 11.3g 100192 38508 S 0.0 0.1 0:01.49 multifdrecv_7 > 3236754 root 20 0 11.3g 100192 38508 S 0.0 0.1 0:01.45 multifdrecv_1 > 3236755 root 20 0 11.3g 100192 38508 S 0.0 0.1 0:01.43 multifdrecv_2 > 3236756 root 20 0 11.3g 100192 38508 S 0.0 0.1 0:01.45 multifdrecv_3 > 3236757 root 20 0 11.3g 100192 38508 S 0.0 0.1 0:01.44 multifdrecv_0 > > So we should still set the multifd channel socket to non-blocking ? Have you looked at why the timeout didn't work? After all, QEMU is not the only application that uses recvmsg() like this, so I wonder whether it's intended or it's a kernel bug that recvmsg() didn't get kicked out. > > > With regards, > > Daniel > > -- > > |: https://berrange.com -o- > > https://www.flickr.com/photos/dberrange :| > > |: https://libvirt.org -o- > > https://fstop138.berrange.com :| > > |: https://entangle-photo.org -o- > > https://www.instagram.com/dberrange :| > > ------------------------------------------------------------------------------------------------------------------------------------- > 本邮件及其附件含有新华三集团的保密信息，仅限于发送给上面地址中列出 > 的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、 > 或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本 > 邮件！ > This e-mail and its attachments contain confidential information from New H3C, which is > intended only for the person or entity whose address is listed above. Any use of the > information contained herein in any way (including, but not limited to, total or partial > disclosure, reproduction, or dissemination) by persons other than the intended > recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender > by phone or email immediately and delete it! -- Peter Xu ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2024-09-25 15:33 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2024-09-20 10:05 [PATCH] migration/multifd: receive channel socket needs to be set to non-blocking Yuchen 2024-09-20 15:53 ` Peter Xu 2024-09-23 1:33 ` 回复: " Yuchen 2024-09-23 16:58 ` Daniel P. Berrangé 2024-09-24 8:25 ` 回复: " Yuchen 2024-09-25 15:32 ` Peter Xu
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).