Re: 回复: [PATCH] migration/multifd: receive channel socket needs to be set to non-blocking

["Daniel P. Berrangé" <berrange@redhat.com>]

On Mon, Sep 23, 2024 at 01:33:13AM +0000, Yuchen wrote:
> 
> 
> > -----邮件原件-----
> > 发件人: Peter Xu <peterx@redhat.com>
> > 发送时间: 2024年9月20日 23:53
> > 收件人: yuchen (CCSPL) <yu.chen@h3c.com>
> > 抄送: farosas@suse.de; qemu-devel@nongnu.org
> > 主题: Re: [PATCH] migration/multifd: receive channel socket needs to be set to
> > non-blocking
> > 
> > On Fri, Sep 20, 2024 at 10:05:42AM +0000, Yuchen wrote:
> > > When the migration network is disconnected, the source qemu can exit
> > > normally with an error, but the destination qemu is always blocked in
> > > recvmsg(), causes the destination qemu main thread to be blocked.
> > >
> > > The destination qemu block stack:
> > > Thread 13 (Thread 0x7f0178bfa640 (LWP 1895906) "multifdrecv_6"):
> > > #0  0x00007f041b5af56f in recvmsg ()
> > > #1  0x000055573ebd0b42 in qio_channel_socket_readv
> > > #2  0x000055573ebce83f in qio_channel_readv
> > > #3  qio_channel_readv_all_eof
> > > #4  0x000055573ebce909 in qio_channel_readv_all
> > > #5  0x000055573eaa1b1f in multifd_recv_thread
> > > #6  0x000055573ec2f0b9 in qemu_thread_start
> > > #7  0x00007f041b52bf7a in start_thread
> > > #8  0x00007f041b5ae600 in clone3
> > >
> > > Thread 1 (Thread 0x7f0410c62240 (LWP 1895156) "kvm"):
> > > #0  0x00007f041b528ae2 in __futex_abstimed_wait_common ()
> > > #1  0x00007f041b5338b8 in __new_sem_wait_slow64.constprop.0
> > > #2  0x000055573ec2fd34 in qemu_sem_wait (sem=0x555742b5a4e0)
> > > #3  0x000055573eaa2f09 in multifd_recv_sync_main ()
> > > #4  0x000055573e7d590d in ram_load_precopy
> > (f=f@entry=0x555742291c20)
> > > #5  0x000055573e7d5cbf in ram_load (opaque=<optimized out>,
> > > version_id=<optimized out>, f=0x555742291c20)
> > > #6  ram_load_entry (f=0x555742291c20, opaque=<optimized out>,
> > > version_id=<optimized out>)
> > > #7  0x000055573ea932e7 in qemu_loadvm_section_part_end
> > > (mis=0x555741136c00, f=0x555742291c20)
> > > #8  qemu_loadvm_state_main (f=f@entry=0x555742291c20,
> > > mis=mis@entry=0x555741136c00)
> > > #9  0x000055573ea94418 in qemu_loadvm_state (f=0x555742291c20,
> > > mode=mode@entry=VMS_MIGRATE)
> > > #10 0x000055573ea88be1 in process_incoming_migration_co
> > > (opaque=<optimized out>)
> > > #11 0x000055573ec43d13 in coroutine_trampoline (i0=<optimized out>,
> > > i1=<optimized out>)
> > > #12 0x00007f041b4f5d90 in ?? () from target:/usr/lib64/libc.so.6
> > > #13 0x00007ffc11890270 in ?? ()
> > > #14 0x0000000000000000 in ?? ()
> > >
> > > Setting the receive channel to non-blocking can solve the problem.
> > 
> > Multifd threads are real threads and there's no coroutine, I'm slightly confused
> > why it needs to use nonblock.
> > 
> > Why recvmsg() didn't get kicked out when disconnect?  Is it a generic Linux
> > kernel are you using?
> > 
> My steps to reproduce:
> ifdown migration network, or disable migration network using iptables. 
> The probability of recurrence of these two methods is very high.
> 
> My test environment uses is linux-5.10.136.
> 
> multifd thread block in kernel:
> # cat /proc/3416190/stack 
> [<0>] wait_woken+0x43/0x80
> [<0>] sk_wait_data+0x123/0x140
> [<0>] tcp_recvmsg+0x4f8/0xa50
> [<0>] inet6_recvmsg+0x5e/0x120
> [<0>] ____sys_recvmsg+0x87/0x180
> [<0>] ___sys_recvmsg+0x82/0x110
> [<0>] __sys_recvmsg+0x56/0xa0
> [<0>] do_syscall_64+0x3d/0x80
> [<0>] entry_SYSCALL_64_after_hwframe+0x61/0xc6
> 
> > I wonder whether that's the expected behavior for sockets.  E.g., we do have
> > multifd/cancel test (test_multifd_tcp_cancel) and I think that runs this path too
> > with it always in block mode as of now..
> > 
> My previous statement may not be accurate. The migration network socket is not disconnected. 
> I use ifdown or iptables to simulate the network card failure. 
> Because the TCP connection was not disconnected, so recvmsg() was blocked.

How long did you wait after doing ifdown ?   TCP is intended to wait if
there is an interruption.... only eventually after relevant TCP timeouts
are hit, it will terminate the connection.  QEMU shouldn't proactively
give up if the TCP conn is still in an active state as reported by the
kernel, even if traffic isn't currently flowing.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|