Re: [PATCH v2 0/1] migration: multifd live migration improvement

[Li Zhang <lizhang@suse.de>]

On 12/7/21 3:16 PM, Daniel P. Berrangé wrote:
> On Tue, Dec 07, 2021 at 02:45:10PM +0100, Li Zhang wrote:
>> On 12/6/21 8:54 PM, Dr. David Alan Gilbert wrote:
>>> * Li Zhang (lizhang@suse.de) wrote:
>>>> When testing live migration with multifd channels (8, 16, or a bigger number)
>>>> and using qemu -incoming (without "defer"), if a network error occurs
>>>> (for example, triggering the kernel SYN flooding detection),
>>>> the migration fails and the guest hangs forever.
>>>>
>>>> The test environment and the command line is as the following:
>>>>
>>>> QEMU verions: QEMU emulator version 6.2.91 (v6.2.0-rc1-47-gc5fbdd60cf)
>>>> Host OS: SLE 15  with kernel: 5.14.5-1-default
>>>> Network Card: mlx5 100Gbps
>>>> Network card: Intel Corporation I350 Gigabit (1Gbps)
>>>>
>>>> Source:
>>>> qemu-system-x86_64 -M q35 -smp 32 -nographic \
>>>>           -serial telnet:10.156.208.153:4321,server,nowait \
>>>>           -m 4096 -enable-kvm -hda /var/lib/libvirt/images/openSUSE-15.3.img \
>>>>           -monitor stdio
>>>> Dest:
>>>> qemu-system-x86_64 -M q35 -smp 32 -nographic \
>>>>           -serial telnet:10.156.208.154:4321,server,nowait \
>>>>           -m 4096 -enable-kvm -hda /var/lib/libvirt/images/openSUSE-15.3.img \
>>>>           -monitor stdio \
>>>>           -incoming tcp:1.0.8.154:4000
>>>>
>>>> (qemu) migrate_set_parameter max-bandwidth 100G
>>>> (qemu) migrate_set_capability multifd on
>>>> (qemu) migrate_set_parameter multifd-channels 16
>>>>
>>>> The guest hangs when executing the command: migrate -d tcp:1.0.8.154:4000.
>>>>
>>>> If a network problem happens, TCP ACK is not received by destination
>>>> and the destination resets the connection with RST.
>>>>
>>>> No.     Time    Source  Destination     Protocol        Length  Info
>>>> 119     1.021169        1.0.8.153       1.0.8.154       TCP     1410    60166 → 4000 [PSH, ACK] Seq=65 Ack=1 Win=62720 Len=1344 TSval=1338662881 TSecr=1399531897
>>>> No.     Time    Source  Destination     Protocol        Length  Info
>>>> 125     1.021181        1.0.8.154       1.0.8.153       TCP     54      4000 → 60166 [RST] Seq=1 Win=0 Len=0
>>>>
>>>> kernel log:
>>>> [334520.229445] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies.  Check SNMP counters.
>>>> [334562.994919] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies.  Check SNMP counters.
>>>> [334695.519927] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies.  Check SNMP counters.
>>>> [334734.689511] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies.  Check SNMP counters.
>>>> [335687.740415] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies.  Check SNMP counters.
>>>> [335730.013598] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies.  Check SNMP counters.
>>> Should we document somewhere how to avoid that?  Is there something we
>>> should be doing in the connection code to avoid it?
>> We should use the command line -incoming defer in QEMU command line instead
>> of -incoming ip:port.
>>
>> And the backlog of the socket will be set as the same as  multifd channels,
>> this problem doesn't happen as far as I test.
>>
>> If we use --incoming ip:port in the QEMU command line, the backlog of the
>> socket is always 1, it will cause the SYN flooding.
> Do we send migration parameters from the src to the dst QEMU ?

No, I don't think we send migration parameters from the src to the dest 
QEMU.

I set migration parameters on both sides from qemu monitor seperately.

> There are a bunch of things that we need to set to the same
> value on the src and dst. If we sent any relevant MigrationParameters
> fields to the dst, when the first/main migration chanel is opened, it
> could validate that it is configured in a way that is compatible with
> the src. If it isn't, it can drop the main channel immediately. This
> would trigger the src to fail the migration and we couldn't get stuck
> setting up the secondary data channels for multifd.

OK,  currently, we have same parameters on both sides if we set them the 
same parameters.

If we use -incoming tcp:ip:port because the multifd is disabled by 
default and backlog is 1 when the socket is created.

Here is the function which set the backlog:

static void
socket_start_incoming_migration_internal(SocketAddress *saddr,
                                          Error **errp)
{
     QIONetListener *listener = qio_net_listener_new();
     MigrationIncomingState *mis = migration_incoming_get_current();
     size_t i;
     int num = 1;

     qio_net_listener_set_name(listener, "migration-socket-listener");

     if (migrate_use_multifd()) {
         num = migrate_multifd_channels();
     }
   ...

}

The process  with -incoming tcp:ip:port is as the following:

1.   Create qemu process with command line -incoming tcp:ip:port

2.   socket_start_incoming_migration_internal  is called and backlog is: 
num=1, multifd is disabled, num = migrate_multifd_channels() is not called

3.   Enable multifd and set multifd parameters, but the backlog is still 
1, because the it couldn't be changed anymore.

4.   Run migration

The process with -incoming defer is as the following:

1. Create qemu process with command line -incoming defer

2. Enable multifd and set multifd parameters

3. Execute the command (qemu) migrate_incoming tcp:ip:port

4. Call socket_start_incoming_migration_internal then the backlog is 
set: num = migrate_multifd_channels();

5. Run migration

>
> Regards,
> Daniel