Re: [PATCH v2 01/10] migration: Increase default number of multifd channels to 16

[Juan Quintela <quintela@redhat.com>]

Daniel P. Berrangé <berrange@redhat.com> wrote:
> On Wed, Dec 18, 2019 at 03:01:10AM +0100, Juan Quintela wrote:
>> We can scale much better with 16, so we can scale to higher numbers.
>
> What was the test scenario showing such scaling ?

On my test hardware, with 2 channels we can saturate around 8Gigabit max,
more than that, and the migration thread is not fast enough to fill the
network bandwidth.

With 8 that is enough to fill whatever we can find.
We used to have a bug where we were getting trouble with more channels
than cores.  That was the initial reason why the default was so low.

So, pros/cons are:
- have low value (2).  We are backwards compatible, but we are not using
  all  bandwith.  Notice that we will dectect the error before 5.0 is
  out and print a good error message.

- have high value (I tested 8 and 16).  Found no performance loss when
  moving to lower bandwidth limits, and clearly we were able to saturate
  the higher speeds (I tested on localhost, so I had big enough bandwidth)


> In the real world I'm sceptical that virt hosts will have
> 16 otherwise idle CPU cores available that are permissible
> to use for migration, or indeed whether they'll have network
> bandwidth available to allow 16 cores to saturate the link.

The problem here is that if you have such a host, and you want to have
high speed migration, you need to configure it.  My measumermets are
that high number of channels don't affect performance with low
bandwidth, but low number of channels affect performance with high
bandwidth speed.

So, if we want to have something that works "automatically" everywhere,
we need to put it to at least 8.  Or we can trust that management app
will do the right thing.

If you are using a low value of bandwidth, the only difference with 16
channels is that you are using a bit more memory (just the space for the
stacks) and that you are having less contention for the locks (but with
low bandwidth you are not having contention anyways).

So,  I think that the question is:
- What does libvirt prefferes
- What does ovirt/openstack preffer
- Do we really want that the user "have" to configure that value

I don't really care one way or another.

Thanks, Juan.

PD.  On next patch submission I will make it be 2 for old machine types,
     it is not difficult and makes the backward compatibility problem go
     away.