Re: [PATCH v3 00/22] Improve scalability of KVM + userfaultfd live migration via annotated memory faults.

[Peter Xu <peterx@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 5133 bytes --]

On Wed, May 03, 2023 at 12:45:07PM -0700, Anish Moorthy wrote:
> On Thu, Apr 27, 2023 at 1:26 PM Peter Xu <peterx@redhat.com> wrote:
> >
> > Thanks (for doing this test, and also to Nadav for all his inputs), and
> > sorry for a late response.
> 
> No need to apologize: anyways, I've got you comfortably beat on being
> late at this point :)
> 
> > These numbers caught my eye, and I'm very curious why even 2 vcpus can
> > scale that bad.
> >
> > I gave it a shot on a test machine and I got something slightly different:
> >
> >   Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz (20 cores, 40 threads)
> >   $ ./demand_paging_test -b 512M -u MINOR -s shmem -v N
> >   |-------+----------+--------|
> >   | n_thr | per-vcpu | total  |
> >   |-------+----------+--------|
> >   |     1 | 39.5K    | 39.5K  |
> >   |     2 | 33.8K    | 67.6K  |
> >   |     4 | 31.8K    | 127.2K |
> >   |     8 | 30.8K    | 246.1K |
> >   |    16 | 21.9K    | 351.0K |
> >   |-------+----------+--------|
> >
> > I used larger ram due to less cores.  I didn't try 32+ vcpus to make sure I
> > don't have two threads content on a core/thread already since I only got 40
> > hardware threads there, but still we can compare with your lower half.
> >
> > When I was testing I noticed bad numbers and another bug on not using
> > NSEC_PER_SEC properly, so I did this before the test:
> >
> > https://lore.kernel.org/all/20230427201112.2164776-1-peterx@redhat.com/
> >
> > I think it means it still doesn't scale that good, however not so bad
> > either - no obvious 1/2 drop on using 2vcpus.  There're still a bunch of
> > paths triggered in the test so I also don't expect it to fully scale
> > linearly.  From my numbers I just didn't see as drastic as yours. I'm not
> > sure whether it's simply broken test number, parameter differences
> > (e.g. you used 64M only per-vcpu), or hardware differences.
> 
> Hmm, I suspect we're dealing with  hardware differences here. I
> rebased my changes onto those two patches you sent up, taking care not
> to clobber them, but even with the repro command you provided my
> results look very different than yours (at least on 1-4 vcpus) on the
> machine I've been testing on (4x AMD EPYC 7B13 64-Core, 2.2GHz).
> 
> (n=20)
> n_thr      per_vcpu       total
> 1            154K              154K
> 2             92k                184K
> 4             71K                285K
> 8             36K                291K
> 16           19K                310K
> 
> Out of interested I tested on another machine (Intel(R) Xeon(R)
> Platinum 8273CL CPU @ 2.20GHz) as well, and results are a bit
> different again
> 
> (n=20)
> n_thr      per_vcpu       total
> 1            115K              115K
> 2             103k              206K
> 4             65K                262K
> 8             39K                319K
> 16           19K                398K

Interesting.

> 
> It is interesting how all three sets of numbers start off different
> but seem to converge around 16 vCPUs. I did check to make sure the
> memory fault exits sped things up in all cases, and that at least
> stays true.
> 
> By the way, I've got a little helper script that I've been using to
> run/average the selftest results (which can vary quite a bit). I've
> attached it below- hopefully it doesn't bounce from the mailing list.
> Just for reference, the invocation to test the command you provided is
> 
> > python dp_runner.py --num_runs 20 --max_cores 16 --percpu_mem 512M

I found that indeed I shouldn't have stopped at 16 vcpus since that's
exactly where it starts to bottleneck. :)

So out of my curiosity I tried to profile 32 vcpus case on my system with
this test case, meanwhile I tried it both with:

  - 1 uffd + 8 readers
  - 32 uffds (so 32 readers)

I've got the flamegraphs attached for both.

It seems that when using >1 uffds the bottleneck is not the spinlock
anymore but something else.

From what I got there, vmx_vcpu_load() gets more highlights than the
spinlocks. I think that's the tlb flush broadcast.

While OTOH indeed when using 1 uffd we can see obviously the overhead of
spinlock contention on either the fault() path or read()/poll() as you and
James rightfully pointed out.

I'm not sure whether my number is caused by special setup, though. After
all I only had 40 threads and I started 32 vcpus + 8 readers and there'll
be contention already between the workloads.

IMHO this means that there's still chance to provide a more generic
userfaultfd scaling solution as long as we can remove the single spinlock
contention on the fault/fault_pending queues.  I'll see whether I can still
explore a bit on the possibility of this and keep you guys updated.  The
general idea here to me is still to make multi-queue out of 1 uffd.

I _think_ this might also be a positive result to your work, because if the
bottleneck is not userfaultfd (as we scale it with creating multiple;
ignoring the split vma effect), then it cannot be resolved by scaling
userfaultfd alone anyway, anymore.  So a general solution, even if existed,
may not work here for kvm, because we'll get stuck somewhere else already.

-- 
Peter Xu

[-- Attachment #2: uffd-1-reader-8.svg --]
[-- Type: image/svg+xml, Size: 427017 bytes --]

[-- Attachment #3: uffd-32-reader-32.svg --]
[-- Type: image/svg+xml, Size: 220618 bytes --]