Re: [PATCHv11 4/4] gnttab: use per-VCPU maptrack free lists

[David Vrabel <david.vrabel@citrix.com>]

On 05/06/15 15:51, Jan Beulich wrote:
>>>> On 02.06.15 at 18:26, <david.vrabel@citrix.com> wrote:
>> Performance analysis of aggregate network throughput with many VMs
>> shows that performance is signficantly limited by contention on the
>> maptrack lock when obtaining/releasing maptrack handles from the free
>> list.
>>
>> Instead of a single free list use a per-VCPU list. This avoids any
>> contention when obtaining a handle.  Handles must be released back to
>> their original list and since this may occur on a different VCPU there
>> is some contention on the destination VCPU's free list tail pointer
>> (but this is much better than a per-domain lock).
>>
>> Increase the default maximum number of maptrack frames by 4 times
>> because: a) struct grant_mapping is now 16 bytes (instead of 8); and
>> b) a guest may not evenly distribute all the grant map operations
>> across the VCPUs (meaning some VCPUs need more maptrack entries than
>> others).
>>
>> Signed-off-by: Malcolm Crossley <malcolm.crossley@citrix.com>
>> Signed-off-by: David Vrabel <david.vrabel@citrix.com>
>> Acked-by: Tim Deegan <tim@xen.org>
> 
> What version was that ack given for?

v9

>> +    /*
>> +     * max_maptrack_frames is per domain so each VCPU gets a share of
>> +     * the maximum, but allow at least one frame per VCPU.
>> +     */
>> +    if ( v->maptrack_frames
>> +         && v->maptrack_frames >= max_maptrack_frames / v->domain->max_vcpus )
>> +        return -1;
> 
> So with e.g. max_maptrack_frames being 256 and ->max_vcpus
> being 129 you'd potentially allow each vCPU to only have exactly
> one page despite there being 127 more to use.

There's a limit to how many wacky combinations we can support with a
single default limit.

With the standard defaults and 129 VCPUs:

Before

131072 entries (256 * 4096 / 8)

After

231168 entries (1024 / 129 * 129 * 4096 / 16)
  1792 entries per vcpu.

David