Re: [RFC PATCH 2/2] gnttab: refactor locking for better scalability

[Keir Fraser <keir.xen@gmail.com>]

On 12/11/2013 07:18, "Matt Wilson" <msw@linux.com> wrote:

>> Is there any concern about writer starvation here? I know our spinlocks
>> aren't 'fair' but our rwlocks are guaranteed to starve out writers if there
>> is a steady continuous stream of readers. Perhaps we should write-bias our
>> rwlock, or at least make that an option. We could get fancier but would
>> probably hurt performance.
> 
> Yes, I'm a little concerned about writer starvation. But so far even
> in the presence of very frequent readers it seems like the infrequent
> writers are able to get the lock when they need to. However, I've not
> tested the iommu=strict path yet. I'm thinking that in that case
> there's just going to be frequent writers, so there's less risk of
> readers starving writers. For what it's worth, when mapcount() gets in
> the picture with persistent grants, I'd expect to see some pretty
> significant performance degradation for map/unmap operations. This was
> also observed in [1] under different circumstances.

The average case isn't the only concern here, but also the worst case, which
could maybe tie up a CPU for unbounded time. Could a malicious guest set up
such a workload? I'm just thinking we don't want to end up with a DoS XSA on
this down the line. :)

> But right now I'm more curious about cache line bouncing between all
> the readers. I've not done any study of inter-arrival times for
> typical workloads (much less some more extreme workloads like we've
> been testing), but lock profiling of grant table operations when a
> spinlock was used showed some pretty long hold times, which should
> translate fairly well to decent rwlock performance. I'm by no means an
> expert in this area so I'm eager to hear the thoughts of others.

In the read-heavy case the only improvement would be with the old
Linux-style biglock (spinlock per CPU; writers must take all spinlocks), or
working out a lock-free scheme for readers (perhaps making use of RCU).

 -- Keir

> I should also mention that some of the improvement I mentioned from
> 3,000 MB/s to 3,600 MB/s was due to avoiding the m2p override spinlock
> in dom0.