[Qemu-devel] Re: Role of qemu_fair_mutex

[Avi Kivity <avi@redhat.com>]

On 01/05/2011 06:44 PM, Anthony Liguori wrote:
> On 01/04/2011 03:39 PM, Marcelo Tosatti wrote:
>> On Tue, Jan 04, 2011 at 08:17:26AM -0600, Anthony Liguori wrote:
>>> On 01/03/2011 04:01 AM, Avi Kivity wrote:
>>>> On 01/03/2011 11:46 AM, Jan Kiszka wrote:
>>>>> Hi,
>>>>>
>>>>> at least in kvm mode, the qemu_fair_mutex seems to have lost its
>>>>> function of balancing qemu_global_mutex access between the 
>>>>> io-thread and
>>>>> vcpus. It's now only taken by the latter, isn't it?
>>>>>
>>>>> This and the fact that qemu-kvm does not use this kind of lock 
>>>>> made me
>>>>> wonder what its role is and if it is still relevant in practice. I'd
>>>>> like to unify the execution models of qemu-kvm and qemu, and this 
>>>>> lock
>>>>> is the most obvious difference (there are surely more subtle ones as
>>>>> well...).
>>>>>
>>>> IIRC it was used for tcg, which has a problem that kvm doesn't
>>>> have: a tcg vcpu needs to hold qemu_mutex when it runs, which
>>>> means there will always be contention on qemu_mutex.  In the
>>>> absence of fairness, the tcg thread could dominate qemu_mutex and
>>>> starve the iothread.
>>> No, it's actually the opposite IIRC.
>>>
>>> TCG relies on the following behavior.   A guest VCPU runs until 1)
>>> it encounters a HLT instruction 2) an event occurs that forces the
>>> TCG execution to break.
>>>
>>> (2) really means that the TCG thread receives a signal.  Usually,
>>> this is the periodic timer signal.
>>>
>>> When the TCG thread, it needs to let the IO thread run for at least
>>> one iteration.  Coordinating the execution of the IO thread such
>>> that it's guaranteed to run at least once and then having it drop
>>> the qemu mutex long enough for the TCG thread to acquire it is the
>>> purpose of the qemu_fair_mutex.
>> Its the vcpu threads that starve the IO thread.
>
> I'm not sure if this is a difference in semantics or if we're not 
> understanding each other.

I think, the latter.

>
> With TCG, the VCPU thread will dominate the qemu_mutex and cause the 
> IO thread to contend heavily on it.
>
> But the IO thread can always force TCG to exit it's loop (and does so 
> when leaving select()).  So the TCG thread make keep the IO thread 
> hungry, but it never "starves" it.

With a pure qemu_mutex_acquire(), tcg does starve out iothread.  
SIG_IPI/cpu_interrupt and qemu_fair_mutex, were introduced to solve this 
starvation; kvm doesn't require them.

>
> OTOH, the TCG thread struggles to hand over execution to the IO thread 
> while making sure that it gets back the qemu_mutex in a timely 
> fashion.  That's the tricky part.  Avi's point is that by giving up 
> the lock at select time, we prevent starvation but my concern is that 
> because the time between select intervals is unbounded (and 
> potentially very, very lock), it's effectively starvation.

It isn't starvation, since the iothread will eventually drain its work.

Suppose we do hand over to tcg while the iothread still has pending 
work.  What now? tcg will not drop the lock voluntarily.  When will the 
iothread complete its work?

Do we immediately interrupt tcg again?  If so, why did we give it the lock?
Do we sleep for a while and then reaquire the lock?  For how long?  
AFAWCT, tcg may be spinning waiting for a completion.

There's simply no scope for an iothread->tcg handoff.  The situation is 
not symmetric, it's more a client/server relationship.

Zooming out for a bit, let's see what out options are:

- the current qemu_fair_mutex/SIG_IPI thing
- a priority lock, which simply encapsulates the current 
qemu_fair_mutex.  tcg is made to drop the lock whenever anyone else 
attempts to acquire it.  No change in behaviour, just coding.
- make tcg take the qemu lock only in helper code; make sure we only do 
tcg things in the tcg thread (like playing with the tlb).  No need for 
special locking, but will reduce tcg throughput somewhat, my estimation 
is measurably but not significantly.

-- 
error compiling committee.c: too many arguments to function