Re: Role of qemu_fair_mutex

[Anthony Liguori <anthony@codemonkey.ws>]

On 01/04/2011 08:27 AM, Avi Kivity wrote:
> On 01/04/2011 04:17 PM, Anthony Liguori wrote:
>> On 01/03/2011 04:01 AM, Avi Kivity wrote:
>>> On 01/03/2011 11:46 AM, Jan Kiszka wrote:
>>>> Hi,
>>>>
>>>> at least in kvm mode, the qemu_fair_mutex seems to have lost its
>>>> function of balancing qemu_global_mutex access between the 
>>>> io-thread and
>>>> vcpus. It's now only taken by the latter, isn't it?
>>>>
>>>> This and the fact that qemu-kvm does not use this kind of lock made me
>>>> wonder what its role is and if it is still relevant in practice. I'd
>>>> like to unify the execution models of qemu-kvm and qemu, and this lock
>>>> is the most obvious difference (there are surely more subtle ones as
>>>> well...).
>>>>
>>>
>>> IIRC it was used for tcg, which has a problem that kvm doesn't have: 
>>> a tcg vcpu needs to hold qemu_mutex when it runs, which means there 
>>> will always be contention on qemu_mutex.  In the absence of 
>>> fairness, the tcg thread could dominate qemu_mutex and starve the 
>>> iothread.
>>
>> No, it's actually the opposite IIRC.
>>
>> TCG relies on the following behavior.   A guest VCPU runs until 1) it 
>> encounters a HLT instruction 2) an event occurs that forces the TCG 
>> execution to break.
>>
>> (2) really means that the TCG thread receives a signal.  Usually, 
>> this is the periodic timer signal.
>
> What about a completion?  an I/O completes, the I/O thread wakes up, 
> needs to acquire the global lock (and force tcg off it) inject and 
> interrupt, and go back to sleep.

I/O completion triggers an fd to become readable.  This will cause 
select to break and the io thread will attempt to acquire the 
qemu_mutex.  When acquiring the mutex in TCG, the io thread will send a 
SIG_IPI to the TCG VCPU thread.


>>
>> When the TCG thread, it needs to let the IO thread run for at least 
>> one iteration.  Coordinating the execution of the IO thread such that 
>> it's guaranteed to run at least once and then having it drop the qemu 
>> mutex long enough for the TCG thread to acquire it is the purpose of 
>> the qemu_fair_mutex.
>
> That doesn't compute - the iothread doesn't hog the global lock (it 
> sleeps most of the time, and drops the lock while sleeping), so the 
> iothread cannot starve out tcg.

The fact that the iothread drops the global lock during sleep is a 
detail that shouldn't affect correctness.  The IO thread is absolutely 
allowed to run for arbitrary periods of time without dropping the qemu 
mutex.

>   On the other hand, tcg does hog the global lock, so it needs to be 
> made to give it up so the iothread can run, for example my completion 
> example.

It's very easy to ask TCG to give up the qemu_mutex by using 
cpu_interrupt().  It will drop the qemu_mutex and it will not attempt to 
acquire it again until you restart the VCPU.

> I think the abstraction we need here is a priority lock, with higher 
> priority given to the iothread.  A lock() operation that takes 
> precedence would atomically signal the current owner to drop the lock.

The I/O thread can reliably acquire the lock whenever it needs to.

If you drop all of the qemu_fair_mutex stuff and leave the qemu_mutex 
getting dropped around select, TCG will generally work reliably.  But 
this is not race free.  Just dropping a lock does not result in reliable 
hand off.

I think a generational counter could work and a condition could work.

Regards,

Anthony Liguori


> Under kvm we'd run a normal mutex, so the it wouldn't need to take the 
> extra mutex.
>