KVM call agenda for Feb 9

KVM call agenda for Feb 9

[Chris Wright]

Please send in any agenda items you are interested in covering.

[Qemu-devel] KVM call agenda for Feb 9

[Chris Wright]

Please send in any agenda items you are interested in covering.

Re: KVM call agenda for Feb 9

[Avi Kivity]

On 02/09/2010 03:28 AM, Chris Wright wrote:
> Please send in any agenda items you are interested in covering.
>    

hpet overhead on large smp guests

I measured hpet consuming about a half a core's worth of cpu on an idle 
Windows 2008 R2 64-way guest.  This is mostly due to futex contention, 
likely from the qemu mutex.

Options:
- ignore, this is about 1% of the entire system (but overhead might 
increase greatly if a workload triggers more hpet accesses)
- push hpet into kernel, with virtio-net, virtio-blk, and kernel-hpet, 
there's little reason to exit into qemu
- rcuify/fine-grain qemu locks

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

[Qemu-devel] Re: KVM call agenda for Feb 9

[Avi Kivity]

On 02/09/2010 03:28 AM, Chris Wright wrote:
> Please send in any agenda items you are interested in covering.
>    

hpet overhead on large smp guests

I measured hpet consuming about a half a core's worth of cpu on an idle 
Windows 2008 R2 64-way guest.  This is mostly due to futex contention, 
likely from the qemu mutex.

Options:
- ignore, this is about 1% of the entire system (but overhead might 
increase greatly if a workload triggers more hpet accesses)
- push hpet into kernel, with virtio-net, virtio-blk, and kernel-hpet, 
there's little reason to exit into qemu
- rcuify/fine-grain qemu locks

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: [Qemu-devel] Re: KVM call agenda for Feb 9

[Alexander Graf]

On 09.02.2010, at 07:56, Avi Kivity wrote:

> On 02/09/2010 03:28 AM, Chris Wright wrote:
>> Please send in any agenda items you are interested in covering.
>>   
> 
> hpet overhead on large smp guests
> 
> I measured hpet consuming about a half a core's worth of cpu on an idle Windows 2008 R2 64-way guest.  This is mostly due to futex contention, likely from the qemu mutex.
> 
> Options:
> - ignore, this is about 1% of the entire system (but overhead might increase greatly if a workload triggers more hpet accesses)
> - push hpet into kernel, with virtio-net, virtio-blk, and kernel-hpet, there's little reason to exit into qemu

That sounds like a good idea to me. It'd make the hpet en par with the lapic as timing source concerning overhead.

> - rcuify/fine-grain qemu locks

And this should be done either way, but is probably not a short-term goal.


Alex

Re: [Qemu-devel] Re: KVM call agenda for Feb 9

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 454 bytes --]

Alexander Graf wrote:
> On 09.02.2010, at 07:56, Avi Kivity wrote:
>> - rcuify/fine-grain qemu locks
> 
> And this should be done either way, but is probably not a short-term goal.
> 

Indeed. We won't get around this longterm as it is a scalability
bottleneck and a killer for RT guest load. We can't push everything into
the kernel. Qemu needs a smart plan how to gradually convert its CPU and
device model to fine-grained locking.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: [Qemu-devel] Re: KVM call agenda for Feb 9

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 454 bytes --]

Alexander Graf wrote:
> On 09.02.2010, at 07:56, Avi Kivity wrote:
>> - rcuify/fine-grain qemu locks
> 
> And this should be done either way, but is probably not a short-term goal.
> 

Indeed. We won't get around this longterm as it is a scalability
bottleneck and a killer for RT guest load. We can't push everything into
the kernel. Qemu needs a smart plan how to gradually convert its CPU and
device model to fine-grained locking.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: [Qemu-devel] Re: KVM call agenda for Feb 9

[Anthony Liguori]

On 02/09/2010 02:52 AM, Jan Kiszka wrote:
> Alexander Graf wrote:
>    
>> On 09.02.2010, at 07:56, Avi Kivity wrote:
>>      
>>> - rcuify/fine-grain qemu locks
>>>        
>> And this should be done either way, but is probably not a short-term goal.
>>
>>      
> Indeed. We won't get around this longterm as it is a scalability
> bottleneck and a killer for RT guest load. We can't push everything into
> the kernel. Qemu needs a smart plan how to gradually convert its CPU and
> device model to fine-grained locking.
>    

The VCPU loops should be easy to convert to lockless operation.  It's 
easier to do this upstream but that requires a functioning IO thread.

For the table based MMIO and PIO dispatch, RCU would be a good locking 
choice since these structures are rarely updated.  The tricky bit is 
that the APIC has to be converted over to lockless before any other 
device can be converted because just about every device wants to inject 
an interrupt.

Regards,

Anthony Liguori

> Jan
>
>    

Re: [Qemu-devel] Re: KVM call agenda for Feb 9

[Anthony Liguori]

On 02/09/2010 02:52 AM, Jan Kiszka wrote:
> Alexander Graf wrote:
>    
>> On 09.02.2010, at 07:56, Avi Kivity wrote:
>>      
>>> - rcuify/fine-grain qemu locks
>>>        
>> And this should be done either way, but is probably not a short-term goal.
>>
>>      
> Indeed. We won't get around this longterm as it is a scalability
> bottleneck and a killer for RT guest load. We can't push everything into
> the kernel. Qemu needs a smart plan how to gradually convert its CPU and
> device model to fine-grained locking.
>    

The VCPU loops should be easy to convert to lockless operation.  It's 
easier to do this upstream but that requires a functioning IO thread.

For the table based MMIO and PIO dispatch, RCU would be a good locking 
choice since these structures are rarely updated.  The tricky bit is 
that the APIC has to be converted over to lockless before any other 
device can be converted because just about every device wants to inject 
an interrupt.

Regards,

Anthony Liguori

> Jan
>
>    

Re: [Qemu-devel] Re: KVM call agenda for Feb 9

[Avi Kivity]

On 02/09/2010 04:18 PM, Anthony Liguori wrote:
> On 02/09/2010 02:52 AM, Jan Kiszka wrote:
>> Alexander Graf wrote:
>>> On 09.02.2010, at 07:56, Avi Kivity wrote:
>>>> - rcuify/fine-grain qemu locks
>>> And this should be done either way, but is probably not a short-term 
>>> goal.
>>>
>> Indeed. We won't get around this longterm as it is a scalability
>> bottleneck and a killer for RT guest load. We can't push everything into
>> the kernel. Qemu needs a smart plan how to gradually convert its CPU and
>> device model to fine-grained locking.
>
> The VCPU loops should be easy to convert to lockless operation.  It's 
> easier to do this upstream but that requires a functioning IO thread.
>
> For the table based MMIO and PIO dispatch, RCU would be a good locking 
> choice since these structures are rarely updated.  The tricky bit is 
> that the APIC has to be converted over to lockless before any other 
> device can be converted because just about every device wants to 
> inject an interrupt.

The problem is that all the internal APIs now have to be threaded.  
Looking at hpet as a simple example, we have qemu_irq_pulse() and 
qemu_mod_timer().  We also have some lock inversion since hpet calls the 
timer but the timers also call hpet.  qemu_irq_pulse() feeds the various 
interrupt controllers; not a problem for kernel irqchip but nontrivial 
for qemu's ioapic and pic.

I'm not saying we should push hpet into the kernel to save userspace 
coding effort; there should be an independent reason to do this.  But I 
don't think threading qemu is going to be anything near easy.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Re: KVM call agenda for Feb 9

[Avi Kivity]

On 02/09/2010 04:18 PM, Anthony Liguori wrote:
> On 02/09/2010 02:52 AM, Jan Kiszka wrote:
>> Alexander Graf wrote:
>>> On 09.02.2010, at 07:56, Avi Kivity wrote:
>>>> - rcuify/fine-grain qemu locks
>>> And this should be done either way, but is probably not a short-term 
>>> goal.
>>>
>> Indeed. We won't get around this longterm as it is a scalability
>> bottleneck and a killer for RT guest load. We can't push everything into
>> the kernel. Qemu needs a smart plan how to gradually convert its CPU and
>> device model to fine-grained locking.
>
> The VCPU loops should be easy to convert to lockless operation.  It's 
> easier to do this upstream but that requires a functioning IO thread.
>
> For the table based MMIO and PIO dispatch, RCU would be a good locking 
> choice since these structures are rarely updated.  The tricky bit is 
> that the APIC has to be converted over to lockless before any other 
> device can be converted because just about every device wants to 
> inject an interrupt.

The problem is that all the internal APIs now have to be threaded.  
Looking at hpet as a simple example, we have qemu_irq_pulse() and 
qemu_mod_timer().  We also have some lock inversion since hpet calls the 
timer but the timers also call hpet.  qemu_irq_pulse() feeds the various 
interrupt controllers; not a problem for kernel irqchip but nontrivial 
for qemu's ioapic and pic.

I'm not saying we should push hpet into the kernel to save userspace 
coding effort; there should be an independent reason to do this.  But I 
don't think threading qemu is going to be anything near easy.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Re: KVM call agenda for Feb 9

[Anthony Liguori]

On 02/09/2010 08:38 AM, Avi Kivity wrote:
> On 02/09/2010 04:18 PM, Anthony Liguori wrote:
>> On 02/09/2010 02:52 AM, Jan Kiszka wrote:
>>> Alexander Graf wrote:
>>>> On 09.02.2010, at 07:56, Avi Kivity wrote:
>>>>> - rcuify/fine-grain qemu locks
>>>> And this should be done either way, but is probably not a 
>>>> short-term goal.
>>>>
>>> Indeed. We won't get around this longterm as it is a scalability
>>> bottleneck and a killer for RT guest load. We can't push everything 
>>> into
>>> the kernel. Qemu needs a smart plan how to gradually convert its CPU 
>>> and
>>> device model to fine-grained locking.
>>
>> The VCPU loops should be easy to convert to lockless operation.  It's 
>> easier to do this upstream but that requires a functioning IO thread.
>>
>> For the table based MMIO and PIO dispatch, RCU would be a good 
>> locking choice since these structures are rarely updated.  The tricky 
>> bit is that the APIC has to be converted over to lockless before any 
>> other device can be converted because just about every device wants 
>> to inject an interrupt.
>
> The problem is that all the internal APIs now have to be threaded.  
> Looking at hpet as a simple example, we have qemu_irq_pulse() and 
> qemu_mod_timer().

Yes.  But just as you mention, we have to approach it on a 
device-by-device basis and incrementally convert over devices.

>   We also have some lock inversion since hpet calls the timer but the 
> timers also call hpet.

Right.  Making the dispatch loop in the IO thread thread safe won't be 
so bad either :-)

>   qemu_irq_pulse() feeds the various interrupt controllers; not a 
> problem for kernel irqchip but nontrivial for qemu's ioapic and pic.
>
> I'm not saying we should push hpet into the kernel to save userspace 
> coding effort; there should be an independent reason to do this.  But 
> I don't think threading qemu is going to be anything near easy.

It's certainly not easy but I don't think it's impossibly hard.

Regards,

Anthony Liguori

Re: [Qemu-devel] Re: KVM call agenda for Feb 9

[Anthony Liguori]

On 02/09/2010 08:38 AM, Avi Kivity wrote:
> On 02/09/2010 04:18 PM, Anthony Liguori wrote:
>> On 02/09/2010 02:52 AM, Jan Kiszka wrote:
>>> Alexander Graf wrote:
>>>> On 09.02.2010, at 07:56, Avi Kivity wrote:
>>>>> - rcuify/fine-grain qemu locks
>>>> And this should be done either way, but is probably not a 
>>>> short-term goal.
>>>>
>>> Indeed. We won't get around this longterm as it is a scalability
>>> bottleneck and a killer for RT guest load. We can't push everything 
>>> into
>>> the kernel. Qemu needs a smart plan how to gradually convert its CPU 
>>> and
>>> device model to fine-grained locking.
>>
>> The VCPU loops should be easy to convert to lockless operation.  It's 
>> easier to do this upstream but that requires a functioning IO thread.
>>
>> For the table based MMIO and PIO dispatch, RCU would be a good 
>> locking choice since these structures are rarely updated.  The tricky 
>> bit is that the APIC has to be converted over to lockless before any 
>> other device can be converted because just about every device wants 
>> to inject an interrupt.
>
> The problem is that all the internal APIs now have to be threaded.  
> Looking at hpet as a simple example, we have qemu_irq_pulse() and 
> qemu_mod_timer().

Yes.  But just as you mention, we have to approach it on a 
device-by-device basis and incrementally convert over devices.

>   We also have some lock inversion since hpet calls the timer but the 
> timers also call hpet.

Right.  Making the dispatch loop in the IO thread thread safe won't be 
so bad either :-)

>   qemu_irq_pulse() feeds the various interrupt controllers; not a 
> problem for kernel irqchip but nontrivial for qemu's ioapic and pic.
>
> I'm not saying we should push hpet into the kernel to save userspace 
> coding effort; there should be an independent reason to do this.  But 
> I don't think threading qemu is going to be anything near easy.

It's certainly not easy but I don't think it's impossibly hard.

Regards,

Anthony Liguori

Re: [Qemu-devel] Re: KVM call agenda for Feb 9

[Avi Kivity]

On 02/09/2010 05:08 PM, Anthony Liguori wrote:
>> I'm not saying we should push hpet into the kernel to save userspace 
>> coding effort; there should be an independent reason to do this.  But 
>> I don't think threading qemu is going to be anything near easy.
>
>
> It's certainly not easy but I don't think it's impossibly hard.

Not impossible - see for example the kernel, the BKL should be removed RSN.

But yes, I think RCU will simplify things for us, and we have all of 
that experience to learn from.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Re: KVM call agenda for Feb 9

[Avi Kivity]

On 02/09/2010 05:08 PM, Anthony Liguori wrote:
>> I'm not saying we should push hpet into the kernel to save userspace 
>> coding effort; there should be an independent reason to do this.  But 
>> I don't think threading qemu is going to be anything near easy.
>
>
> It's certainly not easy but I don't think it's impossibly hard.

Not impossible - see for example the kernel, the BKL should be removed RSN.

But yes, I think RCU will simplify things for us, and we have all of 
that experience to learn from.

-- 
error compiling committee.c: too many arguments to function

Re: KVM call agenda for Feb 9

[Anthony Liguori]

On 02/09/2010 12:56 AM, Avi Kivity wrote:
> On 02/09/2010 03:28 AM, Chris Wright wrote:
>> Please send in any agenda items you are interested in covering.
>
> hpet overhead on large smp guests
>
> I measured hpet consuming about a half a core's worth of cpu on an 
> idle Windows 2008 R2 64-way guest.  This is mostly due to futex 
> contention, likely from the qemu mutex.
>
> Options:
> - ignore, this is about 1% of the entire system (but overhead might 
> increase greatly if a workload triggers more hpet accesses)
> - push hpet into kernel, with virtio-net, virtio-blk, and kernel-hpet, 
> there's little reason to exit into qemu

Security, shamurity, let's just stick all of qemu in the kernel :-)

> - rcuify/fine-grain qemu locks

Should be pretty straight forward.  It would start with removing the 
locking within kvm*.c such that qemu_mutex isn't acquired until we 
dispatch I/O operations.  Then we can add lockless paths for dispatch as 
we convert device models over.

Regards,

Anthony Liguori

[Qemu-devel] Re: KVM call agenda for Feb 9

[Anthony Liguori]

On 02/09/2010 12:56 AM, Avi Kivity wrote:
> On 02/09/2010 03:28 AM, Chris Wright wrote:
>> Please send in any agenda items you are interested in covering.
>
> hpet overhead on large smp guests
>
> I measured hpet consuming about a half a core's worth of cpu on an 
> idle Windows 2008 R2 64-way guest.  This is mostly due to futex 
> contention, likely from the qemu mutex.
>
> Options:
> - ignore, this is about 1% of the entire system (but overhead might 
> increase greatly if a workload triggers more hpet accesses)
> - push hpet into kernel, with virtio-net, virtio-blk, and kernel-hpet, 
> there's little reason to exit into qemu

Security, shamurity, let's just stick all of qemu in the kernel :-)

> - rcuify/fine-grain qemu locks

Should be pretty straight forward.  It would start with removing the 
locking within kvm*.c such that qemu_mutex isn't acquired until we 
dispatch I/O operations.  Then we can add lockless paths for dispatch as 
we convert device models over.

Regards,

Anthony Liguori

Re: KVM call agenda for Feb 9

[Avi Kivity]

On 02/09/2010 08:56 AM, Avi Kivity wrote:
> On 02/09/2010 03:28 AM, Chris Wright wrote:
>> Please send in any agenda items you are interested in covering.
>
> hpet overhead on large smp guests
>
> I measured hpet consuming about a half a core's worth of cpu on an 
> idle Windows 2008 R2 64-way guest.  This is mostly due to futex 
> contention, likely from the qemu mutex.
>
> Options:
> - ignore, this is about 1% of the entire system (but overhead might 
> increase greatly if a workload triggers more hpet accesses)
> - push hpet into kernel, with virtio-net, virtio-blk, and kernel-hpet, 
> there's little reason to exit into qemu
> - rcuify/fine-grain qemu locks
>
- implement hyper-v timers


-- 
error compiling committee.c: too many arguments to function

[Qemu-devel] Re: KVM call agenda for Feb 9

[Avi Kivity]

On 02/09/2010 08:56 AM, Avi Kivity wrote:
> On 02/09/2010 03:28 AM, Chris Wright wrote:
>> Please send in any agenda items you are interested in covering.
>
> hpet overhead on large smp guests
>
> I measured hpet consuming about a half a core's worth of cpu on an 
> idle Windows 2008 R2 64-way guest.  This is mostly due to futex 
> contention, likely from the qemu mutex.
>
> Options:
> - ignore, this is about 1% of the entire system (but overhead might 
> increase greatly if a workload triggers more hpet accesses)
> - push hpet into kernel, with virtio-net, virtio-blk, and kernel-hpet, 
> there's little reason to exit into qemu
> - rcuify/fine-grain qemu locks
>
- implement hyper-v timers


-- 
error compiling committee.c: too many arguments to function