question: HPET for multiple VMs

question: HPET for multiple VMs

[Ryota OZAKI]

Hi all,

Current kvm allows only one VM to use HPET. Is
there a plan to implement a functionality to
allow multiple VMs to use HPET? If so, how
about the status of that?

And I would like to ask right and wrong to
implement the functionality in terms of need
and efficiency (scalability and time accuracy).

Thanks,
ozaki-r

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: question: HPET for multiple VMs

[Avi Kivity]

Ryota OZAKI wrote:
> Hi all,
>
> Current kvm allows only one VM to use HPET. Is
> there a plan to implement a functionality to
> allow multiple VMs to use HPET? If so, how
> about the status of that?
>
>   

If you use the dyntick clock option (the default, IIRC), and a newer 
host kernel, then the kernel provides high-resolution timers, very 
likely using HPET internally or some other high resolution clock and 
event source.

> And I would like to ask right and wrong to
> implement the functionality in terms of need
> and efficiency (scalability and time accuracy).

I think that for newer kernels we already have the desired accuracy.  
We're not always good at exploiting that accuracy; hence the recent 
movement of the PIT implementation from userspace to the kernel.  But 
recent discussion leads me to believe it could have been implemented 
with the userspace PIT as well.

-- 
Any sufficiently difficult bug is indistinguishable from a feature.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: question: HPET for multiple VMs

[Anthony Liguori]

Avi Kivity wrote:
>> And I would like to ask right and wrong to
>> implement the functionality in terms of need
>> and efficiency (scalability and time accuracy).
>>     
>
> I think that for newer kernels we already have the desired accuracy.  
> We're not always good at exploiting that accuracy; hence the recent 
> movement of the PIT implementation from userspace to the kernel.  But 
> recent discussion leads me to believe it could have been implemented 
> with the userspace PIT as well.
>   

What do you think is needed to get the same accuracy in userspace as in 
kernelspace?  Better yet, do you think there is a reasonable kvmctl 
harness we could write to quantify the PIT accuracy?

It's easy enough to count timer interrupts and use compare that to an 
external time source to get some notion of accuracy (on varying 
frequencies of course).  I know you mentioned before that guest CPU 
consumption also comes into play... I'm not quite sure why though so I'm 
not sure how to simulate that.

The nice thing about the CAP infrastructure is we can always move the 
PIT back to userspace.  I'll happily invest some cycles here as I'm a 
big fan of getting rid of unneeded kernel code :-)

Regards,

Anthony Liguori


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: question: HPET for multiple VMs

[Avi Kivity]

Anthony Liguori wrote:
> Avi Kivity wrote:
>>> And I would like to ask right and wrong to
>>> implement the functionality in terms of need
>>> and efficiency (scalability and time accuracy).
>>>     
>>
>> I think that for newer kernels we already have the desired accuracy.  
>> We're not always good at exploiting that accuracy; hence the recent 
>> movement of the PIT implementation from userspace to the kernel.  But 
>> recent discussion leads me to believe it could have been implemented 
>> with the userspace PIT as well.
>>   
>
> What do you think is needed to get the same accuracy in userspace as 
> in kernelspace?  

Some mechanism that allows us to implement kvm_inject_pit_timer_irqs() 
and kvm_pit_timer_intr_post().  Specifically, information about whether 
an interrupt was actually processed, and a window for injecting missed 
ticks.

> Better yet, do you think there is a reasonable kvmctl harness we could 
> write to quantify the PIT accuracy?

kvmctl doesn't implement a pit, so no.  Of course we can test any 
infrastructure for counting missed interrupts.

>
> It's easy enough to count timer interrupts and use compare that to an 
> external time source to get some notion of accuracy (on varying 
> frequencies of course).  I know you mentioned before that guest CPU 
> consumption also comes into play... I'm not quite sure why though so 
> I'm not sure how to simulate that.

It's not so easy, the code is quite tricky since the cpu processes 
vectors, not interrupt lines.  It's also heuristic; if the guest 
programs some random device to share interrupts with the pit, the 
heursitic breaks down.  This never happens in practice, though.

Problems show up when both the guest and host are loaded, as then the 
cpu is timesliced instead of being available on demand.

>
> The nice thing about the CAP infrastructure is we can always move the 
> PIT back to userspace.  I'll happily invest some cycles here as I'm a 
> big fan of getting rid of unneeded kernel code :-)

Yes.

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: question: HPET for multiple VMs

[Dor Laor]

On Sun, 2008-03-23 at 10:30 +0200, Avi Kivity wrote:
> Anthony Liguori wrote:
> > Avi Kivity wrote:
> >>> And I would like to ask right and wrong to
> >>> implement the functionality in terms of need
> >>> and efficiency (scalability and time accuracy).
> >>>     
> >>
> >> I think that for newer kernels we already have the desired accuracy.  
> >> We're not always good at exploiting that accuracy; hence the recent 
> >> movement of the PIT implementation from userspace to the kernel.  But 
> >> recent discussion leads me to believe it could have been implemented 
> >> with the userspace PIT as well.
> >>   
> >
> > What do you think is needed to get the same accuracy in userspace as 
> > in kernelspace?  
> 
> Some mechanism that allows us to implement kvm_inject_pit_timer_irqs() 
> and kvm_pit_timer_intr_post().  Specifically, information about whether 
> an interrupt was actually processed, and a window for injecting missed 
> ticks.
> 
> > Better yet, do you think there is a reasonable kvmctl harness we could 
> > write to quantify the PIT accuracy?
> 
> kvmctl doesn't implement a pit, so no.  Of course we can test any 
> infrastructure for counting missed interrupts.
> 
> >
> > It's easy enough to count timer interrupts and use compare that to an 
> > external time source to get some notion of accuracy (on varying 
> > frequencies of course).  I know you mentioned before that guest CPU 
> > consumption also comes into play... I'm not quite sure why though so 
> > I'm not sure how to simulate that.
> 
> It's not so easy, the code is quite tricky since the cpu processes 
> vectors, not interrupt lines.  It's also heuristic; if the guest 
> programs some random device to share interrupts with the pit, the 
> heursitic breaks down.  This never happens in practice, though.
> 
> Problems show up when both the guest and host are loaded, as then the 
> cpu is timesliced instead of being available on demand.
> 

I have patches that works for the rtc case with acpi windows, just
cleaning and sending for a review.
The same method should work for the pit as well.

> >
> > The nice thing about the CAP infrastructure is we can always move the 
> > PIT back to userspace.  I'll happily invest some cycles here as I'm a 
> > big fan of getting rid of unneeded kernel code :-)
> 
> Yes.
> 


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: question: HPET for multiple VMs

[Ryota OZAKI]

Hi Avi,

> If you use the dyntick clock option (the default, IIRC), and a newer
>  host kernel, then the kernel provides high-resolution timers, very
>  likely using HPET internally or some other high resolution clock and
>  event source.

I see. The dyntick clock seems to be more scalable than
the others. I understood that '-clock hpet' is used for
boosting one VM (becuase hpet gains best performance
on virtio), right?

I would like to try dyntick for my multiple VMs environment.

> I think that for newer kernels we already have the desired accuracy.

Yes. In recent versions of kvm, I didn't experience
any time inaccuracy, although I had only tested under
several VMs. I'll try the more number of VMs, and
if time inaccuracy occurs, I would like to report
that.

Many thanks,
ozaki-r

2008/3/23, Avi Kivity <avi@qumranet.com>:
> Ryota OZAKI wrote:
>  > Hi all,
>  >
>  > Current kvm allows only one VM to use HPET. Is
>  > there a plan to implement a functionality to
>  > allow multiple VMs to use HPET? If so, how
>  > about the status of that?
>  >
>  >
>
>
> If you use the dyntick clock option (the default, IIRC), and a newer
>  host kernel, then the kernel provides high-resolution timers, very
>  likely using HPET internally or some other high resolution clock and
>  event source.
>
>
>  > And I would like to ask right and wrong to
>  > implement the functionality in terms of need
>  > and efficiency (scalability and time accuracy).
>
>
> I think that for newer kernels we already have the desired accuracy.
>  We're not always good at exploiting that accuracy; hence the recent
>  movement of the PIT implementation from userspace to the kernel.  But
>  recent discussion leads me to believe it could have been implemented
>  with the userspace PIT as well.
>
>
>  --
>  Any sufficiently difficult bug is indistinguishable from a feature.
>
>

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: question: HPET for multiple VMs

[Dor Laor]

On Mon, 2008-03-24 at 00:32 +0900, Ryota OZAKI wrote:
> Hi Avi,
> 
> > If you use the dyntick clock option (the default, IIRC), and a newer
> >  host kernel, then the kernel provides high-resolution timers, very
> >  likely using HPET internally or some other high resolution clock and
> >  event source.
> 
> I see. The dyntick clock seems to be more scalable than
> the others. I understood that '-clock hpet' is used for
> boosting one VM (becuase hpet gains best performance
> on virtio), right?
> 
> I would like to try dyntick for my multiple VMs environment.
> 
> > I think that for newer kernels we already have the desired accuracy.
> 
> Yes. In recent versions of kvm, I didn't experience
> any time inaccuracy, although I had only tested under
> several VMs. I'll try the more number of VMs, and
> if time inaccuracy occurs, I would like to report
> that.
> 

The problem is not inaccuracy of guest clock (which we do suffer from in
some guests and there is work in progress to fix). The problem is that
qemu_timer is not accurate, thus the virtio tx timer is too slow leading
to not optimized performance for virtio-net.

Try host kernel >= 2.6.24 with dyntick.

> Many thanks,
> ozaki-r
> 
> 2008/3/23, Avi Kivity <avi@qumranet.com>:
> > Ryota OZAKI wrote:
> >  > Hi all,
> >  >
> >  > Current kvm allows only one VM to use HPET. Is
> >  > there a plan to implement a functionality to
> >  > allow multiple VMs to use HPET? If so, how
> >  > about the status of that?
> >  >
> >  >
> >
> >
> > If you use the dyntick clock option (the default, IIRC), and a newer
> >  host kernel, then the kernel provides high-resolution timers, very
> >  likely using HPET internally or some other high resolution clock and
> >  event source.
> >
> >
> >  > And I would like to ask right and wrong to
> >  > implement the functionality in terms of need
> >  > and efficiency (scalability and time accuracy).
> >
> >
> > I think that for newer kernels we already have the desired accuracy.
> >  We're not always good at exploiting that accuracy; hence the recent
> >  movement of the PIT implementation from userspace to the kernel.  But
> >  recent discussion leads me to believe it could have been implemented
> >  with the userspace PIT as well.
> >
> >
> >  --
> >  Any sufficiently difficult bug is indistinguishable from a feature.
> >
> >
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> kvm-devel mailing list
> kvm-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/kvm-devel


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: question: HPET for multiple VMs

[Ryota OZAKI]

2008/3/24, Dor Laor <dor.laor@gmail.com>:
>
>  On Mon, 2008-03-24 at 00:32 +0900, Ryota OZAKI wrote:
>  > Hi Avi,
>  >
>  > > If you use the dyntick clock option (the default, IIRC), and a newer
>  > >  host kernel, then the kernel provides high-resolution timers, very
>  > >  likely using HPET internally or some other high resolution clock and
>  > >  event source.
>  >
>  > I see. The dyntick clock seems to be more scalable than
>  > the others. I understood that '-clock hpet' is used for
>  > boosting one VM (becuase hpet gains best performance
>  > on virtio), right?
>  >
>  > I would like to try dyntick for my multiple VMs environment.
>  >
>  > > I think that for newer kernels we already have the desired accuracy.
>  >
>  > Yes. In recent versions of kvm, I didn't experience
>  > any time inaccuracy, although I had only tested under
>  > several VMs. I'll try the more number of VMs, and
>  > if time inaccuracy occurs, I would like to report
>  > that.
>  >
>
>
> The problem is not inaccuracy of guest clock (which we do suffer from in
>  some guests and there is work in progress to fix). The problem is that
>  qemu_timer is not accurate, thus the virtio tx timer is too slow leading
>  to not optimized performance for virtio-net.

My apology. As you pointed out, I confused them.

>  Try host kernel >= 2.6.24 with dyntick.

I tried 2.6.25-rc5 with dyntick, but in my evaluation, the tx
performance of virtio-net achieves only a half of rx performance;
tx is 446 Mbps whereas rx is 913 Mbps. Is this best tx performance
at this time, or not? (If that helps, tx with HPET achieves 524
Mbps.)

My experimental setup is:
host cpu: Xeon E5410 2.33 GHz (Quad cores) x 2
host kenrel: linux-2.6.25-rc5
kvm: kvm-63
guest kernel: 'rusty' branch in dor's git  (commit
3fdd01dfef3ab175f7aaf499dd6759205028692c)
tool: iperf

ozaki-r

>
>  > Many thanks,
>  > ozaki-r
>  >
>  > 2008/3/23, Avi Kivity <avi@qumranet.com>:
>  > > Ryota OZAKI wrote:
>  > >  > Hi all,
>  > >  >
>  > >  > Current kvm allows only one VM to use HPET. Is
>  > >  > there a plan to implement a functionality to
>  > >  > allow multiple VMs to use HPET? If so, how
>  > >  > about the status of that?
>  > >  >
>  > >  >
>  > >
>  > >
>  > > If you use the dyntick clock option (the default, IIRC), and a newer
>  > >  host kernel, then the kernel provides high-resolution timers, very
>  > >  likely using HPET internally or some other high resolution clock and
>  > >  event source.
>  > >
>  > >
>  > >  > And I would like to ask right and wrong to
>  > >  > implement the functionality in terms of need
>  > >  > and efficiency (scalability and time accuracy).
>  > >
>  > >
>  > > I think that for newer kernels we already have the desired accuracy.
>  > >  We're not always good at exploiting that accuracy; hence the recent
>  > >  movement of the PIT implementation from userspace to the kernel.  But
>  > >  recent discussion leads me to believe it could have been implemented
>  > >  with the userspace PIT as well.
>  > >
>  > >
>  > >  --
>  > >  Any sufficiently difficult bug is indistinguishable from a feature.
>  > >
>  > >
>  >
>
> > -------------------------------------------------------------------------
>  > This SF.net email is sponsored by: Microsoft
>  > Defy all challenges. Microsoft(R) Visual Studio 2008.
>  > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
>  > _______________________________________________
>  > kvm-devel mailing list
>  > kvm-devel@lists.sourceforge.net
>  > https://lists.sourceforge.net/lists/listinfo/kvm-devel
>
>

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/