Re: Improving hvm IO performance by using self IO emulator (YA io-emu?)

[Tristan Gingold <tgingold@free.fr>]

On Thu, Feb 22, 2007 at 03:23:03PM -0600, Anthony Liguori wrote:
> tgingold@free.fr wrote:
[... overhead ...]
> Yup.  With KVM, there is no scheduler involvement.  qemu does a blocking 
> ioctl to the Linux kernel, and the Linux kernel does a vmrun.  Provided 
> the time slice hasn't been exhausted, Linux returns directly to qemu 
> after a vmexit.
Ok, thank you for the details.

> Xen uses event channels which involved domain switches and 
> select()'ing.  A lot of the time, the path is pretty optimal.  However, 
> quite a bit of the time, you run into worst case scenarios with the 
> various schedulers and the latency sky rockets.
> 
> >Honestly I don't know.  Does anyone have figures ?
> >  
> 
> Yeah, it varies a lot on different hardware.  For reference:
> 
> if round trip to a null int80 syscall is 150 nsec, a round trip vmexit 
> to userspace in KVM may be 2500 nsec.  On bare metal, it may cost 1700 
> nsec to do a PIO operation to a IDE port so 2500 really isn't that bad.
> 
> Xen is usually around there too but every so often, it spikes to 
> something awful (100ks of nsecs) and that skews the average cost.
That explains the latency.

[...]
> >>The big problem with disk emulation isn't IO latency, but the fact that
> >>the IDE emulation can only have one outstanding request at a time.  The
> >>SCSI emulation helps this a lot.
> >>    
> >IIRC, a real IDE can only have one outstanding request too (this may have
> >changed with AHCI).  This is really IIRC :-(
> >  
> 
> You recall correctly.  IDE can only have one type of outstanding DMA 
> request.
So there is something I do not understand: KDM IDE accesses are almost as
fast as bare metal (2500 ns vs 1700 ns).  Is KVM IO performance awful compared
to bare metal ?  If so why ?

[...]
> Removing code from the hypervisor reduces the TCB so it's a win.  Having 
> it in firmware within the HVM domain is even better than having it in 
> dom0 too wrt the TCB.
Ok.

> >>Can you provide more details on how the reflecting works?  Have you
> >>measured the cost of reflection?  Do you just setup a page table that
> >>maps physical memory 1-1 and then reenter the guest?
> >>    
> >Yes, set disable PG, set up flat mode and reenter the guest.
> >Cost not yet measured!
> 
> That would be very useful to measure.  My chief concern would be that 
> disabling PG would be considerably more costly than entering with paging 
> enabled.  That may not be the case on VT today since there is no ASIDs 
> so it would be useful to test on SVM too.
Switching to physical mode shouldn't be slow on ia64 (Sorry I am more
familiar with Xen/ia64).  Anyways this is a detail.

> >>Does the firmware get loaded as an option ROM or is it a special portion
> >>of guest memory that isn't normally reachable?
> >>    
> >IMHO it should come with hvmload.  No needs to make it unreachable.
> >  
> 
> It would be nice to get rid of hvmloader in the long term IMHO.  Any 
> initialization should be done in the BIOS.
Again I am not very familiar with hvmloader and these are implementation
details IMHO.

Tristan.