Re: [Qemu-devel] Re: KQEMU code organization

[Fabrice Bellard <fabrice@bellard.org>]

Anthony Liguori wrote:
> Fabrice Bellard wrote:
>> Anthony Liguori wrote:
>>  
>>> [...]
>>> FWIW, the l1_phys_map table is a current hurdle in getting 
>>> performance. When we use proper accessors to access the virtio_ring, 
>>> we end up taking
>>> a significant performance hit (around 20% on iperf).  I have some simple
>>> patches that implement a page_desc cache that cache the RAM regions in a
>>> linear array.  That helps get most of it back.
>>>
>>> I'd really like to remove the l1_phys_map entirely and replace it with a
>>> sorted list of regions.  I think this would have an overall performance
>>> improvement since its much more cache friendly.  One thing keeping this
>>> from happening is the fact that the data structure is passed up to the
>>> kernel for kqemu.  Eliminating that dependency would be a very good 
>>> thing!
>>>     
>>
>> If the l1_phys_map is a performance bottleneck it means that the
>> internals of QEMU are not properly used. In QEMU/kqemu, it is not
>> accessed to do I/Os : a cache is used thru tlb_table[]. I don't see why
>> KVM cannot use a similar system.
>>   
> 
> This is for device emulation.  KVM doesn't use l1_phys_map() for things 
> like shadow page table accesses.
> 
> In the device emulation, we're currently using stl_phys() and friends.  
> This goes through a full lookup in l1_phys_map.
> 
> Looking at other devices, some use phys_ram_base + PA and stl_raw() 
> which is broken but faster.  A few places call 
> cpu_get_physical_page_desc(), then use phys_ram_base and stl_raw().  
> This is okay but it still requires at least one l1_phys_map lookup per 
> operation in the device (packet receive, io notification, etc.).  I 
> don't think that's going to help much because in our fast paths, we're 
> only doing 2 or 3 stl_phys() operations.
> 
> At least on x86, there are very few regions of RAM.  That makes it very 
> easy to cache.  A TLB style cache seems wrong to me because there are so 
> few RAM regions.  I don't see a better way to do this with the existing 
> APIs.

I see your point. st/ldx_phys() were never optimized in fact.

A first solution would be to use a cache similar to the TLBs. It has the 
advantage is being quite generic and fast. Another solution would be to 
compute a few intervals with are tested before the generic case. These 
intervals would correspond to the main RAM area and would be updated 
each time a new device region is registered.

Does your remark implies that KVM switches back to the QEMU process for 
each I/O ? If so, the l1_phys_map access time should be negligible 
compared to the SVM-VMX/kernel/user context switch !

Fabrice.