Single memory slot

Single memory slot

[Avi Kivity]

One way to improve the gfn_to_pfn() memslot search is to register just 
one slot.  This can only work on 64-bit, since even the smallest guests 
need 4GB of physical address space.  Apart from speeding up 
gfn_to_page(), it would also speed up mmio which must iterate over all 
slots, so a lookup cache cannot help.

This would require quite a bunch of changes:
- modify gfn_to_pfn() to fail gracefully if the page is in the slot but 
unmapped (hole handling)
- modify qemu to reserve the guest physical address space
- modify qemu memory allocation to use MAP_FIXED to allocate memory
- some hack for the vga aliases (mmap an fd multiple times?)
- some hack for the vmx-specific pages (e.g. APIC-access page)

Not sure it's worthwhile, but something to keep in mind if a simple 
cache or sort by size is insufficient due to mmio.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: Single memory slot

[Alexander Graf]

On 15.10.2009, at 09:33, Avi Kivity wrote:

> One way to improve the gfn_to_pfn() memslot search is to register  
> just one slot.  This can only work on 64-bit, since even the  
> smallest guests need 4GB of physical address space.  Apart from  
> speeding up gfn_to_page(), it would also speed up mmio which must  
> iterate over all slots, so a lookup cache cannot help.
>
> This would require quite a bunch of changes:
> - modify gfn_to_pfn() to fail gracefully if the page is in the slot  
> but unmapped (hole handling)
> - modify qemu to reserve the guest physical address space
> - modify qemu memory allocation to use MAP_FIXED to allocate memory
> - some hack for the vga aliases (mmap an fd multiple times?)
> - some hack for the vmx-specific pages (e.g. APIC-access page)
>
> Not sure it's worthwhile, but something to keep in mind if a simple  
> cache or sort by size is insufficient due to mmio.

One thing I've been wondering for quite a while is that slot loop. Why  
do we loop over all possible slots? Couldn't we just remember the max  
extry (usually 1 or 2) and not loop MAX_SLOT_AMOUNT times?

That would be a really easy patch and give instant speed improvements  
for everyone.

Alex

Re: Single memory slot

[Anthony Liguori]

Avi Kivity wrote:
> One way to improve the gfn_to_pfn() memslot search is to register just 
> one slot.  This can only work on 64-bit, since even the smallest 
> guests need 4GB of physical address space.  Apart from speeding up 
> gfn_to_page(), it would also speed up mmio which must iterate over all 
> slots, so a lookup cache cannot help.
>
> This would require quite a bunch of changes:
> - modify gfn_to_pfn() to fail gracefully if the page is in the slot 
> but unmapped (hole handling)
> - modify qemu to reserve the guest physical address space

It could potentially speed up qemu quite a lot too as we would return to 
a model where host va == fixed address + guest pa.  That makes things 
like stl_phys/ldl_phys trivial.

Regards,

Anthony Liguori

Re: Single memory slot

[Marcelo Tosatti]

On Thu, Oct 15, 2009 at 04:33:11PM +0900, Avi Kivity wrote:
> One way to improve the gfn_to_pfn() memslot search is to register just  
> one slot.  This can only work on 64-bit, since even the smallest guests  
> need 4GB of physical address space.  Apart from speeding up  
> gfn_to_page(), it would also speed up mmio which must iterate over all  
> slots, so a lookup cache cannot help.
>
> This would require quite a bunch of changes:
> - modify gfn_to_pfn() to fail gracefully if the page is in the slot but  
> unmapped (hole handling)
> - modify qemu to reserve the guest physical address space
> - modify qemu memory allocation to use MAP_FIXED to allocate memory
> - some hack for the vga aliases (mmap an fd multiple times?)
> - some hack for the vmx-specific pages (e.g. APIC-access page)
>
> Not sure it's worthwhile, but something to keep in mind if a simple  
> cache or sort by size is insufficient due to mmio.

Downside is you lose the ability to write protect a small slot only 
(could mprotect(MAP_READ) the desired area but get_log+write_protect 
must be atomic).

Also if you enable dirty log for the large slot largepages are disabled.

Re: Single memory slot

[Marcelo Tosatti]

On Thu, Oct 15, 2009 at 02:46:38PM +0200, Alexander Graf wrote:
>
> On 15.10.2009, at 09:33, Avi Kivity wrote:
>
>> One way to improve the gfn_to_pfn() memslot search is to register just 
>> one slot.  This can only work on 64-bit, since even the smallest guests 
>> need 4GB of physical address space.  Apart from speeding up 
>> gfn_to_page(), it would also speed up mmio which must iterate over all 
>> slots, so a lookup cache cannot help.
>>
>> This would require quite a bunch of changes:
>> - modify gfn_to_pfn() to fail gracefully if the page is in the slot  
>> but unmapped (hole handling)
>> - modify qemu to reserve the guest physical address space
>> - modify qemu memory allocation to use MAP_FIXED to allocate memory
>> - some hack for the vga aliases (mmap an fd multiple times?)
>> - some hack for the vmx-specific pages (e.g. APIC-access page)
>>
>> Not sure it's worthwhile, but something to keep in mind if a simple  
>> cache or sort by size is insufficient due to mmio.
>
> One thing I've been wondering for quite a while is that slot loop. Why  
> do we loop over all possible slots? Couldn't we just remember the max  
> extry (usually 1 or 2) and not loop MAX_SLOT_AMOUNT times?
>
> That would be a really easy patch and give instant speed improvements  
> for everyone.

gfn_to_memslot_unaliased uses kvm->nmemslots which is the max entry.

Oh, kvm_is_visible_gfn does not. It should just use
gfn_to_memslot_unaliased.

Re: Single memory slot

[Avi Kivity]

On 10/16/2009 03:51 AM, Anthony Liguori wrote:
> Avi Kivity wrote:
>> One way to improve the gfn_to_pfn() memslot search is to register 
>> just one slot.  This can only work on 64-bit, since even the smallest 
>> guests need 4GB of physical address space.  Apart from speeding up 
>> gfn_to_page(), it would also speed up mmio which must iterate over 
>> all slots, so a lookup cache cannot help.
>>
>> This would require quite a bunch of changes:
>> - modify gfn_to_pfn() to fail gracefully if the page is in the slot 
>> but unmapped (hole handling)
>> - modify qemu to reserve the guest physical address space
>
> It could potentially speed up qemu quite a lot too as we would return 
> to a model where host va == fixed address + guest pa.  That makes 
> things like stl_phys/ldl_phys trivial.

This doesn't work on 32-bit, and you still need to perform a lookup for 
mmio.  It just shortens the loop.

Note qemu can't depend on mmio holes being unmapped (you could trap the 
SEGV, but that would be unbearably slow).

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: Single memory slot

[Avi Kivity]

On 10/16/2009 04:46 AM, Marcelo Tosatti wrote:
> On Thu, Oct 15, 2009 at 04:33:11PM +0900, Avi Kivity wrote:
>    
>> One way to improve the gfn_to_pfn() memslot search is to register just
>> one slot.  This can only work on 64-bit, since even the smallest guests
>> need 4GB of physical address space.  Apart from speeding up
>> gfn_to_page(), it would also speed up mmio which must iterate over all
>> slots, so a lookup cache cannot help.
>>
>> This would require quite a bunch of changes:
>> - modify gfn_to_pfn() to fail gracefully if the page is in the slot but
>> unmapped (hole handling)
>> - modify qemu to reserve the guest physical address space
>> - modify qemu memory allocation to use MAP_FIXED to allocate memory
>> - some hack for the vga aliases (mmap an fd multiple times?)
>> - some hack for the vmx-specific pages (e.g. APIC-access page)
>>
>> Not sure it's worthwhile, but something to keep in mind if a simple
>> cache or sort by size is insufficient due to mmio.
>>      
> Downside is you lose the ability to write protect a small slot only
> (could mprotect(MAP_READ) the desired area but get_log+write_protect
> must be atomic).
>
> Also if you enable dirty log for the large slot largepages are disabled.
>    

I guess that shoots this idea down.  We could perhaps only enable it if 
a vnc client is not connected and we don't track vga updates.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.