[RFC] Moving dirty bitmaps to userspace - Double buffering approach

[RFC] Moving dirty bitmaps to userspace - Double buffering approach

[Takuya Yoshikawa]

Hi, I would like to hear your comments about the following plan:

   Moving dirty bitmaps to userspace
     - Double buffering approach

especially I would be glad if I can hear some advice about how
to keep the compatibility.

Thanks in advance,
   Takuya


---
Overview:

Last time, I submitted a patch
   "make get dirty log ioctl return the first dirty page's position"
    http://www.spinics.net/lists/kvm/msg29724.html
and got some new better ideas from Avi.

As a result, I agreed to try to eliminate the bitmap allocation
done in the x86 KVM every time when we execute get dirty log by
using "double buffering" approach.


Here is my plan:

- move the dirty bitmap allocation to userspace

We allocate bitmaps in the userspace and register them by ioctl.
Once a bitmap is registered, we do not touch it from userspace
and let the kernel modify it directly until we switch to the next
bitmap. We use "double buffering" at this switch point: userspace
give the kernel a new bitmap by ioctl and the kernel switch the
bitmap atomically to new one.

After succeeded in this switch, we can read the old bitmap freely
in the userspace and free it if we want: needless to say we can
also reuse it at the next switch.


- implementation details

Although it may be possible to touch the bitmap from the kernel
side without doing kmap, I think kmapping the bitmap is better.
So we may use the following functions paying enough attention to
the preemption control.
   - get_user_pages()
   - kmap_atomic()


- compatibility issues

What I am facing now are the compatibility issues. We have to
support both the userspace and kernel side bitmap allocations
to let the current qemu and KVM work properly.

1. From the kernel side, we have to care bitmap allocations done in both
the kvm_vm_ioctl_set_memory_region() and kvm_vm_ioctl_get_dirty_log().

2. From the userspace side, we have to check the new api's availability
and determine which way we use, e.g. by using check extension ioctl.

The most problematic is 1, kernel side. We have to be able to know
by which way current bitmap allocation is being done using flags or
something. In the case of set memory region, we have to judge whether
we allocate a bitmap, and if not we have to register a bitmap later
by another api: set memory region is not restricted to the dirty log
issues and need more care than get dirty log.

Are there any good ways to solve this kind of problems?

Re: [RFC] Moving dirty bitmaps to userspace - Double buffering approach

[Avi Kivity]

On 03/08/2010 10:22 AM, Takuya Yoshikawa wrote:
> Hi, I would like to hear your comments about the following plan:
>
>   Moving dirty bitmaps to userspace
>     - Double buffering approach
>
> especially I would be glad if I can hear some advice about how
> to keep the compatibility.
>
> Thanks in advance,
>   Takuya
>
>
> ---
> Overview:
>
> Last time, I submitted a patch
>   "make get dirty log ioctl return the first dirty page's position"
>    http://www.spinics.net/lists/kvm/msg29724.html
> and got some new better ideas from Avi.
>
> As a result, I agreed to try to eliminate the bitmap allocation
> done in the x86 KVM every time when we execute get dirty log by
> using "double buffering" approach.
>
>

[...]

> Although it may be possible to touch the bitmap from the kernel
> side without doing kmap, I think kmapping the bitmap is better.
> So we may use the following functions paying enough attention to
> the preemption control.
>   - get_user_pages()
>   - kmap_atomic()

Although direct access is more difficult (you need to implement 
put_user_bit() or similar) I think it is worthwhile.  
get_user_pages_fast() is fast, but nowhere near as fast as 
put_user_bit() (or set_bit_user()), which can be just two instructions 
in the fast path.

>
> - compatibility issues
>
> What I am facing now are the compatibility issues. We have to
> support both the userspace and kernel side bitmap allocations
> to let the current qemu and KVM work properly.
>
> 1. From the kernel side, we have to care bitmap allocations done in both
> the kvm_vm_ioctl_set_memory_region() and kvm_vm_ioctl_get_dirty_log().

One way to handle this is to call do_mmap() from the kernel, so that now 
the bitmap really lives in user space.  This is a bit ugly but I think 
acceptable.  We already do this for KVM_SET_MEMORY_REGION (which was 
replaced by KVM_SET_USER_MEMORY_REGION, which moved allocation to 
userspace).

>
> 2. From the userspace side, we have to check the new api's availability
> and determine which way we use, e.g. by using check extension ioctl.
>
> The most problematic is 1, kernel side. We have to be able to know
> by which way current bitmap allocation is being done using flags or
> something. In the case of set memory region, we have to judge whether
> we allocate a bitmap, and if not we have to register a bitmap later
> by another api: set memory region is not restricted to the dirty log
> issues and need more care than get dirty log.
>
> Are there any good ways to solve this kind of problems?

I believe that do_mmap() will simplify this.

-- 
error compiling committee.c: too many arguments to function

Re: [RFC] Moving dirty bitmaps to userspace - Double buffering approach

[Marcelo Tosatti]

On Mon, Mar 08, 2010 at 05:22:43PM +0900, Takuya Yoshikawa wrote:
> Hi, I would like to hear your comments about the following plan:
> 
>   Moving dirty bitmaps to userspace
>     - Double buffering approach
> 
> especially I would be glad if I can hear some advice about how
> to keep the compatibility.
> 
> Thanks in advance,
>   Takuya
> 
> 
> ---
> Overview:
> 
> Last time, I submitted a patch
>   "make get dirty log ioctl return the first dirty page's position"
>    http://www.spinics.net/lists/kvm/msg29724.html
> and got some new better ideas from Avi.
> 
> As a result, I agreed to try to eliminate the bitmap allocation
> done in the x86 KVM every time when we execute get dirty log by
> using "double buffering" approach.
> 
> 
> Here is my plan:
> 
> - move the dirty bitmap allocation to userspace
> 
> We allocate bitmaps in the userspace and register them by ioctl.
> Once a bitmap is registered, we do not touch it from userspace
> and let the kernel modify it directly until we switch to the next
> bitmap. We use "double buffering" at this switch point: userspace
> give the kernel a new bitmap by ioctl and the kernel switch the
> bitmap atomically to new one.
>
> After succeeded in this switch, we can read the old bitmap freely
> in the userspace and free it if we want: needless to say we can
> also reuse it at the next switch.
> 
> 
> - implementation details
> 
> Although it may be possible to touch the bitmap from the kernel
> side without doing kmap, I think kmapping the bitmap is better.
> So we may use the following functions paying enough attention to
> the preemption control.
>   - get_user_pages()
>   - kmap_atomic()
> 
> 
> - compatibility issues
> 
> What I am facing now are the compatibility issues. We have to
> support both the userspace and kernel side bitmap allocations
> to let the current qemu and KVM work properly.
> 
> 1. From the kernel side, we have to care bitmap allocations done in both
> the kvm_vm_ioctl_set_memory_region() and kvm_vm_ioctl_get_dirty_log().
> 
> 2. From the userspace side, we have to check the new api's availability
> and determine which way we use, e.g. by using check extension ioctl.
> 
> The most problematic is 1, kernel side. We have to be able to know
> by which way current bitmap allocation is being done using flags or
> something. In the case of set memory region, we have to judge whether
> we allocate a bitmap, and if not we have to register a bitmap later
> by another api: set memory region is not restricted to the dirty log
> issues and need more care than get dirty log.
> 
> Are there any good ways to solve this kind of problems?

You can introduce a new get_dirty_log ioctl that passes the address
of the next bitmap in userspace, and use it (after pinning with
get_user_pages), instead of vmalloc'ing.

Re: [RFC] Moving dirty bitmaps to userspace - Double buffering approach

[Avi Kivity]

On 03/15/2010 10:33 AM, Marcelo Tosatti wrote:
>
>> Are there any good ways to solve this kind of problems?
>>      
> You can introduce a new get_dirty_log ioctl that passes the address
> of the next bitmap in userspace, and use it (after pinning with
> get_user_pages), instead of vmalloc'ing.
>
>    

No pinning please, put_user_bit() or set_bit_user().

(can be implemented generically using get_user_pages() and 
kmap_atomic(), but x86 should get an optimized implementation)

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: [RFC] Moving dirty bitmaps to userspace - Double buffering approach

[Takuya Yoshikawa]

Avi Kivity wrote:
> On 03/15/2010 10:33 AM, Marcelo Tosatti wrote:
>>
>>> Are there any good ways to solve this kind of problems?
>>>      
>> You can introduce a new get_dirty_log ioctl that passes the address
>> of the next bitmap in userspace, and use it (after pinning with
>> get_user_pages), instead of vmalloc'ing.
>>

Thank you for your advice!

>>    
> 
> No pinning please, put_user_bit() or set_bit_user().
> 
> (can be implemented generically using get_user_pages() and 
> kmap_atomic(), but x86 should get an optimized implementation)
> 

Given your advice last time, I started this with my colleague.
  -- We were just talking about how to strugle with every architectures.

As your comment, we'll make the generic implementation with optimized one
for x86 first.

Thanks
   Takuya