[Qemu-devel] Do memory mappings need be rebuilt when deleting ioeventfds?

[Qemu-devel] Do memory mappings need be rebuilt when deleting ioeventfds?

[Gonglei (Arei)]

Hi,
  Recently I found that when doing migration on a VM with many Virtio NICs, 
a lot down time was consuming in vm_state_notify(). Further investigation 
shows major consumption is in function memory_region_del_eventfd(). When deletes an 
ioeventfd, in address space transactions commit, it begins with deleting 
all memory mappings in all address spaces, and add each memory mapping again. 
This is time consuming. 

  In my test, each ioeventfd deleing needs about 5ms, within which memory 
mapping rebuilding needs about 4ms. With many Nics and vmchannel in a VM 
doing migrating, there can be many ioeventfds deleting which increasing 
downtime remarkably.

  As far as I can see, memory mappings don't rely on ioeventfds, there is 
no need to destroy and rebuild them when manipulating ioeventfds. I think 
this is for decoupling consideration. Although this simplifies coding and 
design, it scarifies performance.

  So I'd want to know if you have any improve plan on it? Here's my 
assumption: in memory_region_del_eventfd() and memory_region_add_eventfd(), 
we mark that this is a pure ioeventfd operation, so in begin and region_nop 
memory listeners, we eliminate memory mapping destroy and rebuild, thus 
decrease migration down time and improve migration performance.



Best regards,
-Gonglei

Re: [Qemu-devel] Do memory mappings need be rebuilt when deleting ioeventfds?

[Paolo Bonzini]

Il 04/03/2014 11:49, Gonglei (Arei) ha scritto:
> Hi,
>   Recently I found that when doing migration on a VM with many Virtio NICs,
> a lot down time was consuming in vm_state_notify(). Further investigation
> shows major consumption is in function memory_region_del_eventfd(). When deletes an
> ioeventfd, in address space transactions commit, it begins with deleting
> all memory mappings in all address spaces, and add each memory mapping again.
> This is time consuming.
>
>   In my test, each ioeventfd deleing needs about 5ms, within which memory
> mapping rebuilding needs about 4ms. With many Nics and vmchannel in a VM
> doing migrating, there can be many ioeventfds deleting which increasing
> downtime remarkably.
>
>   As far as I can see, memory mappings don't rely on ioeventfds, there is
> no need to destroy and rebuild them when manipulating ioeventfds. I think
> this is for decoupling consideration. Although this simplifies coding and
> design, it scarifies performance.
>
>   So I'd want to know if you have any improve plan on it? Here's my
> assumption: in memory_region_del_eventfd() and memory_region_add_eventfd(),
> we mark that this is a pure ioeventfd operation, so in begin and region_nop
> memory listeners, we eliminate memory mapping destroy and rebuild, thus
> decrease migration down time and improve migration performance.

Precisely, what you suggest won't work because you can have 
memory_region_add/del_eventfd as a part of a larger transaction.  But 
overall this is a good idea.

You can split memory_region_update_pending in two parts, like 
memory_region_update_pending and memory_region_ioeventfd_change_pending. 
  Then, if memory_region_update_pending is false but 
memory_region_ioeventfd_change_pending is true, you can activate an 
optimized path.

Paolo