* [Qemu-devel] [PATCH] Revert "memory: syncronize kvm bitmap using bitmaps operations"
@ 2014-01-29 5:50 Alexey Kardashevskiy
2014-01-29 7:30 ` Paolo Bonzini
0 siblings, 1 reply; 6+ messages in thread
From: Alexey Kardashevskiy @ 2014-01-29 5:50 UTC (permalink / raw)
To: qemu-devel
Cc: Alexey Kardashevskiy, Orit Wasserman, Paolo Bonzini,
Juan Quintela
This reverts commit ae2810c4bb3b383176e8e1b33931b16c01483aab.
This reverts the optimization introduced by the original patch as
it breaks dirty page tracking on systems where
getpagesize() != TARGET_PAGE_SIZE such as POWERPC64.
The cpu_physical_memory_set_dirty_lebitmap() is called from
the kvm_physical_sync_dirty_bitmap() with the bitmap returned by
the KVM's KVM_GET_DIRTY_LOG ioctl where 1 bit is per the system
page. However QEMU's ram_list.dirty_memory maps are allocated to store
1 bit per TARGET_PAGE_SIZE which is hardcoded to 4K.
Since 64K system page size is quite popular configuration on PPC64,
the original patch breaks migration.
Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
include/exec/ram_addr.h | 54 +++++++++++++++++--------------------------------
1 file changed, 18 insertions(+), 36 deletions(-)
diff --git a/include/exec/ram_addr.h b/include/exec/ram_addr.h
index 33c8acc..c6736ed 100644
--- a/include/exec/ram_addr.h
+++ b/include/exec/ram_addr.h
@@ -83,47 +83,29 @@ static inline void cpu_physical_memory_set_dirty_lebitmap(unsigned long *bitmap,
ram_addr_t start,
ram_addr_t pages)
{
- unsigned long i, j;
+ unsigned int i, j;
unsigned long page_number, c;
hwaddr addr;
ram_addr_t ram_addr;
- unsigned long len = (pages + HOST_LONG_BITS - 1) / HOST_LONG_BITS;
+ unsigned int len = (pages + HOST_LONG_BITS - 1) / HOST_LONG_BITS;
unsigned long hpratio = getpagesize() / TARGET_PAGE_SIZE;
- unsigned long page = BIT_WORD(start >> TARGET_PAGE_BITS);
- /* start address is aligned at the start of a word? */
- if (((page * BITS_PER_LONG) << TARGET_PAGE_BITS) == start) {
- long k;
- long nr = BITS_TO_LONGS(pages);
-
- for (k = 0; k < nr; k++) {
- if (bitmap[k]) {
- unsigned long temp = leul_to_cpu(bitmap[k]);
-
- ram_list.dirty_memory[DIRTY_MEMORY_MIGRATION][page + k] |= temp;
- ram_list.dirty_memory[DIRTY_MEMORY_VGA][page + k] |= temp;
- ram_list.dirty_memory[DIRTY_MEMORY_CODE][page + k] |= temp;
- }
- }
- xen_modified_memory(start, pages);
- } else {
- /*
- * bitmap-traveling is faster than memory-traveling (for addr...)
- * especially when most of the memory is not dirty.
- */
- for (i = 0; i < len; i++) {
- if (bitmap[i] != 0) {
- c = leul_to_cpu(bitmap[i]);
- do {
- j = ffsl(c) - 1;
- c &= ~(1ul << j);
- page_number = (i * HOST_LONG_BITS + j) * hpratio;
- addr = page_number * TARGET_PAGE_SIZE;
- ram_addr = start + addr;
- cpu_physical_memory_set_dirty_range(ram_addr,
- TARGET_PAGE_SIZE * hpratio);
- } while (c != 0);
- }
+ /*
+ * bitmap-traveling is faster than memory-traveling (for addr...)
+ * especially when most of the memory is not dirty.
+ */
+ for (i = 0; i < len; i++) {
+ if (bitmap[i] != 0) {
+ c = leul_to_cpu(bitmap[i]);
+ do {
+ j = ffsl(c) - 1;
+ c &= ~(1ul << j);
+ page_number = (i * HOST_LONG_BITS + j) * hpratio;
+ addr = page_number * TARGET_PAGE_SIZE;
+ ram_addr = start + addr;
+ cpu_physical_memory_set_dirty_range(ram_addr,
+ TARGET_PAGE_SIZE * hpratio);
+ } while (c != 0);
}
}
}
--
1.8.4.rc4
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [Qemu-devel] [PATCH] Revert "memory: syncronize kvm bitmap using bitmaps operations"
2014-01-29 5:50 [Qemu-devel] [PATCH] Revert "memory: syncronize kvm bitmap using bitmaps operations" Alexey Kardashevskiy
@ 2014-01-29 7:30 ` Paolo Bonzini
2014-01-29 8:12 ` Alexey Kardashevskiy
0 siblings, 1 reply; 6+ messages in thread
From: Paolo Bonzini @ 2014-01-29 7:30 UTC (permalink / raw)
To: Alexey Kardashevskiy, qemu-devel; +Cc: Orit Wasserman, Juan Quintela
Il 29/01/2014 06:50, Alexey Kardashevskiy ha scritto:
> Since 64K system page size is quite popular configuration on PPC64,
> the original patch breaks migration.
>
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> ---
> include/exec/ram_addr.h | 54 +++++++++++++++++--------------------------------
> 1 file changed, 18 insertions(+), 36 deletions(-)
>
> diff --git a/include/exec/ram_addr.h b/include/exec/ram_addr.h
> index 33c8acc..c6736ed 100644
> --- a/include/exec/ram_addr.h
> +++ b/include/exec/ram_addr.h
> @@ -83,47 +83,29 @@ static inline void cpu_physical_memory_set_dirty_lebitmap(unsigned long *bitmap,
> ram_addr_t start,
> ram_addr_t pages)
> {
> - unsigned long i, j;
> + unsigned int i, j;
> unsigned long page_number, c;
> hwaddr addr;
> ram_addr_t ram_addr;
> - unsigned long len = (pages + HOST_LONG_BITS - 1) / HOST_LONG_BITS;
> + unsigned int len = (pages + HOST_LONG_BITS - 1) / HOST_LONG_BITS;
> unsigned long hpratio = getpagesize() / TARGET_PAGE_SIZE;
> - unsigned long page = BIT_WORD(start >> TARGET_PAGE_BITS);
>
> - /* start address is aligned at the start of a word? */
> - if (((page * BITS_PER_LONG) << TARGET_PAGE_BITS) == start) {
Why not just add " && hpratio == 1" here?
Paolo
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Qemu-devel] [PATCH] Revert "memory: syncronize kvm bitmap using bitmaps operations"
2014-01-29 7:30 ` Paolo Bonzini
@ 2014-01-29 8:12 ` Alexey Kardashevskiy
2014-01-29 10:03 ` Paolo Bonzini
2014-01-29 10:39 ` Juan Quintela
0 siblings, 2 replies; 6+ messages in thread
From: Alexey Kardashevskiy @ 2014-01-29 8:12 UTC (permalink / raw)
To: Paolo Bonzini, qemu-devel; +Cc: Orit Wasserman, Juan Quintela
On 01/29/2014 06:30 PM, Paolo Bonzini wrote:
> Il 29/01/2014 06:50, Alexey Kardashevskiy ha scritto:
>> Since 64K system page size is quite popular configuration on PPC64,
>> the original patch breaks migration.
>>
>> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
>> ---
>> include/exec/ram_addr.h | 54
>> +++++++++++++++++--------------------------------
>> 1 file changed, 18 insertions(+), 36 deletions(-)
>>
>> diff --git a/include/exec/ram_addr.h b/include/exec/ram_addr.h
>> index 33c8acc..c6736ed 100644
>> --- a/include/exec/ram_addr.h
>> +++ b/include/exec/ram_addr.h
>> @@ -83,47 +83,29 @@ static inline void
>> cpu_physical_memory_set_dirty_lebitmap(unsigned long *bitmap,
>> ram_addr_t start,
>> ram_addr_t pages)
>> {
>> - unsigned long i, j;
>> + unsigned int i, j;
>> unsigned long page_number, c;
>> hwaddr addr;
>> ram_addr_t ram_addr;
>> - unsigned long len = (pages + HOST_LONG_BITS - 1) / HOST_LONG_BITS;
>> + unsigned int len = (pages + HOST_LONG_BITS - 1) / HOST_LONG_BITS;
>> unsigned long hpratio = getpagesize() / TARGET_PAGE_SIZE;
>> - unsigned long page = BIT_WORD(start >> TARGET_PAGE_BITS);
>>
>> - /* start address is aligned at the start of a word? */
>> - if (((page * BITS_PER_LONG) << TARGET_PAGE_BITS) == start) {
>
> Why not just add " && hpratio == 1" here?
Or fix dirty map to make it 1 bit per system page size (may be the fix is
coming, who knows, but I am just not ready to do this now). Or do tricks
with bits and support hpratio!=1. I could not choose and decided to revert
it for now :)
Do we really earn a lot here?
--
Alexey
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Qemu-devel] [PATCH] Revert "memory: syncronize kvm bitmap using bitmaps operations"
2014-01-29 8:12 ` Alexey Kardashevskiy
@ 2014-01-29 10:03 ` Paolo Bonzini
2014-01-30 12:07 ` Alexey Kardashevskiy
2014-01-29 10:39 ` Juan Quintela
1 sibling, 1 reply; 6+ messages in thread
From: Paolo Bonzini @ 2014-01-29 10:03 UTC (permalink / raw)
To: Alexey Kardashevskiy, qemu-devel; +Cc: Orit Wasserman, Juan Quintela
Il 29/01/2014 09:12, Alexey Kardashevskiy ha scritto:
> On 01/29/2014 06:30 PM, Paolo Bonzini wrote:
>> Il 29/01/2014 06:50, Alexey Kardashevskiy ha scritto:
>>> Since 64K system page size is quite popular configuration on PPC64,
>>> the original patch breaks migration.
>>>
>>> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
>>> ---
>>> include/exec/ram_addr.h | 54
>>> +++++++++++++++++--------------------------------
>>> 1 file changed, 18 insertions(+), 36 deletions(-)
>>>
>>> diff --git a/include/exec/ram_addr.h b/include/exec/ram_addr.h
>>> index 33c8acc..c6736ed 100644
>>> --- a/include/exec/ram_addr.h
>>> +++ b/include/exec/ram_addr.h
>>> @@ -83,47 +83,29 @@ static inline void
>>> cpu_physical_memory_set_dirty_lebitmap(unsigned long *bitmap,
>>> ram_addr_t start,
>>> ram_addr_t pages)
>>> {
>>> - unsigned long i, j;
>>> + unsigned int i, j;
>>> unsigned long page_number, c;
>>> hwaddr addr;
>>> ram_addr_t ram_addr;
>>> - unsigned long len = (pages + HOST_LONG_BITS - 1) / HOST_LONG_BITS;
>>> + unsigned int len = (pages + HOST_LONG_BITS - 1) / HOST_LONG_BITS;
>>> unsigned long hpratio = getpagesize() / TARGET_PAGE_SIZE;
>>> - unsigned long page = BIT_WORD(start >> TARGET_PAGE_BITS);
>>>
>>> - /* start address is aligned at the start of a word? */
>>> - if (((page * BITS_PER_LONG) << TARGET_PAGE_BITS) == start) {
>>
>> Why not just add " && hpratio == 1" here?
>
> Or fix dirty map to make it 1 bit per system page size (may be the fix is
> coming, who knows, but I am just not ready to do this now). Or do tricks
> with bits and support hpratio!=1. I could not choose and decided to revert
> it for now :)
Can you post the patch that adds " && hpratio == 1"?
> Do we really earn a lot here?
Yes, because this is the only part of migration that runs with the
iothread lock taken. Without Juan's patches you can see large guests
hiccups that last a few seconds.
Paolo
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Qemu-devel] [PATCH] Revert "memory: syncronize kvm bitmap using bitmaps operations"
2014-01-29 10:03 ` Paolo Bonzini
@ 2014-01-30 12:07 ` Alexey Kardashevskiy
0 siblings, 0 replies; 6+ messages in thread
From: Alexey Kardashevskiy @ 2014-01-30 12:07 UTC (permalink / raw)
To: Paolo Bonzini, qemu-devel; +Cc: Orit Wasserman, Juan Quintela
On 01/29/2014 09:03 PM, Paolo Bonzini wrote:
> Il 29/01/2014 09:12, Alexey Kardashevskiy ha scritto:
>> On 01/29/2014 06:30 PM, Paolo Bonzini wrote:
>>> Il 29/01/2014 06:50, Alexey Kardashevskiy ha scritto:
>>>> Since 64K system page size is quite popular configuration on PPC64,
>>>> the original patch breaks migration.
>>>>
>>>> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
>>>> ---
>>>> include/exec/ram_addr.h | 54
>>>> +++++++++++++++++--------------------------------
>>>> 1 file changed, 18 insertions(+), 36 deletions(-)
>>>>
>>>> diff --git a/include/exec/ram_addr.h b/include/exec/ram_addr.h
>>>> index 33c8acc..c6736ed 100644
>>>> --- a/include/exec/ram_addr.h
>>>> +++ b/include/exec/ram_addr.h
>>>> @@ -83,47 +83,29 @@ static inline void
>>>> cpu_physical_memory_set_dirty_lebitmap(unsigned long *bitmap,
>>>> ram_addr_t
>>>> start,
>>>> ram_addr_t
>>>> pages)
>>>> {
>>>> - unsigned long i, j;
>>>> + unsigned int i, j;
>>>> unsigned long page_number, c;
>>>> hwaddr addr;
>>>> ram_addr_t ram_addr;
>>>> - unsigned long len = (pages + HOST_LONG_BITS - 1) / HOST_LONG_BITS;
>>>> + unsigned int len = (pages + HOST_LONG_BITS - 1) / HOST_LONG_BITS;
>>>> unsigned long hpratio = getpagesize() / TARGET_PAGE_SIZE;
>>>> - unsigned long page = BIT_WORD(start >> TARGET_PAGE_BITS);
>>>>
>>>> - /* start address is aligned at the start of a word? */
>>>> - if (((page * BITS_PER_LONG) << TARGET_PAGE_BITS) == start) {
>>>
>>> Why not just add " && hpratio == 1" here?
>>
>> Or fix dirty map to make it 1 bit per system page size (may be the fix is
>> coming, who knows, but I am just not ready to do this now). Or do tricks
>> with bits and support hpratio!=1. I could not choose and decided to revert
>> it for now :)
>
> Can you post the patch that adds " && hpratio == 1"?
Ok, done.
But I still wonder why there are still TARGET_PAGE_SIZE vs. getpagesize()
and why bitmap formats are different between KVM and QEMU - is it just an
old stuff which has to be fixes some day or something more?
>> Do we really earn a lot here?
>
> Yes, because this is the only part of migration that runs with the iothread
> lock taken. Without Juan's patches you can see large guests hiccups that
> last a few seconds.
--
Alexey
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Qemu-devel] [PATCH] Revert "memory: syncronize kvm bitmap using bitmaps operations"
2014-01-29 8:12 ` Alexey Kardashevskiy
2014-01-29 10:03 ` Paolo Bonzini
@ 2014-01-29 10:39 ` Juan Quintela
1 sibling, 0 replies; 6+ messages in thread
From: Juan Quintela @ 2014-01-29 10:39 UTC (permalink / raw)
To: Alexey Kardashevskiy; +Cc: Paolo Bonzini, qemu-devel, Orit Wasserman
Alexey Kardashevskiy <aik@ozlabs.ru> wrote:
> On 01/29/2014 06:30 PM, Paolo Bonzini wrote:
>> Il 29/01/2014 06:50, Alexey Kardashevskiy ha scritto:
>>> Since 64K system page size is quite popular configuration on PPC64,
>>> the original patch breaks migration.
>>>
>>> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
>>> ---
>>> include/exec/ram_addr.h | 54
>>> +++++++++++++++++--------------------------------
>>> 1 file changed, 18 insertions(+), 36 deletions(-)
>>>
>>> diff --git a/include/exec/ram_addr.h b/include/exec/ram_addr.h
>>> index 33c8acc..c6736ed 100644
>>> --- a/include/exec/ram_addr.h
>>> +++ b/include/exec/ram_addr.h
>>> @@ -83,47 +83,29 @@ static inline void
>>> cpu_physical_memory_set_dirty_lebitmap(unsigned long *bitmap,
>>> ram_addr_t start,
>>> ram_addr_t pages)
>>> {
>>> - unsigned long i, j;
>>> + unsigned int i, j;
>>> unsigned long page_number, c;
>>> hwaddr addr;
>>> ram_addr_t ram_addr;
>>> - unsigned long len = (pages + HOST_LONG_BITS - 1) / HOST_LONG_BITS;
>>> + unsigned int len = (pages + HOST_LONG_BITS - 1) / HOST_LONG_BITS;
>>> unsigned long hpratio = getpagesize() / TARGET_PAGE_SIZE;
>>> - unsigned long page = BIT_WORD(start >> TARGET_PAGE_BITS);
>>>
>>> - /* start address is aligned at the start of a word? */
>>> - if (((page * BITS_PER_LONG) << TARGET_PAGE_BITS) == start) {
>>
>> Why not just add " && hpratio == 1" here?
>
>
> Or fix dirty map to make it 1 bit per system page size (may be the fix is
> coming, who knows, but I am just not ready to do this now). Or do tricks
> with bits and support hpratio!=1. I could not choose and decided to revert
> it for now :)
Please, could you test Paolo suggestion?
> Do we really earn a lot here?
With 512GB guests, we used to have stalls of 10 seconds, now they are
lost on the noise.
Thanks for the testing.
Later, Juan.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2014-01-30 12:07 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-01-29 5:50 [Qemu-devel] [PATCH] Revert "memory: syncronize kvm bitmap using bitmaps operations" Alexey Kardashevskiy
2014-01-29 7:30 ` Paolo Bonzini
2014-01-29 8:12 ` Alexey Kardashevskiy
2014-01-29 10:03 ` Paolo Bonzini
2014-01-30 12:07 ` Alexey Kardashevskiy
2014-01-29 10:39 ` Juan Quintela
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).