* [PATCH v3 0/2] migration: Store ram size value
@ 2023-02-08 14:14 Juan Quintela
2023-02-08 14:14 ` [PATCH v3 1/2] migration: Split ram_bytes_total_common() in two functions Juan Quintela
2023-02-08 14:14 ` [PATCH v3 2/2] migration: Calculate ram size once Juan Quintela
0 siblings, 2 replies; 6+ messages in thread
From: Juan Quintela @ 2023-02-08 14:14 UTC (permalink / raw)
To: qemu-devel; +Cc: Juan Quintela, Dr. David Alan Gilbert
Hi
This is just a rebase on top of latest upstream and make
scripts/checkpatch.pl happy.
Please, review.
[v2]
I just resized this patch for latest upstream, we still have the same
trouble for huge guests, we are doing lots of RCU operations that are
not needed at all. As David explained on the previous submission,
ram_mig_ram_block_resized() aborts migration when size changes.
Please review.
[v1]
Current migration code recalculates the amount of RAM each time that
is needed. This calculation requires RCU and other operations.
During migration we disable hot/unplug of memory, so we can store it.
Notice the times difference, and specially that ram_bytes_total()
don't appears anymore in the perf output.
total time: 75852 ms
downtime: 264 ms
setup: 273 ms
transferred ram: 19671939 kbytes
throughput: 2132.28 mbps
remaining ram: 0 kbytes
total ram: 1077936904 kbytes
duplicate: 265170289 pages
skipped: 0 pages
normal: 4316628 pages
normal bytes: 17266512 kbytes
dirty sync count: 4
page size: 4 kbytes
multifd bytes: 17341329 kbytes
pages-per-second: 1236658
precopy ram: 2330608 kbytes
downtime ram: 1 kbytes
37.97% live_migration qemu-system-x86_64 [.] buffer_zero_avx512
10.42% live_migration qemu-system-x86_64 [.] ram_find_and_save_block.part.0
6.67% live_migration qemu-system-x86_64 [.] add_to_iovec
3.71% live_migration qemu-system-x86_64 [.] ram_bytes_total_common
2.79% live_migration qemu-system-x86_64 [.] qemu_ram_is_migratable
2.69% live_migration qemu-system-x86_64 [.] qemu_put_byte.part.0
2.41% live_migration qemu-system-x86_64 [.] bitmap_test_and_clear_atomic
1.55% live_migration qemu-system-x86_64 [.] qemu_put_be32
1.26% live_migration qemu-system-x86_64 [.] find_next_bit
1.07% multifdsend_0 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
1.07% multifdsend_13 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
1.06% multifdsend_6 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
1.05% multifdsend_2 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
1.04% multifdsend_15 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
1.03% multifdsend_12 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
1.02% live_migration qemu-system-x86_64 [.] migrate_ignore_shared
1.01% multifdsend_7 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
1.01% multifdsend_3 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
1.01% multifdsend_10 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.98% live_migration qemu-system-x86_64 [.] ram_save_iterate
0.96% multifdsend_4 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.93% multifdsend_8 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.92% multifdsend_5 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.90% multifdsend_14 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.88% multifdsend_9 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.85% multifdsend_1 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.83% multifdsend_11 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.61% live_migration qemu-system-x86_64 [.] save_zero_page_to_file.part.0
0.48% live_migration [kernel.kallsyms] [k] copy_user_enhanced_fast_string
Migration status: completed
total time: 70033 ms
downtime: 279 ms
setup: 280 ms
transferred ram: 19692747 kbytes
throughput: 2312.82 mbps
remaining ram: 0 kbytes
total ram: 1077936904 kbytes
duplicate: 265164421 pages
skipped: 0 pages
normal: 4322415 pages
normal bytes: 17289660 kbytes
dirty sync count: 3
page size: 4 kbytes
multifd bytes: 17362190 kbytes
pages-per-second: 2523447
precopy ram: 2330555 kbytes
downtime ram: 1 kbytes
43.64% live_migration qemu-system-x86_64 [.] buffer_zero_avx512
11.32% live_migration qemu-system-x86_64 [.] ram_find_and_save_block.part.0
7.60% live_migration qemu-system-x86_64 [.] add_to_iovec
2.95% live_migration qemu-system-x86_64 [.] qemu_put_byte.part.0
2.73% live_migration qemu-system-x86_64 [.] bitmap_test_and_clear_atomic
1.76% live_migration qemu-system-x86_64 [.] qemu_put_be32
1.44% live_migration qemu-system-x86_64 [.] find_next_bit
0.84% multifdsend_1 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.84% multifdsend_7 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.81% multifdsend_15 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.80% multifdsend_4 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.80% multifdsend_3 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.79% multifdsend_12 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.79% multifdsend_14 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.79% multifdsend_11 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.78% multifdsend_13 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.78% live_migration qemu-system-x86_64 [.] ram_save_iterate
0.77% multifdsend_9 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.77% multifdsend_5 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.77% multifdsend_10 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.77% multifdsend_2 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.77% multifdsend_6 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.76% multifdsend_8 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.71% multifdsend_0 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.66% live_migration qemu-system-x86_64 [.] save_zero_page_to_file.part.0
0.62% live_migration qemu-system-x86_64 [.] qemu_ram_is_migratable
0.54% live_migration [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.51% live_migration qemu-system-x86_64 [.] qemu_put_byte
Please, review.
Thanks, Juan.
Juan Quintela (2):
migration: Split ram_bytes_total_common() in two functions
migration: Calculate ram size once
migration/ram.c | 32 +++++++++++++++++++-------------
1 file changed, 19 insertions(+), 13 deletions(-)
--
2.39.1
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH v3 1/2] migration: Split ram_bytes_total_common() in two functions
2023-02-08 14:14 [PATCH v3 0/2] migration: Store ram size value Juan Quintela
@ 2023-02-08 14:14 ` Juan Quintela
2023-02-08 14:54 ` Philippe Mathieu-Daudé
2023-02-08 14:14 ` [PATCH v3 2/2] migration: Calculate ram size once Juan Quintela
1 sibling, 1 reply; 6+ messages in thread
From: Juan Quintela @ 2023-02-08 14:14 UTC (permalink / raw)
To: qemu-devel; +Cc: Juan Quintela, Dr. David Alan Gilbert
It is just a big if in the middle of the function, and we need two
functions anways.
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
migration/ram.c | 25 ++++++++++++++-----------
1 file changed, 14 insertions(+), 11 deletions(-)
diff --git a/migration/ram.c b/migration/ram.c
index b966e148c2..24c96ea822 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -2596,28 +2596,30 @@ void acct_update_position(QEMUFile *f, size_t size, bool zero)
}
}
-static uint64_t ram_bytes_total_common(bool count_ignored)
+static uint64_t ram_bytes_total_with_ignored(void)
{
RAMBlock *block;
uint64_t total = 0;
RCU_READ_LOCK_GUARD();
- if (count_ignored) {
- RAMBLOCK_FOREACH_MIGRATABLE(block) {
- total += block->used_length;
- }
- } else {
- RAMBLOCK_FOREACH_NOT_IGNORED(block) {
- total += block->used_length;
- }
+ RAMBLOCK_FOREACH_MIGRATABLE(block) {
+ total += block->used_length;
}
return total;
}
uint64_t ram_bytes_total(void)
{
- return ram_bytes_total_common(false);
+ RAMBlock *block;
+ uint64_t total = 0;
+
+ RCU_READ_LOCK_GUARD();
+
+ RAMBLOCK_FOREACH_NOT_IGNORED(block) {
+ total += block->used_length;
+ }
+ return total;
}
static void xbzrle_load_setup(void)
@@ -3222,7 +3224,8 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
(*rsp)->pss[RAM_CHANNEL_PRECOPY].pss_channel = f;
WITH_RCU_READ_LOCK_GUARD() {
- qemu_put_be64(f, ram_bytes_total_common(true) | RAM_SAVE_FLAG_MEM_SIZE);
+ qemu_put_be64(f, ram_bytes_total_with_ignored()
+ | RAM_SAVE_FLAG_MEM_SIZE);
RAMBLOCK_FOREACH_MIGRATABLE(block) {
qemu_put_byte(f, strlen(block->idstr));
--
2.39.1
^ permalink raw reply related [flat|nested] 6+ messages in thread* Re: [PATCH v3 1/2] migration: Split ram_bytes_total_common() in two functions
2023-02-08 14:14 ` [PATCH v3 1/2] migration: Split ram_bytes_total_common() in two functions Juan Quintela
@ 2023-02-08 14:54 ` Philippe Mathieu-Daudé
2023-02-08 14:57 ` Juan Quintela
0 siblings, 1 reply; 6+ messages in thread
From: Philippe Mathieu-Daudé @ 2023-02-08 14:54 UTC (permalink / raw)
To: Juan Quintela, qemu-devel; +Cc: Dr. David Alan Gilbert
On 8/2/23 15:14, Juan Quintela wrote:
> It is just a big if in the middle of the function, and we need two
> functions anways.
Typo "anyways".
>
> Signed-off-by: Juan Quintela <quintela@redhat.com>
> ---
> migration/ram.c | 25 ++++++++++++++-----------
> 1 file changed, 14 insertions(+), 11 deletions(-)
> @@ -3222,7 +3224,8 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
> (*rsp)->pss[RAM_CHANNEL_PRECOPY].pss_channel = f;
>
> WITH_RCU_READ_LOCK_GUARD() {
> - qemu_put_be64(f, ram_bytes_total_common(true) | RAM_SAVE_FLAG_MEM_SIZE);
> + qemu_put_be64(f, ram_bytes_total_with_ignored()
> + | RAM_SAVE_FLAG_MEM_SIZE);
Per CODING_STYLE the '|' should go under ram_bytes_total_with_ignored ;)
qemu_put_be64(f, ram_bytes_total_with_ignored()
| RAM_SAVE_FLAG_MEM_SIZE);
> RAMBLOCK_FOREACH_MIGRATABLE(block) {
> qemu_put_byte(f, strlen(block->idstr));
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [PATCH v3 1/2] migration: Split ram_bytes_total_common() in two functions
2023-02-08 14:54 ` Philippe Mathieu-Daudé
@ 2023-02-08 14:57 ` Juan Quintela
0 siblings, 0 replies; 6+ messages in thread
From: Juan Quintela @ 2023-02-08 14:57 UTC (permalink / raw)
To: Philippe Mathieu-Daudé; +Cc: qemu-devel, Dr. David Alan Gilbert
Philippe Mathieu-Daudé <philmd@linaro.org> wrote:
> On 8/2/23 15:14, Juan Quintela wrote:
>> It is just a big if in the middle of the function, and we need two
>> functions anways.
>
> Typo "anyways".
>
>> Signed-off-by: Juan Quintela <quintela@redhat.com>
>> ---
>> migration/ram.c | 25 ++++++++++++++-----------
>> 1 file changed, 14 insertions(+), 11 deletions(-)
>> @@ -3222,7 +3224,8 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
>> (*rsp)->pss[RAM_CHANNEL_PRECOPY].pss_channel = f;
>> WITH_RCU_READ_LOCK_GUARD() {
>> - qemu_put_be64(f, ram_bytes_total_common(true) | RAM_SAVE_FLAG_MEM_SIZE);
>> + qemu_put_be64(f, ram_bytes_total_with_ignored()
>> + | RAM_SAVE_FLAG_MEM_SIZE);
>
> Per CODING_STYLE the '|' should go under ram_bytes_total_with_ignored ;)
I thought so. But emacs autoindent put it there. And as proud follower
of the Emacs cult, emacs is never wrong O:-)
> qemu_put_be64(f, ram_bytes_total_with_ignored()
> | RAM_SAVE_FLAG_MEM_SIZE);
>
>> RAMBLOCK_FOREACH_MIGRATABLE(block) {
>> qemu_put_byte(f, strlen(block->idstr));
I would have to see what I need to change in the indentation code to
make this correct.
BTW, nice catch.
Later, Juan.
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH v3 2/2] migration: Calculate ram size once
2023-02-08 14:14 [PATCH v3 0/2] migration: Store ram size value Juan Quintela
2023-02-08 14:14 ` [PATCH v3 1/2] migration: Split ram_bytes_total_common() in two functions Juan Quintela
@ 2023-02-08 14:14 ` Juan Quintela
2023-02-08 14:54 ` Philippe Mathieu-Daudé
1 sibling, 1 reply; 6+ messages in thread
From: Juan Quintela @ 2023-02-08 14:14 UTC (permalink / raw)
To: qemu-devel; +Cc: Juan Quintela, Dr. David Alan Gilbert
We are recalculating ram size continously, when we know that it don't
change during migration. Create a field in RAMState to track it.
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
migration/ram.c | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/migration/ram.c b/migration/ram.c
index 24c96ea822..bc424b14a7 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -330,6 +330,8 @@ struct RAMState {
PageSearchStatus pss[RAM_CHANNEL_MAX];
/* UFFD file descriptor, used in 'write-tracking' migration */
int uffdio_fd;
+ /* total ram size in bytes */
+ uint64_t ram_bytes_total;
/* Last block that we have visited searching for dirty pages */
RAMBlock *last_seen_block;
/* Last dirty target page we have sent */
@@ -2545,7 +2547,7 @@ static int ram_find_and_save_block(RAMState *rs)
bool again, found;
/* No dirty page as there is zero RAM */
- if (!ram_bytes_total()) {
+ if (!rs->ram_bytes_total) {
return pages;
}
@@ -3004,13 +3006,14 @@ static int ram_state_init(RAMState **rsp)
qemu_mutex_init(&(*rsp)->bitmap_mutex);
qemu_mutex_init(&(*rsp)->src_page_req_mutex);
QSIMPLEQ_INIT(&(*rsp)->src_page_requests);
+ (*rsp)->ram_bytes_total = ram_bytes_total();
/*
* Count the total number of pages used by ram blocks not including any
* gaps due to alignment or unplugs.
* This must match with the initial values of dirty bitmap.
*/
- (*rsp)->migration_dirty_pages = ram_bytes_total() >> TARGET_PAGE_BITS;
+ (*rsp)->migration_dirty_pages = (*rsp)->ram_bytes_total >> TARGET_PAGE_BITS;
ram_state_reset(*rsp);
return 0;
--
2.39.1
^ permalink raw reply related [flat|nested] 6+ messages in thread
end of thread, other threads:[~2023-02-08 14:58 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-02-08 14:14 [PATCH v3 0/2] migration: Store ram size value Juan Quintela
2023-02-08 14:14 ` [PATCH v3 1/2] migration: Split ram_bytes_total_common() in two functions Juan Quintela
2023-02-08 14:54 ` Philippe Mathieu-Daudé
2023-02-08 14:57 ` Juan Quintela
2023-02-08 14:14 ` [PATCH v3 2/2] migration: Calculate ram size once Juan Quintela
2023-02-08 14:54 ` Philippe Mathieu-Daudé
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).