[Qemu-devel] [PATCH v3 0/8] Huge memory fixes

[Qemu-devel] [PATCH v3 0/8] Huge memory fixes

[Juan Quintela]

Hi

The following changes since commit 47ecbdf07ed2c37bdfd2d77137d01bb319ce13da:

  libcacard: build fixes (2012-06-21 20:04:24 +0000)

are available in the git repository at:

  http://repo.or.cz/r/qemu/quintela.git huge_memory-v2

for you to fetch changes up to 565b6c96250305553c21714f3c482d7d408eb4b2:

  Maintain the number of dirty pages (2012-06-22 15:24:14 +0200)


v2:
- Add a define MAX_DELAY (orit suggestion)
- fix handling of dirty pages number (now it is checked that is ok)
- one there, consolidate the handling of the bitmap, limiting access
  to the bitmap to only accessors.

Please review, Juan.


v1:

After a long, long time, this is v2.

This are basically the changes that we have for RHEL, due to the
problems that we have with big memory machines.  I just rebased the
patches and fixed the easy parts:

- buffered_file_limit is gone: we just use 50ms and call it a day

- I let ram_addr_t as a valid type for a counter (no, I still don't
  agree with Anthony on this, but it is not important).

- Print total time of migration always.  Notice that I also print it
  when migration is completed.  Luiz, could you take a look to see if
  I did something worng (probably).

- Moved debug printfs to tracepointns.  Thanks a lot to Stefan for
  helping with it.  Once here, I had to put the traces in the middle
  of trace-events file, if I put them on the end of the file, when I
  enable them, I got generated the previous two tracepoints, instead
  of the ones I just defined.  Stefan is looking on that.  Workaround
  is defining them anywhere else.

- exit from cpu_physical_memory_reset_dirty().  Anthony wanted that I
  created an empty stub for kvm, and maintain the code for tcg.  The
  problem is that we can have both kvm and tcg running from the same
  binary.  Intead of exiting in the middle of the function, I just
  refactored the code out.  Is there an struct where I could add a new
  function pointer for this behaviour?

- exit if we have been too long on ram_save_live() loop.  Anthony
  didn't like this, I will sent a version based on the migration
  thread in the following days.  But just need something working for
  other people to test.

  Notice that I still got "lots" of more than 50ms printf's. (Yes,
  there is a debugging printf there).

- Bitmap handling.  Still all code to count dirty pages, will try to
  get something saner based on bitmap optimizations.

Comments?

Later, Juan.

Juan Quintela (8):
  Add spent time for migration
  Add tracepoints for savevm section start/end
  No need to iterate if we already are over the limit
  Only TCG needs TLB handling
  Only calculate expected_time for stage 2
  Exit loop if we have been there too long
  dirty bitmap: abstract its use
  Maintain the number of dirty pages

 arch_init.c      |   42 ++++++++++++++++++++++++------------------
 cpu-all.h        |    1 +
 exec-obsolete.h  |   50 ++++++++++++++++++++++++++++++--------------------
 exec.c           |   34 ++++++++++++++++++++++------------
 hmp.c            |    2 ++
 migration.c      |   11 +++++++++++
 migration.h      |    1 +
 qapi-schema.json |   12 +++++++++---
 savevm.c         |   11 +++++++++++
 trace-events     |    5 +++++
 10 files changed, 116 insertions(+), 53 deletions(-)

-- 
1.7.10.2

[Qemu-devel] [PATCH 1/8] Add spent time for migration

[Juan Quintela]

We add time spent for migration to the output of "info migrate"
command.  'total_time' means time since the start fo migration if
migration is 'active', and total time of migration if migration is
completed.  As we are also interested in transferred ram when
migration completes, adding all ram statistics

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 hmp.c            |    2 ++
 migration.c      |   11 +++++++++++
 migration.h      |    1 +
 qapi-schema.json |   12 +++++++++---
 4 files changed, 23 insertions(+), 3 deletions(-)

diff --git a/hmp.c b/hmp.c
index b9cec1d..4c6d4ae 100644
--- a/hmp.c
+++ b/hmp.c
@@ -145,6 +145,8 @@ void hmp_info_migrate(Monitor *mon)
                        info->ram->remaining >> 10);
         monitor_printf(mon, "total ram: %" PRIu64 " kbytes\n",
                        info->ram->total >> 10);
+        monitor_printf(mon, "total time: %" PRIu64 " milliseconds\n",
+                       info->ram->total_time);
     }

     if (info->has_disk) {
diff --git a/migration.c b/migration.c
index 3f485d3..599bb6c 100644
--- a/migration.c
+++ b/migration.c
@@ -131,6 +131,8 @@ MigrationInfo *qmp_query_migrate(Error **errp)
         info->ram->transferred = ram_bytes_transferred();
         info->ram->remaining = ram_bytes_remaining();
         info->ram->total = ram_bytes_total();
+        info->ram->total_time = qemu_get_clock_ms(rt_clock)
+            - s->total_time;

         if (blk_mig_active()) {
             info->has_disk = true;
@@ -143,6 +145,13 @@ MigrationInfo *qmp_query_migrate(Error **errp)
     case MIG_STATE_COMPLETED:
         info->has_status = true;
         info->status = g_strdup("completed");
+
+        info->has_ram = true;
+        info->ram = g_malloc0(sizeof(*info->ram));
+        info->ram->transferred = ram_bytes_transferred();
+        info->ram->remaining = 0;
+        info->ram->total = ram_bytes_total();
+        info->ram->total_time = s->total_time;
         break;
     case MIG_STATE_ERROR:
         info->has_status = true;
@@ -260,6 +269,7 @@ static void migrate_fd_put_ready(void *opaque)
         } else {
             migrate_fd_completed(s);
         }
+        s->total_time = qemu_get_clock_ms(rt_clock) - s->total_time;
         if (s->state != MIG_STATE_COMPLETED) {
             if (old_vm_running) {
                 vm_start();
@@ -373,6 +383,7 @@ static MigrationState *migrate_init(int blk, int inc)

     s->bandwidth_limit = bandwidth_limit;
     s->state = MIG_STATE_SETUP;
+    s->total_time = qemu_get_clock_ms(rt_clock);

     return s;
 }
diff --git a/migration.h b/migration.h
index 2e9ca2e..165b27b 100644
--- a/migration.h
+++ b/migration.h
@@ -33,6 +33,7 @@ struct MigrationState
     void *opaque;
     int blk;
     int shared;
+    int64_t total_time;
 };

 void process_incoming_migration(QEMUFile *f);
diff --git a/qapi-schema.json b/qapi-schema.json
index 3b6e346..1ab5dbd 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -260,10 +260,15 @@
 #
 # @total: total amount of bytes involved in the migration process
 #
+# @total_time: tota0l amount of ms since migration started.  If
+#        migration has ended, it returns the total migration
+#        time. (since 1.2)
+#
 # Since: 0.14.0.
 ##
 { 'type': 'MigrationStats',
-  'data': {'transferred': 'int', 'remaining': 'int', 'total': 'int' } }
+  'data': {'transferred': 'int', 'remaining': 'int', 'total': 'int' ,
+           'total_time': 'int' } }

 ##
 # @MigrationInfo
@@ -275,8 +280,9 @@
 #          'cancelled'. If this field is not returned, no migration process
 #          has been initiated
 #
-# @ram: #optional @MigrationStats containing detailed migration status,
-#       only returned if status is 'active'
+# @ram: #optional @MigrationStats containing detailed migration
+#       status, only returned if status is 'active' or
+#       'completed'. 'comppleted' (since 1.2)
 #
 # @disk: #optional @MigrationStats containing detailed disk migration
 #        status, only returned if status is 'active' and it is a block
-- 
1.7.10.2

[Qemu-devel] [PATCH 2/8] Add tracepoints for savevm section start/end

[Juan Quintela]

This allows to know how long each section takes to save.

An awk script like this tells us sections that takes more that 10ms

$1 ~ /savevm_state_iterate_end/ {
	/* Print savevm_section_end line when > 10ms duration */
	if ($2 > 10000) {
		printf("%s times_missing=%u\n", $0, times_missing++);
	}
}

Signed-off-by: Juan Quintela <quintela@redhat.com>

fix ws tracepoints

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 savevm.c     |    8 ++++++++
 trace-events |    5 +++++
 2 files changed, 13 insertions(+)

diff --git a/savevm.c b/savevm.c
index faa8145..40320be 100644
--- a/savevm.c
+++ b/savevm.c
@@ -85,6 +85,7 @@
 #include "cpus.h"
 #include "memory.h"
 #include "qmp-commands.h"
+#include "trace.h"

 #define SELF_ANNOUNCE_ROUNDS 5

@@ -1624,11 +1625,14 @@ int qemu_savevm_state_iterate(QEMUFile *f)
         if (se->save_live_state == NULL)
             continue;

+        trace_savevm_section_start();
         /* Section type */
         qemu_put_byte(f, QEMU_VM_SECTION_PART);
         qemu_put_be32(f, se->section_id);

         ret = se->save_live_state(f, QEMU_VM_SECTION_PART, se->opaque);
+        trace_savevm_section_end(se->section_id);
+
         if (ret <= 0) {
             /* Do not proceed to the next vmstate before this one reported
                completion of the current stage. This serializes the migration
@@ -1658,11 +1662,13 @@ int qemu_savevm_state_complete(QEMUFile *f)
         if (se->save_live_state == NULL)
             continue;

+        trace_savevm_section_start();
         /* Section type */
         qemu_put_byte(f, QEMU_VM_SECTION_END);
         qemu_put_be32(f, se->section_id);

         ret = se->save_live_state(f, QEMU_VM_SECTION_END, se->opaque);
+        trace_savevm_section_end(se->section_id);
         if (ret < 0) {
             return ret;
         }
@@ -1674,6 +1680,7 @@ int qemu_savevm_state_complete(QEMUFile *f)
 	if (se->save_state == NULL && se->vmsd == NULL)
 	    continue;

+        trace_savevm_section_start();
         /* Section type */
         qemu_put_byte(f, QEMU_VM_SECTION_FULL);
         qemu_put_be32(f, se->section_id);
@@ -1687,6 +1694,7 @@ int qemu_savevm_state_complete(QEMUFile *f)
         qemu_put_be32(f, se->version_id);

         vmstate_save(f, se);
+        trace_savevm_section_end(se->section_id);
     }

     qemu_put_byte(f, QEMU_VM_EOF);
diff --git a/trace-events b/trace-events
index 5c82b3a..82c7619 100644
--- a/trace-events
+++ b/trace-events
@@ -782,6 +782,11 @@ displaysurface_resize(void *display_state, void *display_surface, int width, int
 # vga.c
 ppm_save(const char *filename, void *display_surface) "%s surface=%p"

+# savevm.c
+
+savevm_section_start(void) ""
+savevm_section_end(unsigned int section_id) "section_id %u"
+
 # hw/qxl.c
 disable qxl_interface_set_mm_time(int qid, uint32_t mm_time) "%d %d"
 disable qxl_io_write_vga(int qid, const char *mode, uint32_t addr, uint32_t val) "%d %s addr=%u val=%u"
-- 
1.7.10.2

[Qemu-devel] [PATCH 3/8] No need to iterate if we already are over the limit

[Juan Quintela]

If buffers are full, don't iterate, just exit.

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 savevm.c |    3 +++
 1 file changed, 3 insertions(+)

diff --git a/savevm.c b/savevm.c
index 40320be..9101bfb 100644
--- a/savevm.c
+++ b/savevm.c
@@ -1625,6 +1625,9 @@ int qemu_savevm_state_iterate(QEMUFile *f)
         if (se->save_live_state == NULL)
             continue;

+        if (qemu_file_rate_limit(f)) {
+            return 0;
+        }
         trace_savevm_section_start();
         /* Section type */
         qemu_put_byte(f, QEMU_VM_SECTION_PART);
-- 
1.7.10.2

[Qemu-devel] [PATCH 4/8] Only TCG needs TLB handling

[Juan Quintela]

Refactor the code that is only needed for tcg to an static function.
Call that only when tcg is enabled.  We can't refactor to a dummy
function in the kvm case, as qemu can be compiled at the same time
with tcg and kvm.

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 exec.c |   31 +++++++++++++++++++++----------
 1 file changed, 21 insertions(+), 10 deletions(-)

diff --git a/exec.c b/exec.c
index 8244d54..a68b65c 100644
--- a/exec.c
+++ b/exec.c
@@ -1824,11 +1824,29 @@ void tb_flush_jmp_cache(CPUArchState *env, target_ulong addr)
             TB_JMP_PAGE_SIZE * sizeof(TranslationBlock *));
 }

+static void tlb_reset_dirty_range_all(ram_addr_t start, ram_addr_t end,
+                                      uintptr_t length)
+{
+    uintptr_t start1;
+
+    /* we modify the TLB cache so that the dirty bit will be set again
+       when accessing the range */
+    start1 = (uintptr_t)qemu_safe_ram_ptr(start);
+    /* Check that we don't span multiple blocks - this breaks the
+       address comparisons below.  */
+    if ((uintptr_t)qemu_safe_ram_ptr(end - 1) - start1
+            != (end - 1) - start) {
+        abort();
+    }
+    cpu_tlb_reset_dirty_all(start1, length);
+
+}
+
 /* Note: start and end must be within the same ram block.  */
 void cpu_physical_memory_reset_dirty(ram_addr_t start, ram_addr_t end,
                                      int dirty_flags)
 {
-    uintptr_t length, start1;
+    uintptr_t length;

     start &= TARGET_PAGE_MASK;
     end = TARGET_PAGE_ALIGN(end);
@@ -1838,16 +1856,9 @@ void cpu_physical_memory_reset_dirty(ram_addr_t start, ram_addr_t end,
         return;
     cpu_physical_memory_mask_dirty_range(start, length, dirty_flags);

-    /* we modify the TLB cache so that the dirty bit will be set again
-       when accessing the range */
-    start1 = (uintptr_t)qemu_safe_ram_ptr(start);
-    /* Check that we don't span multiple blocks - this breaks the
-       address comparisons below.  */
-    if ((uintptr_t)qemu_safe_ram_ptr(end - 1) - start1
-            != (end - 1) - start) {
-        abort();
+    if (tcg_enabled()) {
+        tlb_reset_dirty_range_all(start, end, length);
     }
-    cpu_tlb_reset_dirty_all(start1, length);
 }

 int cpu_physical_memory_set_dirty_tracking(int enable)
-- 
1.7.10.2

[Qemu-devel] [PATCH 5/8] Only calculate expected_time for stage 2

[Juan Quintela]

ram_save_remaining() is an expensive operation when there is a lot of memory.
So we only call the function when we need it.

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 arch_init.c |   10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/arch_init.c b/arch_init.c
index a9e8b74..424efe7 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -299,7 +299,6 @@ int ram_save_live(QEMUFile *f, int stage, void *opaque)
     ram_addr_t addr;
     uint64_t bytes_transferred_last;
     double bwidth = 0;
-    uint64_t expected_time = 0;
     int ret;

     if (stage < 0) {
@@ -376,9 +375,12 @@ int ram_save_live(QEMUFile *f, int stage, void *opaque)

     qemu_put_be64(f, RAM_SAVE_FLAG_EOS);

-    expected_time = ram_save_remaining() * TARGET_PAGE_SIZE / bwidth;
-
-    return (stage == 2) && (expected_time <= migrate_max_downtime());
+    if (stage == 2) {
+        uint64_t expected_time;
+        expected_time = ram_save_remaining() * TARGET_PAGE_SIZE / bwidth;
+        return expected_time <= migrate_max_downtime();
+    }
+    return 0;
 }

 static inline void *host_from_stream_offset(QEMUFile *f,
-- 
1.7.10.2

[Qemu-devel] [PATCH 6/8] Exit loop if we have been there too long

[Juan Quintela]

Checking each 64 pages is a random magic number as good as any other.
We don't want to test too many times, but on the other hand,
qemu_get_clock_ns() is not so expensive either.  We want to be sure
that we spent less than 50ms (half of buffered_file timer), if we
spent more than 100ms, all the accounting got wrong.

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 arch_init.c |   17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/arch_init.c b/arch_init.c
index 424efe7..7de1abf 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -294,12 +294,15 @@ static void sort_ram_list(void)
     g_free(blocks);
 }

+#define MAX_WAIT 50 /* ms, half buffered_file limit */
+
 int ram_save_live(QEMUFile *f, int stage, void *opaque)
 {
     ram_addr_t addr;
     uint64_t bytes_transferred_last;
     double bwidth = 0;
     int ret;
+    int i;

     if (stage < 0) {
         memory_global_dirty_log_stop();
@@ -339,6 +342,7 @@ int ram_save_live(QEMUFile *f, int stage, void *opaque)
     bytes_transferred_last = bytes_transferred;
     bwidth = qemu_get_clock_ns(rt_clock);

+    i = 0;
     while ((ret = qemu_file_rate_limit(f)) == 0) {
         int bytes_sent;

@@ -347,6 +351,19 @@ int ram_save_live(QEMUFile *f, int stage, void *opaque)
         if (bytes_sent == 0) { /* no more blocks */
             break;
         }
+        /* we want to check in the 1st loop, just in case it was the 1st time
+           and we had to sync the dirty bitmap.
+           qemu_get_clock_ns() is a bit expensive, so we only check each some
+           iterations
+        */
+        if ((i & 63) == 0) {
+            uint64_t t1 = (qemu_get_clock_ns(rt_clock) - bwidth) / 1000000;
+            if (t1 > MAX_WAIT) {
+                DPRINTF("big wait: %ld milliseconds, %d iterations\n", t1, i);
+                break;
+            }
+        }
+        i++;
     }

     if (ret < 0) {
-- 
1.7.10.2

[Qemu-devel] [PATCH 7/8] dirty bitmap: abstract its use

[Juan Quintela]

Always use accessors to read/set the dirty bitmap.

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 exec-obsolete.h |   40 ++++++++++++++++++++--------------------
 exec.c          |    3 +--
 2 files changed, 21 insertions(+), 22 deletions(-)

diff --git a/exec-obsolete.h b/exec-obsolete.h
index 792c831..f8ffce6 100644
--- a/exec-obsolete.h
+++ b/exec-obsolete.h
@@ -45,15 +45,15 @@ int cpu_physical_memory_set_dirty_tracking(int enable);
 #define CODE_DIRTY_FLAG      0x02
 #define MIGRATION_DIRTY_FLAG 0x08

-/* read dirty bit (return 0 or 1) */
-static inline int cpu_physical_memory_is_dirty(ram_addr_t addr)
+static inline int cpu_physical_memory_get_dirty_flags(ram_addr_t addr)
 {
-    return ram_list.phys_dirty[addr >> TARGET_PAGE_BITS] == 0xff;
+    return ram_list.phys_dirty[addr >> TARGET_PAGE_BITS];
 }

-static inline int cpu_physical_memory_get_dirty_flags(ram_addr_t addr)
+/* read dirty bit (return 0 or 1) */
+static inline int cpu_physical_memory_is_dirty(ram_addr_t addr)
 {
-    return ram_list.phys_dirty[addr >> TARGET_PAGE_BITS];
+    return cpu_physical_memory_get_dirty_flags(addr) == 0xff;
 }

 static inline int cpu_physical_memory_get_dirty(ram_addr_t start,
@@ -61,41 +61,45 @@ static inline int cpu_physical_memory_get_dirty(ram_addr_t start,
                                                 int dirty_flags)
 {
     int ret = 0;
-    uint8_t *p;
     ram_addr_t addr, end;

     end = TARGET_PAGE_ALIGN(start + length);
     start &= TARGET_PAGE_MASK;
-    p = ram_list.phys_dirty + (start >> TARGET_PAGE_BITS);
     for (addr = start; addr < end; addr += TARGET_PAGE_SIZE) {
-        ret |= *p++ & dirty_flags;
+        ret |= cpu_physical_memory_get_dirty_flags(addr) & dirty_flags;
     }
     return ret;
 }

+static inline int cpu_physical_memory_set_dirty_flags(ram_addr_t addr,
+                                                      int dirty_flags)
+{
+    return ram_list.phys_dirty[addr >> TARGET_PAGE_BITS] |= dirty_flags;
+}
+
 static inline void cpu_physical_memory_set_dirty(ram_addr_t addr)
 {
-    ram_list.phys_dirty[addr >> TARGET_PAGE_BITS] = 0xff;
+    cpu_physical_memory_set_dirty_flags(addr, 0xff);
 }

-static inline int cpu_physical_memory_set_dirty_flags(ram_addr_t addr,
-                                                      int dirty_flags)
+static inline int cpu_physical_memory_clear_dirty_flags(ram_addr_t addr,
+                                                        int dirty_flags)
 {
-    return ram_list.phys_dirty[addr >> TARGET_PAGE_BITS] |= dirty_flags;
+    int mask = ~dirty_flags;
+
+    return ram_list.phys_dirty[addr >> TARGET_PAGE_BITS] &= mask;
 }

 static inline void cpu_physical_memory_set_dirty_range(ram_addr_t start,
                                                        ram_addr_t length,
                                                        int dirty_flags)
 {
-    uint8_t *p;
     ram_addr_t addr, end;

     end = TARGET_PAGE_ALIGN(start + length);
     start &= TARGET_PAGE_MASK;
-    p = ram_list.phys_dirty + (start >> TARGET_PAGE_BITS);
     for (addr = start; addr < end; addr += TARGET_PAGE_SIZE) {
-        *p++ |= dirty_flags;
+        cpu_physical_memory_set_dirty_flags(addr, dirty_flags);
     }
 }

@@ -103,16 +107,12 @@ static inline void cpu_physical_memory_mask_dirty_range(ram_addr_t start,
                                                         ram_addr_t length,
                                                         int dirty_flags)
 {
-    int mask;
-    uint8_t *p;
     ram_addr_t addr, end;

     end = TARGET_PAGE_ALIGN(start + length);
     start &= TARGET_PAGE_MASK;
-    mask = ~dirty_flags;
-    p = ram_list.phys_dirty + (start >> TARGET_PAGE_BITS);
     for (addr = start; addr < end; addr += TARGET_PAGE_SIZE) {
-        *p++ &= mask;
+        cpu_physical_memory_clear_dirty_flags(addr, dirty_flags);
     }
 }

diff --git a/exec.c b/exec.c
index a68b65c..dd4833d 100644
--- a/exec.c
+++ b/exec.c
@@ -2565,8 +2565,7 @@ ram_addr_t qemu_ram_alloc_from_ptr(ram_addr_t size, void *host,

     ram_list.phys_dirty = g_realloc(ram_list.phys_dirty,
                                        last_ram_offset() >> TARGET_PAGE_BITS);
-    memset(ram_list.phys_dirty + (new_block->offset >> TARGET_PAGE_BITS),
-           0xff, size >> TARGET_PAGE_BITS);
+    cpu_physical_memory_set_dirty_range(new_block->offset, size, 0xff);

     if (kvm_enabled())
         kvm_setup_guest_memory(new_block->host, size);
-- 
1.7.10.2

[Qemu-devel] [PATCH 8/8] Maintain the number of dirty pages

[Juan Quintela]

Calculate the number of dirty pages takes a lot on hosts with lots
of memory.  Just maintain how many pages are dirty.

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 arch_init.c     |   15 +--------------
 cpu-all.h       |    1 +
 exec-obsolete.h |   10 ++++++++++
 3 files changed, 12 insertions(+), 14 deletions(-)

diff --git a/arch_init.c b/arch_init.c
index 7de1abf..545cf8f 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -228,20 +228,7 @@ static uint64_t bytes_transferred;

 static ram_addr_t ram_save_remaining(void)
 {
-    RAMBlock *block;
-    ram_addr_t count = 0;
-
-    QLIST_FOREACH(block, &ram_list.blocks, next) {
-        ram_addr_t addr;
-        for (addr = 0; addr < block->length; addr += TARGET_PAGE_SIZE) {
-            if (memory_region_get_dirty(block->mr, addr, TARGET_PAGE_SIZE,
-                                        DIRTY_MEMORY_MIGRATION)) {
-                count++;
-            }
-        }
-    }
-
-    return count;
+    return ram_list.dirty_pages;
 }

 uint64_t ram_bytes_remaining(void)
diff --git a/cpu-all.h b/cpu-all.h
index 50c8b62..88cedba 100644
--- a/cpu-all.h
+++ b/cpu-all.h
@@ -477,6 +477,7 @@ typedef struct RAMBlock {
 typedef struct RAMList {
     uint8_t *phys_dirty;
     QLIST_HEAD(, RAMBlock) blocks;
+    uint64_t dirty_pages;
 } RAMList;
 extern RAMList ram_list;

diff --git a/exec-obsolete.h b/exec-obsolete.h
index f8ffce6..c099256 100644
--- a/exec-obsolete.h
+++ b/exec-obsolete.h
@@ -74,6 +74,11 @@ static inline int cpu_physical_memory_get_dirty(ram_addr_t start,
 static inline int cpu_physical_memory_set_dirty_flags(ram_addr_t addr,
                                                       int dirty_flags)
 {
+    if ((dirty_flags & MIGRATION_DIRTY_FLAG) &&
+        !cpu_physical_memory_get_dirty(addr, TARGET_PAGE_SIZE,
+                                       MIGRATION_DIRTY_FLAG)) {
+        ram_list.dirty_pages++;
+    }
     return ram_list.phys_dirty[addr >> TARGET_PAGE_BITS] |= dirty_flags;
 }

@@ -87,6 +92,11 @@ static inline int cpu_physical_memory_clear_dirty_flags(ram_addr_t addr,
 {
     int mask = ~dirty_flags;

+    if ((dirty_flags & MIGRATION_DIRTY_FLAG) &&
+        cpu_physical_memory_get_dirty(addr, TARGET_PAGE_SIZE,
+                                      MIGRATION_DIRTY_FLAG)) {
+        ram_list.dirty_pages--;
+    }
     return ram_list.phys_dirty[addr >> TARGET_PAGE_BITS] &= mask;
 }

-- 
1.7.10.2