[PATCH-v2 0/4] KVM: dirty logging optimization

[PATCH-v2 0/4] KVM: dirty logging optimization

[Takuya Yoshikawa]

Hi,

I updated my patch set with your comments reflected.

 - [PATCH 1/4] KVM: fix race condition in kvm_mmu_slot_remove_write_access()
 - [PATCH 2/4] KVM: introduce helper to access lpage_info
 - [PATCH 3/4] KVM: count the number of dirty pages for each memslot
 - [RFC PATCH 4/4] KVM: selective write protection using dirty bitmap

The first one is a small fix and maybe worth applying seperately.

The second and third are preparation for the last one, but I believe
each has its own value.


I measured the time consumed by kvm_mmu_slot_remove_write_access[_mask]()
to decide at which conditions we choose *_mask(). Please check below!

 * I implemented dirty_level_memslot() based on these results.
   See [PATCH 4]



Is it possible to apply [PATCH 1 and 2] first?
It will help me a lot to concentrate on the main part.


  Takuya

===
Test

 - On core 2 duo machine (without EPT)
    => just because there was already test environment.
 - 1GB RAM for the guest
 - Workloads:
   x11perf for frame buffer test
   "find /" loop for live-migration test

 - Time was measured by rdtscll
   kvm mask: our method
   kvm orig: original method


a) Frame buffer
---------------------------------------------------------------

kernel: [ 5584.954209] kvm: nr_dirty_pages/npages = 8/4096
kernel: [ 5584.954240] kvm mask: time(tsc) = 9485

kernel: [ 5992.185125] kvm: nr_dirty_pages/npages = 7/4096
kernel: [ 5992.185389] kvm orig: time(tsc) = 444724

45 times faster

---------------------------------------------------------------

kernel: [ 5588.330585] kvm: nr_dirty_pages/npages = 80/4096
kernel: [ 5588.330623] kvm mask: time(tsc) = 16688

kernel: [ 5996.183982] kvm: nr_dirty_pages/npages = 80/4096
kernel: [ 5996.184275] kvm orig: time(tsc) = 495551

30 times faster

---------------------------------------------------------------

kernel: [ 5578.156641] kvm: nr_dirty_pages/npages = 195/4096
kernel: [ 5578.156690] kvm mask: time(tsc) = 27335

kernel: [ 5985.109688] kvm: nr_dirty_pages/npages = 195/4096
kernel: [ 5985.109820] kvm orig: time(tsc) = 194530

7 times faster

---------------------------------------------------------------

kernel: [ 5605.821017] kvm: nr_dirty_pages/npages = 375/4096
kernel: [ 5605.821077] kvm mask: time(tsc) = 35049

kernel: [ 6014.627767] kvm: nr_dirty_pages/npages = 375/4096
kernel: [ 6014.628190] kvm orig: time(tsc) = 702737

20 times faster

---------------------------------------------------------------

kernel: [ 5573.551459] kvm: nr_dirty_pages/npages = 576/4096
kernel: [ 5573.551532] kvm mask: time(tsc) = 46718

kernel: [ 5981.486862] kvm: nr_dirty_pages/npages = 576/4096
kernel: [ 5981.487036] kvm orig: time(tsc) = 210385

5 times faster

---------------------------------------------------------------



b) Live-migration (1GB RAM)
---------------------------------------------------------------

kernel: [ 5011.599573] kvm: nr_dirty_pages/npages = 52/261888
kernel: [ 5011.599624] kvm mask: time(tsc) = 33712

kernel: [ 4297.465401] kvm: nr_dirty_pages/npages = 50/261888
kernel: [ 4297.465918] kvm orig: time(tsc) = 894593

25 times faster

---------------------------------------------------------------

kernel: [ 5009.045842] kvm: nr_dirty_pages/npages = 120/261888
kernel: [ 5009.045940] kvm mask: time(tsc) = 50575

kernel: [ 4295.438092] kvm: nr_dirty_pages/npages = 106/261888
kernel: [ 4295.438427] kvm orig: time(tsc) = 525140

10 times faster

---------------------------------------------------------------

kernel: [ 5016.914031] kvm: nr_dirty_pages/npages = 251/261888
kernel: [ 5016.914160] kvm mask: time(tsc) = 78750

kernel: [ 4297.265937] kvm: nr_dirty_pages/npages = 260/261888
kernel: [ 4297.266411] kvm orig: time(tsc) = 782446

10 times faster

---------------------------------------------------------------

kernel: [ 5015.717535] kvm: nr_dirty_pages/npages = 526/261888
kernel: [ 5015.717697] kvm mask: time(tsc) = 130137

kernel: [ 4295.370101] kvm: nr_dirty_pages/npages = 596/261888
kernel: [ 4295.370589] kvm orig: time(tsc) = 644805

5 times faster

---------------------------------------------------------------

kernel: [ 5011.693010] kvm: nr_dirty_pages/npages = 1029/261888
kernel: [ 5011.693177] kvm mask: time(tsc) = 219863

kernel: [ 4295.760635] kvm: nr_dirty_pages/npages = 1025/261888
kernel: [ 4295.761078] kvm orig: time(tsc) = 767333

3 times faster

---------------------------------------------------------------

kernel: [ 5014.805403] kvm: nr_dirty_pages/npages = 2219/261888
kernel: [ 5014.805676] kvm mask: time(tsc) = 454314

kernel: [ 4302.186712] kvm: nr_dirty_pages/npages = 1982/261888
kernel: [ 4302.191418] kvm orig: time(tsc) = 2294187

5 times faster

---------------------------------------------------------------

kernel: [ 5015.305794] kvm: nr_dirty_pages/npages = 4143/261888
kernel: [ 5015.306291] kvm mask: time(tsc) = 847175

kernel: [ 4302.068376] kvm: nr_dirty_pages/npages = 3835/261888
kernel: [ 4302.069456] kvm orig: time(tsc) = 1855105

3 times faster

---------------------------------------------------------------

[PATCH 1/4] KVM: fix race condition in kvm_mmu_slot_remove_write_access()

[Takuya Yoshikawa]

Suggested by Avi.

Dropping PT_WRITABLE_MASK is under race condition with the hardware.
Use update_spte() to avoid this.

Signed-off-by: Takuya Yoshikawa <yoshikawa.takuya@oss.ntt.co.jp>
---
 arch/x86/kvm/mmu.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/arch/x86/kvm/mmu.c b/arch/x86/kvm/mmu.c
index bdb9fa9..e716def 100644
--- a/arch/x86/kvm/mmu.c
+++ b/arch/x86/kvm/mmu.c
@@ -3449,7 +3449,7 @@ void kvm_mmu_slot_remove_write_access(struct kvm *kvm, int slot)
 		for (i = 0; i < PT64_ENT_PER_PAGE; ++i)
 			/* avoid RMW */
 			if (is_writable_pte(pt[i]))
-				pt[i] &= ~PT_WRITABLE_MASK;
+				update_spte(&pt[i], pt[i] & ~PT_WRITABLE_MASK);
 	}
 	kvm_flush_remote_tlbs(kvm);
 }
-- 
1.7.1

[PATCH 2/4] KVM: introduce helper to access lpage_info

[Takuya Yoshikawa]

The index calculation for accessing lpage_info appears a few times.
This patch replaces these with a new lpage_idx() helper.

Signed-off-by: Takuya Yoshikawa <yoshikawa.takuya@oss.ntt.co.jp>
---
 arch/x86/kvm/mmu.c |   20 +++++++++++---------
 1 files changed, 11 insertions(+), 9 deletions(-)

diff --git a/arch/x86/kvm/mmu.c b/arch/x86/kvm/mmu.c
index e716def..2139309 100644
--- a/arch/x86/kvm/mmu.c
+++ b/arch/x86/kvm/mmu.c
@@ -476,6 +476,12 @@ static void kvm_mmu_page_set_gfn(struct kvm_mmu_page *sp, int index, gfn_t gfn)
 		sp->gfns[index] = gfn;
 }
 
+static unsigned long lpage_idx(gfn_t gfn, gfn_t base_gfn, int level)
+{
+	return (gfn >> KVM_HPAGE_GFN_SHIFT(level)) -
+	       (base_gfn >> KVM_HPAGE_GFN_SHIFT(level));
+}
+
 /*
  * Return the pointer to the largepage write count for a given
  * gfn, handling slots that are not large page aligned.
@@ -484,10 +490,8 @@ static int *slot_largepage_idx(gfn_t gfn,
 			       struct kvm_memory_slot *slot,
 			       int level)
 {
-	unsigned long idx;
+	unsigned long idx = lpage_idx(gfn, slot->base_gfn, level);
 
-	idx = (gfn >> KVM_HPAGE_GFN_SHIFT(level)) -
-	      (slot->base_gfn >> KVM_HPAGE_GFN_SHIFT(level));
 	return &slot->lpage_info[level - 2][idx].write_count;
 }
 
@@ -591,8 +595,7 @@ static unsigned long *gfn_to_rmap(struct kvm *kvm, gfn_t gfn, int level)
 	if (likely(level == PT_PAGE_TABLE_LEVEL))
 		return &slot->rmap[gfn - slot->base_gfn];
 
-	idx = (gfn >> KVM_HPAGE_GFN_SHIFT(level)) -
-		(slot->base_gfn >> KVM_HPAGE_GFN_SHIFT(level));
+	idx = lpage_idx(gfn, slot->base_gfn, level);
 
 	return &slot->lpage_info[level - 2][idx].rmap_pde;
 }
@@ -887,11 +890,10 @@ static int kvm_handle_hva(struct kvm *kvm, unsigned long hva,
 
 			for (j = 0; j < KVM_NR_PAGE_SIZES - 1; ++j) {
 				unsigned long idx;
-				int sh;
 
-				sh = KVM_HPAGE_GFN_SHIFT(PT_DIRECTORY_LEVEL+j);
-				idx = ((memslot->base_gfn+gfn_offset) >> sh) -
-					(memslot->base_gfn >> sh);
+				idx = lpage_idx(memslot->base_gfn + gfn_offset,
+						memslot->base_gfn,
+						PT_DIRECTORY_LEVEL + j);
 				ret |= handler(kvm,
 					&memslot->lpage_info[j][idx].rmap_pde,
 					data);
-- 
1.7.1

[PATCH 3/4] KVM: count the number of dirty pages for each memslot

[Takuya Yoshikawa]

This patch introduces the counter to hold the number of dirty pages in each
memslot. This eliminate the need to scan dirty bitmap for get_dirty_log.

We will also use this to optimize dirty logging later.

Signed-off-by: Takuya Yoshikawa <yoshikawa.takuya@oss.ntt.co.jp>
---
 arch/x86/kvm/x86.c       |    9 +++------
 include/linux/kvm_host.h |    1 +
 virt/kvm/kvm_main.c      |    6 +++++-
 3 files changed, 9 insertions(+), 7 deletions(-)

diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 410d2d1..e9cf381 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -3199,10 +3199,9 @@ static int kvm_vm_ioctl_reinject(struct kvm *kvm,
 int kvm_vm_ioctl_get_dirty_log(struct kvm *kvm,
 				      struct kvm_dirty_log *log)
 {
-	int r, i;
+	int r;
 	struct kvm_memory_slot *memslot;
 	unsigned long n;
-	unsigned long is_dirty = 0;
 
 	mutex_lock(&kvm->slots_lock);
 
@@ -3217,11 +3216,8 @@ int kvm_vm_ioctl_get_dirty_log(struct kvm *kvm,
 
 	n = kvm_dirty_bitmap_bytes(memslot);
 
-	for (i = 0; !is_dirty && i < n/sizeof(long); i++)
-		is_dirty = memslot->dirty_bitmap[i];
-
 	/* If nothing is dirty, don't bother messing with page tables. */
-	if (is_dirty) {
+	if (memslot->nr_dirty_pages) {
 		struct kvm_memslots *slots, *old_slots;
 		unsigned long *dirty_bitmap;
 
@@ -3236,6 +3232,7 @@ int kvm_vm_ioctl_get_dirty_log(struct kvm *kvm,
 			goto out;
 		memcpy(slots, kvm->memslots, sizeof(struct kvm_memslots));
 		slots->memslots[log->slot].dirty_bitmap = dirty_bitmap;
+		slots->memslots[log->slot].nr_dirty_pages = 0;
 		slots->generation++;
 
 		old_slots = kvm->memslots;
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 4bd663d..ce101b1 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -152,6 +152,7 @@ struct kvm_memory_slot {
 	unsigned long *rmap;
 	unsigned long *dirty_bitmap;
 	unsigned long *dirty_bitmap_head;
+	unsigned long nr_dirty_pages;
 	struct {
 		unsigned long rmap_pde;
 		int write_count;
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index fb93ff9..e1022ad 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -565,6 +565,7 @@ static int kvm_create_dirty_bitmap(struct kvm_memory_slot *memslot)
 		return -ENOMEM;
 
 	memslot->dirty_bitmap_head = memslot->dirty_bitmap;
+	memslot->nr_dirty_pages = 0;
 	return 0;
 }
 
@@ -1388,7 +1389,10 @@ void mark_page_dirty_in_slot(struct kvm *kvm, struct kvm_memory_slot *memslot,
 	if (memslot && memslot->dirty_bitmap) {
 		unsigned long rel_gfn = gfn - memslot->base_gfn;
 
-		generic___set_le_bit(rel_gfn, memslot->dirty_bitmap);
+		if (!generic_test_le_bit(rel_gfn, memslot->dirty_bitmap)) {
+			generic___set_le_bit(rel_gfn, memslot->dirty_bitmap);
+			memslot->nr_dirty_pages++;
+		}
 	}
 }
 
-- 
1.7.1

[RFC PATCH 4/4] KVM: selective write protection using dirty bitmap

[Takuya Yoshikawa]

Lai Jiangshan once tried to rewrite kvm_mmu_slot_remove_write_access() using
rmap: "kvm: rework remove-write-access for a slot"
      http://www.spinics.net/lists/kvm/msg35871.html

One problem pointed out there was that this approach might hurt cache locality
and make things slow down.

But if we restrict the story to dirty logging, we notice that only small
portion of pages are actually needed to be write protected.

So this patch uses his approach with small modification to use switched out
dirty bitmap as a hint to restrict the rmap travel.

We can also use this to selectively write protect pages to reduce unwanted page
faults in the future.

Conditions at which this hack has advantage:
 - few dirty pages compared to shadow pages
 - under reasonable, not pathological, workloads

  Note that slots for frame buffers are also affected by the total number of
  shadow pages in the original implmentation. Actually, I observed that the
  larger RAM became the more time kvm_mmu_slot_remove_write_access() took
  for frame buffer updates. This problem can be solved by our approach.

  Also, in the usual workloads, live-migration does not see so many dirty
  pages which makes this approach not effective.

Performance gain:
  During x11perf, in the condition of nr_dirty_pages/npages = 375/4096,
  our method was 20 times faster than the original.

  Live-migration also got similar improvement, and the effect for low
  workloads was more siginificant.

Signed-off-by: Takuya Yoshikawa <yoshikawa.takuya@oss.ntt.co.jp>
Cc: Lai Jiangshan <laijs@cn.fujitsu.com>
---
 arch/x86/include/asm/kvm_host.h |    2 +
 arch/x86/kvm/mmu.c              |   42 +++++++++++++++++++++++++++++++++++++++
 arch/x86/kvm/x86.c              |   37 ++++++++++++++++++++++++++++++++-
 3 files changed, 79 insertions(+), 2 deletions(-)

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index b04c0fa..bc72c0d 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -617,6 +617,8 @@ void kvm_mmu_set_mask_ptes(u64 user_mask, u64 accessed_mask,
 
 int kvm_mmu_reset_context(struct kvm_vcpu *vcpu);
 void kvm_mmu_slot_remove_write_access(struct kvm *kvm, int slot);
+void kvm_mmu_slot_remove_write_access_mask(struct kvm *kvm,
+		struct kvm_memory_slot *slot, unsigned long *dirty_bitmap);
 void kvm_mmu_zap_all(struct kvm *kvm);
 unsigned int kvm_mmu_calculate_mmu_pages(struct kvm *kvm);
 void kvm_mmu_change_mmu_pages(struct kvm *kvm, unsigned int kvm_nr_mmu_pages);
diff --git a/arch/x86/kvm/mmu.c b/arch/x86/kvm/mmu.c
index 2139309..978e806 100644
--- a/arch/x86/kvm/mmu.c
+++ b/arch/x86/kvm/mmu.c
@@ -3456,6 +3456,48 @@ void kvm_mmu_slot_remove_write_access(struct kvm *kvm, int slot)
 	kvm_flush_remote_tlbs(kvm);
 }
 
+static void remove_write_access_rmapp(struct kvm *kvm, unsigned long *rmapp)
+{
+	u64 *spte = rmap_next(kvm, rmapp, NULL);
+
+	while (spte) {
+		update_spte(spte, *spte & ~PT_WRITABLE_MASK);
+		spte = rmap_next(kvm, rmapp, spte);
+	}
+}
+
+/*
+ * Write protect the pages marked dirty in a given bitmap.
+ */
+void kvm_mmu_slot_remove_write_access_mask(struct kvm *kvm,
+					   struct kvm_memory_slot *slot,
+					   unsigned long *dirty_bitmap)
+{
+	int i;
+	gfn_t gfn_offset;
+	unsigned long idx;
+	long last_idx[KVM_NR_PAGE_SIZES - 1];
+
+	for (i = 0; i < (KVM_NR_PAGE_SIZES - 1); ++i)
+		last_idx[i] = -1;
+
+	for_each_set_bit(gfn_offset, dirty_bitmap, slot->npages) {
+		remove_write_access_rmapp(kvm, &slot->rmap[gfn_offset]);
+
+		for (i = 0; i < (KVM_NR_PAGE_SIZES - 1); ++i) {
+			idx = lpage_idx(slot->base_gfn + gfn_offset,
+					slot->base_gfn, PT_DIRECTORY_LEVEL + i);
+			if (idx == last_idx[i])
+				continue;
+
+			remove_write_access_rmapp(kvm,
+					&slot->lpage_info[i][idx].rmap_pde);
+			last_idx[i] = idx;
+		}
+	}
+	kvm_flush_remote_tlbs(kvm);
+}
+
 void kvm_mmu_zap_all(struct kvm *kvm)
 {
 	struct kvm_mmu_page *sp, *node;
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index e9cf381..222af5e 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -3193,6 +3193,32 @@ static int kvm_vm_ioctl_reinject(struct kvm *kvm,
 	return 0;
 }
 
+enum kvm_dirty_level {
+	KVM_DIRTY_LEVEL_NOTHING,
+	KVM_DIRTY_LEVEL_LIGHT,
+	KVM_DIRTY_LEVEL_HEAVY
+};
+
+/*
+ * Decide which write protection functions we should use.
+ *
+ * Key factors:
+ *  - number of dirty pages
+ *  - number of shadow pages
+ *  - direct mode or shadow mode
+ */
+static enum kvm_dirty_level dirty_level_memslot(struct kvm_memory_slot *memslot)
+{
+	if (!memslot->nr_dirty_pages)
+		return KVM_DIRTY_LEVEL_NOTHING;
+
+	if ((memslot->nr_dirty_pages < 2048) ||
+	    (memslot->nr_dirty_pages < memslot->npages / 64))
+		return KVM_DIRTY_LEVEL_LIGHT;
+
+	return KVM_DIRTY_LEVEL_HEAVY;
+}
+
 /*
  * Get (and clear) the dirty memory log for a memory slot.
  */
@@ -3202,6 +3228,7 @@ int kvm_vm_ioctl_get_dirty_log(struct kvm *kvm,
 	int r;
 	struct kvm_memory_slot *memslot;
 	unsigned long n;
+	enum kvm_dirty_level dirty_level;
 
 	mutex_lock(&kvm->slots_lock);
 
@@ -3217,7 +3244,8 @@ int kvm_vm_ioctl_get_dirty_log(struct kvm *kvm,
 	n = kvm_dirty_bitmap_bytes(memslot);
 
 	/* If nothing is dirty, don't bother messing with page tables. */
-	if (memslot->nr_dirty_pages) {
+	dirty_level = dirty_level_memslot(memslot);
+	if (dirty_level != KVM_DIRTY_LEVEL_NOTHING) {
 		struct kvm_memslots *slots, *old_slots;
 		unsigned long *dirty_bitmap;
 
@@ -3242,7 +3270,12 @@ int kvm_vm_ioctl_get_dirty_log(struct kvm *kvm,
 		kfree(old_slots);
 
 		spin_lock(&kvm->mmu_lock);
-		kvm_mmu_slot_remove_write_access(kvm, log->slot);
+		if (dirty_level == KVM_DIRTY_LEVEL_HEAVY)
+			kvm_mmu_slot_remove_write_access(kvm, log->slot);
+		else
+			kvm_mmu_slot_remove_write_access_mask(kvm,
+						&slots->memslots[log->slot],
+						dirty_bitmap);
 		spin_unlock(&kvm->mmu_lock);
 
 		r = -EFAULT;
-- 
1.7.1

Re: [PATCH-v2 0/4] KVM: dirty logging optimization

[Takuya Yoshikawa]

(2010/11/22 14:40), Takuya Yoshikawa wrote:
> Hi,
>
> I updated my patch set with your comments reflected.

Changelog:
   - used enum
   - lpage_idx issues
   - function names
   - measurements


>
>   - [PATCH 1/4] KVM: fix race condition in kvm_mmu_slot_remove_write_access()
>   - [PATCH 2/4] KVM: introduce helper to access lpage_info
>   - [PATCH 3/4] KVM: count the number of dirty pages for each memslot
>   - [RFC PATCH 4/4] KVM: selective write protection using dirty bitmap
>
> The first one is a small fix and maybe worth applying seperately.
>
> The second and third are preparation for the last one, but I believe
> each has its own value.
>
>
> I measured the time consumed by kvm_mmu_slot_remove_write_access[_mask]()
> to decide at which conditions we choose *_mask(). Please check below!
>
>   * I implemented dirty_level_memslot() based on these results.
>     See [PATCH 4]
>
>
>
> Is it possible to apply [PATCH 1 and 2] first?
> It will help me a lot to concentrate on the main part.
>
>
>    Takuya
>
> ===
> Test
>
>   - On core 2 duo machine (without EPT)
>      =>  just because there was already test environment.
>   - 1GB RAM for the guest
>   - Workloads:
>     x11perf for frame buffer test
>     "find /" loop for live-migration test
>
>   - Time was measured by rdtscll
>     kvm mask: our method
>     kvm orig: original method
>
>
> a) Frame buffer
> ---------------------------------------------------------------
>
> kernel: [ 5584.954209] kvm: nr_dirty_pages/npages = 8/4096
> kernel: [ 5584.954240] kvm mask: time(tsc) = 9485
>
> kernel: [ 5992.185125] kvm: nr_dirty_pages/npages = 7/4096
> kernel: [ 5992.185389] kvm orig: time(tsc) = 444724
>
> 45 times faster
>
> ---------------------------------------------------------------
>
> kernel: [ 5588.330585] kvm: nr_dirty_pages/npages = 80/4096
> kernel: [ 5588.330623] kvm mask: time(tsc) = 16688
>
> kernel: [ 5996.183982] kvm: nr_dirty_pages/npages = 80/4096
> kernel: [ 5996.184275] kvm orig: time(tsc) = 495551
>
> 30 times faster
>
> ---------------------------------------------------------------
>
> kernel: [ 5578.156641] kvm: nr_dirty_pages/npages = 195/4096
> kernel: [ 5578.156690] kvm mask: time(tsc) = 27335
>
> kernel: [ 5985.109688] kvm: nr_dirty_pages/npages = 195/4096
> kernel: [ 5985.109820] kvm orig: time(tsc) = 194530
>
> 7 times faster
>
> ---------------------------------------------------------------
>
> kernel: [ 5605.821017] kvm: nr_dirty_pages/npages = 375/4096
> kernel: [ 5605.821077] kvm mask: time(tsc) = 35049
>
> kernel: [ 6014.627767] kvm: nr_dirty_pages/npages = 375/4096
> kernel: [ 6014.628190] kvm orig: time(tsc) = 702737
>
> 20 times faster
>
> ---------------------------------------------------------------
>
> kernel: [ 5573.551459] kvm: nr_dirty_pages/npages = 576/4096
> kernel: [ 5573.551532] kvm mask: time(tsc) = 46718
>
> kernel: [ 5981.486862] kvm: nr_dirty_pages/npages = 576/4096
> kernel: [ 5981.487036] kvm orig: time(tsc) = 210385
>
> 5 times faster
>
> ---------------------------------------------------------------
>
>
>
> b) Live-migration (1GB RAM)
> ---------------------------------------------------------------
>
> kernel: [ 5011.599573] kvm: nr_dirty_pages/npages = 52/261888
> kernel: [ 5011.599624] kvm mask: time(tsc) = 33712
>
> kernel: [ 4297.465401] kvm: nr_dirty_pages/npages = 50/261888
> kernel: [ 4297.465918] kvm orig: time(tsc) = 894593
>
> 25 times faster
>
> ---------------------------------------------------------------
>
> kernel: [ 5009.045842] kvm: nr_dirty_pages/npages = 120/261888
> kernel: [ 5009.045940] kvm mask: time(tsc) = 50575
>
> kernel: [ 4295.438092] kvm: nr_dirty_pages/npages = 106/261888
> kernel: [ 4295.438427] kvm orig: time(tsc) = 525140
>
> 10 times faster
>
> ---------------------------------------------------------------
>
> kernel: [ 5016.914031] kvm: nr_dirty_pages/npages = 251/261888
> kernel: [ 5016.914160] kvm mask: time(tsc) = 78750
>
> kernel: [ 4297.265937] kvm: nr_dirty_pages/npages = 260/261888
> kernel: [ 4297.266411] kvm orig: time(tsc) = 782446
>
> 10 times faster
>
> ---------------------------------------------------------------
>
> kernel: [ 5015.717535] kvm: nr_dirty_pages/npages = 526/261888
> kernel: [ 5015.717697] kvm mask: time(tsc) = 130137
>
> kernel: [ 4295.370101] kvm: nr_dirty_pages/npages = 596/261888
> kernel: [ 4295.370589] kvm orig: time(tsc) = 644805
>
> 5 times faster
>
> ---------------------------------------------------------------
>
> kernel: [ 5011.693010] kvm: nr_dirty_pages/npages = 1029/261888
> kernel: [ 5011.693177] kvm mask: time(tsc) = 219863
>
> kernel: [ 4295.760635] kvm: nr_dirty_pages/npages = 1025/261888
> kernel: [ 4295.761078] kvm orig: time(tsc) = 767333
>
> 3 times faster
>
> ---------------------------------------------------------------
>
> kernel: [ 5014.805403] kvm: nr_dirty_pages/npages = 2219/261888
> kernel: [ 5014.805676] kvm mask: time(tsc) = 454314
>
> kernel: [ 4302.186712] kvm: nr_dirty_pages/npages = 1982/261888
> kernel: [ 4302.191418] kvm orig: time(tsc) = 2294187
>
> 5 times faster
>
> ---------------------------------------------------------------
>
> kernel: [ 5015.305794] kvm: nr_dirty_pages/npages = 4143/261888
> kernel: [ 5015.306291] kvm mask: time(tsc) = 847175
>
> kernel: [ 4302.068376] kvm: nr_dirty_pages/npages = 3835/261888
> kernel: [ 4302.069456] kvm orig: time(tsc) = 1855105
>
> 3 times faster
>
> ---------------------------------------------------------------
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH-v2 0/4] KVM: dirty logging optimization

[Takuya Yoshikawa]

(2010/11/22 14:40), Takuya Yoshikawa wrote:
> ===

I'm going to do additional measurements this weekend.
  - Especially with NPT.

If you have any request about which cases you want to see,
please let me know!


Thanks,
   Takuya


> Test
>
>   - On core 2 duo machine (without EPT)
>      =>  just because there was already test environment.
>   - 1GB RAM for the guest
>   - Workloads:
>     x11perf for frame buffer test
>     "find /" loop for live-migration test
>
>   - Time was measured by rdtscll
>     kvm mask: our method
>     kvm orig: original method
>
>
> a) Frame buffer
> ---------------------------------------------------------------
>
> kernel: [ 5584.954209] kvm: nr_dirty_pages/npages = 8/4096
> kernel: [ 5584.954240] kvm mask: time(tsc) = 9485
>
> kernel: [ 5992.185125] kvm: nr_dirty_pages/npages = 7/4096
> kernel: [ 5992.185389] kvm orig: time(tsc) = 444724
>
> 45 times faster
>
> ---------------------------------------------------------------
>
> kernel: [ 5588.330585] kvm: nr_dirty_pages/npages = 80/4096
> kernel: [ 5588.330623] kvm mask: time(tsc) = 16688
>
> kernel: [ 5996.183982] kvm: nr_dirty_pages/npages = 80/4096
> kernel: [ 5996.184275] kvm orig: time(tsc) = 495551
>
> 30 times faster
>
> ---------------------------------------------------------------
>
> kernel: [ 5578.156641] kvm: nr_dirty_pages/npages = 195/4096
> kernel: [ 5578.156690] kvm mask: time(tsc) = 27335
>
> kernel: [ 5985.109688] kvm: nr_dirty_pages/npages = 195/4096
> kernel: [ 5985.109820] kvm orig: time(tsc) = 194530
>
> 7 times faster
>
> ---------------------------------------------------------------
>
> kernel: [ 5605.821017] kvm: nr_dirty_pages/npages = 375/4096
> kernel: [ 5605.821077] kvm mask: time(tsc) = 35049
>
> kernel: [ 6014.627767] kvm: nr_dirty_pages/npages = 375/4096
> kernel: [ 6014.628190] kvm orig: time(tsc) = 702737
>
> 20 times faster
>
> ---------------------------------------------------------------
>
> kernel: [ 5573.551459] kvm: nr_dirty_pages/npages = 576/4096
> kernel: [ 5573.551532] kvm mask: time(tsc) = 46718
>
> kernel: [ 5981.486862] kvm: nr_dirty_pages/npages = 576/4096
> kernel: [ 5981.487036] kvm orig: time(tsc) = 210385
>
> 5 times faster
>
> ---------------------------------------------------------------
>
>
>
> b) Live-migration (1GB RAM)
> ---------------------------------------------------------------
>
> kernel: [ 5011.599573] kvm: nr_dirty_pages/npages = 52/261888
> kernel: [ 5011.599624] kvm mask: time(tsc) = 33712
>
> kernel: [ 4297.465401] kvm: nr_dirty_pages/npages = 50/261888
> kernel: [ 4297.465918] kvm orig: time(tsc) = 894593
>
> 25 times faster
>
> ---------------------------------------------------------------
>
> kernel: [ 5009.045842] kvm: nr_dirty_pages/npages = 120/261888
> kernel: [ 5009.045940] kvm mask: time(tsc) = 50575
>
> kernel: [ 4295.438092] kvm: nr_dirty_pages/npages = 106/261888
> kernel: [ 4295.438427] kvm orig: time(tsc) = 525140
>
> 10 times faster
>
> ---------------------------------------------------------------
>
> kernel: [ 5016.914031] kvm: nr_dirty_pages/npages = 251/261888
> kernel: [ 5016.914160] kvm mask: time(tsc) = 78750
>
> kernel: [ 4297.265937] kvm: nr_dirty_pages/npages = 260/261888
> kernel: [ 4297.266411] kvm orig: time(tsc) = 782446
>
> 10 times faster
>
> ---------------------------------------------------------------
>
> kernel: [ 5015.717535] kvm: nr_dirty_pages/npages = 526/261888
> kernel: [ 5015.717697] kvm mask: time(tsc) = 130137
>
> kernel: [ 4295.370101] kvm: nr_dirty_pages/npages = 596/261888
> kernel: [ 4295.370589] kvm orig: time(tsc) = 644805
>
> 5 times faster
>
> ---------------------------------------------------------------
>
> kernel: [ 5011.693010] kvm: nr_dirty_pages/npages = 1029/261888
> kernel: [ 5011.693177] kvm mask: time(tsc) = 219863
>
> kernel: [ 4295.760635] kvm: nr_dirty_pages/npages = 1025/261888
> kernel: [ 4295.761078] kvm orig: time(tsc) = 767333
>
> 3 times faster
>
> ---------------------------------------------------------------
>
> kernel: [ 5014.805403] kvm: nr_dirty_pages/npages = 2219/261888
> kernel: [ 5014.805676] kvm mask: time(tsc) = 454314
>
> kernel: [ 4302.186712] kvm: nr_dirty_pages/npages = 1982/261888
> kernel: [ 4302.191418] kvm orig: time(tsc) = 2294187
>
> 5 times faster
>
> ---------------------------------------------------------------
>
> kernel: [ 5015.305794] kvm: nr_dirty_pages/npages = 4143/261888
> kernel: [ 5015.306291] kvm mask: time(tsc) = 847175
>
> kernel: [ 4302.068376] kvm: nr_dirty_pages/npages = 3835/261888
> kernel: [ 4302.069456] kvm orig: time(tsc) = 1855105
>
> 3 times faster
>
> ---------------------------------------------------------------
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html