* Re: [RFC][PATCH 0/12] KVM, x86, ppc, asm-generic: moving dirty bitmaps
2010-05-04 12:56 [RFC][PATCH 0/12] KVM, x86, ppc, asm-generic: moving dirty bitmaps Takuya Yoshikawa
@ 2010-05-10 12:06 ` Avi Kivity
2010-05-10 12:26 ` Takuya Yoshikawa
` (5 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Avi Kivity @ 2010-05-10 12:06 UTC (permalink / raw)
To: kvm-ia64
On 05/04/2010 03:56 PM, Takuya Yoshikawa wrote:
> [Performance test]
>
> We measured the tsc needed to the ioctl()s for getting dirty logs in
> kernel.
>
> Test environment
>
> AMD Phenom(tm) 9850 Quad-Core Processor with 8GB memory
>
>
> 1. GUI test (running Ubuntu guest in graphical mode)
>
> sudo qemu-system-x86_64 -hda dirtylog_test.img -boot c -m 4192 -net ...
>
> We show a relatively stable part to compare how much time is needed
> for the basic parts of dirty log ioctl.
>
> get.org get.opt switch.opt
>
> slots[7].len2768 278379 66398 64024
> slots[8].len2768 181246 270 160
> slots[7].len2768 263961 64673 64494
> slots[8].len2768 181655 265 160
> slots[7].len2768 263736 64701 64610
> slots[8].len2768 182785 267 160
> slots[7].len2768 260925 65360 65042
> slots[8].len2768 182579 264 160
> slots[7].len2768 267823 65915 65682
> slots[8].len2768 186350 271 160
>
> At a glance, we know our optimization improved significantly compared
> to the original get dirty log ioctl. This is true for both get.opt and
> switch.opt. This has a really big impact for the personal KVM users who
> drive KVM in GUI mode on their usual PCs.
>
> Next, we notice that switch.opt improved a hundred nano seconds or so for
> these slots. Although this may sound a bit tiny improvement, we can feel
> this as a difference of GUI's responses like mouse reactions.
>
100 ns... this is a bit on the low side (and if you can measure it
interactively you have much better reflexes than I).
> To feel the difference, please try GUI on your PC with our patch series!
>
No doubt get.org -> get.opt is measurable, but get.opt->switch.opt is
problematic. Have you tried profiling to see where the time is spent
(well I can guess, clearing the write access from the sptes).
>
> 2. Live-migration test (4GB guest, write loop with 1GB buf)
>
> We also did a live-migration test.
>
> get.org get.opt switch.opt
>
> slots[0].lene5360 797383 261144 222181
> slots[1].len757047808 2186721 1965244 1842824
> slots[2].lenc7534208 1433562 1012723 1031213
> slots[3].len\x131072 216858 331 331
> slots[4].len\x131072 121635 225 164
> slots[5].len\x131072 120863 356 164
> slots[6].len\x16777216 121746 1133 156
> slots[7].len2768 120415 230 278
> slots[8].len2768 120368 216 149
> slots[0].lene5360 806497 194710 223582
> slots[1].len757047808 2142922 1878025 1895369
> slots[2].lenc7534208 1386512 1021309 1000345
> slots[3].len\x131072 221118 459 296
> slots[4].len\x131072 121516 272 166
> slots[5].len\x131072 122652 244 173
> slots[6].len\x16777216 123226 99185 149
> slots[7].len2768 121803 457 505
> slots[8].len2768 121586 216 155
> slots[0].lene5360 766113 211317 213179
> slots[1].len757047808 2155662 1974790 1842361
> slots[2].lenc7534208 1481411 1020004 1031352
> slots[3].len\x131072 223100 351 295
> slots[4].len\x131072 122982 436 164
> slots[5].len\x131072 122100 300 503
> slots[6].len\x16777216 123653 779 151
> slots[7].len2768 122617 284 157
> slots[8].len2768 122737 253 149
>
> For slots other than 0,1,2 we can see the similar improvement.
>
> Considering the fact that switch.opt does not depend on the bitmap length
> except for kvm_mmu_slot_remove_write_access(), this is the cause of some
> usec to msec time consumption: there might be some context switches.
>
> But note that this was done with the workload which dirtied the memory
> endlessly during the live-migration.
>
> In usual workload, the number of dirty pages varies a lot for each iteration
> and we should gain really a lot for relatively clean cases.
>
Can you post such a test, for an idle large guest?
--
error compiling committee.c: too many arguments to function
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [RFC][PATCH 0/12] KVM, x86, ppc, asm-generic: moving dirty bitmaps
2010-05-04 12:56 [RFC][PATCH 0/12] KVM, x86, ppc, asm-generic: moving dirty bitmaps Takuya Yoshikawa
2010-05-10 12:06 ` Avi Kivity
@ 2010-05-10 12:26 ` Takuya Yoshikawa
2010-05-11 10:11 ` Takuya Yoshikawa
` (4 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Takuya Yoshikawa @ 2010-05-10 12:26 UTC (permalink / raw)
To: kvm-ia64
>>
>> get.org get.opt switch.opt
>>
>> slots[7].len2768 278379 66398 64024
>> slots[8].len2768 181246 270 160
>> slots[7].len2768 263961 64673 64494
>> slots[8].len2768 181655 265 160
>> slots[7].len2768 263736 64701 64610
>> slots[8].len2768 182785 267 160
>> slots[7].len2768 260925 65360 65042
>> slots[8].len2768 182579 264 160
>> slots[7].len2768 267823 65915 65682
>> slots[8].len2768 186350 271 160
>>
>> At a glance, we know our optimization improved significantly compared
>> to the original get dirty log ioctl. This is true for both get.opt and
>> switch.opt. This has a really big impact for the personal KVM users who
>> drive KVM in GUI mode on their usual PCs.
>>
>> Next, we notice that switch.opt improved a hundred nano seconds or so for
>> these slots. Although this may sound a bit tiny improvement, we can feel
>> this as a difference of GUI's responses like mouse reactions.
>
> 100 ns... this is a bit on the low side (and if you can measure it
> interactively you have much better reflexes than I).
>
>> To feel the difference, please try GUI on your PC with our patch series!
>
> No doubt get.org -> get.opt is measurable, but get.opt->switch.opt is
> problematic. Have you tried profiling to see where the time is spent
> (well I can guess, clearing the write access from the sptes).
Sorry but no, and I agree with your guess.
Anyway, I want to do some profiling to confirm this guess.
BTW, If we only think about performance improvement of time, optimized
get(get.opt) may be enough at this stage.
But if we consider the future expansion like using user allocated bitmaps,
new API's introduced for switch.opt won't become waste, I think, because we
need a structure to get and export bitmap addresses.
>>
>> In usual workload, the number of dirty pages varies a lot for each
>> iteration
>> and we should gain really a lot for relatively clean cases.
>
> Can you post such a test, for an idle large guest?
OK, I'll do!
>
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [RFC][PATCH 0/12] KVM, x86, ppc, asm-generic: moving dirty bitmaps
2010-05-04 12:56 [RFC][PATCH 0/12] KVM, x86, ppc, asm-generic: moving dirty bitmaps Takuya Yoshikawa
2010-05-10 12:06 ` Avi Kivity
2010-05-10 12:26 ` Takuya Yoshikawa
@ 2010-05-11 10:11 ` Takuya Yoshikawa
2010-05-11 15:55 ` Alexander Graf
` (3 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Takuya Yoshikawa @ 2010-05-11 10:11 UTC (permalink / raw)
To: kvm-ia64
>>>
>>> In usual workload, the number of dirty pages varies a lot for each
>>> iteration
>>> and we should gain really a lot for relatively clean cases.
>>
>> Can you post such a test, for an idle large guest?
>
> OK, I'll do!
Result of "low workload test" (running top during migration) first,
4GB guest
picked up slots[1](len757047808) only
*****************************************
get.org get.opt switch.opt
1060875 310292 190335
1076754 301295 188600
655504 318284 196029
529769 301471 325
694796 70216 221172
651868 353073 196184
543339 312865 213236
1061938 72785 203090
689527 323901 249519
621364 323881 473
1063671 70703 192958
915903 336318 174008
1046462 332384 782
1037942 72783 190655
680122 318305 243544
688156 314935 193526
558658 265934 190550
652454 372135 196270
660140 68613 352
1101947 378642 186575
... ... ...
*****************************************
As expected we've got the difference more clearly.
In this case, switch.opt reduced 1/3 (.1 msec) compared to get.opt
for each iteration.
And when the slot is cleaner, the ratio is bigger.
>
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [RFC][PATCH 0/12] KVM, x86, ppc, asm-generic: moving dirty bitmaps
2010-05-04 12:56 [RFC][PATCH 0/12] KVM, x86, ppc, asm-generic: moving dirty bitmaps Takuya Yoshikawa
` (2 preceding siblings ...)
2010-05-11 10:11 ` Takuya Yoshikawa
@ 2010-05-11 15:55 ` Alexander Graf
2010-05-12 9:19 ` Takuya Yoshikawa
` (2 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Alexander Graf @ 2010-05-11 15:55 UTC (permalink / raw)
To: kvm-ia64
Takuya Yoshikawa wrote:
> Hi, sorry for sending from my personal account.
> The following series are all from me:
>
> From: Takuya Yoshikawa <yoshikawa.takuya@oss.ntt.co.jp>
>
> The 3rd version of "moving dirty bitmaps to user space".
>
> From this version, we add x86 and ppc and asm-generic people to CC lists.
>
>
> [To KVM people]
>
> Sorry for being late to reply your comments.
>
> Avi,
> - I've wrote an answer to your question in patch 5/12: drivers/vhost/vhost.c .
>
> - I've considered to change the set_bit_user_non_atomic to an inline function,
> but did not change because the other helpers in the uaccess.h are written as
> macros. Anyway, I hope that x86 people will give us appropriate suggestions
> about this.
>
> - I thought that documenting about making bitmaps 64-bit aligned will be
> written when we add an API to register user-allocated bitmaps. So probably
> in the next series.
>
> Avi, Alex,
> - Could you check the ia64 and ppc parts, please? I tried to keep the logical
> changes as small as possible.
>
> I personally tried to build these with cross compilers. For ia64, I could check
> build success with my patch series. But book3s, even without my patch series,
> it failed with the following errors:
>
> arch/powerpc/kvm/book3s_paired_singles.c: In function 'kvmppc_emulate_paired_single':
> arch/powerpc/kvm/book3s_paired_singles.c:1289: error: the frame size of 2288 bytes is larger than 2048 bytes
> make[1]: *** [arch/powerpc/kvm/book3s_paired_singles.o] Error 1
> make: *** [arch/powerpc/kvm] Error 2
>
This is bad. I haven't encountered that one at all so far, but I guess
my compiler version is different from yours. Sigh.
>
> About changelog: there are two main changes from the 2nd version:
> 1. I changed the treatment of clean slots (see patch 1/12).
> This was already applied today, thanks!
> 2. I changed the switch API. (see patch 11/12).
>
> To show this API's advantage, I also did a test (see the end of this mail).
>
>
> [To x86 people]
>
> Hi, Thomas, Ingo, Peter,
>
> Please review the patches 4,5/12. Because this is the first experience for
> me to send patches to x86, please tell me if this lacks anything.
>
>
> [To ppc people]
>
> Hi, Benjamin, Paul, Alex,
>
> Please see the patches 6,7/12. I first say sorry for that I've not tested these
> yet. In that sense, these may not be in the quality for precise reviews. But I
> will be happy if you would give me any comments.
>
> Alex, could you help me? Though I have a plan to get PPC box in the future,
> currently I cannot test these.
>
Could you please point me to a git tree where everything's readily
applied? That would make testing a lot easier.
Alex
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [RFC][PATCH 0/12] KVM, x86, ppc, asm-generic: moving dirty bitmaps
2010-05-04 12:56 [RFC][PATCH 0/12] KVM, x86, ppc, asm-generic: moving dirty bitmaps Takuya Yoshikawa
` (3 preceding siblings ...)
2010-05-11 15:55 ` Alexander Graf
@ 2010-05-12 9:19 ` Takuya Yoshikawa
2010-05-13 11:47 ` Avi Kivity
2010-05-17 9:06 ` Takuya Yoshikawa
6 siblings, 0 replies; 8+ messages in thread
From: Takuya Yoshikawa @ 2010-05-12 9:19 UTC (permalink / raw)
To: kvm-ia64
>> [To ppc people]
>>
>> Hi, Benjamin, Paul, Alex,
>>
>> Please see the patches 6,7/12. I first say sorry for that I've not tested these
>> yet. In that sense, these may not be in the quality for precise reviews. But I
>> will be happy if you would give me any comments.
>>
>> Alex, could you help me? Though I have a plan to get PPC box in the future,
>> currently I cannot test these.
>>
>
> Could you please point me to a git tree where everything's readily
> applied? That would make testing a lot easier.
OK, I'll prepare one. Probably on sourceforge or somewhere like Kemari.
Thanks,
Takuya
>
> Alex
>
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [RFC][PATCH 0/12] KVM, x86, ppc, asm-generic: moving dirty bitmaps
2010-05-04 12:56 [RFC][PATCH 0/12] KVM, x86, ppc, asm-generic: moving dirty bitmaps Takuya Yoshikawa
` (4 preceding siblings ...)
2010-05-12 9:19 ` Takuya Yoshikawa
@ 2010-05-13 11:47 ` Avi Kivity
2010-05-17 9:06 ` Takuya Yoshikawa
6 siblings, 0 replies; 8+ messages in thread
From: Avi Kivity @ 2010-05-13 11:47 UTC (permalink / raw)
To: kvm-ia64
On 05/10/2010 03:26 PM, Takuya Yoshikawa wrote:
>> No doubt get.org -> get.opt is measurable, but get.opt->switch.opt is
>> problematic. Have you tried profiling to see where the time is spent
>> (well I can guess, clearing the write access from the sptes).
>
>
> Sorry but no, and I agree with your guess.
> Anyway, I want to do some profiling to confirm this guess.
>
>
> BTW, If we only think about performance improvement of time, optimized
> get(get.opt) may be enough at this stage.
>
> But if we consider the future expansion like using user allocated
> bitmaps,
> new API's introduced for switch.opt won't become waste, I think,
> because we
> need a structure to get and export bitmap addresses.
User allocated bitmaps have the advantage of reducing pinned memory.
However we have plenty more pinned memory allocated in memory slots, so
by itself, user allocated bitmaps don't justify this change.
Perhaps if we optimize memory slot write protection (I have some ideas
about this) we can make the performance improvement more pronounced.
--
error compiling committee.c: too many arguments to function
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [RFC][PATCH 0/12] KVM, x86, ppc, asm-generic: moving dirty bitmaps
2010-05-04 12:56 [RFC][PATCH 0/12] KVM, x86, ppc, asm-generic: moving dirty bitmaps Takuya Yoshikawa
` (5 preceding siblings ...)
2010-05-13 11:47 ` Avi Kivity
@ 2010-05-17 9:06 ` Takuya Yoshikawa
6 siblings, 0 replies; 8+ messages in thread
From: Takuya Yoshikawa @ 2010-05-17 9:06 UTC (permalink / raw)
To: kvm-ia64
> User allocated bitmaps have the advantage of reducing pinned memory.
> However we have plenty more pinned memory allocated in memory slots, so
> by itself, user allocated bitmaps don't justify this change.
In that sense, what do you think about the question I sent last week?
== REPOST 1 = >>
>> mark_page_dirty is called with the mmu_lock spinlock held in set_spte.
>> Must find a way to move it outside of the spinlock section.
>>
>
> Oh, it's a serious problem. I have to consider it.
Avi, Marcelo,
Sorry but I have to say that mmu_lock spin_lock problem was completely out of
my mind. Although I looked through the code, it seems not easy to move the
set_bit_user to outside of spinlock section without breaking the semantics of
its protection.
So this may take some time to solve.
But personally, I want to do something for x86's "vmallc() every time" problem
even though moving dirty bitmaps to user space cannot be achieved soon.
In that sense, do you mind if we do double buffering without moving dirty bitmaps to
user space?
I know that the resource for vmalloc() is precious for x86 but even now, at the timing
of get_dirty_log, we use the same amount of memory as double buffering.
== 1 END =
>
> Perhaps if we optimize memory slot write protection (I have some ideas
> about this) we can make the performance improvement more pronounced.
>
It's really nice!
Even now we can measure the performance improvement by introducing switch ioctl
when guest is relatively idle, so the combination will be really effective!
== REPOST 2 = >>
>> Can you post such a test, for an idle large guest?
>
> OK, I'll do!
Result of "low workload test" (running top during migration) first,
4GB guest
picked up slots[1](len757047808) only
*****************************************
get.org get.opt switch.opt
1060875 310292 190335
1076754 301295 188600
655504 318284 196029
529769 301471 325
694796 70216 221172
651868 353073 196184
543339 312865 213236
1061938 72785 203090
689527 323901 249519
621364 323881 473
1063671 70703 192958
915903 336318 174008
1046462 332384 782
1037942 72783 190655
680122 318305 243544
688156 314935 193526
558658 265934 190550
652454 372135 196270
660140 68613 352
1101947 378642 186575
... ... ...
*****************************************
As expected we've got the difference more clearly.
In this case, switch.opt reduced 1/3 (.1 msec) compared to get.opt
for each iteration.
And when the slot is cleaner, the ratio is bigger.
== 2 END =
^ permalink raw reply [flat|nested] 8+ messages in thread