* [nVMX Test] Build kernel (defconfig) on L2, w/ VMCS Shadowing enabled on L0
@ 2013-05-13 4:58 Kashyap Chamarthy
2013-05-13 6:10 ` Ren, Yongjie
0 siblings, 1 reply; 18+ messages in thread
From: Kashyap Chamarthy @ 2013-05-13 4:58 UTC (permalink / raw)
To: kvm@vger.kernel.org
Heya,
A simple test below:
Setup Info:
-------------
L0:
- 4 pCPU, 8G pMEM.
- Version:
$ uname -r; rpm -q qemu-kvm libvirt-daemon-kvm libguestfs
3.10.0-0.rc0.git26.1.fc20.x86_64
qemu-kvm-1.4.1-1.fc19.x86_64
libvirt-daemon-kvm-1.0.5-2.fc19.x86_64
libguestfs-1.21.35-1.fc19.x86_64
- KVM parameters:
$ cat /sys/module/kvm_intel/parameters/nested
Y
$ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
Y
$ cat /sys/module/kvm_intel/parameters/enable_apicv
N
$ cat /sys/module/kvm_intel/parameters/ept
Y
L1:
- 4 vCPU, 6G vMEM.
- Version:
$ uname -r; rpm -q qemu-kvm libvirt-daemon-kvm libguestfs
3.10.0-0.rc0.git26.1.fc20.x86_64
qemu-kvm-1.4.1-1.fc19.x86_64
libvirt-daemon-kvm-1.0.5-2.fc19.x86_64
libguestfs-1.21.35-1.fc19.x86_64
L2:
- 3 vCPU, 3G vMEM.
- Version:
$ uname -r
3.9.1-301.fc19.x86_64
qemu-kvm command line for L0 and L1) here:
https://raw.github.com/kashyapc/nvmx-haswell/master/SETUP-nVMX.rst
Build Kernel w/ defconfig on L2:
----------------------------------------
$ yum install git gcc -y
$ yum-builddep kernel
$ git clone \
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
$ git describe
v3.10-rc1
$ cd linux
$ time make defconfig
[...]
real 0m30.027s
user 0m9.716s
sys 0m9.686s
$ time make
[...]
real 63m16.320s
user 32m28.038s
sys 32m21.190s
I should perhaps try a few possibilities here for MEM, CPU intensive workloads:
- Build Kernel w/ defconfig, VMCS Shadowing enabled on L0.
- Build Kernel w/ defconfig, VMCS Shadowind disabled on L0.
Some more for I/O ?
If you have further suggestions, please advise.
^ permalink raw reply [flat|nested] 18+ messages in thread* RE: [nVMX Test] Build kernel (defconfig) on L2, w/ VMCS Shadowing enabled on L0
2013-05-13 4:58 [nVMX Test] Build kernel (defconfig) on L2, w/ VMCS Shadowing enabled on L0 Kashyap Chamarthy
@ 2013-05-13 6:10 ` Ren, Yongjie
2013-05-13 6:21 ` Kashyap Chamarthy
0 siblings, 1 reply; 18+ messages in thread
From: Ren, Yongjie @ 2013-05-13 6:10 UTC (permalink / raw)
To: Kashyap Chamarthy, kvm@vger.kernel.org
> -----Original Message-----
> From: kvm-owner@vger.kernel.org [mailto:kvm-owner@vger.kernel.org]
> On Behalf Of Kashyap Chamarthy
> Sent: Monday, May 13, 2013 12:59 PM
> To: kvm@vger.kernel.org
> Subject: [nVMX Test] Build kernel (defconfig) on L2, w/ VMCS Shadowing
> enabled on L0
>
> I should perhaps try a few possibilities here for MEM, CPU intensive
> workloads:
>
> - Build Kernel w/ defconfig, VMCS Shadowing enabled on L0.
> - Build Kernel w/ defconfig, VMCS Shadowind disabled on L0.
>
> Some more for I/O ?
>
> If you have further suggestions, please advise.
>
Thanks for the detailed configurations and steps about the nVMX test.
I have some suggestions if you want to test more:
1. you can make L2 guest have 4vCPU and 4Mem
2. as L2 is SMP system, you can try 'time make -j 4' instead of 'time make'.
3. some performance comparison when enable/disable VMCS shadowing
4. some other benchmarks (e.g. UnixBench, SuperPI) ?
Best Regards,
Yongjie (Jay)
^ permalink raw reply [flat|nested] 18+ messages in thread* Re: [nVMX Test] Build kernel (defconfig) on L2, w/ VMCS Shadowing enabled on L0
2013-05-13 6:10 ` Ren, Yongjie
@ 2013-05-13 6:21 ` Kashyap Chamarthy
2013-05-13 7:07 ` Kashyap Chamarthy
0 siblings, 1 reply; 18+ messages in thread
From: Kashyap Chamarthy @ 2013-05-13 6:21 UTC (permalink / raw)
To: Ren, Yongjie; +Cc: kvm@vger.kernel.org
> Thanks for the detailed configurations and steps about the nVMX test.
> I have some suggestions if you want to test more:
> 1. you can make L2 guest have 4vCPU and 4Mem
Yeah, will try it. L0 has a max of 8G MEM.
> 2. as L2 is SMP system, you can try 'time make -j 4' instead of 'time make'.
This is my next step :) .
> 3. some performance comparison when enable/disable VMCS shadowing
Yes, as we speak, Kernel is building in L2 w/ VMCS Shadowing disabled
on L0. I'll do two runs.
> 4. some other benchmarks (e.g. UnixBench, SuperPI) ?
Yeah, I was researching some benchmark tools, and came across the
above. I've never tried them before. If they provide some meaningful
results, I'll give them a shot.
Thanks for your response.
/kashyap
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [nVMX Test] Build kernel (defconfig) on L2, w/ VMCS Shadowing enabled on L0
2013-05-13 6:21 ` Kashyap Chamarthy
@ 2013-05-13 7:07 ` Kashyap Chamarthy
2013-05-13 7:16 ` Abel Gordon
2013-05-13 7:30 ` Ren, Yongjie
0 siblings, 2 replies; 18+ messages in thread
From: Kashyap Chamarthy @ 2013-05-13 7:07 UTC (permalink / raw)
To: Ren, Yongjie; +Cc: kvm@vger.kernel.org
On Mon, May 13, 2013 at 11:51 AM, Kashyap Chamarthy
<kashyap.cv@gmail.com> wrote:
>> Thanks for the detailed configurations and steps about the nVMX test.
>> I have some suggestions if you want to test more:
>> 1. you can make L2 guest have 4vCPU and 4Mem
>
> Yeah, will try it. L0 has a max of 8G MEM.
>
>> 2. as L2 is SMP system, you can try 'time make -j 4' instead of 'time make'.
>
> This is my next step :) .
>
>> 3. some performance comparison when enable/disable VMCS shadowing
>
> Yes, as we speak, Kernel is building in L2 w/ VMCS Shadowing disabled
> on L0. I'll do two runs.
Interesting:
1/ With VMCS Shadowing *disabled* :
- On L0 (Side note: I did reboot the host, once I disabled shadow VMCS)
====
$ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
N
====
- Building Kernel on L2:
====
$ time make
.....
real 36m33.361s
user 18m19.003s
sys 17m6.208s
====
2/ With VMCS Shadowing *enabled* :
- On L0 (Side note: I did reboot the host, once I enabled shadow VMCS)
====
$ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
Y
====
- Building Kernel on L2:
====
$ time make
.....
real 62m53.853s
user 32m32.724s
sys 32m16.430s
====
Comments ?
/kashyap
^ permalink raw reply [flat|nested] 18+ messages in thread* Re: [nVMX Test] Build kernel (defconfig) on L2, w/ VMCS Shadowing enabled on L0
2013-05-13 7:07 ` Kashyap Chamarthy
@ 2013-05-13 7:16 ` Abel Gordon
2013-05-13 7:31 ` Kashyap Chamarthy
2013-05-13 7:30 ` Ren, Yongjie
1 sibling, 1 reply; 18+ messages in thread
From: Abel Gordon @ 2013-05-13 7:16 UTC (permalink / raw)
To: Kashyap Chamarthy; +Cc: kvm@vger.kernel.org, kvm-owner, Ren, Yongjie
Kashyap Chamarthy <kashyap.cv@gmail.com> wrote on 13/05/2013 10:07:02 AM:
> 1/ With VMCS Shadowing *disabled* :
>
> - On L0 (Side note: I did reboot the host, once I disabled shadow VMCS)
> ====
> $ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
> N
> ====
>
> - Building Kernel on L2:
> ====
> $ time make
> .....
> real 36m33.361s
> user 18m19.003s
> sys 17m6.208s
> ====
>
>
> 2/ With VMCS Shadowing *enabled* :
>
> - On L0 (Side note: I did reboot the host, once I enabled shadow VMCS)
> ====
> $ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
> Y
> ====
>
> - Building Kernel on L2:
> ====
> $ time make
> .....
> real 62m53.853s
> user 32m32.724s
> sys 32m16.430s
> ====
>
> Comments ?
wow... according to this info shadow-vmcs is *significantly*
slowing nested while it should be the opposite.
Did you monitor the exit rate with and without shadow-vmcs ?
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [nVMX Test] Build kernel (defconfig) on L2, w/ VMCS Shadowing enabled on L0
2013-05-13 7:16 ` Abel Gordon
@ 2013-05-13 7:31 ` Kashyap Chamarthy
2013-05-13 7:51 ` Abel Gordon
0 siblings, 1 reply; 18+ messages in thread
From: Kashyap Chamarthy @ 2013-05-13 7:31 UTC (permalink / raw)
To: Abel Gordon; +Cc: kvm@vger.kernel.org, kvm-owner, Ren, Yongjie
On Mon, May 13, 2013 at 12:46 PM, Abel Gordon <ABELG@il.ibm.com> wrote:
>
>
> Kashyap Chamarthy <kashyap.cv@gmail.com> wrote on 13/05/2013 10:07:02 AM:
>
>> 1/ With VMCS Shadowing *disabled* :
>>
>> - On L0 (Side note: I did reboot the host, once I disabled shadow VMCS)
>> ====
>> $ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
>> N
>> ====
>>
>> - Building Kernel on L2:
>> ====
>> $ time make
>> .....
>> real 36m33.361s
>> user 18m19.003s
>> sys 17m6.208s
>> ====
>>
>>
>> 2/ With VMCS Shadowing *enabled* :
>>
>> - On L0 (Side note: I did reboot the host, once I enabled shadow VMCS)
>> ====
>> $ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
>> Y
>> ====
>>
>> - Building Kernel on L2:
>> ====
>> $ time make
>> .....
>> real 62m53.853s
>> user 32m32.724s
>> sys 32m16.430s
>> ====
>>
>> Comments ?
>
> wow... according to this info shadow-vmcs is *significantly*
> slowing nested while it should be the opposite.
Indeed.
> Did you monitor the exit rate with and without shadow-vmcs ?
I'm afraid, I'm not too familiar w/ the internals of KVM, still
learning. By exit rate, I presume, VMX exit ?
Can you please suggest a way to capture these exits, so I could get
reliable information & post the notes here ? qemu/scripts/kvm_stat ?
/kashyap
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [nVMX Test] Build kernel (defconfig) on L2, w/ VMCS Shadowing enabled on L0
2013-05-13 7:31 ` Kashyap Chamarthy
@ 2013-05-13 7:51 ` Abel Gordon
2013-05-13 8:11 ` Kashyap Chamarthy
0 siblings, 1 reply; 18+ messages in thread
From: Abel Gordon @ 2013-05-13 7:51 UTC (permalink / raw)
To: Kashyap Chamarthy; +Cc: kvm@vger.kernel.org, kvm-owner, Ren, Yongjie
Kashyap Chamarthy <kashyap.cv@gmail.com> wrote on 13/05/2013 10:31:12 AM:
> I'm afraid, I'm not too familiar w/ the internals of KVM, still
> learning. By exit rate, I presume, VMX exit ?
>
> Can you please suggest a way to capture these exits, so I could get
> reliable information & post the notes here ? qemu/scripts/kvm_stat ?
Yep, kvm_stat is the simplest way to monitor the exits frequency.
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [nVMX Test] Build kernel (defconfig) on L2, w/ VMCS Shadowing enabled on L0
2013-05-13 7:51 ` Abel Gordon
@ 2013-05-13 8:11 ` Kashyap Chamarthy
2013-05-13 8:34 ` Gleb Natapov
0 siblings, 1 reply; 18+ messages in thread
From: Kashyap Chamarthy @ 2013-05-13 8:11 UTC (permalink / raw)
To: Abel Gordon; +Cc: kvm@vger.kernel.org, kvm-owner, Ren, Yongjie
[-- Attachment #1: Type: text/plain, Size: 163 bytes --]
>
> Yep, kvm_stat is the simplest way to monitor the exits frequency.
Attached is kvm_stat results from L1, captured from several seconds of
intervals.
/kashyap
[-- Attachment #2: kvm-stat-results-L1.txt --]
[-- Type: text/plain, Size: 15440 bytes --]
========================================================================================
kvm_entry 7957 2047
kvm_exit 7959 2045
kvm_page_fault 6237 1649
kvm_inj_exception 2074 552
kvm_emulate_insn 1162 311
kvm_apic 983 258
vcpu_match_mmio 972 258
kvm_mmio 971 257
kvm_cr 933 235
kvm_inj_virq 468 123
kvm_eoi 455 120
kvm_apic_accept_irq 450 120
kvm_pv_eoi 413 111
kvm_apic_ipi 74 21
kvm_cpuid 64 16
kvm_fpu 21 5
kvm_msi_set_irq 1 0
kvm_pio 1 0
kvm_exit(EXCEPTION_NMI) 1115 0
kvm_exit(EXTERNAL_INTERRUPT) 223 0
kvm_exit(CR_ACCESS) 177 0
kvm_exit(HLT) 24 0
kvm_exit(CPUID) 16 0
kvm_exit(PENDING_INTERRUPT) 16 0
kvm_exit(INVLPG) 11 0
kvm_exit(TRIPLE_FAULT) 6 0
kvm_exit(APIC_ACCESS) 6 0
kvm_exit(MWAIT_INSTRUCTION) 5 0
kvm_exit(EPT_VIOLATION) 5 0
kvm_exit(TASK_SWITCH) 5 0
kvm_exit(VMREAD) 5 0
kvm_exit(RDPMC) 5 0
kvm_exit(VMRESUME) 5 0
kvm_exit(EPT_MISCONFIG) 5 0
kvm_exit(VMOFF) 5 0
kvm_exit(DR_ACCESS) 4 0
kvm_exit(IO_INSTRUCTION) 4 0
kvm_exit(VMCLEAR) 4 0
kvm_exit(VMON) 4 0
kvm_exit(MCE_DURING_VMENTRY) 3 0
kvm_exit(INVALID_STATE) 3 0
kvm_exit(TPR_BELOW_THRESHOLD) 2 0
kvm_exit(MONITOR_INSTRUCTION) 2 0
========================================================================================
kvm statistics
kvm_exit 2310005 29825
kvm_entry 2309999 29823
kvm_page_fault 2082037 26953
kvm_inj_exception 874914 11517
kvm_emulate_insn 236606 2991
kvm_mmio 212606 2789
vcpu_match_mmio 212247 2784
kvm_apic 211898 2779
kvm_inj_virq 110141 1419
kvm_eoi 108906 1410
kvm_apic_accept_irq 108903 1410
kvm_pv_eoi 102577 1338
kvm_cr 71455 804
kvm_apic_ipi 13427 165
kvm_cpuid 9242 76
kvm_fpu 3103 23
kvm_userspace_exit 360 5
kvm_msi_set_irq 244 1
kvm_pio 210 0
kvm_exit(EXCEPTION_NMI) 1115 0
kvm_exit(EXTERNAL_INTERRUPT) 223 0
kvm_exit(CR_ACCESS) 177 0
kvm_msr 36 0
kvm_exit(HLT) 24 0
kvm_exit(CPUID) 16 0
kvm_exit(PENDING_INTERRUPT) 16 0
kvm_exit(INVLPG) 11 0
kvm_exit(TRIPLE_FAULT) 6 0
kvm_exit(APIC_ACCESS) 6 0
kvm_exit(MWAIT_INSTRUCTION) 5 0
kvm_exit(EPT_VIOLATION) 5 0
kvm_exit(TASK_SWITCH) 5 0
kvm_exit(VMREAD) 5 0
kvm_exit(RDPMC) 5 0
kvm_exit(VMRESUME) 5 0
kvm_exit(EPT_MISCONFIG) 5 0
kvm_exit(VMOFF) 5 0
kvm_exit(DR_ACCESS) 4 0
kvm_exit(IO_INSTRUCTION) 4 0
kvm_exit(VMCLEAR) 4 0
kvm_exit(VMON) 4 0
kvm_exit(MCE_DURING_VMENTRY) 3 0
========================================================================================
kvm statistics
kvm_entry 3209449 30926
kvm_exit 3209455 30925
kvm_page_fault 2893256 27941
kvm_inj_exception 1219807 11986
kvm_emulate_insn 326476 3056
kvm_mmio 295425 2855
vcpu_match_mmio 294916 2851
kvm_apic 294417 2847
kvm_inj_virq 153427 1452
kvm_eoi 151889 1442
kvm_apic_accept_irq 151886 1442
kvm_pv_eoi 142938 1374
kvm_cr 98692 821
kvm_apic_ipi 18426 178
kvm_cpuid 11926 76
kvm_fpu 4019 24
kvm_userspace_exit 510 5
kvm_msi_set_irq 326 1
kvm_pio 278 1
kvm_exit(EXCEPTION_NMI) 1115 0
kvm_exit(EXTERNAL_INTERRUPT) 223 0
kvm_exit(CR_ACCESS) 177 0
kvm_msr 36 0
kvm_exit(HLT) 24 0
kvm_exit(CPUID) 16 0
kvm_exit(PENDING_INTERRUPT) 16 0
kvm_exit(INVLPG) 11 0
========================================================================================
kvm statistics
kvm_exit 3987803 31654
kvm_entry 3987802 31653
kvm_page_fault 3594347 28600
kvm_inj_exception 1518801 12154
kvm_emulate_insn 403726 3194
kvm_mmio 367160 2975
vcpu_match_mmio 366516 2970
kvm_apic 365882 2966
kvm_inj_virq 191666 1511
kvm_eoi 189844 1494
kvm_apic_accept_irq 189842 1494
kvm_pv_eoi 179007 1421
kvm_cr 121367 839
kvm_apic_ipi 22612 182
kvm_cpuid 13978 76
kvm_fpu 4790 33
kvm_userspace_exit 645 5
kvm_msi_set_irq 412 3
kvm_pio 351 3
kvm_exit(EXCEPTION_NMI) 1115 0
kvm_exit(EXTERNAL_INTERRUPT) 223 0
kvm_exit(CR_ACCESS) 177 0
kvm_msr 36 0
kvm_exit(HLT) 24 0
kvm_exit(CPUID) 16 0
kvm_exit(PENDING_INTERRUPT) 16 0
kvm_exit(INVLPG) 11 0
kvm_exit(TRIPLE_FAULT) 6 0
kvm_exit(APIC_ACCESS) 6 0
kvm_exit(MWAIT_INSTRUCTION) 5 0
kvm_exit(EPT_VIOLATION) 5 0
kvm_exit(TASK_SWITCH) 5 0
kvm_exit(VMREAD) 5 0
kvm_exit(RDPMC) 5 0
kvm_exit(VMRESUME) 5 0
kvm_exit(EPT_MISCONFIG) 5 0
kvm_exit(VMOFF) 5 0
kvm_exit(DR_ACCESS) 4 0
kvm_exit(IO_INSTRUCTION) 4 0
kvm_exit(VMCLEAR) 4 0
kvm_exit(VMON) 4 0
kvm_exit(MCE_DURING_VMENTRY) 3 0
kvm_exit(INVALID_STATE) 3 0
kvm_exit(TPR_BELOW_THRESHOLD) 2 0
kvm_exit(MONITOR_INSTRUCTION) 2 0
========================================================================================
kvm statistics
kvm_exit 5249496 33215
kvm_entry 5249490 33215
kvm_page_fault 4734454 30198
kvm_inj_exception 2000987 12870
kvm_emulate_insn 530007 3224
kvm_mmio 481442 2926
vcpu_match_mmio 480603 2921
kvm_apic 479773 2915
kvm_inj_virq 250609 1493
kvm_apic_accept_irq 248190 1480
kvm_eoi 248191 1480
kvm_pv_eoi 233890 1373
kvm_cr 159896 928
kvm_apic_ipi 29680 173
kvm_cpuid 18530 114
kvm_fpu 6251 35
kvm_userspace_exit 840 5
kvm_msi_set_irq 516 2
kvm_pio 436 2
kvm_exit(EXCEPTION_NMI) 1115 0
kvm_exit(EXTERNAL_INTERRUPT) 223 0
kvm_exit(CR_ACCESS) 177 0
kvm_msr 36 0
kvm_exit(HLT) 24 0
kvm_exit(CPUID) 16 0
kvm_exit(PENDING_INTERRUPT) 16 0
kvm_exit(INVLPG) 11 0
kvm_exit(TRIPLE_FAULT) 6 0
kvm_exit(APIC_ACCESS) 6 0
kvm_exit(MWAIT_INSTRUCTION) 5 0
kvm_exit(EPT_VIOLATION) 5 0
kvm_exit(TASK_SWITCH) 5 0
kvm_exit(VMREAD) 5 0
kvm_exit(RDPMC) 5 0
kvm_exit(VMRESUME) 5 0
kvm_exit(EPT_MISCONFIG) 5 0
kvm_exit(VMOFF) 5 0
kvm_exit(DR_ACCESS) 4 0
kvm_exit(IO_INSTRUCTION) 4 0
kvm_exit(VMCLEAR) 4 0
kvm_exit(VMON) 4 0
kvm_exit(MCE_DURING_VMENTRY) 3 0
kvm_exit(INVALID_STATE) 3 0
kvm_exit(TPR_BELOW_THRESHOLD) 2 0
kvm_exit(MONITOR_INSTRUCTION) 2 0
========================================================================================
kvm statistics
kvm_entry 6740066 27607
kvm_exit 6740066 27605
kvm_page_fault 6079438 24813
kvm_inj_exception 2563167 10330
kvm_emulate_insn 684814 2963
kvm_mmio 618861 2625
vcpu_match_mmio 617782 2620
kvm_apic 616713 2615
kvm_inj_virq 323299 1420
kvm_eoi 320100 1409
kvm_apic_accept_irq 320098 1409
kvm_pv_eoi 301514 1331
kvm_cr 203989 748
kvm_apic_ipi 38221 151
kvm_cpuid 25318 134
kvm_fpu 8388 43
kvm_userspace_exit 1080 5
kvm_msi_set_irq 691 2
kvm_pio 593 2
kvm_exit(EXCEPTION_NMI) 1115 0
kvm_exit(EXTERNAL_INTERRUPT) 223 0
kvm_exit(CR_ACCESS) 177 0
kvm_msr 36 0
kvm_exit(HLT) 24 0
kvm_exit(CPUID) 16 0
kvm_exit(PENDING_INTERRUPT) 16 0
kvm_exit(INVLPG) 11 0
kvm_exit(TRIPLE_FAULT) 6 0
kvm_exit(APIC_ACCESS) 6 0
kvm_exit(MWAIT_INSTRUCTION) 5 0
kvm_exit(EPT_VIOLATION) 5 0
kvm_exit(TASK_SWITCH) 5 0
kvm_exit(VMREAD) 5 0
kvm_exit(RDPMC) 5 0
kvm_exit(VMRESUME) 5 0
kvm_exit(EPT_MISCONFIG) 5 0
kvm_exit(VMOFF) 5 0
kvm_exit(DR_ACCESS) 4 0
kvm_exit(IO_INSTRUCTION) 4 0
kvm_exit(VMCLEAR) 4 0
kvm_exit(VMON) 4 0
kvm_exit(MCE_DURING_VMENTRY) 3 0
kvm_exit(INVALID_STATE) 3 0
kvm_exit(TPR_BELOW_THRESHOLD) 2 0
kvm_exit(MONITOR_INSTRUCTION) 2 0
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [nVMX Test] Build kernel (defconfig) on L2, w/ VMCS Shadowing enabled on L0
2013-05-13 8:11 ` Kashyap Chamarthy
@ 2013-05-13 8:34 ` Gleb Natapov
2013-05-13 12:30 ` Kashyap Chamarthy
0 siblings, 1 reply; 18+ messages in thread
From: Gleb Natapov @ 2013-05-13 8:34 UTC (permalink / raw)
To: Kashyap Chamarthy; +Cc: Abel Gordon, kvm@vger.kernel.org, Ren, Yongjie
On Mon, May 13, 2013 at 01:41:28PM +0530, Kashyap Chamarthy wrote:
> >
> > Yep, kvm_stat is the simplest way to monitor the exits frequency.
>
> Attached is kvm_stat results from L1, captured from several seconds of
> intervals.
>
You should run it on L0 while compilation is running in L2 once for each
of two cased: with and without shadow vmcs.
--
Gleb.
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [nVMX Test] Build kernel (defconfig) on L2, w/ VMCS Shadowing enabled on L0
2013-05-13 8:34 ` Gleb Natapov
@ 2013-05-13 12:30 ` Kashyap Chamarthy
2013-05-13 13:02 ` Abel Gordon
0 siblings, 1 reply; 18+ messages in thread
From: Kashyap Chamarthy @ 2013-05-13 12:30 UTC (permalink / raw)
To: Gleb Natapov; +Cc: Abel Gordon, kvm@vger.kernel.org, Ren, Yongjie
Thanks Gleb. Here we go:
This time results appear more closer to what Ren mentioned in his
previous email - i.e. enabling VMCS Shadowing didn't "degrade", the
Kernel compile time in L2.
1] VMCS Shadowing *enabled* on L0; Kernel build on L2,
3.10.0-0.rc0.git26.1.fc20.x86_64
Kernel build time on L2
------------------------------
$ time make
[...]
real 31m0.386s
user 16m43.655s
sys 14m6.451s
kvm_stat intervals on L1
-------------------------------
- http://kashyapc.fedorapeople.org/virt/kvm_stat-VMCS-Shadowing/kvm_stat-L0-VMCS-Shadowing-enabled.txt
- NOTE: The below statistics are from intervals ranging from
30 seconds to 6,10,13.. minutes.
2] VMCS Shadowing *disabled* on L0; Kernel build on L2,
3.10.0-0.rc0.git26.1.fc20.x86_64
Kernel build time on L2
------------------------------
$ time make
[...]
real 40m56.010s
user 20m19.061s
sys 19m57.723s
kvm_stat intervals on L1
-------------------------------
- http://kashyapc.fedorapeople.org/virt/kvm_stat-VMCS-Shadowing/kvm_stat-L0-VMCS-Shadowing-disabled.txt
- NOTE: The below statistics are from intervals ranging from
30 seconds to 6,10,13.. minutes.
Looks better ?
(Note to self: Maybe I should run these more number of times to
monitor consistency)
Thanks,
Kashyap.
^ permalink raw reply [flat|nested] 18+ messages in thread* Re: [nVMX Test] Build kernel (defconfig) on L2, w/ VMCS Shadowing enabled on L0
2013-05-13 12:30 ` Kashyap Chamarthy
@ 2013-05-13 13:02 ` Abel Gordon
2013-05-13 15:13 ` Kashyap Chamarthy
0 siblings, 1 reply; 18+ messages in thread
From: Abel Gordon @ 2013-05-13 13:02 UTC (permalink / raw)
To: Kashyap Chamarthy; +Cc: Gleb Natapov, kvm@vger.kernel.org, Ren, Yongjie
Kashyap Chamarthy <kashyap.cv@gmail.com> wrote on 13/05/2013 03:30:03 PM:
>
> 1] VMCS Shadowing *enabled* on L0; Kernel build on L2,
> 3.10.0-0.rc0.git26.1.fc20.x86_64
>
> Kernel build time on L2
> ------------------------------
>
> $ time make
> [...]
> real 31m0.386s
> user 16m43.655s
> sys 14m6.451s
>
> kvm_stat intervals on L1
> -------------------------------
>
> - http://kashyapc.fedorapeople.org/virt/kvm_stat-VMCS-Shadowing/
> kvm_stat-L0-VMCS-Shadowing-enabled.txt
>
> - NOTE: The below statistics are from intervals ranging from
> 30 seconds to 6,10,13.. minutes.
>
>
> 2] VMCS Shadowing *disabled* on L0; Kernel build on L2,
> 3.10.0-0.rc0.git26.1.fc20.x86_64
>
> Kernel build time on L2
> ------------------------------
> $ time make
> [...]
> real 40m56.010s
> user 20m19.061s
> sys 19m57.723s
>
> kvm_stat intervals on L1
> -------------------------------
>
> - http://kashyapc.fedorapeople.org/virt/kvm_stat-VMCS-Shadowing/
> kvm_stat-L0-VMCS-Shadowing-disabled.txt
>
> - NOTE: The below statistics are from intervals ranging from
> 30 seconds to 6,10,13.. minutes.
>
> Looks better ?
Much better :) Now seems the results were improved by 25% (reduced 10m of
40m)
> (Note to self: Maybe I should run these more number of times to
> monitor consistency)
Yes, that's a good point.
You could start with a short make and repeat the experiments to verify
the results are consistent. Once you have reproducible results with a
short make you can move to back to a long make.
Note that if you just run make then the source files will be loaded
from disk only the first run (lot of I/O, which is slow with nested).
Next runs may load the files from the page-cache and you may notice a
performance improvement.
Maybe, that's why your results were not like we expected in the first
time.
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [nVMX Test] Build kernel (defconfig) on L2, w/ VMCS Shadowing enabled on L0
2013-05-13 13:02 ` Abel Gordon
@ 2013-05-13 15:13 ` Kashyap Chamarthy
2013-05-13 15:19 ` Abel Gordon
0 siblings, 1 reply; 18+ messages in thread
From: Kashyap Chamarthy @ 2013-05-13 15:13 UTC (permalink / raw)
To: Abel Gordon; +Cc: Gleb Natapov, kvm@vger.kernel.org, Ren, Yongjie
>>
>> - NOTE: The below statistics are from intervals ranging from
>> 30 seconds to 6,10,13.. minutes.
>>
>> Looks better ?
>
> Much better :) Now seems the results were improved by 25% (reduced 10m of
> 40m)
:) I wonder how it differed the first time.
>
>> (Note to self: Maybe I should run these more number of times to
>> monitor consistency)
>
> Yes, that's a good point.
> You could start with a short make and repeat the experiments to verify
> the results are consistent. Once you have reproducible results with a
> short make you can move to back to a long make.
Just to be clear:
- Short make - Subsequent runs of 'make 'after the first run (i.e.
from page-cache).
- Long make - 1st run of 'make' , after a 'make clean' (i.e. from disk).
I presume that above is what you meant.
>
> Note that if you just run make then the source files will be loaded
> from disk only the first run (lot of I/O, which is slow with nested).
> Next runs may load the files from the page-cache and you may notice a
> performance improvement.
Glad that you mention it..
> Maybe, that's why your results were not like we expected in the first
> time.
>
..I did perform a "make clean" before doing "make" on each run. I
note it in the above kvm_stat files.
Re-running make (on L2), from page-cache gives me the below statistics:
----------------------------------------------
[test@nguest-02 linux]$ time make
make[1]: Nothing to be done for `all'.
make[1]: Nothing to be done for `relocs'.
CHK include/generated/uapi/linux/version.h
CHK include/generated/utsrelease.h
CALL scripts/checksyscalls.sh
CC scripts/mod/devicetable-offsets.s
GEN scripts/mod/devicetable-offsets.h
HOSTCC scripts/mod/file2alias.o
HOSTLD scripts/mod/modpost
CHK include/generated/compile.h
make[3]: `arch/x86/realmode/rm/realmode.bin' is up to date.
Kernel: arch/x86/boot/bzImage is ready (#6)
Building modules, stage 2.
MODPOST 10 modules
real 1m38.276s
user 0m36.806s
sys 0m55.493s
[test@nguest-02 linux]$
----------------------------------------------
2nd-run:
-----------------------
real 1m37.392s
user 0m36.587s
sys 0m55.296s
-----------------------
I'll run a short script for longer duration.
Thanks Abel.
/kashyap
^ permalink raw reply [flat|nested] 18+ messages in thread* Re: [nVMX Test] Build kernel (defconfig) on L2, w/ VMCS Shadowing enabled on L0
2013-05-13 15:13 ` Kashyap Chamarthy
@ 2013-05-13 15:19 ` Abel Gordon
2013-05-13 15:29 ` Kashyap Chamarthy
0 siblings, 1 reply; 18+ messages in thread
From: Abel Gordon @ 2013-05-13 15:19 UTC (permalink / raw)
To: Kashyap Chamarthy; +Cc: Gleb Natapov, kvm@vger.kernel.org, Ren, Yongjie
Kashyap Chamarthy <kashyap.cv@gmail.com> wrote on 13/05/2013 06:13:23 PM:
> Just to be clear:
>
> - Short make - Subsequent runs of 'make 'after the first run (i.e.
> from page-cache).
> - Long make - 1st run of 'make' , after a 'make clean' (i.e. from
disk).
>
> I presume that above is what you meant.
Not exactly. With short a make I was a referring to a run that does not
take
more than few minutes and with a long a make I was referring to a run
that may take an hour.
>
> >
> > Note that if you just run make then the source files will be loaded
> > from disk only the first run (lot of I/O, which is slow with nested).
> > Next runs may load the files from the page-cache and you may notice a
> > performance improvement.
>
> Glad that you mention it..
>
> > Maybe, that's why your results were not like we expected in the first
> > time.
> >
>
> ..I did perform a "make clean" before doing "make" on each run. I
> note it in the above kvm_stat files.
Note that "make clean" clears the output but the source files may still
be in the page-cache (if they were loaded into the cache by a previous
run).
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [nVMX Test] Build kernel (defconfig) on L2, w/ VMCS Shadowing enabled on L0
2013-05-13 15:19 ` Abel Gordon
@ 2013-05-13 15:29 ` Kashyap Chamarthy
2013-05-13 18:55 ` Kashyap Chamarthy
0 siblings, 1 reply; 18+ messages in thread
From: Kashyap Chamarthy @ 2013-05-13 15:29 UTC (permalink / raw)
To: Abel Gordon; +Cc: Gleb Natapov, kvm@vger.kernel.org, Ren, Yongjie
>
> Not exactly. With short a make I was a referring to a run that does not
> take
> more than few minutes and with a long a make I was referring to a run
> that may take an hour.
Ah, sorry for misinterpreting it. I'll try to spend more time on it.
> Note that "make clean" clears the output but the source files may still
> be in the page-cache (if they were loaded into the cache by a previous
> run).
Noted. Thank you.
/kashyap
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [nVMX Test] Build kernel (defconfig) on L2, w/ VMCS Shadowing enabled on L0
2013-05-13 15:29 ` Kashyap Chamarthy
@ 2013-05-13 18:55 ` Kashyap Chamarthy
2013-05-15 4:17 ` Kashyap Chamarthy
0 siblings, 1 reply; 18+ messages in thread
From: Kashyap Chamarthy @ 2013-05-13 18:55 UTC (permalink / raw)
To: Abel Gordon; +Cc: Gleb Natapov, kvm@vger.kernel.org, Ren, Yongjie
Another point I totally forgot to note - these tests were
performed"debug" enabled kernels from Fedora's Rawhide.
I'll also try w/ Nodebug ones --
http://fedoraproject.org/wiki/RawhideKernelNodebug
/kashyap
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [nVMX Test] Build kernel (defconfig) on L2, w/ VMCS Shadowing enabled on L0
2013-05-13 18:55 ` Kashyap Chamarthy
@ 2013-05-15 4:17 ` Kashyap Chamarthy
0 siblings, 0 replies; 18+ messages in thread
From: Kashyap Chamarthy @ 2013-05-15 4:17 UTC (permalink / raw)
To: Abel Gordon; +Cc: Gleb Natapov, kvm@vger.kernel.org, Ren, Yongjie
A small update:
Yesterday, I also ran a simple libguestfs test[1], which creates a
light weight appliance (i.e. an L2) guest , with and without nesting
to see some over-head.
Summary:
-------------
L0 (bare-metal):
- guestfish run: ~ 3 seconds
L1 (guest):
- w/o nesting, guestfish run: ~ 52 seconds
- w/ nesting, guestfish run: ~ 8 seconds (Almost 6 1/2 times
speed improvement)
To clarify, by "w/ nesting" what I meant is, L1's qemu-kvm cli will have:
-cpu Core2Duo,+vmx # or whatever the model is.
And, "w/o nesting": No -cpu parameter in L1's qemu-kvm cli.
Details:
---------
On libguestfs mailing list --
https://www.redhat.com/archives/libguestfs/2013-May/msg00033.html
[1] The baseline measurement mentioned there --
http://libguestfs.org/guestfs-performance.1.html
Thanks.
Details: https://www.redhat.com/archives/libguestfs/2013-May/msg00033.html
^ permalink raw reply [flat|nested] 18+ messages in thread
* RE: [nVMX Test] Build kernel (defconfig) on L2, w/ VMCS Shadowing enabled on L0
2013-05-13 7:07 ` Kashyap Chamarthy
2013-05-13 7:16 ` Abel Gordon
@ 2013-05-13 7:30 ` Ren, Yongjie
2013-05-13 7:38 ` Kashyap Chamarthy
1 sibling, 1 reply; 18+ messages in thread
From: Ren, Yongjie @ 2013-05-13 7:30 UTC (permalink / raw)
To: Kashyap Chamarthy; +Cc: kvm@vger.kernel.org
> -----Original Message-----
> From: Kashyap Chamarthy [mailto:kashyap.cv@gmail.com]
> Sent: Monday, May 13, 2013 3:07 PM
> To: Ren, Yongjie
> Cc: kvm@vger.kernel.org
> Subject: Re: [nVMX Test] Build kernel (defconfig) on L2, w/ VMCS
> Shadowing enabled on L0
>
> On Mon, May 13, 2013 at 11:51 AM, Kashyap Chamarthy
> <kashyap.cv@gmail.com> wrote:
> >> Thanks for the detailed configurations and steps about the nVMX test.
> >> I have some suggestions if you want to test more:
> >> 1. you can make L2 guest have 4vCPU and 4Mem
> >
> > Yeah, will try it. L0 has a max of 8G MEM.
> >
> >> 2. as L2 is SMP system, you can try 'time make -j 4' instead of 'time
> make'.
> >
> > This is my next step :) .
> >
> >> 3. some performance comparison when enable/disable VMCS
> shadowing
> >
> > Yes, as we speak, Kernel is building in L2 w/ VMCS Shadowing disabled
> > on L0. I'll do two runs.
>
> Interesting:
>
> 1/ With VMCS Shadowing *disabled* :
>
> - On L0 (Side note: I did reboot the host, once I disabled shadow VMCS)
> ====
> $ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
> N
> ====
>
> - Building Kernel on L2:
> ====
> $ time make
> .....
> real 36m33.361s
> user 18m19.003s
> sys 17m6.208s
> ====
>
>
> 2/ With VMCS Shadowing *enabled* :
>
> - On L0 (Side note: I did reboot the host, once I enabled shadow VMCS)
> ====
> $ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
> Y
> ====
>
> - Building Kernel on L2:
> ====
> $ time make
> .....
> real 62m53.853s
> user 32m32.724s
> sys 32m16.430s
> ====
>
> Comments ?
>
My test result is different from yours. In my test several days ago,
when enabling shadow VMCS, I can get ~4% performance improvement
in kernel build test in L2. Hardware is also a Haswell machine.
shadow VMCS disabled, build time: 649 seconds
shadow VMCS enabled, build time: 627 seconds.
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [nVMX Test] Build kernel (defconfig) on L2, w/ VMCS Shadowing enabled on L0
2013-05-13 7:30 ` Ren, Yongjie
@ 2013-05-13 7:38 ` Kashyap Chamarthy
0 siblings, 0 replies; 18+ messages in thread
From: Kashyap Chamarthy @ 2013-05-13 7:38 UTC (permalink / raw)
To: Ren, Yongjie; +Cc: kvm@vger.kernel.org
On Mon, May 13, 2013 at 1:00 PM, Ren, Yongjie <yongjie.ren@intel.com> wrote:
>> -----Original Message-----
>> From: Kashyap Chamarthy [mailto:kashyap.cv@gmail.com]
>> Sent: Monday, May 13, 2013 3:07 PM
>> To: Ren, Yongjie
>> Cc: kvm@vger.kernel.org
>> Subject: Re: [nVMX Test] Build kernel (defconfig) on L2, w/ VMCS
>> Shadowing enabled on L0
>>
>> On Mon, May 13, 2013 at 11:51 AM, Kashyap Chamarthy
>> <kashyap.cv@gmail.com> wrote:
>>
>> Comments ?
>>
> My test result is different from yours. In my test several days ago,
> when enabling shadow VMCS, I can get ~4% performance improvement
> in kernel build test in L2. Hardware is also a Haswell machine.
> shadow VMCS disabled, build time: 649 seconds
> shadow VMCS enabled, build time: 627 seconds.
Let me try it on a new L2 guest to see if I can reproduce the behavior
I reported reliably.
Meantime, I noted my entire procedure
https://github.com/kashyapc/nvmx-haswell/blob/master/SETUP-nVMX.rst
Do you see any anomalies ?
/kashyap
^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2013-05-15 4:17 UTC | newest]
Thread overview: 18+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-05-13 4:58 [nVMX Test] Build kernel (defconfig) on L2, w/ VMCS Shadowing enabled on L0 Kashyap Chamarthy
2013-05-13 6:10 ` Ren, Yongjie
2013-05-13 6:21 ` Kashyap Chamarthy
2013-05-13 7:07 ` Kashyap Chamarthy
2013-05-13 7:16 ` Abel Gordon
2013-05-13 7:31 ` Kashyap Chamarthy
2013-05-13 7:51 ` Abel Gordon
2013-05-13 8:11 ` Kashyap Chamarthy
2013-05-13 8:34 ` Gleb Natapov
2013-05-13 12:30 ` Kashyap Chamarthy
2013-05-13 13:02 ` Abel Gordon
2013-05-13 15:13 ` Kashyap Chamarthy
2013-05-13 15:19 ` Abel Gordon
2013-05-13 15:29 ` Kashyap Chamarthy
2013-05-13 18:55 ` Kashyap Chamarthy
2013-05-15 4:17 ` Kashyap Chamarthy
2013-05-13 7:30 ` Ren, Yongjie
2013-05-13 7:38 ` Kashyap Chamarthy
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.