Why so many vm exits caused by ept violation

Why so many vm exits caused by ept violation

[lidong chen]

I run my application both on xen and kvm, and I found the performance
on kvm is worse than xen.

kvm: 2.6.32-60.el6.x86_64 redhat 6 bate 2
xen: xen3.4.2

then i stat the vm exit reason, and i found the ept violation reason,
kvm is much more than xen. the static result is :

(XEN) 100000 times on cpu 0,
reason, times
0, 388
1, 8636
7, 1902
12, 18292
18, 6065
28, 8474
30, 16590
44, 39653

(KVM) 100000 times on cpu 0,
reason, times
0, 998
1, 7115
7, 3335
12, 1953
28, 1988
30, 22324
44, 20224
48, 42063

by used xen, it didn't have ept violation, because my application
first malloc all memory,and initialize it. so i think kvm is also
didn't need vm exits by ept violation .

the exit_qualification of ept violation  is 181 or 182, i found the
code run abnormality while deal with ept violation, the error happed
because the return of slot is NULL

unsigned long gfn_to_hva(struct kvm *kvm, gfn_t gfn)
{
	struct kvm_memory_slot *slot;

	gfn = unalias_gfn_instantiation(kvm, gfn);
	slot = gfn_to_memslot_unaliased(kvm, gfn);

                /* the error happed at this , the return of slot is NULL */

	if (!slot || slot->flags & KVM_MEMSLOT_INVALID)
		return bad_hva();
	return (slot->userspace_addr + (gfn - slot->base_gfn) * PAGE_SIZE);
}

then i printk the gfa below:

__ratelimit: 387 callbacks suppressed
error gpa is 4273995776
error gpa is 4273995776
error gpa is 4273995776
error gpa is 4273995776
error gpa is 4273995776
error gpa is 782336
error gpa is 782336
error gpa is 782336
error gpa is 782336
error gpa is 782336
__ratelimit: 387 callbacks suppressed
error gpa is 782336
error gpa is 782336
error gpa is 782336
error gpa is 782336
error gpa is 782336
error gpa is 782336
error gpa is 782336
error gpa is 782336
error gpa is 782336
error gpa is 782336


why access this address cause vm exit in kvm, but didn't cause vm exit in xen.

Re: Why so many vm exits caused by ept violation

[Avi Kivity]

On 11/08/2010 02:15 PM, lidong chen wrote:
> I run my application both on xen and kvm, and I found the performance
> on kvm is worse than xen.
>
> kvm: 2.6.32-60.el6.x86_64 redhat 6 bate 2
> xen: xen3.4.2
>
> then i stat the vm exit reason, and i found the ept violation reason,
> kvm is much more than xen. the static result is :
>
> __ratelimit: 387 callbacks suppressed
> error gpa is 4273995776
> error gpa is 4273995776
> error gpa is 4273995776
> error gpa is 4273995776
> error gpa is 4273995776

fec00000, this is the IOAPIC.

What workload is this?  Are you using virtio?

> error gpa is 782336
> error gpa is 782336
> error gpa is 782336
> error gpa is 782336
> error gpa is 782336
> __ratelimit: 387 callbacks suppressed
> error gpa is 782336
> error gpa is 782336
> error gpa is 782336
> error gpa is 782336
> error gpa is 782336
> error gpa is 782336
> error gpa is 782336
> error gpa is 782336
> error gpa is 782336
> error gpa is 782336
>

bf000, this is the display.  What display mode are you using? is the 
workload display intensive?

Please provide reports using

   $ trace-cmd record -e kvm:kvm_page_fault
   ^C
   $ trace-cmd report


-- 
error compiling committee.c: too many arguments to function

Re: Why so many vm exits caused by ept violation

[lidong chen]

> What workload is this?  Are you using virtio?
yes,i using virtio and vhost. the cpu rate is 70%.

I found it maybe caused by  2M HugePages. if  didn't use  2M
HugePages, there are no vm exits caused by ept violation.

I will test the performance without 2M Hugepages later, and see why
HugePage cause such a result.


2010/11/8 Avi Kivity <avi@redhat.com>:
> On 11/08/2010 02:15 PM, lidong chen wrote:
>>
>> I run my application both on xen and kvm, and I found the performance
>> on kvm is worse than xen.
>>
>> kvm: 2.6.32-60.el6.x86_64 redhat 6 bate 2
>> xen: xen3.4.2
>>
>> then i stat the vm exit reason, and i found the ept violation reason,
>> kvm is much more than xen. the static result is :
>>
>> __ratelimit: 387 callbacks suppressed
>> error gpa is 4273995776
>> error gpa is 4273995776
>> error gpa is 4273995776
>> error gpa is 4273995776
>> error gpa is 4273995776
>
> fec00000, this is the IOAPIC.
>
> What workload is this?  Are you using virtio?
>
>> error gpa is 782336
>> error gpa is 782336
>> error gpa is 782336
>> error gpa is 782336
>> error gpa is 782336
>> __ratelimit: 387 callbacks suppressed
>> error gpa is 782336
>> error gpa is 782336
>> error gpa is 782336
>> error gpa is 782336
>> error gpa is 782336
>> error gpa is 782336
>> error gpa is 782336
>> error gpa is 782336
>> error gpa is 782336
>> error gpa is 782336
>>
>
> bf000, this is the display.  What display mode are you using? is the
> workload display intensive?
>
> Please provide reports using
>
>  $ trace-cmd record -e kvm:kvm_page_fault
>  ^C
>  $ trace-cmd report
>
>
> --
> error compiling committee.c: too many arguments to function
>
>

Re: Why so many vm exits caused by ept violation

[lidong chen]

sorry,i take mistake,

the ept violation is not caused by 2M hugepages, but caused by e1000
emulated by qemu.
I test uesd 2M hugepages and virio, the performance is better than
e1000. but is still worse than xen. I will continue to find out other
reason.



so, there are two question:
1.why have so many ept violation when used e1000?


2.when used virtio, there ara also such error happen.is this normal?
I add the printk in function tdp_page_fault below.
        pfn = gfn_to_pfn(vcpu->kvm, gfn);
        if (is_error_pfn(pfn)) {
                kvm_release_pfn_clean(pfn);
                if( printk_ratelimit() ) {
                        printk(KERN_EMERG "error gpa is %ld\n", gpa);
                }
                return is_fault_pfn(pfn) ? -EFAULT : 1;
        }

and result is:
[root@rhel6-kvm1 ~]# dmesg -c
__ratelimit: 387 callbacks suppressed
error gpa is 4273995776
error gpa is 4273995776
error gpa is 4273995776
error gpa is 4273995776
error gpa is 4273995776



2010/11/9 lidong chen <chen.lidong.kernel@gmail.com>:
>> What workload is this?  Are you using virtio?
> yes,i using virtio and vhost. the cpu rate is 70%.
>
> I found it maybe caused by  2M HugePages. if  didn't use  2M
> HugePages, there are no vm exits caused by ept violation.
>
> I will test the performance without 2M Hugepages later, and see why
> HugePage cause such a result.
>
>
> 2010/11/8 Avi Kivity <avi@redhat.com>:
>> On 11/08/2010 02:15 PM, lidong chen wrote:
>>>
>>> I run my application both on xen and kvm, and I found the performance
>>> on kvm is worse than xen.
>>>
>>> kvm: 2.6.32-60.el6.x86_64 redhat 6 bate 2
>>> xen: xen3.4.2
>>>
>>> then i stat the vm exit reason, and i found the ept violation reason,
>>> kvm is much more than xen. the static result is :
>>>
>>> __ratelimit: 387 callbacks suppressed
>>> error gpa is 4273995776
>>> error gpa is 4273995776
>>> error gpa is 4273995776
>>> error gpa is 4273995776
>>> error gpa is 4273995776
>>
>> fec00000, this is the IOAPIC.
>>
>> What workload is this?  Are you using virtio?
>>
>>> error gpa is 782336
>>> error gpa is 782336
>>> error gpa is 782336
>>> error gpa is 782336
>>> error gpa is 782336
>>> __ratelimit: 387 callbacks suppressed
>>> error gpa is 782336
>>> error gpa is 782336
>>> error gpa is 782336
>>> error gpa is 782336
>>> error gpa is 782336
>>> error gpa is 782336
>>> error gpa is 782336
>>> error gpa is 782336
>>> error gpa is 782336
>>> error gpa is 782336
>>>
>>
>> bf000, this is the display.  What display mode are you using? is the
>> workload display intensive?
>>
>> Please provide reports using
>>
>>  $ trace-cmd record -e kvm:kvm_page_fault
>>  ^C
>>  $ trace-cmd report
>>
>>
>> --
>> error compiling committee.c: too many arguments to function
>>
>>
>

Re: Why so many vm exits caused by ept violation

[Gleb Natapov]

On Tue, Nov 09, 2010 at 03:37:44PM +0800, lidong chen wrote:
> sorry,i take mistake,
> 
> the ept violation is not caused by 2M hugepages, but caused by e1000
> emulated by qemu.
> I test uesd 2M hugepages and virio, the performance is better than
> e1000. but is still worse than xen. I will continue to find out other
> reason.
> 
> 
> 
> so, there are two question:
> 1.why have so many ept violation when used e1000?
> 
> 
> 2.when used virtio, there ara also such error happen.is this normal?
> I add the printk in function tdp_page_fault below.
>         pfn = gfn_to_pfn(vcpu->kvm, gfn);
>         if (is_error_pfn(pfn)) {
>                 kvm_release_pfn_clean(pfn);
>                 if( printk_ratelimit() ) {
>                         printk(KERN_EMERG "error gpa is %ld\n", gpa);
>                 }
>                 return is_fault_pfn(pfn) ? -EFAULT : 1;
>         }
> 
> and result is:
> [root@rhel6-kvm1 ~]# dmesg -c
> __ratelimit: 387 callbacks suppressed
> error gpa is 4273995776
> error gpa is 4273995776
> error gpa is 4273995776
> error gpa is 4273995776
> error gpa is 4273995776
> 
You should really print this in hex :) This is 0xfec00000 AKA
IOAPIC. Try to configure virtio to use MSI.

--
			Gleb.

Re: Why so many vm exits caused by ept violation

[lidong chen]

how to configure virtio to use MSI? I didn't found out it.
thanks.

2010/11/9 Gleb Natapov <gleb@redhat.com>:
> On Tue, Nov 09, 2010 at 03:37:44PM +0800, lidong chen wrote:
>> sorry,i take mistake,
>>
>> the ept violation is not caused by 2M hugepages, but caused by e1000
>> emulated by qemu.
>> I test uesd 2M hugepages and virio, the performance is better than
>> e1000. but is still worse than xen. I will continue to find out other
>> reason.
>>
>>
>>
>> so, there are two question:
>> 1.why have so many ept violation when used e1000?
>>
>>
>> 2.when used virtio, there ara also such error happen.is this normal?
>> I add the printk in function tdp_page_fault below.
>>         pfn = gfn_to_pfn(vcpu->kvm, gfn);
>>         if (is_error_pfn(pfn)) {
>>                 kvm_release_pfn_clean(pfn);
>>                 if( printk_ratelimit() ) {
>>                         printk(KERN_EMERG "error gpa is %ld\n", gpa);
>>                 }
>>                 return is_fault_pfn(pfn) ? -EFAULT : 1;
>>         }
>>
>> and result is:
>> [root@rhel6-kvm1 ~]# dmesg -c
>> __ratelimit: 387 callbacks suppressed
>> error gpa is 4273995776
>> error gpa is 4273995776
>> error gpa is 4273995776
>> error gpa is 4273995776
>> error gpa is 4273995776
>>
> You should really print this in hex :) This is 0xfec00000 AKA
> IOAPIC. Try to configure virtio to use MSI.
>
> --
>                        Gleb.
>

Re: Why so many vm exits caused by ept violation

[Gleb Natapov]

On Tue, Nov 09, 2010 at 05:04:15PM +0800, lidong chen wrote:
> how to configure virtio to use MSI? I didn't found out it.
> thanks.
> 
It looks like it should be enabled by default. What is you guest?

> 2010/11/9 Gleb Natapov <gleb@redhat.com>:
> > On Tue, Nov 09, 2010 at 03:37:44PM +0800, lidong chen wrote:
> >> sorry,i take mistake,
> >>
> >> the ept violation is not caused by 2M hugepages, but caused by e1000
> >> emulated by qemu.
> >> I test uesd 2M hugepages and virio, the performance is better than
> >> e1000. but is still worse than xen. I will continue to find out other
> >> reason.
> >>
> >>
> >>
> >> so, there are two question:
> >> 1.why have so many ept violation when used e1000?
> >>
> >>
> >> 2.when used virtio, there ara also such error happen.is this normal?
> >> I add the printk in function tdp_page_fault below.
> >>         pfn = gfn_to_pfn(vcpu->kvm, gfn);
> >>         if (is_error_pfn(pfn)) {
> >>                 kvm_release_pfn_clean(pfn);
> >>                 if( printk_ratelimit() ) {
> >>                         printk(KERN_EMERG "error gpa is %ld\n", gpa);
> >>                 }
> >>                 return is_fault_pfn(pfn) ? -EFAULT : 1;
> >>         }
> >>
> >> and result is:
> >> [root@rhel6-kvm1 ~]# dmesg -c
> >> __ratelimit: 387 callbacks suppressed
> >> error gpa is 4273995776
> >> error gpa is 4273995776
> >> error gpa is 4273995776
> >> error gpa is 4273995776
> >> error gpa is 4273995776
> >>
> > You should really print this in hex :) This is 0xfec00000 AKA
> > IOAPIC. Try to configure virtio to use MSI.
> >
> > --
> >                        Gleb.
> >

--
			Gleb.

Re: Why so many vm exits caused by ept violation

[lidong chen]

the version of guest os is 2.6.16.60-0.21-bigsmp.

the virtio driver we use from RHEL5.5.

2010/11/9 Gleb Natapov <gleb@redhat.com>:
> On Tue, Nov 09, 2010 at 05:04:15PM +0800, lidong chen wrote:
>> how to configure virtio to use MSI? I didn't found out it.
>> thanks.
>>
> It looks like it should be enabled by default. What is you guest?
>
>> 2010/11/9 Gleb Natapov <gleb@redhat.com>:
>> > On Tue, Nov 09, 2010 at 03:37:44PM +0800, lidong chen wrote:
>> >> sorry,i take mistake,
>> >>
>> >> the ept violation is not caused by 2M hugepages, but caused by e1000
>> >> emulated by qemu.
>> >> I test uesd 2M hugepages and virio, the performance is better than
>> >> e1000. but is still worse than xen. I will continue to find out other
>> >> reason.
>> >>
>> >>
>> >>
>> >> so, there are two question:
>> >> 1.why have so many ept violation when used e1000?
>> >>
>> >>
>> >> 2.when used virtio, there ara also such error happen.is this normal?
>> >> I add the printk in function tdp_page_fault below.
>> >>         pfn = gfn_to_pfn(vcpu->kvm, gfn);
>> >>         if (is_error_pfn(pfn)) {
>> >>                 kvm_release_pfn_clean(pfn);
>> >>                 if( printk_ratelimit() ) {
>> >>                         printk(KERN_EMERG "error gpa is %ld\n", gpa);
>> >>                 }
>> >>                 return is_fault_pfn(pfn) ? -EFAULT : 1;
>> >>         }
>> >>
>> >> and result is:
>> >> [root@rhel6-kvm1 ~]# dmesg -c
>> >> __ratelimit: 387 callbacks suppressed
>> >> error gpa is 4273995776
>> >> error gpa is 4273995776
>> >> error gpa is 4273995776
>> >> error gpa is 4273995776
>> >> error gpa is 4273995776
>> >>
>> > You should really print this in hex :) This is 0xfec00000 AKA
>> > IOAPIC. Try to configure virtio to use MSI.
>> >
>> > --
>> >                        Gleb.
>> >
>
> --
>                        Gleb.
>

Re: Why so many vm exits caused by ept violation

[Gleb Natapov]

On Tue, Nov 09, 2010 at 06:24:06PM +0800, lidong chen wrote:
> the version of guest os is 2.6.16.60-0.21-bigsmp.
> 
> the virtio driver we use from RHEL5.5.
> 
This guest does not support MSI.

> 2010/11/9 Gleb Natapov <gleb@redhat.com>:
> > On Tue, Nov 09, 2010 at 05:04:15PM +0800, lidong chen wrote:
> >> how to configure virtio to use MSI? I didn't found out it.
> >> thanks.
> >>
> > It looks like it should be enabled by default. What is you guest?
> >
> >> 2010/11/9 Gleb Natapov <gleb@redhat.com>:
> >> > On Tue, Nov 09, 2010 at 03:37:44PM +0800, lidong chen wrote:
> >> >> sorry,i take mistake,
> >> >>
> >> >> the ept violation is not caused by 2M hugepages, but caused by e1000
> >> >> emulated by qemu.
> >> >> I test uesd 2M hugepages and virio, the performance is better than
> >> >> e1000. but is still worse than xen. I will continue to find out other
> >> >> reason.
> >> >>
> >> >>
> >> >>
> >> >> so, there are two question:
> >> >> 1.why have so many ept violation when used e1000?
> >> >>
> >> >>
> >> >> 2.when used virtio, there ara also such error happen.is this normal?
> >> >> I add the printk in function tdp_page_fault below.
> >> >>         pfn = gfn_to_pfn(vcpu->kvm, gfn);
> >> >>         if (is_error_pfn(pfn)) {
> >> >>                 kvm_release_pfn_clean(pfn);
> >> >>                 if( printk_ratelimit() ) {
> >> >>                         printk(KERN_EMERG "error gpa is %ld\n", gpa);
> >> >>                 }
> >> >>                 return is_fault_pfn(pfn) ? -EFAULT : 1;
> >> >>         }
> >> >>
> >> >> and result is:
> >> >> [root@rhel6-kvm1 ~]# dmesg -c
> >> >> __ratelimit: 387 callbacks suppressed
> >> >> error gpa is 4273995776
> >> >> error gpa is 4273995776
> >> >> error gpa is 4273995776
> >> >> error gpa is 4273995776
> >> >> error gpa is 4273995776
> >> >>
> >> > You should really print this in hex :) This is 0xfec00000 AKA
> >> > IOAPIC. Try to configure virtio to use MSI.
> >> >
> >> > --
> >> >                        Gleb.
> >> >
> >
> > --
> >                        Gleb.
> >

--
			Gleb.

Re: Why so many vm exits caused by ept violation

[lidong chen]

after slove the ept violation problem, i found  the vm exits times is
still more than xen.

and i found most of them is caused by io instruction. then i stat the
port number, most of them is caused by virio_net.

        qemu-kvm-18727 [006] 31297.698935: kvm_pio: pio_read at 0xc093
size 1 count 1
        qemu-kvm-18727 [006] 31297.698945: kvm_pio: pio_read at 0xc0f3
size 1 count 1
        qemu-kvm-18846 [001] 31297.698962: kvm_pio: pio_read at 0xc073
size 1 count 1
        qemu-kvm-18846 [001] 31297.698968: kvm_pio: pio_read at 0xc093
size 1 count 1
        qemu-kvm-18846 [001] 31297.698973: kvm_pio: pio_read at 0xc0f3
size 1 count 1
        qemu-kvm-18481 [006] 31297.699000: kvm_pio: pio_read at 0xc073
size 1 count 1
        qemu-kvm-18481 [006] 31297.699008: kvm_pio: pio_read at 0xc093
size 1 count 1
        qemu-kvm-18846 [001] 31297.699011: kvm_pio: pio_read at 0xc053
size 1 count 1
        qemu-kvm-18846 [001] 31297.699017: kvm_pio: pio_read at 0xc0b3
size 1 count 1
        qemu-kvm-18846 [001] 31297.699023: kvm_pio: pio_read at 0xc0d3
size 1 count 1
        qemu-kvm-18481 [006] 31297.699025: kvm_pio: pio_read at 0xc053
size 1 count 1
        qemu-kvm-18481 [006] 31297.699030: kvm_pio: pio_read at 0xc0b3
size 1 count 1
        qemu-kvm-18481 [006] 31297.699057: kvm_pio: pio_read at 0xc0d3
size 1 count 1
        qemu-kvm-18481 [006] 31297.699075: kvm_pio: pio_read at 0xc053
size 1 count 1
        qemu-kvm-18481 [006] 31297.699081: kvm_pio: pio_read at 0xc0b3
size 1 count 1
        qemu-kvm-18481 [006] 31297.699084: kvm_pio: pio_read at 0xc0d3
size 1 count 1

00:04.0 Ethernet controller: Unknown device 1af4:1000
	Subsystem: Unknown device 1af4:0001
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
	Interrupt: pin A routed to IRQ 11
	Region 0: I/O ports at c040 [size=32]
	Region 1: Memory at f2020000 (32-bit, non-prefetchable) [size=4K]
	Expansion ROM at f2030000 [disabled] [size=64K]
	Capabilities: [40] MSI-X: Enable- Mask- TabSize=3
		Vector table: BAR=1 offset=00000000
		PBA: BAR=1 offset=00000800

00:05.0 Ethernet controller: Unknown device 1af4:1000
	Subsystem: Unknown device 1af4:0001
	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
	Interrupt: pin A routed to IRQ 10
	Region 0: I/O ports at c060 [size=32]
	Region 1: Memory at f2040000 (32-bit, non-prefetchable) [size=4K]
	Expansion ROM at f2050000 [disabled] [size=64K]
	Capabilities: [40] MSI-X: Enable- Mask- TabSize=3
		Vector table: BAR=1 offset=00000000
		PBA: BAR=1 offset=00000800

00:06.0 Ethernet controller: Unknown device 1af4:1000
	Subsystem: Unknown device 1af4:0001
	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
	Interrupt: pin A routed to IRQ 10
	Region 0: I/O ports at c080 [size=32]
	Region 1: Memory at f2060000 (32-bit, non-prefetchable) [size=4K]
	Expansion ROM at f2070000 [disabled] [size=64K]
	Capabilities: [40] MSI-X: Enable- Mask- TabSize=3
		Vector table: BAR=1 offset=00000000
		PBA: BAR=1 offset=00000800

00:07.0 Ethernet controller: Unknown device 1af4:1000
	Subsystem: Unknown device 1af4:0001
	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
	Interrupt: pin A routed to IRQ 11
	Region 0: I/O ports at c0a0 [size=32]
	Region 1: Memory at f2080000 (32-bit, non-prefetchable) [size=4K]
	Expansion ROM at f2090000 [disabled] [size=64K]
	Capabilities: [40] MSI-X: Enable- Mask- TabSize=3
		Vector table: BAR=1 offset=00000000
		PBA: BAR=1 offset=00000800

00:08.0 Ethernet controller: Unknown device 1af4:1000
	Subsystem: Unknown device 1af4:0001
	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
	Interrupt: pin A routed to IRQ 11
	Region 0: I/O ports at c0c0 [size=32]
	Region 1: Memory at f20a0000 (32-bit, non-prefetchable) [size=4K]
	Expansion ROM at f20b0000 [disabled] [size=64K]
	Capabilities: [40] MSI-X: Enable- Mask- TabSize=3
		Vector table: BAR=1 offset=00000000
		PBA: BAR=1 offset=00000800

the driver version is RHLE5.5. and i want move the virtio driver of
RHLE 6 to RHLE5.5.
have anyone already done this work?





2010/11/9 Gleb Natapov <gleb@redhat.com>:
> On Tue, Nov 09, 2010 at 06:24:06PM +0800, lidong chen wrote:
>> the version of guest os is 2.6.16.60-0.21-bigsmp.
>>
>> the virtio driver we use from RHEL5.5.
>>
> This guest does not support MSI.
>
>> 2010/11/9 Gleb Natapov <gleb@redhat.com>:
>> > On Tue, Nov 09, 2010 at 05:04:15PM +0800, lidong chen wrote:
>> >> how to configure virtio to use MSI? I didn't found out it.
>> >> thanks.
>> >>
>> > It looks like it should be enabled by default. What is you guest?
>> >
>> >> 2010/11/9 Gleb Natapov <gleb@redhat.com>:
>> >> > On Tue, Nov 09, 2010 at 03:37:44PM +0800, lidong chen wrote:
>> >> >> sorry,i take mistake,
>> >> >>
>> >> >> the ept violation is not caused by 2M hugepages, but caused by e1000
>> >> >> emulated by qemu.
>> >> >> I test uesd 2M hugepages and virio, the performance is better than
>> >> >> e1000. but is still worse than xen. I will continue to find out other
>> >> >> reason.
>> >> >>
>> >> >>
>> >> >>
>> >> >> so, there are two question:
>> >> >> 1.why have so many ept violation when used e1000?
>> >> >>
>> >> >>
>> >> >> 2.when used virtio, there ara also such error happen.is this normal?
>> >> >> I add the printk in function tdp_page_fault below.
>> >> >>         pfn = gfn_to_pfn(vcpu->kvm, gfn);
>> >> >>         if (is_error_pfn(pfn)) {
>> >> >>                 kvm_release_pfn_clean(pfn);
>> >> >>                 if( printk_ratelimit() ) {
>> >> >>                         printk(KERN_EMERG "error gpa is %ld\n", gpa);
>> >> >>                 }
>> >> >>                 return is_fault_pfn(pfn) ? -EFAULT : 1;
>> >> >>         }
>> >> >>
>> >> >> and result is:
>> >> >> [root@rhel6-kvm1 ~]# dmesg -c
>> >> >> __ratelimit: 387 callbacks suppressed
>> >> >> error gpa is 4273995776
>> >> >> error gpa is 4273995776
>> >> >> error gpa is 4273995776
>> >> >> error gpa is 4273995776
>> >> >> error gpa is 4273995776
>> >> >>
>> >> > You should really print this in hex :) This is 0xfec00000 AKA
>> >> > IOAPIC. Try to configure virtio to use MSI.
>> >> >
>> >> > --
>> >> >                        Gleb.
>> >> >
>> >
>> > --
>> >                        Gleb.
>> >
>
> --
>                        Gleb.
>

Re: Why so many vm exits caused by ept violation

[Avi Kivity]

On 11/10/2010 09:09 AM, lidong chen wrote:
> after slove the ept violation problem, i found  the vm exits times is
> still more than xen.
>
> and i found most of them is caused by io instruction. then i stat the
> port number, most of them is caused by virio_net.

This is also solved by the msi-capable virtio driver.

-- 
error compiling committee.c: too many arguments to function

Re: Why so many vm exits caused by ept violation

[lidong chen]

i use the virtio dirvier of rhle6, the guest os version is 2.6.16.

the vm exits which caused by io instruction and apic access reduce a lot.
the total number of vm exits decreased by 50% :)

but the ept violation seem increase on cpu 0, cpu 1, cpu 2, cpu3.

(KVM)100000 times on cpu 2,
0, 1989
1, 9937
7, 7951
12, 2709
28, 20
30, 18536
44, 29058
48, 29800

and not all cpu have this problem: cpu4,cpu5,cpu6,cpu7
100000 times on cpu 5,
0, 13934
1, 35955
7, 26
12, 3841
28, 11224
30, 29389
44, 5631

I bind vcpu like this:
      virsh vcpupin brd1vm4 0 7
    virsh vcpupin brd3vm4 0 0
    virsh vcpupin brd3vm4 1 4
    virsh vcpupin brd5vm4 0 1
    virsh vcpupin brd5vm4 1 5
    virsh vcpupin brd9vm4 0 2
    virsh vcpupin brd9vm4 1 6
    virsh vcpupin brd11vm4 0 3
    virsh vcpupin brd11vm4 1 7

the /proc/interrupt of guest os is below:
           CPU0       CPU1
  0:    1896802          0    IO-APIC-edge  timer
  1:          8          0    IO-APIC-edge  i8042
  4:         14          0    IO-APIC-edge  serial
  8:          0          0    IO-APIC-edge  rtc
  9:          0          0   IO-APIC-level  acpi
 10:          0          0   IO-APIC-level  virtio1, virtio2, virtio5
 11:          1          0   IO-APIC-level  virtio0, virtio3, virtio4
 12:        104          0    IO-APIC-edge  i8042
 50:          1          0       PCI-MSI-X  virtio2-output
 58:          0          0       PCI-MSI-X  virtio3-config
 66:    2046985          0       PCI-MSI-X  virtio3-input
 74:          2          0       PCI-MSI-X  virtio3-output
 82:          0          0       PCI-MSI-X  virtio4-config
 90:        217          0       PCI-MSI-X  virtio4-input
 98:          0          0       PCI-MSI-X  virtio4-output
177:          0          0       PCI-MSI-X  virtio0-config
185:     341831          0       PCI-MSI-X  virtio0-input
193:          1          0       PCI-MSI-X  virtio0-output
201:          0          0       PCI-MSI-X  virtio1-config
209:     188747          0       PCI-MSI-X  virtio1-input
217:          1          0       PCI-MSI-X  virtio1-output
225:          0          0       PCI-MSI-X  virtio2-config
233:    2204149          0       PCI-MSI-X  virtio2-input
NMI:    1455767    1426226
LOC:    1896099    1896637
ERR:          0
MIS:          0

the application on all vcpu is the same. so i think the new pv driver
caused ept violation .


2010/11/10 Avi Kivity <avi@redhat.com>:
> On 11/10/2010 09:09 AM, lidong chen wrote:
>>
>> after slove the ept violation problem, i found  the vm exits times is
>> still more than xen.
>>
>> and i found most of them is caused by io instruction. then i stat the
>> port number, most of them is caused by virio_net.
>
> This is also solved by the msi-capable virtio driver.
>
> --
> error compiling committee.c: too many arguments to function
>
>

Re: Why so many vm exits caused by ept violation

[lidong chen]

the gpa caused ept violation is below:

most of them is F2020000.(4060217344)

error gpa is 4060217344
error gpa is 4060217344
error gpa is 4060217344
error gpa is 4060217344
error gpa is 4060217344
error gpa is 4060217344
error gpa is 4060217344
error gpa is 4060217344
error gpa is 4060217344
error gpa is 4060217344

other is F20A0000(4060741632).

error gpa is 4060741632
error gpa is 4060741632


2010/11/15, lidong chen <chen.lidong.kernel@gmail.com>:
> i use the virtio dirvier of rhle6, the guest os version is 2.6.16.
>
> the vm exits which caused by io instruction and apic access reduce a lot.
> the total number of vm exits decreased by 50% :)
>
> but the ept violation seem increase on cpu 0, cpu 1, cpu 2, cpu3.
>
> (KVM)100000 times on cpu 2,
> 0, 1989
> 1, 9937
> 7, 7951
> 12, 2709
> 28, 20
> 30, 18536
> 44, 29058
> 48, 29800
>
> and not all cpu have this problem: cpu4,cpu5,cpu6,cpu7
> 100000 times on cpu 5,
> 0, 13934
> 1, 35955
> 7, 26
> 12, 3841
> 28, 11224
> 30, 29389
> 44, 5631
>
> I bind vcpu like this:
>       virsh vcpupin brd1vm4 0 7
>     virsh vcpupin brd3vm4 0 0
>     virsh vcpupin brd3vm4 1 4
>     virsh vcpupin brd5vm4 0 1
>     virsh vcpupin brd5vm4 1 5
>     virsh vcpupin brd9vm4 0 2
>     virsh vcpupin brd9vm4 1 6
>     virsh vcpupin brd11vm4 0 3
>     virsh vcpupin brd11vm4 1 7
>
> the /proc/interrupt of guest os is below:
>            CPU0       CPU1
>   0:    1896802          0    IO-APIC-edge  timer
>   1:          8          0    IO-APIC-edge  i8042
>   4:         14          0    IO-APIC-edge  serial
>   8:          0          0    IO-APIC-edge  rtc
>   9:          0          0   IO-APIC-level  acpi
>  10:          0          0   IO-APIC-level  virtio1, virtio2, virtio5
>  11:          1          0   IO-APIC-level  virtio0, virtio3, virtio4
>  12:        104          0    IO-APIC-edge  i8042
>  50:          1          0       PCI-MSI-X  virtio2-output
>  58:          0          0       PCI-MSI-X  virtio3-config
>  66:    2046985          0       PCI-MSI-X  virtio3-input
>  74:          2          0       PCI-MSI-X  virtio3-output
>  82:          0          0       PCI-MSI-X  virtio4-config
>  90:        217          0       PCI-MSI-X  virtio4-input
>  98:          0          0       PCI-MSI-X  virtio4-output
> 177:          0          0       PCI-MSI-X  virtio0-config
> 185:     341831          0       PCI-MSI-X  virtio0-input
> 193:          1          0       PCI-MSI-X  virtio0-output
> 201:          0          0       PCI-MSI-X  virtio1-config
> 209:     188747          0       PCI-MSI-X  virtio1-input
> 217:          1          0       PCI-MSI-X  virtio1-output
> 225:          0          0       PCI-MSI-X  virtio2-config
> 233:    2204149          0       PCI-MSI-X  virtio2-input
> NMI:    1455767    1426226
> LOC:    1896099    1896637
> ERR:          0
> MIS:          0
>
> the application on all vcpu is the same. so i think the new pv driver
> caused ept violation .
>
>
> 2010/11/10 Avi Kivity <avi@redhat.com>:
>> On 11/10/2010 09:09 AM, lidong chen wrote:
>>>
>>> after slove the ept violation problem, i found  the vm exits times is
>>> still more than xen.
>>>
>>> and i found most of them is caused by io instruction. then i stat the
>>> port number, most of them is caused by virio_net.
>>
>> This is also solved by the msi-capable virtio driver.
>>
>> --
>> error compiling committee.c: too many arguments to function
>>
>>
>

Re: Why so many vm exits caused by ept violation

[lidong chen]

the address is the Region 1 of virtio_net.

why virtio_net use this address caused ept violation?

00:04.0 Ethernet controller: Unknown device 1af4:1000
	Subsystem: Unknown device 1af4:0001
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
	Interrupt: pin A routed to IRQ 11
	Region 0: I/O ports at c040 [size=32]
	Region 1: Memory at f2020000 (32-bit, non-prefetchable) [size=4K]
	Expansion ROM at f2030000 [disabled] [size=64K]
	Capabilities: [40] MSI-X: Enable+ Mask- TabSize=3
		Vector table: BAR=1 offset=00000000
		PBA: BAR=1 offset=00000800

00:08.0 Ethernet controller: Unknown device 1af4:1000
	Subsystem: Unknown device 1af4:0001
	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
	Interrupt: pin A routed to IRQ 11
	Region 0: I/O ports at c0c0 [size=32]
	Region 1: Memory at f20a0000 (32-bit, non-prefetchable) [size=4K]
	Expansion ROM at f20b0000 [disabled] [size=64K]
	Capabilities: [40] MSI-X: Enable+ Mask- TabSize=3
		Vector table: BAR=1 offset=00000000
		PBA: BAR=1 offset=00000800

2010/11/15, lidong chen <chen.lidong.kernel@gmail.com>:
> the gpa caused ept violation is below:
>
> most of them is F2020000.(4060217344)
>
> error gpa is 4060217344
> error gpa is 4060217344
> error gpa is 4060217344
> error gpa is 4060217344
> error gpa is 4060217344
> error gpa is 4060217344
> error gpa is 4060217344
> error gpa is 4060217344
> error gpa is 4060217344
> error gpa is 4060217344
>
> other is F20A0000(4060741632).
>
> error gpa is 4060741632
> error gpa is 4060741632
>
>
> 2010/11/15, lidong chen <chen.lidong.kernel@gmail.com>:
>> i use the virtio dirvier of rhle6, the guest os version is 2.6.16.
>>
>> the vm exits which caused by io instruction and apic access reduce a lot.
>> the total number of vm exits decreased by 50% :)
>>
>> but the ept violation seem increase on cpu 0, cpu 1, cpu 2, cpu3.
>>
>> (KVM)100000 times on cpu 2,
>> 0, 1989
>> 1, 9937
>> 7, 7951
>> 12, 2709
>> 28, 20
>> 30, 18536
>> 44, 29058
>> 48, 29800
>>
>> and not all cpu have this problem: cpu4,cpu5,cpu6,cpu7
>> 100000 times on cpu 5,
>> 0, 13934
>> 1, 35955
>> 7, 26
>> 12, 3841
>> 28, 11224
>> 30, 29389
>> 44, 5631
>>
>> I bind vcpu like this:
>>       virsh vcpupin brd1vm4 0 7
>>     virsh vcpupin brd3vm4 0 0
>>     virsh vcpupin brd3vm4 1 4
>>     virsh vcpupin brd5vm4 0 1
>>     virsh vcpupin brd5vm4 1 5
>>     virsh vcpupin brd9vm4 0 2
>>     virsh vcpupin brd9vm4 1 6
>>     virsh vcpupin brd11vm4 0 3
>>     virsh vcpupin brd11vm4 1 7
>>
>> the /proc/interrupt of guest os is below:
>>            CPU0       CPU1
>>   0:    1896802          0    IO-APIC-edge  timer
>>   1:          8          0    IO-APIC-edge  i8042
>>   4:         14          0    IO-APIC-edge  serial
>>   8:          0          0    IO-APIC-edge  rtc
>>   9:          0          0   IO-APIC-level  acpi
>>  10:          0          0   IO-APIC-level  virtio1, virtio2, virtio5
>>  11:          1          0   IO-APIC-level  virtio0, virtio3, virtio4
>>  12:        104          0    IO-APIC-edge  i8042
>>  50:          1          0       PCI-MSI-X  virtio2-output
>>  58:          0          0       PCI-MSI-X  virtio3-config
>>  66:    2046985          0       PCI-MSI-X  virtio3-input
>>  74:          2          0       PCI-MSI-X  virtio3-output
>>  82:          0          0       PCI-MSI-X  virtio4-config
>>  90:        217          0       PCI-MSI-X  virtio4-input
>>  98:          0          0       PCI-MSI-X  virtio4-output
>> 177:          0          0       PCI-MSI-X  virtio0-config
>> 185:     341831          0       PCI-MSI-X  virtio0-input
>> 193:          1          0       PCI-MSI-X  virtio0-output
>> 201:          0          0       PCI-MSI-X  virtio1-config
>> 209:     188747          0       PCI-MSI-X  virtio1-input
>> 217:          1          0       PCI-MSI-X  virtio1-output
>> 225:          0          0       PCI-MSI-X  virtio2-config
>> 233:    2204149          0       PCI-MSI-X  virtio2-input
>> NMI:    1455767    1426226
>> LOC:    1896099    1896637
>> ERR:          0
>> MIS:          0
>>
>> the application on all vcpu is the same. so i think the new pv driver
>> caused ept violation .
>>
>>
>> 2010/11/10 Avi Kivity <avi@redhat.com>:
>>> On 11/10/2010 09:09 AM, lidong chen wrote:
>>>>
>>>> after slove the ept violation problem, i found  the vm exits times is
>>>> still more than xen.
>>>>
>>>> and i found most of them is caused by io instruction. then i stat the
>>>> port number, most of them is caused by virio_net.
>>>
>>> This is also solved by the msi-capable virtio driver.
>>>
>>> --
>>> error compiling committee.c: too many arguments to function
>>>
>>>
>>
>

Re: Why so many vm exits caused by ept violation

[Avi Kivity]

On 11/15/2010 09:24 AM, lidong chen wrote:
> the address is the Region 1 of virtio_net.
>
> why virtio_net use this address caused ept violation?

It's probably the MSIX mask bit.  Older kernels program this bit twice 
on every interrupt.  Newer kernels do this much less frequently.  Try 
with a new kernel and see.

-- 
error compiling committee.c: too many arguments to function

Re: Why so many vm exits caused by ept violation

[lidong chen]

2010/11/15, Avi Kivity <avi@redhat.com>:
> On 11/15/2010 09:24 AM, lidong chen wrote:
>> the address is the Region 1 of virtio_net.
>>
>> why virtio_net use this address caused ept violation?
>
> It's probably the MSIX mask bit.  Older kernels program this bit twice
> on every interrupt.  Newer kernels do this much less frequently.  Try
> with a new kernel and see.
>
> --
> error compiling committee.c: too many arguments to function
>
>

Re: Why so many vm exits caused by ept violation

[lidong chen]

I think the address maybe initialized in function virtio_pci_probe.
err = pci_request_regions(pci_dev, "virtio-pci");

but i did not know when used this address.


2010/11/15, lidong chen <chen.lidong.kernel@gmail.com>:
> 2010/11/15, Avi Kivity <avi@redhat.com>:
>> On 11/15/2010 09:24 AM, lidong chen wrote:
>>> the address is the Region 1 of virtio_net.
>>>
>>> why virtio_net use this address caused ept violation?
>>
>> It's probably the MSIX mask bit.  Older kernels program this bit twice
>> on every interrupt.  Newer kernels do this much less frequently.  Try
>> with a new kernel and see.
>>
>> --
>> error compiling committee.c: too many arguments to function
>>
>>
>

Re: Why so many vm exits caused by ept violation

[lidong chen]

it ready caused by the the MSIX mask bit.

so i disable the msix in the newer virtio driver, and i found it have
better performent than msix enabled. the cpu rate of host os reduce.

the reason maybe is ept violation.

I couldn't use newer kernels. because the purpose i used kvm is run
old software on new hardware.

but i think this is a common problem, maybe newer kernel also exist.

i think we should not enable msi for virtio.

if anybody also interest on this problem, you can do the same test.

I disalbe msi by modify the function  vp_find_vqs like this:

static int vp_find_vqs(struct virtio_device *vdev, unsigned nvqs,
		       struct virtqueue *vqs[],
		       vq_callback_t *callbacks[],
		       const char *names[])
{

#if 0
	int err;

	/* Try MSI-X with one vector per queue. */
	err = vp_try_to_find_vqs(vdev, nvqs, vqs, callbacks, names, true, true);
	if (!err)
		return 0;
	/* Fallback: MSI-X with one vector for config, one shared for queues. */
	err = vp_try_to_find_vqs(vdev, nvqs, vqs, callbacks, names,
				 true, false);
	if (!err)
		return 0;
	/* Finally fall back to regular interrupts. */
#endif

	return vp_try_to_find_vqs(vdev, nvqs, vqs, callbacks, names,
				  false, false);
}



2010/11/15, lidong chen <chen.lidong.kernel@gmail.com>:
> I think the address maybe initialized in function virtio_pci_probe.
> err = pci_request_regions(pci_dev, "virtio-pci");
>
> but i did not know when used this address.
>
>
> 2010/11/15, lidong chen <chen.lidong.kernel@gmail.com>:
>> 2010/11/15, Avi Kivity <avi@redhat.com>:
>>> On 11/15/2010 09:24 AM, lidong chen wrote:
>>>> the address is the Region 1 of virtio_net.
>>>>
>>>> why virtio_net use this address caused ept violation?
>>>
>>> It's probably the MSIX mask bit.  Older kernels program this bit twice
>>> on every interrupt.  Newer kernels do this much less frequently.  Try
>>> with a new kernel and see.
>>>
>>> --
>>> error compiling committee.c: too many arguments to function
>>>
>>>
>>
>