lapic & npt

lapic & npt

[Gerd Hoffmann]

  Hi,

I've just noticed that xenner doesn't work for 64bit xen guest kernels
on my new shiny barcelona box.  The VM crashes when trying to access the
lapic.  lapic setup is done before idt setup, register dump looks like
init state, thus it most likely is a triple fault resetting the vcpu.
Turning off npt fixes it.

xenner maps the lapic to ffff820000000000 (64bit guests) or fe900000
(32bit guests).  32bit works fine even with npt enabled.

host kernel is fedora 9 with kvm-69 modules.

ideas anyone?

cheers,
  Gerd

-- 
http://kraxel.fedorapeople.org/xenner/

Re: lapic & npt

[Avi Kivity]

Gerd Hoffmann wrote:
>   Hi,
>
> I've just noticed that xenner doesn't work for 64bit xen guest kernels
> on my new shiny barcelona box.  The VM crashes when trying to access the
> lapic.  lapic setup is done before idt setup, register dump looks like
> init state, thus it most likely is a triple fault resetting the vcpu.
> Turning off npt fixes it.
>
> xenner maps the lapic to ffff820000000000 (64bit guests) or fe900000
> (32bit guests).  32bit works fine even with npt enabled.
>
> host kernel is fedora 9 with kvm-69 modules.
>
> ideas anyone?
>
>   

Turn on logging in lapic.c.  See if something turns up.

If not, the page tables are probably set up incorrectly, but in a way 
that kvm doesn't notice.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Vringfd

[Dong, Eddie]

Rusty/Anthony/Avi and all:
	I did a read at Rusty's April 18 post of vringfd patch, though I
didn't find explicit code for how to call get_user_skb_frags, it seems
to be clear the user only need to trigger a "flush_tx" command. All the
rest will be automatically turned on, fetching the ring and send to
bridge etc, which is perfect from performance point of view.
	Now I am working on enabling VMDq which provide HW pre-sorting
mechanism base on MAC address, it can definetely take the advantage of
vringfd with additional advantage of 0-copy in receive side, which makes
VMDq enabled virtio-net fly with 0-copy for both transmit & receive. The
only thing an VMDq agency (vringfd similar components in kernel) may
bother user level is initialization and notification given that vringfd
already solved the ring operations, since KVM virtual interrupt module
is in kernel side, it is pretty easy to make user free of receiving
packet at run time.
	Any comments?
Thanks, eddie

Re: lapic & npt

[Gerd Hoffmann]

Avi Kivity wrote:
> Gerd Hoffmann wrote:
>>   Hi,
>>
>> I've just noticed that xenner doesn't work for 64bit xen guest kernels
>> on my new shiny barcelona box.  The VM crashes when trying to access the
>> lapic.  lapic setup is done before idt setup, register dump looks like
>> init state, thus it most likely is a triple fault resetting the vcpu.
>> Turning off npt fixes it.
>>
>> xenner maps the lapic to ffff820000000000 (64bit guests) or fe900000
>> (32bit guests).  32bit works fine even with npt enabled.
>>
>> host kernel is fedora 9 with kvm-69 modules.
>>
>> ideas anyone?
> 
> Turn on logging in lapic.c.  See if something turns up.
> 
> If not, the page tables are probably set up incorrectly, but in a way
> that kvm doesn't notice.

Figured meanwhile the instruction accessing the lapic register somehow
beames me into my gpf handler.  With a quite strange rsp value.  Hmm.
To be analyzed further after kvm forum I guess, unless someone has a
bright idea ...

cheers,
  Gerd

-- 
http://kraxel.fedorapeople.org/xenner/

Re: lapic & npt

[Avi Kivity]

Gerd Hoffmann wrote:
> Avi Kivity wrote:
>   
>> Gerd Hoffmann wrote:
>>     
>>>   Hi,
>>>
>>> I've just noticed that xenner doesn't work for 64bit xen guest kernels
>>> on my new shiny barcelona box.  The VM crashes when trying to access the
>>> lapic.  lapic setup is done before idt setup, register dump looks like
>>> init state, thus it most likely is a triple fault resetting the vcpu.
>>> Turning off npt fixes it.
>>>
>>> xenner maps the lapic to ffff820000000000 (64bit guests) or fe900000
>>> (32bit guests).  32bit works fine even with npt enabled.
>>>
>>> host kernel is fedora 9 with kvm-69 modules.
>>>
>>> ideas anyone?
>>>       
>> Turn on logging in lapic.c.  See if something turns up.
>>
>> If not, the page tables are probably set up incorrectly, but in a way
>> that kvm doesn't notice.
>>     
>
> Figured meanwhile the instruction accessing the lapic register somehow
> beames me into my gpf handler.  With a quite strange rsp value.  Hmm.
> To be analyzed further after kvm forum I guess, unless someone has a
> bright idea ...
>   

x86-64 will gpf if an address is not canonical.  ffff820000000000 is 
canonical, but perhaps a type killed a zero somewhere?

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: lapic & npt

[Gerd Hoffmann]

  Hi,

>> Figured meanwhile the instruction accessing the lapic register somehow
>> beames me into my gpf handler.  With a quite strange rsp value.  Hmm.
>> To be analyzed further after kvm forum I guess, unless someone has a
>> bright idea ...

> x86-64 will gpf if an address is not canonical.  ffff820000000000 is
> canonical, but perhaps a type killed a zero somewhere?

Doesn't look like.  Tried to analyze that.  Compiled lastest kvm.git
kernel (to get kvmtrace support), booted, and noticed "kvm-amd npt=0"
doesn't work at all ...

cheers,
  Gerd

-- 
http://kraxel.fedorapeople.org/xenner/

Re: lapic & npt

[Gerd Hoffmann]

[-- Attachment #1: Type: text/plain, Size: 1394 bytes --]

Avi Kivity wrote:
> Gerd Hoffmann wrote:
>>   Hi,
>>
>> I've just noticed that xenner doesn't work for 64bit xen guest kernels
>> on my new shiny barcelona box.  The VM crashes when trying to access the
>> lapic.  lapic setup is done before idt setup, register dump looks like
>> init state, thus it most likely is a triple fault resetting the vcpu.
>> Turning off npt fixes it.
>>
>> xenner maps the lapic to ffff820000000000 (64bit guests) or fe900000
>> (32bit guests).  32bit works fine even with npt enabled.
>>
>> host kernel is fedora 9 with kvm-69 modules.
>>
>> ideas anyone?
> 
> Turn on logging in lapic.c.  See if something turns up.

It doesn't come that far according to kvmtrace.

> If not, the page tables are probably set up incorrectly, but in a way
> that kvm doesn't notice.

Dunno.  Tried kvmtrace and got the results attached without and with
NTP.  Cutted down to the important sequence, starting with the apic MSR
read.  After the msr access there are some page faults (some page table
pages are allocated and used to map the apic, thus likely triggering
shadow pt updates).  Then the apic access (id register).

With npt kvm doesn't see the apic access at all.  Also the TDP_FAULT
line looks fishy to me.  The "virt = ... " isn't a (guest) virtual address.

btw: the iowrite to 0x00ea is console output.

ideas anyone?
  Gerd

-- 
http://kraxel.fedorapeople.org/xenner/

[-- Attachment #2: no-npt --]
[-- Type: text/plain, Size: 1705 bytes --]

1673786946412 (+    6068)  VMEXIT        vcpu = 0x00000000  pid = 0x0000102d [ exitcode = 0x0000007c, rip = 0xffff8300 0000a1d2 ]
0 (+       0)  MSR_READ      vcpu = 0x00000000  pid = 0x0000102d [ MSR# = 0x0000001b, data = 0x00000000 fee00900 ]
1673786950748 (+    4336)  VMENTRY       vcpu = 0x00000000  pid = 0x0000102d
1673786957068 (+    6320)  VMEXIT        vcpu = 0x00000000  pid = 0x0000102d [ exitcode = 0x00000003, rip = 0xffff8300 00006ceb ]
0 (+       0)  CR_READ       vcpu = 0x00000000  pid = 0x0000102d [ CR# = 3, value = 0x00000000 017a3000 ]
1673786966522 (+    9454)  VMENTRY       vcpu = 0x00000000  pid = 0x0000102d
1673786973678 (+    7156)  VMEXIT        vcpu = 0x00000000  pid = 0x0000102d [ exitcode = 0x0000004e, rip = 0xffff8300 00004023 ]
0 (+       0)  PAGE_FAULT    vcpu = 0x00000000  pid = 0x0000102d [ errorcode = 0x00000002, virt = 0xffff8300 00048000 ]
1673787002404 (+   28726)  VMENTRY       vcpu = 0x00000000  pid = 0x0000102d
1673787034034 (+   31630)  VMEXIT        vcpu = 0x00000000  pid = 0x0000102d [ exitcode = 0x0000004e, rip = 0xffff8300 00004023 ]
0 (+       0)  PAGE_FAULT    vcpu = 0x00000000  pid = 0x0000102d [ errorcode = 0x00000002, virt = 0xffff8300 00049000 ]
1673787049388 (+   15354)  VMENTRY       vcpu = 0x00000000  pid = 0x0000102d
1673787080890 (+   31502)  VMEXIT        vcpu = 0x00000000  pid = 0x0000102d [ exitcode = 0x0000004e, rip = 0xffff8300 0000a1ef ]
0 (+       0)  PAGE_FAULT    vcpu = 0x00000000  pid = 0x0000102d [ errorcode = 0x00000000, virt = 0xffff8200 00000020 ]
0 (+       0)  APIC_ACCESS   vcpu = 0x00000000  pid = 0x0000102d [ offset = 0x00000020 ]
1673787096270 (+   15380)  VMENTRY       vcpu = 0x00000000  pid = 0x0000102d

[-- Attachment #3: npt --]
[-- Type: text/plain, Size: 1603 bytes --]

1834321779170 (+    6312)  VMEXIT        vcpu = 0x00000000  pid = 0x00001080 [ exitcode = 0x0000007c, rip = 0xffff8300 0000a1d2 ]
0 (+       0)  MSR_READ      vcpu = 0x00000000  pid = 0x00001080 [ MSR# = 0x0000001b, data = 0x00000000 fee00900 ]
1834321783872 (+    4702)  VMENTRY       vcpu = 0x00000000  pid = 0x00001080
1834321793790 (+    9918)  VMEXIT        vcpu = 0x00000000  pid = 0x00001080 [ exitcode = 0x00000400, rip = 0xffff8300 00004023 ]
0 (+       0)  TDP_FAULT     vcpu = 0x00000000  pid = 0x00001080 [ errorcode = 0x00000006, virt = 0x00000000 00048000 ]
1834321818582 (+   24792)  VMENTRY       vcpu = 0x00000000  pid = 0x00001080
1834321851480 (+   32898)  VMEXIT        vcpu = 0x00000000  pid = 0x00001080 [ exitcode = 0x00000400, rip = 0xffff8300 00004023 ]
0 (+       0)  TDP_FAULT     vcpu = 0x00000000  pid = 0x00001080 [ errorcode = 0x00000006, virt = 0x00000000 00049000 ]
1834321870858 (+   19378)  VMENTRY       vcpu = 0x00000000  pid = 0x00001080
1834321906650 (+   35792)  VMEXIT        vcpu = 0x00000000  pid = 0x00001080 [ exitcode = 0x00000400, rip = 0xffff8300 000053b1 ]
0 (+       0)  TDP_FAULT     vcpu = 0x00000000  pid = 0x00001080 [ errorcode = 0x00000004, virt = 0x00000000 0000d000 ]
1834321912818 (+    6168)  VMENTRY       vcpu = 0x00000000  pid = 0x00001080
1834321931382 (+   18564)  VMEXIT        vcpu = 0x00000000  pid = 0x00001080 [ exitcode = 0x0000007b, rip = 0xffff8300 0000b256 ]
0 (+       0)  IO_WRITE      vcpu = 0x00000000  pid = 0x00001080 [ port = 0x00ea, size = 1 ]
1834322090822 (+  159440)  VMENTRY       vcpu = 0x00000000  pid = 0x00001080