Nvidia, Xen, and Vt-d

[Michael J Coss <mjcoss@alcatel-lucent.com>]

If this is the right list I apologies in advance, but I am a kernel 
developer, and I'm not afraid of getting my hands dirty so...

Like a number of others, I want to play with Vt-d and graphics pass
through.  But first I need to just get Xen up and running. with a dom0
and an operational X server.  I've been trying various combinations
looking for some way to have a working dom0.  The hardware is a ASUS
p6t7 w/ 2 Nvidia GTX 295 graphics cards.  Each card occupies 2 pci 
slots, and have 2 GPUs plus quite a bit of memory.

I keep getting the follow X server crash when I boot Xen and a dom0 kerenl.

> [54149.822514] X: Corrupted page table at address b61a1000
> [54149.822519] *pdpt = 0000000027690001
> [54149.822525] Bad pagetable: 000f [#1] SMP
> [54149.822531] last sysfs file: 
> /sys/devices/pci0000:00/0000:00:1c.5/0000:10:00.0/resource
> [54149.822535] Modules linked in: nvidia(P) [last unloaded: nvidia]
> [54149.822544]
> [54149.822548] Pid: 7148, comm: X Tainted: P           (2.6.30-rc3-tip 
> #2) System Product Name
> [54149.822552] EIP: 0073:[<b622da45>] EFLAGS: 00210202 CPU: 0
> [54149.822555] EIP is at 0xb622da45
> [54149.822558] EAX: b61a1000 EBX: b6580950 ECX: 0979a410 EDX: 00000001
> [54149.822561] ESI: 00000020 EDI: 097961e0 EBP: 0979a410 ESP: bfd3cec8
> [54149.822565]  DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b
> [54149.822568] Process X (pid: 7148, ti=e0c38000 task=e1615d60 
> task.ti=e0c38000)
> [54149.822571]
> [54149.822574] EIP: [<b622da45>] 0xb622da45 SS:ESP 007b:bfd3cec8
> [54149.822583] ---[ end trace 410a0a71e695876c ]---

Running strace on X, I see that it's crashing doing a series of mmap2.
The other odd thing is that if I cat the card files
(/proc/driver/nvidia/cards/0), it reports the video bios as ??:??:??:??. 
  It looks like either the Nvidia driver is poking around the system
using non-standard interfaces or somehow the presented pci interface to
the cards is broken somehow.  Turning on or off the Vt-d support in the
bios makes no difference, and if I boot the system without the Xen
hypervisor, the video bios is reported correctly, and the driver works
just fine.

I've compared the lspci output with and without the Xen hypervisor, and 
there are a number of discrepancies.  Notably in the capabilities listed 
for the graphics cards.  Capability 78 is misreported as Express (v1) 
instead of Express (v2).  I'm not sure what to make of this.

Unfortunately, I don't have any other older Nvidia cards to see if they
behave differently.

I've tried 3.4.1, and the lastest xen-unstable.  I've tried the pv-ops
git tree as I really need a 2.6.31 dom0 kernel for other reasons, but
for the moment I'd just like to get to the point where I have Xen, and
dom0 and Nvidia playing nicely with one another, so I can move on to
working on Vt-d and graphic pass through.

Any suggestions?

---Michael J Coss