* PV PCI pass-through with ixgbe ?
@ 2009-04-06 17:04 Fischer, Anna
2009-04-07 9:48 ` Rolf Neugebauer
0 siblings, 1 reply; 2+ messages in thread
From: Fischer, Anna @ 2009-04-06 17:04 UTC (permalink / raw)
To: xen-devel@lists.xensource.com
Is it possible to use MSI-X in a PV Xen DomU (2.6.18.8) for a PCI pass-through device? I am running an Intel ixgbe NIC in DomU, but I do not seem to be able to run it with MSI-X enabled. In fact, I do not get it to work in any interrupt mode. The DomU/Dom0 kernel is compiled with PCI_MSI=y and I have msi=1 set in Xen's boot options.
I do seem to get an interrupt clash when running up the ixgbe device driver in DomU. Also, packets seem to be corrupted when I capture with tcpdump on the interfaces.
tcpdump -i eth1 -xx gives me this:
01:46:56.770226 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00 (oui Ethernet) Null Information, send seq 0, rcv seq 0, Flags [Command], length 46
0x0000: 0000 0000 0000 0000 0000 0000 0000 0000
0x0010: 0000 0000 0000 0000 0000 0000 0000 0000
0x0020: 0000 0000 0000 0000 0000 0000 0000 0000
0x0030: 0000 0000 0000 0000 0000 0000
01:46:56.973564 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00 (oui Ethernet) Null Information, send seq 0, rcv seq 0, Flags [Command], length 46
0x0000: 0000 0000 0000 0000 0000 0000 0000 0000
0x0010: 0000 0000 0000 0000 0000 0000 0000 0000
0x0020: 0000 0000 0000 0000 0000 0000 0000 0000
0x0030: 0000 0000 0000 0000 0000 0000
01:46:57.176902 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00 (oui Ethernet) Null Information, send seq 0, rcv seq 0, Flags [Command], length 46
0x0000: 0000 0000 0000 0000 0000 0000 0000 0000
0x0010: 0000 0000 0000 0000 0000 0000 0000 0000
0x0020: 0000 0000 0000 0000 0000 0000 0000 0000
0x0030: 0000 0000 0000 0000 0000 0000
01:46:57.380238 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00 (oui Ethernet) Null Information, send seq 0, rcv seq 0, Flags [Command], length 46
0x0000: 0000 0000 0000 0000 0000 0000 0000 0000
0x0010: 0000 0000 0000 0000 0000 0000 0000 0000
0x0020: 0000 0000 0000 0000 0000 0000 0000 0000
0x0030: 0000 0000 0000 0000 0000 0000
I have tried to run DomU/DomU with the irqpoll boot parameter but when I do that then the DomU fails to boot and hangs with a kernel panic:
Intel(R) 10 Gigabit PCI Express Network Driver - version 1.3.31.5
Copyright (c) 1999-2008 Intel Corporation.
PCI: Enabling device 0000:02:00.0 (0000 -> 0003)
ixgbe: Interrupt Type set to 1
ixgbe: 0000:02:00.0: ixgbe_check_options: Multiple queues are not supported while MSI-X is disabled. Disabling Multiple Queues.
ixgbe: 0000:02:00.0: ixgbe_check_options: RSS is not supported while multiple queues are disabled. Disabling RSS.
ixgbe: 0000:02:00.0: ixgbe_init_interrupt_scheme: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1
ixgbe: 0000:02:00.0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) 00:21:f7:58:01:6d
ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection
PCI: Enabling device 0000:02:00.1 (0000 -> 0003)
BUG: unable to handle kernel NULL pointer dereference at virtual address 0000002d
printing eip:
c01fccaa
2c8fa000 -> *pde = 00000000:60b62001
2cfc1000 -> *pme = 00000000:00000000
Oops: 0002 [#1]
SMP
Modules linked in: ixgbe aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd usbcore
CPU: 0
EIP: 0061:[<c01fccaa>] Not tainted VLI
EFLAGS: 00210246 (2.6.18.8-xen #5)
EIP is at pci_enable_msix+0x1da/0x510
eax: ed717800 ebx: 00000011 ecx: 00000000 edx: 00000004
esi: 00000000 edi: ed717800 ebp: ec951f80 esp: ecab9d54
ds: 007b es: 007b ss: 0069
Process modprobe (pid: 1566, ti=ecab8000 task=eccd2e10 task.ti=ecab8000)
Stack: 00000098 ecfd4400 c036d220 c015f905 ecfd4400 ecfd4400 00000004 ee0e639a
00000000 00000000 00000008 ec951f60 000000b2 00000002 00030003 0000007b
0000007b 00000000 c017682e ec951f80 00000004 00000010 ecfd4400 00000000
Call Trace:
[<c015f905>] __kzalloc+0x15/0x50
[<ee0e639a>] ixgbe_alloc_queues+0x8a/0x580 [ixgbe]
[<c017682e>] __kmalloc+0xbe/0xf0
[<ee0e6a13>] ixgbe_init_interrupt_scheme+0x183/0x340 [ixgbe]
[<ee0e3b81>] ixgbe_read_pci_cfg_word+0x21/0x30 [ixgbe]
[<ee0e8c17>] ixgbe_probe+0x6e7/0x9e0 [ixgbe]
[<c017682e>] __kmalloc+0xbe/0xf0
[<c01f5ef6>] pci_device_probe+0x56/0x80
[<c023ecd4>] driver_probe_device+0x44/0xc0
[<c023ee52>] __driver_attach+0x82/0x90
[<c023e64a>] bus_for_each_dev+0x3a/0x60
[<c023ec16>] driver_attach+0x16/0x20
[<c023edd0>] __driver_attach+0x0/0x90
[<c023e2bc>] bus_add_driver+0x8c/0x140
[<c01f6097>] __pci_register_driver+0x47/0x70
[<c0143b08>] sys_init_module+0x148/0x1b40
[<c01ea55f>] prio_tree_insert+0x1f/0x260
[<c01187fc>] do_page_fault+0x10c/0xc6f
[<c010acc0>] sys_mmap2+0xd0/0xe0
[<c01059bf>] syscall_call+0x7/0xb
Code: 00 89 54 24 38 ba ed ff ff ff 89 4c 24 3c 0b 4c 24 38 0f 85 2d 01 00 00 8b 4c 24 28 89 d6 89 8f b0 01 00 00 e9 62 fe ff ff 89 f8 <89> 5b 1c e8 3e f2 ff ff 8b 7c 24 18 89 44 24 54 85 ff 0f 8e 48
EIP: [<c01fccaa>] pci_enable_msix+0x1da/0x510 SS:ESP 0069:ecab9d54
udevd-event[1564]: run_program: '/sbin/modprobe' abnormal exit
I have also tried to use legacy interrupts by setting InterruptType to 0:
Intel(R) 10 Gigabit PCI Express Network Driver - version 1.3.31.5-lro
Copyright (c) 1999-2008 Intel Corporation.
PCI: Enabling device 0000:02:00.0 (0000 -> 0003)
PCI: Setting latency timer of device 0000:02:00.0 to 64
ixgbe: Interrupt Type set to 0
ixgbe: Multiple Queue Support Disabled
ixgbe: Receive-Side Scaling (RSS) set to 0
ixgbe: Virtual Machine Device Queues (VMDQ) set to 0
ixgbe: 0000:02:00.0: ixgbe_init_interrupt_scheme: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1
ixgbe: 0000:02:00.0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) 00:21:f7:58:01:6d
ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection
PCI: Enabling device 0000:02:00.1 (0000 -> 0003)
PCI: Setting latency timer of device 0000:02:00.1 to 64
ixgbe: Interrupt Type set to 0
ixgbe: Multiple Queue Support Disabled
ixgbe: Receive-Side Scaling (RSS) set to 0
ixgbe: Virtual Machine Device Queues (VMDQ) set to 0
ixgbe: 0000:02:00.1: ixgbe_init_interrupt_scheme: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1
ixgbe: 0000:02:00.1: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) 00:21:f7:58:01:6c
ixgbe: eth1: ixgbe_probe: Intel(R) 10 Gigabit Network Connection
Without MSI and MSI-X the driver loads properly. However, I still end up with corrupted packets on tcpdump, and additionally I am getting a kernel crash when the guest is under high network load:
BUG: unable to handle kernel NULL pointer dereference at virtual address 00000034
printing eip:
2c934000 -> *pde = 00000000:60f1a001
2c8e7000 -> *pme = 00000000:00000000
Oops: 0002 [#1]
SMP
Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod ixgbe aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd usbcore
CPU: 0
EIP: 0061:[<c0158df7>] Not tainted VLI
EFLAGS: 00010082 (2.6.18.8-xen #5)
BUG: unable to handle kernel paging request at virtual address 245c8bf1
printing eip:
c0145df6
2c934000 -> *pde = 00000000:60f1a001
2c8e7000 -> *pme = 00000000:00000000
Oops: 0002 [#2]
SMP
Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod ixgbe aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd usbcore
CPU: 0
EIP: 0061:[<c0145df6>] Not tainted VLI
EFLAGS: 00010002 (2.6.18.8-xen #5)
BUG: unable to handle kernel paging request at virtual address 245c8bf1
printing eip:
c0145df6
2c934000 -> *pde = 00000000:60f1a001
2c8e7000 -> *pme = 00000000:00000000
Recursive die() failure, output suppressed
<0>Kernel panic - not syncing: Fatal exception in interrupt
BUG: warning at /home/user/Download/linux-2.6.18-xen.hg/arch/i386/kernel/smp-xen.c:511/smp_call_function()
[<c0112541>] <1>BUG: unable to handle kernel paging request at virtual address 245c8bf1
printing eip:
c0145df6
2c934000 -> *pde = 00000000:60f1a001
2c8e7000 -> *pme = 00000000:00000000
Oops: 0002 [#3]
SMP
Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod ixgbe aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd usbcore
CPU: 0
EIP: 0061:[<c0145df6>] Not tainted VLI
EFLAGS: 00010002 (2.6.18.8-xen #5)
BUG: unable to handle kernel paging request at virtual address 245c8bf1
printing eip:
c0145df6
2c934000 -> *pde = 00000000:60f1a001
2c8e7000 -> *pme = 00000000:00000000
Oops: 0002 [#4]
SMP
Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod ixgbe aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd usbcore
CPU: 0
EIP: 0061:[<c0145df6>] Not tainted VLI
EFLAGS: 00010002 (2.6.18.8-xen #5)
BUG: unable to handle kernel paging request at virtual address 245c8bf1
printing eip:
c0145df6
2c934000 -> *pde = 00000000:60f1a001
2c8e7000 -> *pme = 00000000:00000000
Recursive die() failure, output suppressed
<0>Kernel panic - not syncing: Fatal exception in interrupt
<1>BUG: unable to handle kernel NULL pointer dereference at virtual address 00000064
printing eip:
c0158df7
2d2e1000 -> *pde = 00000000:6051f001
2d2e2000 -> *pme = 00000000:00000000
Oops: 0002 [#5]
SMP
Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod ixgbe aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd usbcore
CPU: 1
EIP: 0061:[<c0158df7>] Not tainted VLI
EFLAGS: 00010282 (2.6.18.8-xen #5)
BUG: unable to handle kernel paging request at virtual address 245c8bf1
printing eip:
c0145df6
2d2e1000 -> *pde = 00000000:6051f001
2d2e2000 -> *pme = 00000000:00000000
Oops: 0002 [#6]
SMP
Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod ixgbe aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd usbcore
CPU: 1
EIP: 0061:[<c0145df6>] Not tainted VLI
EFLAGS: 00010002 (2.6.18.8-xen #5)
BUG: unable to handle kernel paging request at virtual address 245c8bf1
printing eip:
c0145df6
2d2e1000 -> *pde = 00000000:6051f001
2d2e2000 -> *pme = 00000000:00000000
Recursive die() failure, output suppressed
What are the correct interrupt settings for using ixgbe in PV pass-through mode? I have tried all three possible modes but I do not seem to get the device working properly under high network load.
I am using Xen 3.3.1.
release : 2.6.18.8-xen
version : #5 SMP Thu Mar 5 04:14:30 PST 2009
machine : i686
nr_cpus : 2
nr_nodes : 1
cores_per_socket : 2
threads_per_core : 1
cpu_mhz : 2200
virt_caps : hvm
total_memory : 2039
free_memory : 6
node_to_cpu : node0:0-1
node_to_memory : node0:6
xen_major : 3
xen_minor : 3
xen_extra : .1
xen_caps : xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xf5800000
xen_changeset : unavailable
cc_compiler : gcc version 4.1.2 20070925 (Red Hat 4.1.2-33)
cc_compile_by : root
cc_compile_domain : localdomain
cc_compile_date : Sun Mar 8 05:40:51 PDT 2009
xend_config_format : 4
I am using PCI pass-through to assign the NIC to my PV guest, so this is the part of my config file:
pci = [ '02:00.0','02:00.1' ]
I am also hiding the PCI devices from Dom0, so my Dom0 GRUB config looks like this:
title Xen 3.3
root (hd0,1)
kernel /boot/xen-3.3.1.gz console=vga msi=1
module /boot/vmlinuz-2.6.18.8-xen root=LABEL=/ ro console=tty0 rhgb quiet pciback.permissive pciback.hide=(02:00.0)(02:00.1) reassigndev=0000:02:00.0,0000:02:00.1
module /boot/initrd-2.6.18.8-xen.img
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: PV PCI pass-through with ixgbe ?
2009-04-06 17:04 PV PCI pass-through with ixgbe ? Fischer, Anna
@ 2009-04-07 9:48 ` Rolf Neugebauer
0 siblings, 0 replies; 2+ messages in thread
From: Rolf Neugebauer @ 2009-04-07 9:48 UTC (permalink / raw)
To: Fischer, Anna; +Cc: xen-devel@lists.xensource.com
This worked for me for a fairly recent version of xen-unstable (cs:
19429:6dcccd2dded6). I used the corresponding -xen kernel for domU. I
used a freshly compiled version of the ixgbe driver (1.3.56.5-NAPI).
MSI-X worked fine, though I've only tried ping, not bulk data transfers.
packets looked fine when I looked at it with tcpdump.
there were quite a few changes to the MSI code recently.
Rolf
Fischer, Anna wrote:
> Is it possible to use MSI-X in a PV Xen DomU (2.6.18.8) for a PCI pass-through device? I am running an Intel ixgbe NIC in DomU, but I do not seem to be able to run it with MSI-X enabled. In fact, I do not get it to work in any interrupt mode. The DomU/Dom0 kernel is compiled with PCI_MSI=y and I have msi=1 set in Xen's boot options.
>
> I do seem to get an interrupt clash when running up the ixgbe device driver in DomU. Also, packets seem to be corrupted when I capture with tcpdump on the interfaces.
>
> tcpdump -i eth1 -xx gives me this:
>
> 01:46:56.770226 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00 (oui Ethernet) Null Information, send seq 0, rcv seq 0, Flags [Command], length 46
> 0x0000: 0000 0000 0000 0000 0000 0000 0000 0000
> 0x0010: 0000 0000 0000 0000 0000 0000 0000 0000
> 0x0020: 0000 0000 0000 0000 0000 0000 0000 0000
> 0x0030: 0000 0000 0000 0000 0000 0000
> 01:46:56.973564 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00 (oui Ethernet) Null Information, send seq 0, rcv seq 0, Flags [Command], length 46
> 0x0000: 0000 0000 0000 0000 0000 0000 0000 0000
> 0x0010: 0000 0000 0000 0000 0000 0000 0000 0000
> 0x0020: 0000 0000 0000 0000 0000 0000 0000 0000
> 0x0030: 0000 0000 0000 0000 0000 0000
> 01:46:57.176902 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00 (oui Ethernet) Null Information, send seq 0, rcv seq 0, Flags [Command], length 46
> 0x0000: 0000 0000 0000 0000 0000 0000 0000 0000
> 0x0010: 0000 0000 0000 0000 0000 0000 0000 0000
> 0x0020: 0000 0000 0000 0000 0000 0000 0000 0000
> 0x0030: 0000 0000 0000 0000 0000 0000
> 01:46:57.380238 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00 (oui Ethernet) Null Information, send seq 0, rcv seq 0, Flags [Command], length 46
> 0x0000: 0000 0000 0000 0000 0000 0000 0000 0000
> 0x0010: 0000 0000 0000 0000 0000 0000 0000 0000
> 0x0020: 0000 0000 0000 0000 0000 0000 0000 0000
> 0x0030: 0000 0000 0000 0000 0000 0000
>
>
> I have tried to run DomU/DomU with the irqpoll boot parameter but when I do that then the DomU fails to boot and hangs with a kernel panic:
>
> Intel(R) 10 Gigabit PCI Express Network Driver - version 1.3.31.5
> Copyright (c) 1999-2008 Intel Corporation.
> PCI: Enabling device 0000:02:00.0 (0000 -> 0003)
> ixgbe: Interrupt Type set to 1
> ixgbe: 0000:02:00.0: ixgbe_check_options: Multiple queues are not supported while MSI-X is disabled. Disabling Multiple Queues.
> ixgbe: 0000:02:00.0: ixgbe_check_options: RSS is not supported while multiple queues are disabled. Disabling RSS.
> ixgbe: 0000:02:00.0: ixgbe_init_interrupt_scheme: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1
> ixgbe: 0000:02:00.0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) 00:21:f7:58:01:6d
> ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection
> PCI: Enabling device 0000:02:00.1 (0000 -> 0003)
> BUG: unable to handle kernel NULL pointer dereference at virtual address 0000002d
> printing eip:
> c01fccaa
> 2c8fa000 -> *pde = 00000000:60b62001
> 2cfc1000 -> *pme = 00000000:00000000
> Oops: 0002 [#1]
> SMP
> Modules linked in: ixgbe aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd usbcore
> CPU: 0
> EIP: 0061:[<c01fccaa>] Not tainted VLI
> EFLAGS: 00210246 (2.6.18.8-xen #5)
> EIP is at pci_enable_msix+0x1da/0x510
> eax: ed717800 ebx: 00000011 ecx: 00000000 edx: 00000004
> esi: 00000000 edi: ed717800 ebp: ec951f80 esp: ecab9d54
> ds: 007b es: 007b ss: 0069
> Process modprobe (pid: 1566, ti=ecab8000 task=eccd2e10 task.ti=ecab8000)
> Stack: 00000098 ecfd4400 c036d220 c015f905 ecfd4400 ecfd4400 00000004 ee0e639a
> 00000000 00000000 00000008 ec951f60 000000b2 00000002 00030003 0000007b
> 0000007b 00000000 c017682e ec951f80 00000004 00000010 ecfd4400 00000000
> Call Trace:
> [<c015f905>] __kzalloc+0x15/0x50
> [<ee0e639a>] ixgbe_alloc_queues+0x8a/0x580 [ixgbe]
> [<c017682e>] __kmalloc+0xbe/0xf0
> [<ee0e6a13>] ixgbe_init_interrupt_scheme+0x183/0x340 [ixgbe]
> [<ee0e3b81>] ixgbe_read_pci_cfg_word+0x21/0x30 [ixgbe]
> [<ee0e8c17>] ixgbe_probe+0x6e7/0x9e0 [ixgbe]
> [<c017682e>] __kmalloc+0xbe/0xf0
> [<c01f5ef6>] pci_device_probe+0x56/0x80
> [<c023ecd4>] driver_probe_device+0x44/0xc0
> [<c023ee52>] __driver_attach+0x82/0x90
> [<c023e64a>] bus_for_each_dev+0x3a/0x60
> [<c023ec16>] driver_attach+0x16/0x20
> [<c023edd0>] __driver_attach+0x0/0x90
> [<c023e2bc>] bus_add_driver+0x8c/0x140
> [<c01f6097>] __pci_register_driver+0x47/0x70
> [<c0143b08>] sys_init_module+0x148/0x1b40
> [<c01ea55f>] prio_tree_insert+0x1f/0x260
> [<c01187fc>] do_page_fault+0x10c/0xc6f
> [<c010acc0>] sys_mmap2+0xd0/0xe0
> [<c01059bf>] syscall_call+0x7/0xb
> Code: 00 89 54 24 38 ba ed ff ff ff 89 4c 24 3c 0b 4c 24 38 0f 85 2d 01 00 00 8b 4c 24 28 89 d6 89 8f b0 01 00 00 e9 62 fe ff ff 89 f8 <89> 5b 1c e8 3e f2 ff ff 8b 7c 24 18 89 44 24 54 85 ff 0f 8e 48
> EIP: [<c01fccaa>] pci_enable_msix+0x1da/0x510 SS:ESP 0069:ecab9d54
> udevd-event[1564]: run_program: '/sbin/modprobe' abnormal exit
>
>
> I have also tried to use legacy interrupts by setting InterruptType to 0:
>
> Intel(R) 10 Gigabit PCI Express Network Driver - version 1.3.31.5-lro
> Copyright (c) 1999-2008 Intel Corporation.
> PCI: Enabling device 0000:02:00.0 (0000 -> 0003)
> PCI: Setting latency timer of device 0000:02:00.0 to 64
> ixgbe: Interrupt Type set to 0
> ixgbe: Multiple Queue Support Disabled
> ixgbe: Receive-Side Scaling (RSS) set to 0
> ixgbe: Virtual Machine Device Queues (VMDQ) set to 0
> ixgbe: 0000:02:00.0: ixgbe_init_interrupt_scheme: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1
> ixgbe: 0000:02:00.0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) 00:21:f7:58:01:6d
> ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection
> PCI: Enabling device 0000:02:00.1 (0000 -> 0003)
> PCI: Setting latency timer of device 0000:02:00.1 to 64
> ixgbe: Interrupt Type set to 0
> ixgbe: Multiple Queue Support Disabled
> ixgbe: Receive-Side Scaling (RSS) set to 0
> ixgbe: Virtual Machine Device Queues (VMDQ) set to 0
> ixgbe: 0000:02:00.1: ixgbe_init_interrupt_scheme: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1
> ixgbe: 0000:02:00.1: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) 00:21:f7:58:01:6c
> ixgbe: eth1: ixgbe_probe: Intel(R) 10 Gigabit Network Connection
>
> Without MSI and MSI-X the driver loads properly. However, I still end up with corrupted packets on tcpdump, and additionally I am getting a kernel crash when the guest is under high network load:
>
> BUG: unable to handle kernel NULL pointer dereference at virtual address 00000034
> printing eip:
> 2c934000 -> *pde = 00000000:60f1a001
> 2c8e7000 -> *pme = 00000000:00000000
> Oops: 0002 [#1]
> SMP
> Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod ixgbe aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd usbcore
> CPU: 0
> EIP: 0061:[<c0158df7>] Not tainted VLI
> EFLAGS: 00010082 (2.6.18.8-xen #5)
> BUG: unable to handle kernel paging request at virtual address 245c8bf1
> printing eip:
> c0145df6
> 2c934000 -> *pde = 00000000:60f1a001
> 2c8e7000 -> *pme = 00000000:00000000
> Oops: 0002 [#2]
> SMP
> Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod ixgbe aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd usbcore
> CPU: 0
> EIP: 0061:[<c0145df6>] Not tainted VLI
> EFLAGS: 00010002 (2.6.18.8-xen #5)
> BUG: unable to handle kernel paging request at virtual address 245c8bf1
> printing eip:
> c0145df6
> 2c934000 -> *pde = 00000000:60f1a001
> 2c8e7000 -> *pme = 00000000:00000000
> Recursive die() failure, output suppressed
> <0>Kernel panic - not syncing: Fatal exception in interrupt
> BUG: warning at /home/user/Download/linux-2.6.18-xen.hg/arch/i386/kernel/smp-xen.c:511/smp_call_function()
> [<c0112541>] <1>BUG: unable to handle kernel paging request at virtual address 245c8bf1
> printing eip:
> c0145df6
> 2c934000 -> *pde = 00000000:60f1a001
> 2c8e7000 -> *pme = 00000000:00000000
> Oops: 0002 [#3]
> SMP
> Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod ixgbe aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd usbcore
> CPU: 0
> EIP: 0061:[<c0145df6>] Not tainted VLI
> EFLAGS: 00010002 (2.6.18.8-xen #5)
> BUG: unable to handle kernel paging request at virtual address 245c8bf1
> printing eip:
> c0145df6
> 2c934000 -> *pde = 00000000:60f1a001
> 2c8e7000 -> *pme = 00000000:00000000
> Oops: 0002 [#4]
> SMP
> Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod ixgbe aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd usbcore
> CPU: 0
> EIP: 0061:[<c0145df6>] Not tainted VLI
> EFLAGS: 00010002 (2.6.18.8-xen #5)
> BUG: unable to handle kernel paging request at virtual address 245c8bf1
> printing eip:
> c0145df6
> 2c934000 -> *pde = 00000000:60f1a001
> 2c8e7000 -> *pme = 00000000:00000000
> Recursive die() failure, output suppressed
> <0>Kernel panic - not syncing: Fatal exception in interrupt
> <1>BUG: unable to handle kernel NULL pointer dereference at virtual address 00000064
> printing eip:
> c0158df7
> 2d2e1000 -> *pde = 00000000:6051f001
> 2d2e2000 -> *pme = 00000000:00000000
> Oops: 0002 [#5]
> SMP
> Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod ixgbe aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd usbcore
> CPU: 1
> EIP: 0061:[<c0158df7>] Not tainted VLI
> EFLAGS: 00010282 (2.6.18.8-xen #5)
> BUG: unable to handle kernel paging request at virtual address 245c8bf1
> printing eip:
> c0145df6
> 2d2e1000 -> *pde = 00000000:6051f001
> 2d2e2000 -> *pme = 00000000:00000000
> Oops: 0002 [#6]
> SMP
> Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod ixgbe aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd usbcore
> CPU: 1
> EIP: 0061:[<c0145df6>] Not tainted VLI
> EFLAGS: 00010002 (2.6.18.8-xen #5)
> BUG: unable to handle kernel paging request at virtual address 245c8bf1
> printing eip:
> c0145df6
> 2d2e1000 -> *pde = 00000000:6051f001
> 2d2e2000 -> *pme = 00000000:00000000
> Recursive die() failure, output suppressed
>
> What are the correct interrupt settings for using ixgbe in PV pass-through mode? I have tried all three possible modes but I do not seem to get the device working properly under high network load.
>
> I am using Xen 3.3.1.
>
> release : 2.6.18.8-xen
> version : #5 SMP Thu Mar 5 04:14:30 PST 2009
> machine : i686
> nr_cpus : 2
> nr_nodes : 1
> cores_per_socket : 2
> threads_per_core : 1
> cpu_mhz : 2200
> virt_caps : hvm
> total_memory : 2039
> free_memory : 6
> node_to_cpu : node0:0-1
> node_to_memory : node0:6
> xen_major : 3
> xen_minor : 3
> xen_extra : .1
> xen_caps : xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p
> xen_scheduler : credit
> xen_pagesize : 4096
> platform_params : virt_start=0xf5800000
> xen_changeset : unavailable
> cc_compiler : gcc version 4.1.2 20070925 (Red Hat 4.1.2-33)
> cc_compile_by : root
> cc_compile_domain : localdomain
> cc_compile_date : Sun Mar 8 05:40:51 PDT 2009
> xend_config_format : 4
>
> I am using PCI pass-through to assign the NIC to my PV guest, so this is the part of my config file:
>
> pci = [ '02:00.0','02:00.1' ]
>
> I am also hiding the PCI devices from Dom0, so my Dom0 GRUB config looks like this:
>
> title Xen 3.3
> root (hd0,1)
> kernel /boot/xen-3.3.1.gz console=vga msi=1
> module /boot/vmlinuz-2.6.18.8-xen root=LABEL=/ ro console=tty0 rhgb quiet pciback.permissive pciback.hide=(02:00.0)(02:00.1) reassigndev=0000:02:00.0,0000:02:00.1
> module /boot/initrd-2.6.18.8-xen.img
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2009-04-07 9:48 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2009-04-06 17:04 PV PCI pass-through with ixgbe ? Fischer, Anna
2009-04-07 9:48 ` Rolf Neugebauer
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.