From mboxrd@z Thu Jan 1 00:00:00 1970 From: Rolf Neugebauer Subject: Re: PV PCI pass-through with ixgbe ? Date: Tue, 07 Apr 2009 10:48:43 +0100 Message-ID: <49DB217B.3000005@netronome.com> References: <0199E0D51A61344794750DC57738F58E66B941F2F8@GVW1118EXC.americas.hpqcorp.net> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <0199E0D51A61344794750DC57738F58E66B941F2F8@GVW1118EXC.americas.hpqcorp.net> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "Fischer, Anna" Cc: "xen-devel@lists.xensource.com" List-Id: xen-devel@lists.xenproject.org This worked for me for a fairly recent version of xen-unstable (cs: 19429:6dcccd2dded6). I used the corresponding -xen kernel for domU. I used a freshly compiled version of the ixgbe driver (1.3.56.5-NAPI). MSI-X worked fine, though I've only tried ping, not bulk data transfers. packets looked fine when I looked at it with tcpdump. there were quite a few changes to the MSI code recently. Rolf Fischer, Anna wrote: > Is it possible to use MSI-X in a PV Xen DomU (2.6.18.8) for a PCI pass-through device? I am running an Intel ixgbe NIC in DomU, but I do not seem to be able to run it with MSI-X enabled. In fact, I do not get it to work in any interrupt mode. The DomU/Dom0 kernel is compiled with PCI_MSI=y and I have msi=1 set in Xen's boot options. > > I do seem to get an interrupt clash when running up the ixgbe device driver in DomU. Also, packets seem to be corrupted when I capture with tcpdump on the interfaces. > > tcpdump -i eth1 -xx gives me this: > > 01:46:56.770226 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00 (oui Ethernet) Null Information, send seq 0, rcv seq 0, Flags [Command], length 46 > 0x0000: 0000 0000 0000 0000 0000 0000 0000 0000 > 0x0010: 0000 0000 0000 0000 0000 0000 0000 0000 > 0x0020: 0000 0000 0000 0000 0000 0000 0000 0000 > 0x0030: 0000 0000 0000 0000 0000 0000 > 01:46:56.973564 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00 (oui Ethernet) Null Information, send seq 0, rcv seq 0, Flags [Command], length 46 > 0x0000: 0000 0000 0000 0000 0000 0000 0000 0000 > 0x0010: 0000 0000 0000 0000 0000 0000 0000 0000 > 0x0020: 0000 0000 0000 0000 0000 0000 0000 0000 > 0x0030: 0000 0000 0000 0000 0000 0000 > 01:46:57.176902 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00 (oui Ethernet) Null Information, send seq 0, rcv seq 0, Flags [Command], length 46 > 0x0000: 0000 0000 0000 0000 0000 0000 0000 0000 > 0x0010: 0000 0000 0000 0000 0000 0000 0000 0000 > 0x0020: 0000 0000 0000 0000 0000 0000 0000 0000 > 0x0030: 0000 0000 0000 0000 0000 0000 > 01:46:57.380238 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00 (oui Ethernet) Null Information, send seq 0, rcv seq 0, Flags [Command], length 46 > 0x0000: 0000 0000 0000 0000 0000 0000 0000 0000 > 0x0010: 0000 0000 0000 0000 0000 0000 0000 0000 > 0x0020: 0000 0000 0000 0000 0000 0000 0000 0000 > 0x0030: 0000 0000 0000 0000 0000 0000 > > > I have tried to run DomU/DomU with the irqpoll boot parameter but when I do that then the DomU fails to boot and hangs with a kernel panic: > > Intel(R) 10 Gigabit PCI Express Network Driver - version 1.3.31.5 > Copyright (c) 1999-2008 Intel Corporation. > PCI: Enabling device 0000:02:00.0 (0000 -> 0003) > ixgbe: Interrupt Type set to 1 > ixgbe: 0000:02:00.0: ixgbe_check_options: Multiple queues are not supported while MSI-X is disabled. Disabling Multiple Queues. > ixgbe: 0000:02:00.0: ixgbe_check_options: RSS is not supported while multiple queues are disabled. Disabling RSS. > ixgbe: 0000:02:00.0: ixgbe_init_interrupt_scheme: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1 > ixgbe: 0000:02:00.0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) 00:21:f7:58:01:6d > ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection > PCI: Enabling device 0000:02:00.1 (0000 -> 0003) > BUG: unable to handle kernel NULL pointer dereference at virtual address 0000002d > printing eip: > c01fccaa > 2c8fa000 -> *pde = 00000000:60b62001 > 2cfc1000 -> *pme = 00000000:00000000 > Oops: 0002 [#1] > SMP > Modules linked in: ixgbe aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd usbcore > CPU: 0 > EIP: 0061:[] Not tainted VLI > EFLAGS: 00210246 (2.6.18.8-xen #5) > EIP is at pci_enable_msix+0x1da/0x510 > eax: ed717800 ebx: 00000011 ecx: 00000000 edx: 00000004 > esi: 00000000 edi: ed717800 ebp: ec951f80 esp: ecab9d54 > ds: 007b es: 007b ss: 0069 > Process modprobe (pid: 1566, ti=ecab8000 task=eccd2e10 task.ti=ecab8000) > Stack: 00000098 ecfd4400 c036d220 c015f905 ecfd4400 ecfd4400 00000004 ee0e639a > 00000000 00000000 00000008 ec951f60 000000b2 00000002 00030003 0000007b > 0000007b 00000000 c017682e ec951f80 00000004 00000010 ecfd4400 00000000 > Call Trace: > [] __kzalloc+0x15/0x50 > [] ixgbe_alloc_queues+0x8a/0x580 [ixgbe] > [] __kmalloc+0xbe/0xf0 > [] ixgbe_init_interrupt_scheme+0x183/0x340 [ixgbe] > [] ixgbe_read_pci_cfg_word+0x21/0x30 [ixgbe] > [] ixgbe_probe+0x6e7/0x9e0 [ixgbe] > [] __kmalloc+0xbe/0xf0 > [] pci_device_probe+0x56/0x80 > [] driver_probe_device+0x44/0xc0 > [] __driver_attach+0x82/0x90 > [] bus_for_each_dev+0x3a/0x60 > [] driver_attach+0x16/0x20 > [] __driver_attach+0x0/0x90 > [] bus_add_driver+0x8c/0x140 > [] __pci_register_driver+0x47/0x70 > [] sys_init_module+0x148/0x1b40 > [] prio_tree_insert+0x1f/0x260 > [] do_page_fault+0x10c/0xc6f > [] sys_mmap2+0xd0/0xe0 > [] syscall_call+0x7/0xb > Code: 00 89 54 24 38 ba ed ff ff ff 89 4c 24 3c 0b 4c 24 38 0f 85 2d 01 00 00 8b 4c 24 28 89 d6 89 8f b0 01 00 00 e9 62 fe ff ff 89 f8 <89> 5b 1c e8 3e f2 ff ff 8b 7c 24 18 89 44 24 54 85 ff 0f 8e 48 > EIP: [] pci_enable_msix+0x1da/0x510 SS:ESP 0069:ecab9d54 > udevd-event[1564]: run_program: '/sbin/modprobe' abnormal exit > > > I have also tried to use legacy interrupts by setting InterruptType to 0: > > Intel(R) 10 Gigabit PCI Express Network Driver - version 1.3.31.5-lro > Copyright (c) 1999-2008 Intel Corporation. > PCI: Enabling device 0000:02:00.0 (0000 -> 0003) > PCI: Setting latency timer of device 0000:02:00.0 to 64 > ixgbe: Interrupt Type set to 0 > ixgbe: Multiple Queue Support Disabled > ixgbe: Receive-Side Scaling (RSS) set to 0 > ixgbe: Virtual Machine Device Queues (VMDQ) set to 0 > ixgbe: 0000:02:00.0: ixgbe_init_interrupt_scheme: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1 > ixgbe: 0000:02:00.0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) 00:21:f7:58:01:6d > ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection > PCI: Enabling device 0000:02:00.1 (0000 -> 0003) > PCI: Setting latency timer of device 0000:02:00.1 to 64 > ixgbe: Interrupt Type set to 0 > ixgbe: Multiple Queue Support Disabled > ixgbe: Receive-Side Scaling (RSS) set to 0 > ixgbe: Virtual Machine Device Queues (VMDQ) set to 0 > ixgbe: 0000:02:00.1: ixgbe_init_interrupt_scheme: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1 > ixgbe: 0000:02:00.1: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) 00:21:f7:58:01:6c > ixgbe: eth1: ixgbe_probe: Intel(R) 10 Gigabit Network Connection > > Without MSI and MSI-X the driver loads properly. However, I still end up with corrupted packets on tcpdump, and additionally I am getting a kernel crash when the guest is under high network load: > > BUG: unable to handle kernel NULL pointer dereference at virtual address 00000034 > printing eip: > 2c934000 -> *pde = 00000000:60f1a001 > 2c8e7000 -> *pme = 00000000:00000000 > Oops: 0002 [#1] > SMP > Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod ixgbe aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd usbcore > CPU: 0 > EIP: 0061:[] Not tainted VLI > EFLAGS: 00010082 (2.6.18.8-xen #5) > BUG: unable to handle kernel paging request at virtual address 245c8bf1 > printing eip: > c0145df6 > 2c934000 -> *pde = 00000000:60f1a001 > 2c8e7000 -> *pme = 00000000:00000000 > Oops: 0002 [#2] > SMP > Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod ixgbe aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd usbcore > CPU: 0 > EIP: 0061:[] Not tainted VLI > EFLAGS: 00010002 (2.6.18.8-xen #5) > BUG: unable to handle kernel paging request at virtual address 245c8bf1 > printing eip: > c0145df6 > 2c934000 -> *pde = 00000000:60f1a001 > 2c8e7000 -> *pme = 00000000:00000000 > Recursive die() failure, output suppressed > <0>Kernel panic - not syncing: Fatal exception in interrupt > BUG: warning at /home/user/Download/linux-2.6.18-xen.hg/arch/i386/kernel/smp-xen.c:511/smp_call_function() > [] <1>BUG: unable to handle kernel paging request at virtual address 245c8bf1 > printing eip: > c0145df6 > 2c934000 -> *pde = 00000000:60f1a001 > 2c8e7000 -> *pme = 00000000:00000000 > Oops: 0002 [#3] > SMP > Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod ixgbe aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd usbcore > CPU: 0 > EIP: 0061:[] Not tainted VLI > EFLAGS: 00010002 (2.6.18.8-xen #5) > BUG: unable to handle kernel paging request at virtual address 245c8bf1 > printing eip: > c0145df6 > 2c934000 -> *pde = 00000000:60f1a001 > 2c8e7000 -> *pme = 00000000:00000000 > Oops: 0002 [#4] > SMP > Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod ixgbe aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd usbcore > CPU: 0 > EIP: 0061:[] Not tainted VLI > EFLAGS: 00010002 (2.6.18.8-xen #5) > BUG: unable to handle kernel paging request at virtual address 245c8bf1 > printing eip: > c0145df6 > 2c934000 -> *pde = 00000000:60f1a001 > 2c8e7000 -> *pme = 00000000:00000000 > Recursive die() failure, output suppressed > <0>Kernel panic - not syncing: Fatal exception in interrupt > <1>BUG: unable to handle kernel NULL pointer dereference at virtual address 00000064 > printing eip: > c0158df7 > 2d2e1000 -> *pde = 00000000:6051f001 > 2d2e2000 -> *pme = 00000000:00000000 > Oops: 0002 [#5] > SMP > Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod ixgbe aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd usbcore > CPU: 1 > EIP: 0061:[] Not tainted VLI > EFLAGS: 00010282 (2.6.18.8-xen #5) > BUG: unable to handle kernel paging request at virtual address 245c8bf1 > printing eip: > c0145df6 > 2d2e1000 -> *pde = 00000000:6051f001 > 2d2e2000 -> *pme = 00000000:00000000 > Oops: 0002 [#6] > SMP > Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod ixgbe aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd usbcore > CPU: 1 > EIP: 0061:[] Not tainted VLI > EFLAGS: 00010002 (2.6.18.8-xen #5) > BUG: unable to handle kernel paging request at virtual address 245c8bf1 > printing eip: > c0145df6 > 2d2e1000 -> *pde = 00000000:6051f001 > 2d2e2000 -> *pme = 00000000:00000000 > Recursive die() failure, output suppressed > > What are the correct interrupt settings for using ixgbe in PV pass-through mode? I have tried all three possible modes but I do not seem to get the device working properly under high network load. > > I am using Xen 3.3.1. > > release : 2.6.18.8-xen > version : #5 SMP Thu Mar 5 04:14:30 PST 2009 > machine : i686 > nr_cpus : 2 > nr_nodes : 1 > cores_per_socket : 2 > threads_per_core : 1 > cpu_mhz : 2200 > virt_caps : hvm > total_memory : 2039 > free_memory : 6 > node_to_cpu : node0:0-1 > node_to_memory : node0:6 > xen_major : 3 > xen_minor : 3 > xen_extra : .1 > xen_caps : xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p > xen_scheduler : credit > xen_pagesize : 4096 > platform_params : virt_start=0xf5800000 > xen_changeset : unavailable > cc_compiler : gcc version 4.1.2 20070925 (Red Hat 4.1.2-33) > cc_compile_by : root > cc_compile_domain : localdomain > cc_compile_date : Sun Mar 8 05:40:51 PDT 2009 > xend_config_format : 4 > > I am using PCI pass-through to assign the NIC to my PV guest, so this is the part of my config file: > > pci = [ '02:00.0','02:00.1' ] > > I am also hiding the PCI devices from Dom0, so my Dom0 GRUB config looks like this: > > title Xen 3.3 > root (hd0,1) > kernel /boot/xen-3.3.1.gz console=vga msi=1 > module /boot/vmlinuz-2.6.18.8-xen root=LABEL=/ ro console=tty0 rhgb quiet pciback.permissive pciback.hide=(02:00.0)(02:00.1) reassigndev=0000:02:00.0,0000:02:00.1 > module /boot/initrd-2.6.18.8-xen.img > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel