From: Rolf Neugebauer <rolf.neugebauer@netronome.com>
To: "Fischer, Anna" <anna.fischer@hp.com>
Cc: "xen-devel@lists.xensource.com" <xen-devel@lists.xensource.com>
Subject: Re: PV PCI pass-through with ixgbe ?
Date: Tue, 07 Apr 2009 10:48:43 +0100 [thread overview]
Message-ID: <49DB217B.3000005@netronome.com> (raw)
In-Reply-To: <0199E0D51A61344794750DC57738F58E66B941F2F8@GVW1118EXC.americas.hpqcorp.net>
This worked for me for a fairly recent version of xen-unstable (cs:
19429:6dcccd2dded6). I used the corresponding -xen kernel for domU. I
used a freshly compiled version of the ixgbe driver (1.3.56.5-NAPI).
MSI-X worked fine, though I've only tried ping, not bulk data transfers.
packets looked fine when I looked at it with tcpdump.
there were quite a few changes to the MSI code recently.
Rolf
Fischer, Anna wrote:
> Is it possible to use MSI-X in a PV Xen DomU (2.6.18.8) for a PCI pass-through device? I am running an Intel ixgbe NIC in DomU, but I do not seem to be able to run it with MSI-X enabled. In fact, I do not get it to work in any interrupt mode. The DomU/Dom0 kernel is compiled with PCI_MSI=y and I have msi=1 set in Xen's boot options.
>
> I do seem to get an interrupt clash when running up the ixgbe device driver in DomU. Also, packets seem to be corrupted when I capture with tcpdump on the interfaces.
>
> tcpdump -i eth1 -xx gives me this:
>
> 01:46:56.770226 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00 (oui Ethernet) Null Information, send seq 0, rcv seq 0, Flags [Command], length 46
> 0x0000: 0000 0000 0000 0000 0000 0000 0000 0000
> 0x0010: 0000 0000 0000 0000 0000 0000 0000 0000
> 0x0020: 0000 0000 0000 0000 0000 0000 0000 0000
> 0x0030: 0000 0000 0000 0000 0000 0000
> 01:46:56.973564 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00 (oui Ethernet) Null Information, send seq 0, rcv seq 0, Flags [Command], length 46
> 0x0000: 0000 0000 0000 0000 0000 0000 0000 0000
> 0x0010: 0000 0000 0000 0000 0000 0000 0000 0000
> 0x0020: 0000 0000 0000 0000 0000 0000 0000 0000
> 0x0030: 0000 0000 0000 0000 0000 0000
> 01:46:57.176902 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00 (oui Ethernet) Null Information, send seq 0, rcv seq 0, Flags [Command], length 46
> 0x0000: 0000 0000 0000 0000 0000 0000 0000 0000
> 0x0010: 0000 0000 0000 0000 0000 0000 0000 0000
> 0x0020: 0000 0000 0000 0000 0000 0000 0000 0000
> 0x0030: 0000 0000 0000 0000 0000 0000
> 01:46:57.380238 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00 (oui Ethernet) Null Information, send seq 0, rcv seq 0, Flags [Command], length 46
> 0x0000: 0000 0000 0000 0000 0000 0000 0000 0000
> 0x0010: 0000 0000 0000 0000 0000 0000 0000 0000
> 0x0020: 0000 0000 0000 0000 0000 0000 0000 0000
> 0x0030: 0000 0000 0000 0000 0000 0000
>
>
> I have tried to run DomU/DomU with the irqpoll boot parameter but when I do that then the DomU fails to boot and hangs with a kernel panic:
>
> Intel(R) 10 Gigabit PCI Express Network Driver - version 1.3.31.5
> Copyright (c) 1999-2008 Intel Corporation.
> PCI: Enabling device 0000:02:00.0 (0000 -> 0003)
> ixgbe: Interrupt Type set to 1
> ixgbe: 0000:02:00.0: ixgbe_check_options: Multiple queues are not supported while MSI-X is disabled. Disabling Multiple Queues.
> ixgbe: 0000:02:00.0: ixgbe_check_options: RSS is not supported while multiple queues are disabled. Disabling RSS.
> ixgbe: 0000:02:00.0: ixgbe_init_interrupt_scheme: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1
> ixgbe: 0000:02:00.0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) 00:21:f7:58:01:6d
> ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection
> PCI: Enabling device 0000:02:00.1 (0000 -> 0003)
> BUG: unable to handle kernel NULL pointer dereference at virtual address 0000002d
> printing eip:
> c01fccaa
> 2c8fa000 -> *pde = 00000000:60b62001
> 2cfc1000 -> *pme = 00000000:00000000
> Oops: 0002 [#1]
> SMP
> Modules linked in: ixgbe aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd usbcore
> CPU: 0
> EIP: 0061:[<c01fccaa>] Not tainted VLI
> EFLAGS: 00210246 (2.6.18.8-xen #5)
> EIP is at pci_enable_msix+0x1da/0x510
> eax: ed717800 ebx: 00000011 ecx: 00000000 edx: 00000004
> esi: 00000000 edi: ed717800 ebp: ec951f80 esp: ecab9d54
> ds: 007b es: 007b ss: 0069
> Process modprobe (pid: 1566, ti=ecab8000 task=eccd2e10 task.ti=ecab8000)
> Stack: 00000098 ecfd4400 c036d220 c015f905 ecfd4400 ecfd4400 00000004 ee0e639a
> 00000000 00000000 00000008 ec951f60 000000b2 00000002 00030003 0000007b
> 0000007b 00000000 c017682e ec951f80 00000004 00000010 ecfd4400 00000000
> Call Trace:
> [<c015f905>] __kzalloc+0x15/0x50
> [<ee0e639a>] ixgbe_alloc_queues+0x8a/0x580 [ixgbe]
> [<c017682e>] __kmalloc+0xbe/0xf0
> [<ee0e6a13>] ixgbe_init_interrupt_scheme+0x183/0x340 [ixgbe]
> [<ee0e3b81>] ixgbe_read_pci_cfg_word+0x21/0x30 [ixgbe]
> [<ee0e8c17>] ixgbe_probe+0x6e7/0x9e0 [ixgbe]
> [<c017682e>] __kmalloc+0xbe/0xf0
> [<c01f5ef6>] pci_device_probe+0x56/0x80
> [<c023ecd4>] driver_probe_device+0x44/0xc0
> [<c023ee52>] __driver_attach+0x82/0x90
> [<c023e64a>] bus_for_each_dev+0x3a/0x60
> [<c023ec16>] driver_attach+0x16/0x20
> [<c023edd0>] __driver_attach+0x0/0x90
> [<c023e2bc>] bus_add_driver+0x8c/0x140
> [<c01f6097>] __pci_register_driver+0x47/0x70
> [<c0143b08>] sys_init_module+0x148/0x1b40
> [<c01ea55f>] prio_tree_insert+0x1f/0x260
> [<c01187fc>] do_page_fault+0x10c/0xc6f
> [<c010acc0>] sys_mmap2+0xd0/0xe0
> [<c01059bf>] syscall_call+0x7/0xb
> Code: 00 89 54 24 38 ba ed ff ff ff 89 4c 24 3c 0b 4c 24 38 0f 85 2d 01 00 00 8b 4c 24 28 89 d6 89 8f b0 01 00 00 e9 62 fe ff ff 89 f8 <89> 5b 1c e8 3e f2 ff ff 8b 7c 24 18 89 44 24 54 85 ff 0f 8e 48
> EIP: [<c01fccaa>] pci_enable_msix+0x1da/0x510 SS:ESP 0069:ecab9d54
> udevd-event[1564]: run_program: '/sbin/modprobe' abnormal exit
>
>
> I have also tried to use legacy interrupts by setting InterruptType to 0:
>
> Intel(R) 10 Gigabit PCI Express Network Driver - version 1.3.31.5-lro
> Copyright (c) 1999-2008 Intel Corporation.
> PCI: Enabling device 0000:02:00.0 (0000 -> 0003)
> PCI: Setting latency timer of device 0000:02:00.0 to 64
> ixgbe: Interrupt Type set to 0
> ixgbe: Multiple Queue Support Disabled
> ixgbe: Receive-Side Scaling (RSS) set to 0
> ixgbe: Virtual Machine Device Queues (VMDQ) set to 0
> ixgbe: 0000:02:00.0: ixgbe_init_interrupt_scheme: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1
> ixgbe: 0000:02:00.0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) 00:21:f7:58:01:6d
> ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection
> PCI: Enabling device 0000:02:00.1 (0000 -> 0003)
> PCI: Setting latency timer of device 0000:02:00.1 to 64
> ixgbe: Interrupt Type set to 0
> ixgbe: Multiple Queue Support Disabled
> ixgbe: Receive-Side Scaling (RSS) set to 0
> ixgbe: Virtual Machine Device Queues (VMDQ) set to 0
> ixgbe: 0000:02:00.1: ixgbe_init_interrupt_scheme: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1
> ixgbe: 0000:02:00.1: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) 00:21:f7:58:01:6c
> ixgbe: eth1: ixgbe_probe: Intel(R) 10 Gigabit Network Connection
>
> Without MSI and MSI-X the driver loads properly. However, I still end up with corrupted packets on tcpdump, and additionally I am getting a kernel crash when the guest is under high network load:
>
> BUG: unable to handle kernel NULL pointer dereference at virtual address 00000034
> printing eip:
> 2c934000 -> *pde = 00000000:60f1a001
> 2c8e7000 -> *pme = 00000000:00000000
> Oops: 0002 [#1]
> SMP
> Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod ixgbe aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd usbcore
> CPU: 0
> EIP: 0061:[<c0158df7>] Not tainted VLI
> EFLAGS: 00010082 (2.6.18.8-xen #5)
> BUG: unable to handle kernel paging request at virtual address 245c8bf1
> printing eip:
> c0145df6
> 2c934000 -> *pde = 00000000:60f1a001
> 2c8e7000 -> *pme = 00000000:00000000
> Oops: 0002 [#2]
> SMP
> Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod ixgbe aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd usbcore
> CPU: 0
> EIP: 0061:[<c0145df6>] Not tainted VLI
> EFLAGS: 00010002 (2.6.18.8-xen #5)
> BUG: unable to handle kernel paging request at virtual address 245c8bf1
> printing eip:
> c0145df6
> 2c934000 -> *pde = 00000000:60f1a001
> 2c8e7000 -> *pme = 00000000:00000000
> Recursive die() failure, output suppressed
> <0>Kernel panic - not syncing: Fatal exception in interrupt
> BUG: warning at /home/user/Download/linux-2.6.18-xen.hg/arch/i386/kernel/smp-xen.c:511/smp_call_function()
> [<c0112541>] <1>BUG: unable to handle kernel paging request at virtual address 245c8bf1
> printing eip:
> c0145df6
> 2c934000 -> *pde = 00000000:60f1a001
> 2c8e7000 -> *pme = 00000000:00000000
> Oops: 0002 [#3]
> SMP
> Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod ixgbe aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd usbcore
> CPU: 0
> EIP: 0061:[<c0145df6>] Not tainted VLI
> EFLAGS: 00010002 (2.6.18.8-xen #5)
> BUG: unable to handle kernel paging request at virtual address 245c8bf1
> printing eip:
> c0145df6
> 2c934000 -> *pde = 00000000:60f1a001
> 2c8e7000 -> *pme = 00000000:00000000
> Oops: 0002 [#4]
> SMP
> Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod ixgbe aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd usbcore
> CPU: 0
> EIP: 0061:[<c0145df6>] Not tainted VLI
> EFLAGS: 00010002 (2.6.18.8-xen #5)
> BUG: unable to handle kernel paging request at virtual address 245c8bf1
> printing eip:
> c0145df6
> 2c934000 -> *pde = 00000000:60f1a001
> 2c8e7000 -> *pme = 00000000:00000000
> Recursive die() failure, output suppressed
> <0>Kernel panic - not syncing: Fatal exception in interrupt
> <1>BUG: unable to handle kernel NULL pointer dereference at virtual address 00000064
> printing eip:
> c0158df7
> 2d2e1000 -> *pde = 00000000:6051f001
> 2d2e2000 -> *pme = 00000000:00000000
> Oops: 0002 [#5]
> SMP
> Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod ixgbe aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd usbcore
> CPU: 1
> EIP: 0061:[<c0158df7>] Not tainted VLI
> EFLAGS: 00010282 (2.6.18.8-xen #5)
> BUG: unable to handle kernel paging request at virtual address 245c8bf1
> printing eip:
> c0145df6
> 2d2e1000 -> *pde = 00000000:6051f001
> 2d2e2000 -> *pme = 00000000:00000000
> Oops: 0002 [#6]
> SMP
> Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod ixgbe aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd usbcore
> CPU: 1
> EIP: 0061:[<c0145df6>] Not tainted VLI
> EFLAGS: 00010002 (2.6.18.8-xen #5)
> BUG: unable to handle kernel paging request at virtual address 245c8bf1
> printing eip:
> c0145df6
> 2d2e1000 -> *pde = 00000000:6051f001
> 2d2e2000 -> *pme = 00000000:00000000
> Recursive die() failure, output suppressed
>
> What are the correct interrupt settings for using ixgbe in PV pass-through mode? I have tried all three possible modes but I do not seem to get the device working properly under high network load.
>
> I am using Xen 3.3.1.
>
> release : 2.6.18.8-xen
> version : #5 SMP Thu Mar 5 04:14:30 PST 2009
> machine : i686
> nr_cpus : 2
> nr_nodes : 1
> cores_per_socket : 2
> threads_per_core : 1
> cpu_mhz : 2200
> virt_caps : hvm
> total_memory : 2039
> free_memory : 6
> node_to_cpu : node0:0-1
> node_to_memory : node0:6
> xen_major : 3
> xen_minor : 3
> xen_extra : .1
> xen_caps : xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p
> xen_scheduler : credit
> xen_pagesize : 4096
> platform_params : virt_start=0xf5800000
> xen_changeset : unavailable
> cc_compiler : gcc version 4.1.2 20070925 (Red Hat 4.1.2-33)
> cc_compile_by : root
> cc_compile_domain : localdomain
> cc_compile_date : Sun Mar 8 05:40:51 PDT 2009
> xend_config_format : 4
>
> I am using PCI pass-through to assign the NIC to my PV guest, so this is the part of my config file:
>
> pci = [ '02:00.0','02:00.1' ]
>
> I am also hiding the PCI devices from Dom0, so my Dom0 GRUB config looks like this:
>
> title Xen 3.3
> root (hd0,1)
> kernel /boot/xen-3.3.1.gz console=vga msi=1
> module /boot/vmlinuz-2.6.18.8-xen root=LABEL=/ ro console=tty0 rhgb quiet pciback.permissive pciback.hide=(02:00.0)(02:00.1) reassigndev=0000:02:00.0,0000:02:00.1
> module /boot/initrd-2.6.18.8-xen.img
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
prev parent reply other threads:[~2009-04-07 9:48 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2009-04-06 17:04 PV PCI pass-through with ixgbe ? Fischer, Anna
2009-04-07 9:48 ` Rolf Neugebauer [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=49DB217B.3000005@netronome.com \
--to=rolf.neugebauer@netronome.com \
--cc=anna.fischer@hp.com \
--cc=xen-devel@lists.xensource.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.