Re: VT-d not working for FreeBSD 9.0 guest

[Shesha Sreenivasamurthy <sheshas@gmail.com>]

Alex Williamson <alex.williamson <at> redhat.com> writes:

> 
> On Wed, 2012-05-16 at 17:49 +0000, Shesha Sreenivasamurthy wrote:
> > I'm using FreeBSD 9.0 (FreeBSD freebsd9-i386 9.0-RELEASE FreeBSD 9.0-RELEASE)
> > as my guest OS with qemu-kvm (qemu-kvm-1.1-rc2/x86_64-softmmu/
> > qemu-system-x86_64) running on centos 6.2 (CentOS release 6.2) with
> > 2.6.32-220.el6.x86_64 kernel. I am using Intel's 82599EB 10-Gigabit hardware
> > on the host, which I am trying to export into the FreeBSD guest OS. The guest
> > driver (version 2.4.4) will claim it.  However, the status is shown as
> > "no carrier" an therefore cannot ping any other system.
> > 
> > $ ifconfig ix0
> > ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> >         
> > options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,
> > VLAN_HWCSUM,TSO4,VLAN_HWTSO>
> >         ether 90:e2:ba:0d:3d:b0
> >         inet 4.4.4.1 netmask 0xffffff00 broadcast 4.4.4.255
> >         nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> >         media: Ethernet autoselect
> >         status: no carrier
> > 
> > On the host side, before the ixgbe is loaded in the guest, /proc/interrupts
> > show the following entry for kvm
> > 
> > # cat /proc/interrupts | grep kvm
> >   30:          0          0          0          0          0          0
> >   0          0          0          0          0          0          0
> >   0          0          0 IR-IO-APIC-fasteoi kvm_assigned_intx_device
> > 
> > However, after the driver is loaded, the above entry is gone but two new
> > entries show up.
> > 
> > # cat /proc/interrupts | grep kvm
> >   76:          0          0          0          0          0          0
> >   0          0          0          0          0          0          0
> >   0          0          0          IR-PCI-MSI-edge   kvm_assigned_msix_device
> >   77:          0          0          0          0          0          0
> >   0          0          0          0          0          0          0
> >   0          0          0          IR-PCI-MSI-edge  kvm_assigned_msix_device.
> 
> This is expected, the driver enabled MSI-X.  If you have an equivalent
> of lspci, you should be able to see before that MSI/MSI-X is disabled
> and after MSI-X is enabled with 2 vectors.  Do you ever get interrupts
> on these lines?
> 
> > To make sure, the my host configurations are fine, I loaded linux guest, with
> > exact same qemu params and host configurations. It works just fine.
> > 
> > What I noticed in case linux guest is, to start with same entry is seen in
> > /proc/interrupts. However, after the ixgbe is loaded in the guest, I see only
> > one entry instead of two as in case of freebsd.
> > 
> > # cat /proc/interrupts | grep kvm
> >   30:          0          0          0          0          0          0
> >   0          0          0          0          0          0          0
> >   0          0          0       IR-IO-APIC-fasteoi   kvm_assigned_intx_device
> > 
> > # cat /proc/interrupts | grep kvm
> >   76:          0          0          0          0          0          0
> >   0          0          0          0          0          0          0
> >   0          0          0          IR-PCI-MSI-edg   kvm_assigned_msi_device
> 
> Linux may be enabling MSI instead of MSI-X, that would be Ok.  Check
> lspci in the guest to see what Linux thinks it's using.
> 
> > Similar problem was reported earlier too: 
> > http://comments.gmane.org/gmane.comp.emulators.kvm.devel/83806
> > 
> > Any ideas or advice will be highly valued
> 
> Right, we fixed how we do MSI-X vector allocation so we can better track
> how this driver works.  That allows an 82576 (igb) device to work, but
> there might still be a FreeBSD ixgbe driver issue that's not allowing it
> to see the link state.  Thanks,
> 
> Alex
> 
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo <at> vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

I tried the same card on a natively running freebsd9.0 and used the same
drivers and it works fine.

It was installed using the same ISO from which I created the vm disk image,
used the same driver and it works fine. I have captured the output of loading
ixgbe driver in a VM with VT-d and loading the same driver on freebsd running
natively on a system.

===
VM
===
ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.4.4> port
0xc040-0xc05f mem 0xfea80000-0xfeafffff,0xfebd0000-0xfebd3fff irq 11
at device 3.0 on pci0
ix0: Using MSIX interrupts with 2 vectors
ixgbe_init_mac_link_ops_82599
ixgbe_identify_phy_generic
ixgbe_identify_phy_generic: -17
ixgbe_identify_sfp_module_generic
---------------
type: 0xF
addr: 0x0
id: 0x3
sfp_type: 0x5
sfp_setup_needed: 0x1
revision: 0x0
media_type: 0x0
autoneg_advertised: 0x0
smart_speed: 0x1
smart_speed_active: 0x0
multispeed_fiber: 0x1
reset_if_overtemp: 0x0
---------------
ixgbe_identify_phy_82599: 0
ixgbe_init_mac_link_ops_82599
ixgbe_init_mac_link_ops_82599
Enter ixgbe_check_mac_link_generic
LINKS orig: 2C080001, new: 2C080001
link_up_wait_to_complete: 0
link_up: 1
IXGBE_LINK_SPEED_10GB_FULL: 0x80, IXGBE_LINK_SPEED_1GB_FULL: 0x20
IXGBE_LINK_SPEED_100_FULL: 0x8, IXGBE_LINK_SPEED_UNKNOWN: 0x0
speed: 0x80
ixgbe_identify_phy_generic
ixgbe_identify_phy_generic: 0
ixgbe_identify_phy_82599: 0
ixgbe_identify_sfp_module_generic
---------------
type: 0xF
addr: 0x0
id: 0x3
sfp_type: 0x5
sfp_setup_needed: 0x0
revision: 0x0
media_type: 0x1
autoneg_advertised: 0x0
smart_speed: 0x1
smart_speed_active: 0x0
multispeed_fiber: 0x1
reset_if_overtemp: 0x0
---------------
ix0: Ethernet address: 90:e2:ba:0d:3d:b0
Enter ixgbe_get_bus_info_generic
link_status: 0x1082, bus_width: 0x0, bus_sped: 0x0
NEW link_status: 0x1082, bus_width: 0x8, bus_sped: 0x1388
ix0: PCI Express Bus: Speed 5.0Gb/s Width x8

======
NATIVE
======

ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.4.4> port
0xe880-0xe89f mem 0xf8e80000-0xf8efffff,0xf8e7c000-0xf8e7ffff irq 30
at device 0.0 on pci3
ix0: Using MSIX interrupts with 9 vectors
ix0: RX Descriptors exceed system mbuf max, using default instead!
ixgbe_init_mac_link_ops_82599
ixgbe_identify_phy_generic
ixgbe_identify_phy_generic: -17
ixgbe_identify_sfp_module_generic
---------------
type: 0xF
addr: 0x0
id: 0x3
sfp_type: 0x5
sfp_setup_needed: 0x1
revision: 0x0
media_type: 0x0
autoneg_advertised: 0x0
smart_speed: 0x1
smart_speed_active: 0x0
multispeed_fiber: 0x1
reset_if_overtemp: 0x0
---------------
ixgbe_identify_phy_82599: 0
ixgbe_init_mac_link_ops_82599
ixgbe_init_mac_link_ops_82599
Enter ixgbe_check_mac_link_generic
LINKS orig: 2C080001, new: 2C080001
link_up_wait_to_complete: 0
link_up: 0
IXGBE_LINK_SPEED_10GB_FULL: 0x80, IXGBE_LINK_SPEED_1GB_FULL: 0x20
IXGBE_LINK_SPEED_100_FULL: 0x8, IXGBE_LINK_SPEED_UNKNOWN: 0x0
speed: 0x80
ixgbe_identify_phy_generic
ixgbe_identify_phy_generic: 0
ixgbe_identify_phy_82599: 0
ixgbe_identify_sfp_module_generic
---------------
type: 0xF
addr: 0x0
id: 0x3
sfp_type: 0x5
sfp_setup_needed: 0x0
revision: 0x0
media_type: 0x1
autoneg_advertised: 0x0
smart_speed: 0x1
smart_speed_active: 0x0
multispeed_fiber: 0x1
reset_if_overtemp: 0x0
---------------
ix0: Ethernet address: 90:e2:ba:0d:3e:5c
Enter ixgbe_get_bus_info_generic
link_status: 0x1082, bus_width: 0x0, bus_sped: 0x0
NEW link_status: 0x1082, bus_width: 0x8, bus_sped: 0x1388
ix0: PCI Express Bus: Speed 5.0Gb/s Width x8
===