Re: VT-d not working for FreeBSD 9.0 guest

[Alex Williamson <alex.williamson@redhat.com>]

On Wed, 2012-05-16 at 17:49 +0000, Shesha Sreenivasamurthy wrote:
> I'm using FreeBSD 9.0 (FreeBSD freebsd9-i386 9.0-RELEASE FreeBSD 9.0-RELEASE)
> as my guest OS with qemu-kvm (qemu-kvm-1.1-rc2/x86_64-softmmu/
> qemu-system-x86_64) running on centos 6.2 (CentOS release 6.2) with
> 2.6.32-220.el6.x86_64 kernel. I am using Intel's 82599EB 10-Gigabit hardware
> on the host, which I am trying to export into the FreeBSD guest OS. The guest
> driver (version 2.4.4) will claim it.  However, the status is shown as
> "no carrier" an therefore cannot ping any other system.
> 
> $ ifconfig ix0
> ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>         
> options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,
> VLAN_HWCSUM,TSO4,VLAN_HWTSO>
>         ether 90:e2:ba:0d:3d:b0
>         inet 4.4.4.1 netmask 0xffffff00 broadcast 4.4.4.255
>         nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>         media: Ethernet autoselect
>         status: no carrier
> 
> On the host side, before the ixgbe is loaded in the guest, /proc/interrupts
> show the following entry for kvm
> 
> # cat /proc/interrupts | grep kvm
>   30:          0          0          0          0          0          0
>   0          0          0          0          0          0          0
>   0          0          0 IR-IO-APIC-fasteoi kvm_assigned_intx_device
> 
> However, after the driver is loaded, the above entry is gone but two new
> entries show up.
> 
> # cat /proc/interrupts | grep kvm
>   76:          0          0          0          0          0          0
>   0          0          0          0          0          0          0
>   0          0          0          IR-PCI-MSI-edge   kvm_assigned_msix_device
>   77:          0          0          0          0          0          0
>   0          0          0          0          0          0          0
>   0          0          0          IR-PCI-MSI-edge  kvm_assigned_msix_device.

This is expected, the driver enabled MSI-X.  If you have an equivalent
of lspci, you should be able to see before that MSI/MSI-X is disabled
and after MSI-X is enabled with 2 vectors.  Do you ever get interrupts
on these lines?

> To make sure, the my host configurations are fine, I loaded linux guest, with
> exact same qemu params and host configurations. It works just fine.
> 
> What I noticed in case linux guest is, to start with same entry is seen in
> /proc/interrupts. However, after the ixgbe is loaded in the guest, I see only
> one entry instead of two as in case of freebsd.
> 
> # cat /proc/interrupts | grep kvm
>   30:          0          0          0          0          0          0
>   0          0          0          0          0          0          0
>   0          0          0       IR-IO-APIC-fasteoi   kvm_assigned_intx_device
> 
> # cat /proc/interrupts | grep kvm
>   76:          0          0          0          0          0          0
>   0          0          0          0          0          0          0
>   0          0          0          IR-PCI-MSI-edg   kvm_assigned_msi_device

Linux may be enabling MSI instead of MSI-X, that would be Ok.  Check
lspci in the guest to see what Linux thinks it's using.

> Similar problem was reported earlier too: 
> http://comments.gmane.org/gmane.comp.emulators.kvm.devel/83806
> 
> Any ideas or advice will be highly valued

Right, we fixed how we do MSI-X vector allocation so we can better track
how this driver works.  That allows an 82576 (igb) device to work, but
there might still be a FreeBSD ixgbe driver issue that's not allowing it
to see the link state.  Thanks,

Alex