From mboxrd@z Thu Jan 1 00:00:00 1970 From: Alex Williamson Subject: Re: FreeBSD guest with VTD NIC not passing traffic Date: Tue, 03 Jan 2012 20:21:02 -0700 Message-ID: <1325647262.4305.47.camel@bling.home> References: Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Cc: kvm@vger.kernel.org To: Shashidhar Patil Return-path: Received: from mx1.redhat.com ([209.132.183.28]:13798 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754978Ab2ADDVE (ORCPT ); Tue, 3 Jan 2012 22:21:04 -0500 In-Reply-To: Sender: kvm-owner@vger.kernel.org List-ID: On Mon, 2011-12-19 at 19:49 +0530, Shashidhar Patil wrote: > Hi, > I am running Ubuntu 10.10 (amd64) on a 2 socket nehalem based > server with IOH 5520. 5520 supports VTD. > I enabled DMAR with intel_iommu=on. The box has intel 82599 adapter > which I assigned through VT-D to FreeBSD 8.2 running > as guest os. The ixgbe driver detects the device and the driver > successfully configures the device. But the link > never comes up. It looks like link up/down interrupts are not > delivered. Then I checked kvm interrupt assignment and as expected > kvm could not make MSI-X entries for the VT-d guest. So no output from > "grep kvm /proc/interrupt". By enabling some debugs in the > qemu-kvm I figured out that the MSI-x updates are not received > properly. It does look like Linux updates MSI-X table in a batch > fashion > which qemu-kvm gets in one shot and every thing works fine in case of > linux. In case of FreeBSD PCIE updates come /MSI-X entry > which qemu-kvm can't make use. That's right, Linux and Windows both seem to setup the MSI-X table then enable it in one shot, so we only trigger the interrupt programming when the enable bit is set. We don't trigger changes on writes to the MSI-X table... not very accurate emulation of mask bits. > To overcome this I compiled latest intel ixgbe driver with MSI-X > diabled. This time MSI interrupt got allocated both in the guest > and in the qemu-kvm (host kernel). Still I could not get link UP. I > modified the FreeBSD driver to poll for Link in some local_timer > task. Link comes up but no traffic flows. The MAC statistics show > packets received but packets do not reach the guest. > DMAing of packet may be failing. I could not find out the reason. > The same happens with legacy interrupt allocated in the guest. I even > tried qemu-kvm prefer_msi=off. > > I think there are two problems1 > 1. Interrupt delivery either because of interrupt remapping failure or > all out interrupt allocation failure in qemu-kvm > 2. Packets not getting DMAed to the guest possibly because of some DMAR issue. > > Before doing VT-d I made sure connections are fine and the adapter > works fine in bare metal Linux. > I also tried VT-d of 82599 with Linux as guest (both 32 bits(PAE and > non-PAE) and amd64) and it just works magically. Unknowns: does the intel ixgbe driver work in FreeBSD with MSI-X disabled, does polling for link UP work. The way you describe MSI-X interrupt setup for FreeBSD is likely something we don't handle correctly. You might do a hack in device-assignment.c to call assigned_dev_update_msix from msix_mmio_writel any time we write to the vector control dword. We really have no support for either function mask or per vector mask right now. I doubt you're having DMAR issues or you'd likely be seeing errors in the host dmesg. > Is this a known issue I am hitting with non-Linux guest oses and VT-d > ? Appreciate any help in debugging this problem. > > -Shashidhar > > Linux kernel - 2.6.35 > kvm - 0.14.1 It'd be helpful to test on something newer too. Thanks, Alex