From mboxrd@z Thu Jan 1 00:00:00 1970 From: Alex Williamson Subject: Re: FreeBSD guest with VTD NIC not passing traffic Date: Fri, 13 Jan 2012 14:05:45 -0700 Message-ID: <1326488745.5945.42.camel@bling.home> References: <1325647262.4305.47.camel@bling.home> <4F109B65.8020702@web.de> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Cc: Shashidhar Patil , kvm@vger.kernel.org To: Jan Kiszka Return-path: Received: from mx1.redhat.com ([209.132.183.28]:64845 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1759180Ab2AMVFs (ORCPT ); Fri, 13 Jan 2012 16:05:48 -0500 In-Reply-To: <4F109B65.8020702@web.de> Sender: kvm-owner@vger.kernel.org List-ID: On Fri, 2012-01-13 at 22:00 +0100, Jan Kiszka wrote: > On 2012-01-04 04:21, Alex Williamson wrote: > > On Mon, 2011-12-19 at 19:49 +0530, Shashidhar Patil wrote: > >> Hi, > >> I am running Ubuntu 10.10 (amd64) on a 2 socket nehalem based > >> server with IOH 5520. 5520 supports VTD. > >> I enabled DMAR with intel_iommu=on. The box has intel 82599 adapter > >> which I assigned through VT-D to FreeBSD 8.2 running > >> as guest os. The ixgbe driver detects the device and the driver > >> successfully configures the device. But the link > >> never comes up. It looks like link up/down interrupts are not > >> delivered. Then I checked kvm interrupt assignment and as expected > >> kvm could not make MSI-X entries for the VT-d guest. So no output from > >> "grep kvm /proc/interrupt". By enabling some debugs in the > >> qemu-kvm I figured out that the MSI-x updates are not received > >> properly. It does look like Linux updates MSI-X table in a batch > >> fashion > >> which qemu-kvm gets in one shot and every thing works fine in case of > >> linux. In case of FreeBSD PCIE updates come /MSI-X entry > >> which qemu-kvm can't make use. > > > > That's right, Linux and Windows both seem to setup the MSI-X table then > > enable it in one shot, so we only trigger the interrupt programming when > > the enable bit is set. We don't trigger changes on writes to the MSI-X > > table... not very accurate emulation of mask bits. > > According to the PCI spec, updates that happen while a vector is > unmasked, need not be considered by the hardware (thus the hypervisor > here). Is that the scenario here? I'm assuming the vector is masked in the MSI-X table. So Linux/Windows do: a) program MSI-X table b) enable MSI-X in capability register Whereas FreeBSD does: a) enable MSI-X in capability register (vectors masked in table) b) program and unmask individual vectors Thanks, Alex