From mboxrd@z Thu Jan 1 00:00:00 1970 From: Martin Gerhard Loschwitz Subject: 2.6.29-rc8 / KVM 84: Problems with PCI Device Throughpassing Date: Tue, 17 Mar 2009 19:13:35 +0100 Message-ID: <49BFE84F.6080502@linbit.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit To: kvm@vger.kernel.org Return-path: Received: from mail09.linbit.com ([212.69.161.110]:47466 "EHLO mail09.linbit.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1757387AbZCQSYx (ORCPT ); Tue, 17 Mar 2009 14:24:53 -0400 Received: from [10.9.9.63] (unknown [86.59.100.100]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail09.linbit.com (LINBIT Mail Daemon) with ESMTPS id ECA15106923B for ; Tue, 17 Mar 2009 19:13:35 +0100 (CET) Sender: kvm-owner@vger.kernel.org List-ID: Hi, I am writing to this list as I believe to have found a KVM bug and hope that my information might be helpful to you (additionally, I of course hope to receive a patch or some such to fix my problem). First of all, here is what I am trying to achieve: I have a 4-port CologneChip ISDN card (compatible with any BeroNet Card out there), and I would like to pass that card through to a KVM VM. I've read almost every single description about how to a- chieve this, and I am seeing very strange effects that I believe to be unnormal. This finally led to me thinking there might be a bug or something in KVM. The software I use for this: Debian GNU/Linux Lenny, self-built 2.6.29-rc8 (after having tried multiple other kernel versions) and self-built KVM 84. Intel Dual-Core Xeon, my chipset supports VT-d, VT-d is enabled in the BIOS. The machine is an x86_64 installation. KVM modules are loaded with default options enabled, I tried with numerous settings meanwhile but did not have any luck. The ISDN card shows up with two devices in the system: 01:00.0 PCI bridge: PLX Technology, Inc. PEX 8111 PCI Express-to-PCI Bridge (rev 21) 02:00.0 ISDN controller: Cologne Chip Designs GmbH ISDN network Controller [HFC-4S] (rev 01) None of these two cards share an IRQ with another device, so that I believe that an IRQ-sharing-related problem is outruled. I explicitly disabled all drivers that could attach themselves to these devices via module blacklisting; none of these drivers is at boot-time or anytime later claimed by any active driver. Here is the situation I see before starting the KVM VM that is to access the ISDN card: Output of lspci -t, relevant parts: -[0000:00]-+-00.0 +-01.0-[0000:01-02]----00.0-[0000:02]----00.0 Output of lspci -n, relevant parts: 01:00.0 0604: 10b5:8111 (rev 21) 02:00.0 0204: 1397:08b4 (rev 01) Output of lspci -vvv, relevant parts: 01:00.0 PCI bridge: PLX Technology, Inc. PEX 8111 PCI Express-to-PCI Bridge (rev 21) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- TAbort- Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 2 Flags: PMEClk- DSI- D1+ D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold-) Status: D0 PME-Enable- DSel=0 DScale=0 PME- Capabilities: [50] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0 Enable- Address: 0000000000000000 Data: 0000 Capabilities: [60] Express (v1) PCI/PCI-X Bridge, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- BrConfRtry- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <1us, L1 <16us ClockPM- Suprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- Capabilities: [100] Power Budgeting Kernel modules: shpchp 02:00.0 ISDN controller: Cologne Chip Designs GmbH ISDN network Controller [HFC-4S] (rev 01) Subsystem: Cologne Chip Designs GmbH Device b762 Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- SERR-
Starting the VM with these configuration settings leads to very interesting effects. First of all, I see this in dmesg: [ 7660.248124] pci 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16 Now see how the output of the numerous lspci-commands changed: Output of lspci -t, relevant parts. -[0000:00]-+-00.0 +-01.0-[0000:01-02]--+-[0000:02]---00.0 | \-[0000:01]---00.0 Output of lspci -n: Unchanged. Output of lspci -vvv, relevant parts: 01:00.0 PCI bridge: PLX Technology, Inc. PEX 8111 PCI Express-to-PCI Bridge (rev ff) (prog-if ff) !!! Unknown header type 7f 02:00.0 ISDN controller: Cologne Chip Designs GmbH ISDN network Controller [HFC-4S] (rev ff) (prog-if ff) !!! Unknown header type 7f Kernel modules: hfcmulti Output of lspci -H1, relevant parts: None. Both devices are gone. Of course, the device is not there in the VM file and stopping KVM will not lead to me being able to use the devices in the bare system again ... the only way to get back the devices' functionality is a reboot, the windows-way to do it, so to say. I actually ran out of ideas what might be wrong here after having tried a dozen numerous configurations and setups, which is why I hope that you might be able to help me out. Please let me know if you need any further information about how to debug this problem, or if you have an idea what might be wrong. Best Regards Martin -- : Martin G. Loschwitz Tel +43-1-8178292-63 : : LINBIT Information Technologies GmbH Fax +43-1-8178292-82 : : Vivenotgasse 48, 1120 Vienna, Austria http://www.linbit.com :