From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dana Goyette Subject: Re: genirq: Flags mismatch irq 17. 00000000 (vfio-intx(0000:07:04.0)) vs. 00000000 (vfio-intx(0000:01:00.1)) Date: Fri, 10 Jan 2014 14:45:33 -0800 Message-ID: References: <1389375197.3209.321.camel@bling.home> <1389388347.3209.344.camel@bling.home> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit To: kvm@vger.kernel.org Return-path: Received: from plane.gmane.org ([80.91.229.3]:52519 "EHLO plane.gmane.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750983AbaAJWpu (ORCPT ); Fri, 10 Jan 2014 17:45:50 -0500 Received: from list by plane.gmane.org with local (Exim 4.69) (envelope-from ) id 1W1kpk-0005AP-Sn for kvm@vger.kernel.org; Fri, 10 Jan 2014 23:45:44 +0100 Received: from 64.89.225.142 ([64.89.225.142]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 10 Jan 2014 23:45:44 +0100 Received: from DanaGoyette by 64.89.225.142 with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 10 Jan 2014 23:45:44 +0100 In-Reply-To: <1389388347.3209.344.camel@bling.home> Sender: kvm-owner@vger.kernel.org List-ID: On 01/10/2014 01:12 PM, Alex Williamson wrote: > On Fri, 2014-01-10 at 12:18 -0800, Dana Goyette wrote: >> On 01/10/2014 09:33 AM, Alex Williamson wrote: >>> On Fri, 2014-01-10 at 08:23 -0800, Dana Goyette wrote: >>>> I'm using the SuperMicro X10SAT, kernel 3.13-rc5, with ACS override on >>>> Intel root ports. >>>> >>>> I'm trying to pass several devices to the same guest: >>>> >>>> 01:00.0 -- [1002:68be] Radeon HD 5750 >>>> 01:00.1 -- [1002:aa58] HDMI Audio (not really needed) >>>> 07:04.0 -- [13f6:8788] Xonar D1/DX sound card, behind PEX8112 >>>> 09:00.0 -- [1912:0014] Renesas uPD720201 (USB 3.0) >>>> >>>> When trying to start qemu with various combinations of those devices: >>>> vfio: Error: Failed to setup INTx fd: Device or resource busy >>>> >>>> Sound card conflicts with HDMI audio: >>>> genirq: Flags mismatch irq 17. >>>> 00000000 (vfio-intx(0000:07:04.0)) vs. >>>> 00000000 (vfio-intx(0000:01:00.1)) >>>> >>>> USB controller conflicts with video card: >>>> genirq: Flags mismatch irq 16. >>>> 00000000 (vfio-intx(0000:09:00.0)) vs. >>>> 00000000 (vfio-intx(0000:01:00.0)) >>>> >>>> On the ArchLinux forums, I was told that this means each device wants >>>> the interrupt line to itself. >>>> >>>> Oddly, the host locks up if I launch Xorg with 'radeon' and >>>> 'snd-virtuoso' both loaded. (Primary video is the Intel graphics.) >>>> >>>> These devices all work fine together using pci-assign, but pci-assign >>>> requires ejecting the Radeon before VM shutdown. >>>> >>>> VFIO allows me to start the VM if I forward only the sound card and the >>>> video card, but I really need the USB controller, as well. >>>> >>>> What can I do to forward those three devices via VFIO? >>> >>> Change slots until the interrupts don't conflict, maybe there's an >>> IO-APIC option you can enable in the BIOS, or get devices that properly >>> support PCI2.3 INTx disable. pci-assign tries to work around this by >>> using MSI to emulate INTx, but I'm not convinced that this doesn't >>> create more problems than it solves or I'd try to do the same with vfio. >>> The Xorg problem might also suggest there's an existing interrupt >>> routing problem on the system. It is surprising that you have so many >>> devices that doesn't support INTx disable, can you try running this >>> script on them: >>> >>> #!/bin/sh >>> # Usage $0 , ex: 9:00.0 >>> >>> INTX=$(( 0x400 )) >>> ORIG=$(( 0x$(setpci -s $1 4.w) )) >>> >>> if [ $(( $INTX & $ORIG )) -ne 0 ]; then >>> echo "INTx disable supported and enabled on $1" >>> exit 0 >>> fi >>> >>> NEW=$(printf %04x $(( $INTX | $ORIG ))) >>> setpci -s $1 4.w=$NEW >>> NEW=$(( 0x$(setpci -s $1 4.w) )) >>> >>> if [ $(( $INTX & $NEW )) -ne 0 ]; then >>> echo "INTx disable support available on $1" >>> else >>> echo "INTx disable support NOT available on $1" >>> fi >>> >>> NEW=$(printf %04x $ORIG) >>> setpci -s $1 4.w=$NEW >>> >> After cold boot, without any attempts to run VM: >> >> Video card: >> INTx disable support available on 01:00.0 >> >> HDMI Audio: >> INTx disable supported and enabled on 01:00.1 >> >> USB: >> INTx disable supported and enabled on 09:00.0 >> >> PEX8112 (sound card sits behind this): >> INTx disable support available on 06:00.0) >> >> Sound card: >> INTx disable support NOT available on 07:04.0 >> >> If the sound card is the only one lacking this feature, then it seems >> odd that the USB controller is the one I have to remove for the VM to boot. > > Please report the output of: > > $ cat /sys/module/vfio_pci/parameters/nointxmask > > I don't understand why your flags are 0 for all the devices, only the > sound card should have that. > >> The lspci -nnvv listing for my current sound card is quite sparse: >> >> 07:04.0 Multimedia audio controller [0401]: C-Media Electronics Inc >> CMI8788 [Oxygen HD Audio] [13f6:8788] >> Subsystem: ASUSTeK Computer Inc. Virtuoso 100 (Xonar D1) [1043:834f] >> Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- >> Stepping- SERR+ FastB2B- DisINTx- >> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- >> SERR- > Interrupt: pin A routed to IRQ 10 >> Region 0: I/O ports at c000 [disabled] [size=256] >> Capabilities: [c0] Power Management version 2 >> Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) >> Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- >> >> I'm willing to purchase a different sound card if that will fix the >> problem. Does lspci (-v or -vv) tell you if INTx Disable is available? > > No, lspci only reports the state of the DisINTx bit, not whether it's > supported. > > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Looks like I had 'nointxmask' set to 1 for some reason (in /etc/modprobe.d/custom.conf). With nointxmask NOT set, the conflict messages change: genirq: Flags mismatch irq 17. 00000000 (vfio-intx(0000:07:04.0)) vs. 00000080 (vfio-intx(0000:01:00.1)) The latter is the HDMI audio. Removing it from the VM allows the VM to start. It would still be nice for the message to say something more descriptive. The flag meanings are not obvious, and when both sides are the same, it doesn't seem like a "mismatch". Incidentally, the genirq message omits the bus ID for PCI serial devices: genirq: Flags mismatch irq 19. 00000000 (vfio-intx(0000:0c:00.0)) vs. 00000080 (serial) That's with the following devices (and likely only with nointxmask): 0c:00.0 -- [8086:1533] Intel Corporation I210 Gigabit Network Connection 00:16.3 -- [8086:8c3d] Intel C220 Series Chipset Family KT Controller