PCI passthrough (shared IRQ) not working / contradicting information

PCI passthrough (shared IRQ) not working / contradicting information

[Binarus]

Hi all,

a while ago, I have set up a KVM virtual machine with PCI passthrough. The device which is passed through is an AVM Fritz!card PCI v2.1.

This works, but there is a drawback: At that time, KVM was not able to do PCI passthrough for PCI devices with shared interrupt. So I had to find out which other devices were using the same interrupt, and had to inactivate them by handling them over to the PCI stub driver.

Then, one day, I have been very excited when I was reading that with kernel 3.4, KVM should be able to do PCI passthrough even if the respective device shares its IRQ with other devices. Thus, two day ago, I took the time, got vanilla kernel 3.9, configured, compiled and installed, and got qemu 1.4.1, configured, compiled and installed.

Now, I am disappointed that the passthrough still doesn't work if I leave the other devices active, probably due to a silly error on my side. It still works if I inactivate the other devices which share their IRQ with the AVM card, but that actually was not the goal.

Furthermore, the last two sentences on the following page suggest that PCI passthrough of devices with shared IRQs is still not possible: http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM. So there is a contradiction between what has been stated when kernel 3.4 came out and what linux-kvm.org says.

Could anybody please tell me if PCI passthrough with shared IRQs is possible or not? Does anybody know why the AVM Fritz!card won't work without inactivating other devices? What mistake could lead to the current situation (passthrough works if other devices are inactivated, and does not work if not)?

This is my command line for starting the VM:

/usr/local/bin/qemu-system-x86_64 -machine type=pc,accel=kvm -cpu host -smp cores=1,threads=2,sockets=1 -rtc base=localtime,clock=host,driftfix=none -drive file=/dev/sda6,if=virtio,cache=none,index=0 -drive file=/dev/sdb,if=virtio,cache=none,index=1 -cdrom /dev/sr0 -pidfile ./qemu-garak.pid -boot c -k de -m 4096 -device pci-assign,host=01:05.0 -daemonize -usb -usbdevice "tablet" -name garak -net nic,vlan=0,model=virtio,macaddr=02:01:01:01:01:01 -net tap,vlan=0,ifname=virtnet1,script=/etc/qemu-ifup,downscript=/etc/qemu-ifup -vnc :1

This is the error message which I get if I don't inactivate the other devices:

qemu-system-x86_64: -device pci-assign,host=01:05.0: Failed to assign irq for "(null)": Input/output error
qemu-system-x86_64: -device pci-assign,host=01:05.0: Perhaps you are assigning a device that shares an IRQ with another device?
qemu-system-x86_64: -device pci-assign,host=01:05.0: Device 'kvm-pci-assign' could not be initialized

I don't understand why the first line talks about the device "(null)" (I would have excepted a device name or a PCI bus device number here), and I don't understand the second line (because that is exactly what kernels from 3.4 on should be able to do, right?).

By the way, my method for making the other devices which share their IRQ with the Fritz!card is the following:

modprobe -r kvm-intel
modprobe -r kvm

echo "1244 0a00" > /sys/bus/pci/drivers/pci-stub/new_id
echo 0000:01:05.0 > /sys/bus/pci/devices/0000:01:05.0/driver/unbind
echo 0000:01:05.0 > /sys/bus/pci/drivers/pci-stub/bind

echo "8086 3a39" > /sys/bus/pci/drivers/pci-stub/new_id
echo 0000:00:1a.2 > /sys/bus/pci/devices/0000:00:1a.2/driver/unbind
echo 0000:00:1a.2 > /sys/bus/pci/drivers/pci-stub/bind

echo "8086 3a35" > /sys/bus/pci/drivers/pci-stub/new_id
echo 0000:00:1d.1 > /sys/bus/pci/devices/0000:00:1d.1/driver/unbind
echo 0000:00:1d.1 > /sys/bus/pci/drivers/pci-stub/bind

modprobe kvm-intel


Thank you very much for any help,

Binarus

Re: PCI passthrough (shared IRQ) not working / contradicting information

[Jan Kiszka]

On 2013-05-03 09:33, Binarus wrote:
> Hi all,
> 
> a while ago, I have set up a KVM virtual machine with PCI passthrough.
> The device which is passed through is an AVM Fritz!card PCI v2.1.
> 
> This works, but there is a drawback: At that time, KVM was not able to
> do PCI passthrough for PCI devices with shared interrupt. So I had to
> find out which other devices were using the same interrupt, and had to
> inactivate them by handling them over to the PCI stub driver.
> 
> Then, one day, I have been very excited when I was reading that with
> kernel 3.4, KVM should be able to do PCI passthrough even if the
> respective device shares its IRQ with other devices. Thus, two day ago,
> I took the time, got vanilla kernel 3.9, configured, compiled and
> installed, and got qemu 1.4.1, configured, compiled and installed.
> 
> Now, I am disappointed that the passthrough still doesn't work if I
> leave the other devices active, probably due to a silly error on my
> side. It still works if I inactivate the other devices which share their
> IRQ with the AVM card, but that actually was not the goal.
> 
> Furthermore, the last two sentences on the following page suggest that
> PCI passthrough of devices with shared IRQs is still not possible:
> http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM. So
> there is a contradiction between what has been stated when kernel 3.4
> came out and what linux-kvm.org says.
> 
> Could anybody please tell me if PCI passthrough with shared IRQs is
> possible or not? Does anybody know why the AVM Fritz!card won't work
> without inactivating other devices? What mistake could lead to the
> current situation (passthrough works if other devices are inactivated,
> and does not work if not)?

The Fritz!CARD is very old and likely not PCI 2.3 compliant. But that is
required to share legacy interrupts.

Alex, does VFIO provide any convenient diagnostics if INTx sharing is
possible or not? If not, maybe we should add it at least over there.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux

Re: PCI passthrough (shared IRQ) not working / contradicting information

[Alex Williamson]

On Fri, 2013-05-03 at 10:35 +0200, Jan Kiszka wrote:
> On 2013-05-03 09:33, Binarus wrote:
> > Hi all,
> > 
> > a while ago, I have set up a KVM virtual machine with PCI passthrough.
> > The device which is passed through is an AVM Fritz!card PCI v2.1.
> > 
> > This works, but there is a drawback: At that time, KVM was not able to
> > do PCI passthrough for PCI devices with shared interrupt. So I had to
> > find out which other devices were using the same interrupt, and had to
> > inactivate them by handling them over to the PCI stub driver.
> > 
> > Then, one day, I have been very excited when I was reading that with
> > kernel 3.4, KVM should be able to do PCI passthrough even if the
> > respective device shares its IRQ with other devices. Thus, two day ago,
> > I took the time, got vanilla kernel 3.9, configured, compiled and
> > installed, and got qemu 1.4.1, configured, compiled and installed.
> > 
> > Now, I am disappointed that the passthrough still doesn't work if I
> > leave the other devices active, probably due to a silly error on my
> > side. It still works if I inactivate the other devices which share their
> > IRQ with the AVM card, but that actually was not the goal.
> > 
> > Furthermore, the last two sentences on the following page suggest that
> > PCI passthrough of devices with shared IRQs is still not possible:
> > http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM. So
> > there is a contradiction between what has been stated when kernel 3.4
> > came out and what linux-kvm.org says.
> > 
> > Could anybody please tell me if PCI passthrough with shared IRQs is
> > possible or not? Does anybody know why the AVM Fritz!card won't work
> > without inactivating other devices? What mistake could lead to the
> > current situation (passthrough works if other devices are inactivated,
> > and does not work if not)?
> 
> The Fritz!CARD is very old and likely not PCI 2.3 compliant. But that is
> required to share legacy interrupts.

Yep.  Binarus, the issue is that shared interrupts require hardware
support to mask the interrupt at the device.  This was introduced as
part of the PCI 2.3 standard.  If the device does not support this then
exclusive interrupts are still required.

> Alex, does VFIO provide any convenient diagnostics if INTx sharing is
> possible or not? If not, maybe we should add it at least over there.

It does not.  It's actually easier to test outside of qemu as vfio will
emulate PCI2.3 support.  Here's a simple shell script requiring only
sudo and setpci:

#!/bin/sh

DESC=$(lspci -s $1)

if [ "x$DESC" == "x" ]; then
	echo "Device $1 not found"
	exit 1
fi

ORIGCMD=$(printf 0x%04x $(setpci -s $1 COMMAND))
NEWCMD=$(printf 0x%04x $(( $ORIGCMD ^ 0x400 )))
setpci -s $1 COMMAND=$NEWCMD
NEWCMD=$(printf 0x%04x $(setpci -s $1 COMMAND))

if [ $ORIGCMD == $NEWCMD ]; then
	echo "Device $1 does NOT support PCI 2.3 INTx disable"
	exit 1
fi

setpci -s $1 COMMAND=$ORIGCMD
echo "Device $1 supports PCI 2.3 INTx disable"

Re: PCI passthrough (shared IRQ) not working / contradicting information

[Jan Kiszka]

On 2013-05-03 16:37, Alex Williamson wrote:
> On Fri, 2013-05-03 at 10:35 +0200, Jan Kiszka wrote:
>> On 2013-05-03 09:33, Binarus wrote:
>>> Hi all,
>>>
>>> a while ago, I have set up a KVM virtual machine with PCI passthrough.
>>> The device which is passed through is an AVM Fritz!card PCI v2.1.
>>>
>>> This works, but there is a drawback: At that time, KVM was not able to
>>> do PCI passthrough for PCI devices with shared interrupt. So I had to
>>> find out which other devices were using the same interrupt, and had to
>>> inactivate them by handling them over to the PCI stub driver.
>>>
>>> Then, one day, I have been very excited when I was reading that with
>>> kernel 3.4, KVM should be able to do PCI passthrough even if the
>>> respective device shares its IRQ with other devices. Thus, two day ago,
>>> I took the time, got vanilla kernel 3.9, configured, compiled and
>>> installed, and got qemu 1.4.1, configured, compiled and installed.
>>>
>>> Now, I am disappointed that the passthrough still doesn't work if I
>>> leave the other devices active, probably due to a silly error on my
>>> side. It still works if I inactivate the other devices which share their
>>> IRQ with the AVM card, but that actually was not the goal.
>>>
>>> Furthermore, the last two sentences on the following page suggest that
>>> PCI passthrough of devices with shared IRQs is still not possible:
>>> http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM. So
>>> there is a contradiction between what has been stated when kernel 3.4
>>> came out and what linux-kvm.org says.
>>>
>>> Could anybody please tell me if PCI passthrough with shared IRQs is
>>> possible or not? Does anybody know why the AVM Fritz!card won't work
>>> without inactivating other devices? What mistake could lead to the
>>> current situation (passthrough works if other devices are inactivated,
>>> and does not work if not)?
>>
>> The Fritz!CARD is very old and likely not PCI 2.3 compliant. But that is
>> required to share legacy interrupts.
> 
> Yep.  Binarus, the issue is that shared interrupts require hardware
> support to mask the interrupt at the device.  This was introduced as
> part of the PCI 2.3 standard.  If the device does not support this then
> exclusive interrupts are still required.
> 
>> Alex, does VFIO provide any convenient diagnostics if INTx sharing is
>> possible or not? If not, maybe we should add it at least over there.
> 
> It does not.  It's actually easier to test outside of qemu as vfio will
> emulate PCI2.3 support.  Here's a simple shell script requiring only
> sudo and setpci:
> 
> #!/bin/sh
> 
> DESC=$(lspci -s $1)
> 
> if [ "x$DESC" == "x" ]; then
> 	echo "Device $1 not found"
> 	exit 1
> fi
> 
> ORIGCMD=$(printf 0x%04x $(setpci -s $1 COMMAND))
> NEWCMD=$(printf 0x%04x $(( $ORIGCMD ^ 0x400 )))
> setpci -s $1 COMMAND=$NEWCMD
> NEWCMD=$(printf 0x%04x $(setpci -s $1 COMMAND))
> 
> if [ $ORIGCMD == $NEWCMD ]; then
> 	echo "Device $1 does NOT support PCI 2.3 INTx disable"
> 	exit 1
> fi
> 
> setpci -s $1 COMMAND=$ORIGCMD
> echo "Device $1 supports PCI 2.3 INTx disable"
> 

Easier for the implementers POV... ;)

Well, please push it at least to our wiki page.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux