RE: vfio issue in qemu 2.5

["Goutham GS" <goutham@zadarastorage.com>]

Hello Alex,

Sorry, I got the timestamps wrong yesterday.

I see the following messages in kern.log at the time of the issue:

Jan 31 03:46:39 qa2-sn2 kernel: [228419.858857] vfio-pci 0000:04:01.4:
enabling device (0000 -> 0002)
Jan 31 03:46:39 qa2-sn2 kernel: [228419.970492] vfio-pci 0000:04:02.7:
enabling device (0000 -> 0002)
Jan 31 03:46:39 qa2-sn2 kernel: [228420.082435] IOMMU: no free domain ids
Jan 31 03:46:39 qa2-sn2 kernel: [228420.124440] IOMMU: no free domain ids

Does this say anything?

Regards,
Goutham.

-----Original Message-----
From: Goutham GS [mailto:goutham@zadarastorage.com] 
Sent: Thursday, February 4, 2016 11:53 PM
To: 'Alex Williamson' <alex.williamson@redhat.com>
Cc: kvm@vger.kernel.org
Subject: RE: vfio issue in qemu 2.5

Hello Alex,

Thanks for your quick response.

Unfortunately we are tied to this kernel. Probably we can move to 3.19, if
we are sure of the benefits . Not sure.

Regarding dmesg, there is nothing on the host and the VM never came up to
the point where we could collect dmesg.

Regards,
Goutham.

-----Original Message-----
From: Alex Williamson [mailto:alex.williamson@redhat.com]
Sent: Thursday, February 4, 2016 10:32 PM
To: Goutham GS <goutham@zadarastorage.com>
Cc: kvm@vger.kernel.org
Subject: Re: vfio issue in qemu 2.5

On Thu, 4 Feb 2016 20:31:17 +0530
"Goutham GS" <goutham@zadarastorage.com> wrote:

> Hi All,
> 
> We are facing a vfio issue on qemu 2.5. Really appreciate any help or 
> pointers. Details are as below:
> 
> We are using qemu 2.5 compiled out of git commit
> 0b0571dd246871f18b7d64b5279511e91e2a7bf6 and are using Linux Kernel
> 3.18.19 for both host and the VM. We are also using KVM VM with 
> pci-assign'ed SRIOV VF interfaces.
> 
> The issue happens once in a while when a running VM is rebooted. On 
> boot, the VM hits the following error and stops.
> 
> qemu-system-x86_64: -device
> vfio-pci,host=04:00.7,id=hostdev2,bus=pci.0,addr=0x9: vfio: failed to 
> set iommu for container: Bad address
> qemu-system-x86_64: -device
> vfio-pci,host=04:00.7,id=hostdev2,bus=pci.0,addr=0x9: vfio: failed to 
> setup container for group 40
> qemu-system-x86_64: -device
> vfio-pci,host=04:00.7,id=hostdev2,bus=pci.0,addr=0x9: vfio: failed to 
> get group 40
> qemu-system-x86_64: -device
> vfio-pci,host=04:00.7,id=hostdev2,bus=pci.0,addr=0x9: Device 
> initialization failed
> 
> Strange thing is, once this error is hit, no further VMs can be 
> spawned on the host and all of them run into the same problem.
> However a reboot of the host appears to solve the issue.
> 
> I have attached the relevant logs.

Is it possible to try a newer kernel on the host?  "Bad address" is -EFAULT,
but I'm not actually able to spot a return path for the VFIO_SET_IOMMU ioctl
that returns -EFAULT.  Is there anything in dmesg when this triggers?
Thanks,

Alex