kernel BUG at drivers/iommu/intel-iommu.c:732! , kernel 4.14.39

kernel BUG at drivers/iommu/intel-iommu.c:732! , kernel 4.14.39

[plexo rama]

[-- Attachment #1.1: Type: text/plain, Size: 3960 bytes --]

Hi all,


I receive the following kernel dump when trying to load QLogic qede in-tree
driver for a dual-port

*QLogic* Corp. FastLinQ QL41000 Series 10/25/40/50GbE Controller (rev 02).
One of the ports (45:00.0) is bound to the vfio-pci driver for
pci-passthrough to a KVM/QEMU guest. Id like to use the PF of the second
port on the host. I've unbound the pci device (45:00.1) from vfio-pci, but when
I modprobe qede, I receive this:




[1290644.615499] QLogic FastLinQ 4xxxx Core Module qed 8.10.11.21

[1290644.619128] qede_init: QLogic FastLinQ 4xxxx Ethernet Driver qede
8.10.10.21

[1290644.619347] qede 0000:45:00.1: enabling device (0400 -> 0402)

[1290644.632341] ------------[ cut here ]------------

[1290644.632344] kernel BUG at drivers/iommu/intel-iommu.c:732!

[1290644.632352] invalid opcode: 0000 [#1] SMP PTI

[1290644.632353] Modules linked in: qede(+) qed nbd vfio_pci
vfio_iommu_type1 vfio vhost_net vhost tap tcp_lp fuse dummy vfio_virqfd
ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat tun
bridge stp llc ebtable_filter ebtables team_mode_loadbalance
x86_pkg_temp_thermal coretemp kvm_intel cdc_ether kvm ioatdma irqbypass
usbnet dca shpchp hed nfsd efivarfs xfs libcrc32c team i40e ipmi_si
ipmi_devintf ipmi_msghandler [last unloaded: qed]

[1290644.632387] CPU: 24 PID: 106151 Comm: kworker/24:2 Not tainted 4.14.39
#1

[1290644.632389] Hardware name: Lenovo ThinkSystem SR850
-[7X19CTO1WW]-/-[7X19CTO1WW]-, BIOS -[TEE120N-1.20]- 10/16/2017

[1290644.632397] Workqueue: events work_for_cpu_fn

[1290644.632399] task: ffff8aec3a830180 task.stack: ffff9bc39a284000

[1290644.632409] RIP: 0010:domain_get_iommu+0x45/0x60

[1290644.632410] RSP: 0018:ffff9bc39a287b28 EFLAGS: 00010202

[1290644.632412] RAX: 0000000000000000 RBX: ffff8bacb72770a0 RCX:
0000000000000000

[1290644.632413] RDX: 0000000000000000 RSI: 000000b2e0d70000 RDI:
ffff8b4c9ff02140

[1290644.632414] RBP: 0000000000000000 R08: ffffffffffffffff R09:
ffff8ae0e0d70000

[1290644.632415] R10: 0000000000000004 R11: ffff8aee3ffded00 R12:
000000b2e0d70000

[1290644.632416] R13: ffff8b4c9ff02140 R14: ffffffffffffffff R15:
0000000000010000

[1290644.632417] FS:  0000000000000000(0000) GS:ffff8aecbf200000(0000)
knlGS:0000000000000000

[1290644.632418] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033

[1290644.632420] CR2: 00007f3ecd5f6000 CR3: 0000000254c0a002 CR4:
00000000007626e0

[1290644.632420] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000

[1290644.632421] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400

[1290644.632422] PKRU: 55555554

[1290644.632423] Call Trace:

[1290644.632427]  __intel_map_single+0x5c/0x170

[1290644.632430]  intel_alloc_coherent+0xa8/0x120

[1290644.632446]  qed_ilt_blk_alloc+0xea/0x1f0 [qed]

[1290644.632455]  ? __kmalloc+0xe5/0x1e0

[1290644.632459]  qed_cxt_tables_alloc+0xb6/0x560 [qed]

[1290644.632465]  qed_resc_alloc+0x235/0x510 [qed]

[1290644.632472]  qed_slowpath_start+0x14b/0x6e0 [qed]

[1290644.632481]  __qede_probe.isra.31+0xe3/0x600 [qede]

[1290644.632487]  ? update_curr+0xac/0x140

[1290644.632491]  local_pci_probe+0x19/0x50

[1290644.632494]  work_for_cpu_fn+0xb/0x20

[1290644.632497]  process_one_work+0x133/0x340

[1290644.632499]  worker_thread+0x1ac/0x3b0

[1290644.632502]  kthread+0xf7/0x130

[1290644.632505]  ? process_one_work+0x340/0x340

[1290644.632506]  ? __kthread_parkme+0x60/0x60

[1290644.632516]  ret_from_fork+0x1f/0x30

[1290644.632518] Code: 48 83 c7 08 31 c0 eb 0b 48 83 c7 04 8b 4f fc 85 c9
75 1a 83 c0 01 39 d0 75 ee 31 c0 c3 31 d2 48 8b 05 08 a0 3e 01 48 8b 04 10
c3 <0f> 0b 85 c0 78 e9 48 98 48 8d 14 c5 00 00 00 00 eb e2 66 0f 1f

[1290644.632541] RIP: domain_get_iommu+0x45/0x60 RSP: ffff9bc39a287b28

[1290644.632544] ---[ end trace 80f7ccb19898ef42 ]---




What am I doing wrong? How can I make use of the second port on my host?


I'd really appreciate any help or suggestions how to make this work.


Thanks!

- Plexo

[-- Attachment #1.2: Type: text/html, Size: 14264 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: kernel BUG at drivers/iommu/intel-iommu.c:732! , kernel 4.14.39

[Alex Williamson]

On Tue, 22 May 2018 13:30:20 +0200
plexo rama <plexorama-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

> Hi all,
> 
> 
> I receive the following kernel dump when trying to load QLogic qede in-tree
> driver for a dual-port
> 
> *QLogic* Corp. FastLinQ QL41000 Series 10/25/40/50GbE Controller (rev 02).
> One of the ports (45:00.0) is bound to the vfio-pci driver for
> pci-passthrough to a KVM/QEMU guest. Id like to use the PF of the second
> port on the host. I've unbound the pci device (45:00.1) from vfio-pci, but when
> I modprobe qede, I receive this:

AIUI, this device does not provide a PCIe ACS capability, therefore the
kernel assumes the functions of the device are not isolated and places
them in the same IOMMU group.  The vendor has also not indicated that
the PCIe functions are DMA isolated nor provided quirks to indicate any
sort of isolation.  vfio-pci requires DMA isolation between users,
whether they be separate userspace instances or between kernel and
users.  Via the process above, assuming 45:00.0 is in use by a user
instance when it occurs, you've compromised the isolation.  Generally
the response to this is a BUG in the vfio code, but in the case of this
driver it attempts to map DMA in the probe routine and therefore
encounters a different BUG where one device within the group has a
mapping through the IOMMU API and the other is attempting to map
through the DMA API, where the domains within intel-iommu are
incompatible.

In short, the device you're using doesn't support the DMA isolation
necessary for the use case you're trying to impose on it and the kernel
protects itself by preventing it, harshly.  Perhaps ask QLogic if
there's actually ACS equivalent isolation provided between PCIe
functions of the device and we can implement quirks for it.  Otherwise
you'll need a separate card to use for the host.  Thanks,

Alex