vfio-pci + no-kvm-irqchip = oops

vfio-pci + no-kvm-irqchip = oops

[Jan Kiszka]

Hi Alex,

just tried vfio-pci with user-space irqchip (qemu-system-x86_64 -device
vfio-pci,host=... -enable-kvm -no-kvm-irqchip). This ends up in the
following oops:

[   61.908453] BUG: unable to handle kernel NULL pointer dereference at 0000000000000128
[   61.908462] IP: [<ffffffffa0146d87>] kvm_irq_map_gsi+0x7c/0xd7 [kvm]
[   61.908488] PGD 0 
[   61.908491] Oops: 0000 [#1] PREEMPT SMP 
[   61.908496] Modules linked in: vfio_iommu_type1 vfio_pci vfio vfio_virqfd xt_tcpudp xt_pkttype xt_limit fuse af_packet snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_raw ipt_REJECT nf_reject_ipv4 iptable_raw iptable_filter ip6table_mangle nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_ipv4 nf_defrag_ipv4 ip_tables xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables ipv6 dm_mod snd_hda_codec_generic vhost_net vhost tun kvm_intel snd_hda_intel kvm snd_hda_controller snd_hda_codec i2c_i801 lpc_ich sg snd_hda_core snd_pcm mfd_core snd_timer snd evdev psmouse soundcore pcspkr serio_raw e1000 intel_agp button intel_gtt virtio_scsi fan thermal_sys ata_generic ahci libahci
[   61.908563] CPU: 2 PID: 5322 Comm: qemu-system-x86 Not tainted 4.1.0-rc6-dbg+ #95
[   61.908568] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.8.1-0-g4adadbd-20150316_085822-nilsson.home.kraxel.org 04/01/2014
[   61.908574] task: ffff880031fe6a10 ti: ffff880017460000 task.ti: ffff880017460000
[   61.908578] RIP: 0010:[<ffffffffa0146d87>]  [<ffffffffa0146d87>] kvm_irq_map_gsi+0x7c/0xd7 [kvm]
[   61.908589] RSP: 0018:ffff880017463c58  EFLAGS: 00010046
[   61.908592] RAX: 0000000000000000 RBX: ffff880031f94000 RCX: 000000000081c000
[   61.908596] RDX: 0000000000000001 RSI: ffff880031f94388 RDI: 0000000000000046
[   61.908600] RBP: ffff880017463c78 R08: ffffffff821d0f38 R09: 0000000000000000
[   61.908603] R10: ffff880031f94c98 R11: 0000000000000246 R12: ffff880017463c98
[   61.908607] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88001a95de00
[   61.908613] FS:  00007f05e2c3aae0(0000) GS:ffff88003fd00000(0000) knlGS:0000000000000000
[   61.908618] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   61.908634] CR2: 0000000000000128 CR3: 000000001a8ce000 CR4: 00000000001427a0
[   61.908641] DR0: ffffffff8278f3d8 DR1: 0000000000000000 DR2: 0000000000000000
[   61.908646] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600
[   61.908651] Stack:
[   61.908654]  ffff88001a95de00 ffff880031f94238 ffff880031f94388 ffff880031f94c60
[   61.908662]  ffff880017463d78 ffffffffa0145a74 ffff880017463d08 ffffffff81089fcc
[   61.908669]  0000000000000001 000000000006d950 0000000200000001 ffffffff82159f50
[   61.908676] Call Trace:
[   61.908696]  [<ffffffffa0145a74>] irqfd_update+0x2a/0xaf [kvm]
[   61.908727]  [<ffffffff81089fcc>] ? __lock_acquire+0xa1f/0x12d6
[   61.908739]  [<ffffffffa01466c2>] ? kvm_irqfd+0x486/0x5d7 [kvm]
[   61.908750]  [<ffffffffa0146709>] kvm_irqfd+0x4cd/0x5d7 [kvm]
[   61.908761]  [<ffffffffa01466c2>] ? kvm_irqfd+0x486/0x5d7 [kvm]
[   61.908772]  [<ffffffffa01444a3>] kvm_vm_ioctl+0x35d/0x662 [kvm]
[   61.908783]  [<ffffffff813034b6>] ? debug_smp_processor_id+0x17/0x19
[   61.908793]  [<ffffffff8117913b>] do_vfs_ioctl+0x3bb/0x47a
[   61.908798]  [<ffffffff81182fbf>] ? __fget+0x5/0x186
[   61.908803]  [<ffffffff811831cc>] ? __fget_light+0x65/0x75
[   61.908808]  [<ffffffff81183a32>] ? __fd_install+0x9a/0xa6
[   61.908814]  [<ffffffff8117924d>] SyS_ioctl+0x53/0x81
[   61.908825]  [<ffffffff8152f4ee>] system_call_fastpath+0x12/0x76
[   61.908830] Code: 00 e8 73 ff f3 e0 85 c0 75 1f 48 c7 c2 ff 3d 18 a0 be 35 00 00 00 48 c7 c7 28 3e 18 a0 c6 05 91 a1 04 00 01 e8 a6 0b f4 e0 31 c0 <45> 3b b5 28 01 00 00 73 49 4b 8b 94 f5 30 01 00 00 48 85 d2 74 
[   61.908875] RIP  [<ffffffffa0146d87>] kvm_irq_map_gsi+0x7c/0xd7 [kvm]
[   61.908887]  RSP <ffff880017463c58>
[   61.908890] CR2: 0000000000000128

This test was in QEMU, ie. nested, but the oops is reproducible on real
hw as well. And on older kernels, e.g. 3.18.

Known issue? Some idea what goes wrong?

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: vfio-pci + no-kvm-irqchip = oops

[Alex Williamson]

On Thu, 2015-06-11 at 15:37 +0200, Jan Kiszka wrote:
> Hi Alex,
> 
> just tried vfio-pci with user-space irqchip (qemu-system-x86_64 -device
> vfio-pci,host=... -enable-kvm -no-kvm-irqchip). This ends up in the
> following oops:
> 
> [   61.908453] BUG: unable to handle kernel NULL pointer dereference at 0000000000000128
> [   61.908462] IP: [<ffffffffa0146d87>] kvm_irq_map_gsi+0x7c/0xd7 [kvm]
> [   61.908488] PGD 0 
> [   61.908491] Oops: 0000 [#1] PREEMPT SMP 
> [   61.908496] Modules linked in: vfio_iommu_type1 vfio_pci vfio vfio_virqfd xt_tcpudp xt_pkttype xt_limit fuse af_packet snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_raw ipt_REJECT nf_reject_ipv4 iptable_raw iptable_filter ip6table_mangle nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_ipv4 nf_defrag_ipv4 ip_tables xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables ipv6 dm_mod snd_hda_codec_generic vhost_net vhost tun kvm_intel snd_hda_intel kvm snd_hda_controller snd_hda_codec i2c_i801 lpc_ich sg snd_hda_core snd_pcm mfd_core snd_timer snd evdev psmouse soundcore pcspkr serio_raw e1000 intel_agp button intel_gtt virtio_scsi fan thermal_sys ata_generic ahci libahci
> [   61.908563] CPU: 2 PID: 5322 Comm: qemu-system-x86 Not tainted 4.1.0-rc6-dbg+ #95
> [   61.908568] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.8.1-0-g4adadbd-20150316_085822-nilsson.home.kraxel.org 04/01/2014
> [   61.908574] task: ffff880031fe6a10 ti: ffff880017460000 task.ti: ffff880017460000
> [   61.908578] RIP: 0010:[<ffffffffa0146d87>]  [<ffffffffa0146d87>] kvm_irq_map_gsi+0x7c/0xd7 [kvm]
> [   61.908589] RSP: 0018:ffff880017463c58  EFLAGS: 00010046
> [   61.908592] RAX: 0000000000000000 RBX: ffff880031f94000 RCX: 000000000081c000
> [   61.908596] RDX: 0000000000000001 RSI: ffff880031f94388 RDI: 0000000000000046
> [   61.908600] RBP: ffff880017463c78 R08: ffffffff821d0f38 R09: 0000000000000000
> [   61.908603] R10: ffff880031f94c98 R11: 0000000000000246 R12: ffff880017463c98
> [   61.908607] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88001a95de00
> [   61.908613] FS:  00007f05e2c3aae0(0000) GS:ffff88003fd00000(0000) knlGS:0000000000000000
> [   61.908618] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [   61.908634] CR2: 0000000000000128 CR3: 000000001a8ce000 CR4: 00000000001427a0
> [   61.908641] DR0: ffffffff8278f3d8 DR1: 0000000000000000 DR2: 0000000000000000
> [   61.908646] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600
> [   61.908651] Stack:
> [   61.908654]  ffff88001a95de00 ffff880031f94238 ffff880031f94388 ffff880031f94c60
> [   61.908662]  ffff880017463d78 ffffffffa0145a74 ffff880017463d08 ffffffff81089fcc
> [   61.908669]  0000000000000001 000000000006d950 0000000200000001 ffffffff82159f50
> [   61.908676] Call Trace:
> [   61.908696]  [<ffffffffa0145a74>] irqfd_update+0x2a/0xaf [kvm]
> [   61.908727]  [<ffffffff81089fcc>] ? __lock_acquire+0xa1f/0x12d6
> [   61.908739]  [<ffffffffa01466c2>] ? kvm_irqfd+0x486/0x5d7 [kvm]
> [   61.908750]  [<ffffffffa0146709>] kvm_irqfd+0x4cd/0x5d7 [kvm]
> [   61.908761]  [<ffffffffa01466c2>] ? kvm_irqfd+0x486/0x5d7 [kvm]
> [   61.908772]  [<ffffffffa01444a3>] kvm_vm_ioctl+0x35d/0x662 [kvm]
> [   61.908783]  [<ffffffff813034b6>] ? debug_smp_processor_id+0x17/0x19
> [   61.908793]  [<ffffffff8117913b>] do_vfs_ioctl+0x3bb/0x47a
> [   61.908798]  [<ffffffff81182fbf>] ? __fget+0x5/0x186
> [   61.908803]  [<ffffffff811831cc>] ? __fget_light+0x65/0x75
> [   61.908808]  [<ffffffff81183a32>] ? __fd_install+0x9a/0xa6
> [   61.908814]  [<ffffffff8117924d>] SyS_ioctl+0x53/0x81
> [   61.908825]  [<ffffffff8152f4ee>] system_call_fastpath+0x12/0x76
> [   61.908830] Code: 00 e8 73 ff f3 e0 85 c0 75 1f 48 c7 c2 ff 3d 18 a0 be 35 00 00 00 48 c7 c7 28 3e 18 a0 c6 05 91 a1 04 00 01 e8 a6 0b f4 e0 31 c0 <45> 3b b5 28 01 00 00 73 49 4b 8b 94 f5 30 01 00 00 48 85 d2 74 
> [   61.908875] RIP  [<ffffffffa0146d87>] kvm_irq_map_gsi+0x7c/0xd7 [kvm]
> [   61.908887]  RSP <ffff880017463c58>
> [   61.908890] CR2: 0000000000000128
> 
> This test was in QEMU, ie. nested, but the oops is reproducible on real
> hw as well. And on older kernels, e.g. 3.18.
> 
> Known issue? Some idea what goes wrong?

Hi Jan,

I suppose it works without -enable-kvm, right?  I thought we had the
right set of flags to avoid routing interrupts through KVM without
irqchip, but apparently not.  Seems like two issues, a) fix the vfio
logic to not attempt this path without irqchip, b) KVM shouldn't allow
this to oops.  It might be sufficient to only fix b), but QEMU would
probably complain more than it should about setup failures.  Thanks,

Alex