[Bug 196685] New: System becomes unresponsive after intel_iommu kernel error

[Bug 196685] New: System becomes unresponsive after intel_iommu kernel error

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=196685

            Bug ID: 196685
           Summary: System becomes unresponsive after intel_iommu kernel
                    error
           Product: Virtualization
           Version: unspecified
    Kernel Version: 4.12.7
          Hardware: Intel
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: kvm
          Assignee: virtualization_kvm@kernel-bugs.osdl.org
          Reporter: jaszhix@gmail.com
        Regression: No

Created attachment 257969
  --> https://bugzilla.kernel.org/attachment.cgi?id=257969&action=edit
Kernel error in /var/log/kern.log

I using a qemu virtual machine with VGA pass through. I have two Intel NICs in
use by my motherboard, and I am passing through one of them so I can network
with the VM over gigabit ethernet. I have been using this without issues for
several months, but recently my system started becoming unstable. Before each
crash, the network connection would abruptly end, and I suspect that is related
to this. My desktop environment locks up, and after logging into my user
account in another terminal, it is flooded with UFW block messages and no
commands using sudo return anything. A kernel error appearing to be caused by
intel_iommu is left in the kernel log. After this has happened yesterday, I ran
memtest86 and my memory is okay. I am using an engineering sample CPU that is a
down-clocked Intel E5 2680 v4. I haven't had any issues with this chip except
recently when using VT-d.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 196685] System becomes unresponsive after intel_iommu kernel error

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=196685

--- Comment #1 from Jason Hicks (jaszhix@gmail.com) ---
Created attachment 257971
  --> https://bugzilla.kernel.org/attachment.cgi?id=257971&action=edit
lspci -vt

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 196685] System becomes unresponsive after intel_iommu kernel error

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=196685

--- Comment #2 from Jason Hicks (jaszhix@gmail.com) ---
Created attachment 257973
  --> https://bugzilla.kernel.org/attachment.cgi?id=257973&action=edit
inxi -F

NVIDIA Device 1b81 is a Geforce GTX 1070 which is passed through to the VM.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 196685] System becomes unresponsive after intel_iommu kernel error

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=196685

--- Comment #3 from Jason Hicks (jaszhix@gmail.com) ---
Created attachment 257975
  --> https://bugzilla.kernel.org/attachment.cgi?id=257975&action=edit
iommu groups

Output of `for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1
-mindepth 1 -type d); do echo "IOMMU group $(basename "$iommu_group")"; for
device in $(\ls -1 "$iommu_group"/devices/); do echo -n $'\t'; lspci -nns
"$device"; done; done`

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 196685] System becomes unresponsive after intel_iommu kernel error

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=196685

--- Comment #4 from Jason Hicks (jaszhix@gmail.com) ---
Created attachment 257977
  --> https://bugzilla.kernel.org/attachment.cgi?id=257977&action=edit
dmesg | grep DMAR

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 196685] System becomes unresponsive after intel_iommu kernel error

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=196685

changbin.du@hotmail.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |changbin.du@hotmail.com

--- Comment #5 from changbin.du@hotmail.com ---
(In reply to Jason Hicks from comment #0)
> Created attachment 257969 [details]
> Kernel error in /var/log/kern.log
> 
> I using a qemu virtual machine with VGA pass through. I have two Intel NICs
> in use by my motherboard, and I am passing through one of them so I can
> network with the VM over gigabit ethernet. I have been using this without
> issues for several months, but recently my system started becoming unstable.
> Before each crash, the network connection would abruptly end, and I suspect
> that is related to this. My desktop environment locks up, and after logging
> into my user account in another terminal, it is flooded with UFW block
> messages and no commands using sudo return anything. A kernel error
> appearing to be caused by intel_iommu is left in the kernel log. After this
> has happened yesterday, I ran memtest86 and my memory is okay. I am using an
> engineering sample CPU that is a down-clocked Intel E5 2680 v4. I haven't
> had any issues with this chip except recently when using VT-d.

Checking your dmesg, there are several kernel oops from different modules. Can
you share a full dmesg start from boot, to get the original error?

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 196685] System becomes unresponsive after intel_iommu kernel error

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=196685

--- Comment #6 from Jason Hicks (jaszhix@gmail.com) ---
Created attachment 257979
  --> https://bugzilla.kernel.org/attachment.cgi?id=257979&action=edit
dmesg Aug 16

Log from boot to the error event.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 196685] System becomes unresponsive after intel_iommu kernel error

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=196685

--- Comment #7 from Jason Hicks (jaszhix@gmail.com) ---
I should also add I am using Docker, so there are a lot of bridged network
connections originating from it. I usually can work in the VM for a few hours
before it crashes.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 196685] System becomes unresponsive after intel_iommu kernel error

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=196685

--- Comment #8 from changbin.du@hotmail.com ---
If my code match yours, at least there 2 issues:
First, kvm detected a warnning condition when delivering irq to guest.
Second, the WARN caused a paging request oops when executing the trap handler
of the UD0 instrunction.

[36330.176867] BUG: unable to handle kernel paging request at ffffffffc0633f92
[36330.176888] IP: report_bug+0x94/0x120
[36330.176891] PGD 168c20c067 
[36330.176893] P4D 168c20c067 
[36330.176895] PUD 168c20e067 
[36330.176898] PMD 2008194067 
[36330.176901] PTE 80000020017bc161
[36330.176904] 
[36330.176911] Oops: 0003 [#1] SMP
[36330.176913] Modules linked in: xt_recent xt_nat veth nls_utf8 udf crc_itu_t
rfcomm ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink
xfrm_user xfrm_algo iptable_nat nf_nat_ipv4 br_netfilter overlay
vmw_vsock_vmci_transport vsock vmw_vmci bridge stp llc bnep binfmt_misc
nls_iso8859_1 snd_hda_codec_hdmi nvidia_uvm(POE) arc4 intel_rapl
x86_pkg_temp_thermal intel_powerclamp coretemp snd_hda_codec_realtek
snd_hda_codec_generic crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc
snd_hda_intel snd_hda_codec btusb btrtl btbcm btintel aesni_intel iwlmvm
bluetooth nvidia_drm(POE) aes_x86_64 nvidia_modeset(POE) snd_hda_core
crypto_simd input_leds usblp snd_hwdep hid_sony glue_helper joydev ff_memless
ecdh_generic cryptd nvidia(POE) mac80211 snd_pcm intel_cstate drm_kms_helper
snd_seq_midi
[36330.176991]  snd_seq_midi_event intel_rapl_perf drm snd_rawmidi iwlwifi
fb_sys_fops mxm_wmi snd_seq syscopyarea sysfillrect sysimgblt snd_seq_device
snd_timer cfg80211 snd mei_me mei lpc_ich soundcore shpchp wmi mac_hid
kvm_intel kvm ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 xt_hl ip6t_rt
nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT nf_reject_ipv4 nf_log_ipv4
nf_log_common xt_LOG xt_multiport xt_limit xt_tcpudp xt_addrtype
nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack ip6table_filter ip6_tables
nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat
nf_conntrack_ftp nf_conntrack libcrc32c iptable_filter ip_tables x_tables
parport_pc ppdev lp parport autofs4 btrfs xor raid6_pq dm_mirror dm_region_hash
dm_log vhost_net vhost tap vfio_pci vfio_virqfd irqbypass vfio_iommu_type1 vfio
hid_logitech_hidpp
[36330.177074]  hid_logitech_dj hid_generic usbhid hid uas usb_storage igb
e1000e dca ptp ahci i2c_algo_bit pps_core libahci
[36330.177099] CPU: 13 PID: 8711 Comm: qemu-system-x86 Tainted: P           OE 
 4.12.7-041207-generic #201708160856
[36330.177103] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X99
Taichi, BIOS P1.10 06/29/2016
[36330.177107] task: ffff94808147c380 task.stack: ffffa46262f7c000
[36330.177114] RIP: 0010:report_bug+0x94/0x120
[36330.177118] RSP: 0018:ffffa46262f7f7a0 EFLAGS: 00010202
[36330.177123] RAX: 0000000000000907 RBX: ffffa46262f7f908 RCX:
ffffffffc0633f88
[36330.177127] RDX: 0000000000000001 RSI: 0000000000001359 RDI:
0000000000000001
[36330.177130] RBP: ffffa46262f7f7c0 R08: ffffa46262f80000 R09:
00000000000003ce
[36330.177134] R10: ffffffff83206a80 R11: fffff803741fea0f R12:
ffffffffc0614073
[36330.177137] R13: ffffffffc0633a98 R14: 0000000000000004 R15:
ffffa46262f7f908
[36330.177153] FS:  00007f5b08c67700(0000) GS:ffff94841f140000(0000)
knlGS:00000000003f2000
[36330.177157] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[36330.177161] CR2: ffffffffc0633f92 CR3: 0000001c883de000 CR4:
00000000003426e0
[36330.177165] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[36330.177168] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[36330.177171] Call Trace:
[36330.177186]  ? vmx_deliver_posted_interrupt+0xe3/0xf0 [kvm_intel]
[36330.177196]  fixup_bug+0x2e/0x50
[36330.177203]  do_trap+0x119/0x150
[36330.177209]  do_error_trap+0x89/0x110
[36330.177218]  ? vmx_deliver_posted_interrupt+0xe3/0xf0 [kvm_intel]
[36330.177224]  ? check_preempt_curr+0x74/0x80
[36330.177229]  ? ttwu_do_wakeup+0x1e/0x140
[36330.177233]  ? ttwu_do_activate+0x77/0x80
[36330.177240]  do_invalid_op+0x20/0x30
[36330.177247]  invalid_op+0x1e/0x30
[36330.177255] RIP: 0010:vmx_deliver_posted_interrupt+0xe3/0xf0 [kvm_intel]
[36330.177259] RSP: 0018:ffffa46262f7f9b0 EFLAGS: 00010202
[36330.177263] RAX: 00000c0000f20003 RBX: ffff948081448000 RCX:
0000000000000000
[36330.177267] RDX: ffff94808814f1e0 RSI: 00000000000000d2 RDI:
ffff948081448000
[36330.177270] RBP: ffffa46262f7f9b8 R08: 0000000000000000 R09:
0000000000000000
[36330.177273] R10: fffff8037400cf10 R11: fffff803741fea0f R12:
00000000000000d2
[36330.177277] R13: ffff94831fcf6700 R14: 0000000000000000 R15:
ffff948081448000
[36330.177326]  __apic_accept_irq+0x25b/0x300 [kvm]
[36330.177351]  kvm_apic_set_irq+0x2a/0x30 [kvm]
[36330.177374]  kvm_irq_delivery_to_apic+0x20b/0x2a0 [kvm]
[36330.177396]  kvm_lapic_reg_write+0x112/0x640 [kvm]
[36330.177417]  kvm_hv_vapic_msr_write+0x34/0x70 [kvm]
[36330.177439]  kvm_hv_set_msr_common+0x303/0x9d0 [kvm]
[36330.177461]  kvm_set_msr_common+0x64c/0xc30 [kvm]
[36330.177484]  ? x86_emulate_insn+0x161/0xdd0 [kvm]
[36330.177491]  vmx_set_msr+0xba/0x7d0 [kvm_intel]
[36330.177510]  kvm_set_msr+0x41/0x70 [kvm]
[36330.177516]  handle_wrmsr+0x59/0x140 [kvm_intel]
[36330.177521]  vmx_handle_exit+0xab/0xb40 [kvm_intel]
[36330.177527]  ? vmx_vcpu_run+0x2d7/0x460 [kvm_intel]
[36330.177547]  vcpu_enter_guest+0x440/0x1120 [kvm]
[36330.177567]  ? x86_emulate_instruction+0x586/0x6e0 [kvm]
[36330.177573]  ? __set_task_blocked+0x3c/0xa0
[36330.177594]  kvm_arch_vcpu_ioctl_run+0x210/0x3f0 [kvm]
[36330.177612]  ? kvm_arch_vcpu_ioctl_run+0x210/0x3f0 [kvm]
[36330.177628]  kvm_vcpu_ioctl+0x2a6/0x620 [kvm]
[36330.177633]  ? do_futex+0x30a/0x510
[36330.177638]  do_vfs_ioctl+0xa5/0x600
[36330.177657]  ? kvm_on_user_return+0x6f/0xa0 [kvm]
[36330.177661]  SyS_ioctl+0x79/0x90
[36330.177665]  entry_SYSCALL_64_fastpath+0x1e/0xa9
[36330.177668] RIP: 0033:0x7f5b27557f07
[36330.177670] RSP: 002b:00007f5b08c66908 EFLAGS: 00000246 ORIG_RAX:
0000000000000010
[36330.177674] RAX: ffffffffffffffda RBX: 000000000000ae80 RCX:
00007f5b27557f07
[36330.177676] RDX: 0000000000000000 RSI: 000000000000ae80 RDI:
0000000000000012
[36330.177678] RBP: 0000009fdb7405a0 R08: 0000009fd9118550 R09:
00000000000000ff
[36330.177680] R10: 0000000000000001 R11: 0000000000000246 R12:
0000000000000000
[36330.177682] R13: 00007f5b2c6e3000 R14: 0000000000000000 R15:
0000009fdb7405a0
[36330.177685] Code: 74 59 0f b7 41 0a 4c 63 69 04 0f b7 71 08 89 c7 49 01 cd
83 e7 01 a8 02 74 15 66 85 ff 74 10 a8 04 ba 01 00 00 00 75 26 83 c8 04 <66> 89
41 0a 66 85 ff 74 49 0f b6 49 0b 4c 89 e2 45 31 c9 49 89 
[36330.177735] RIP: report_bug+0x94/0x120 RSP: ffffa46262f7f7a0
[36330.177737] CR2: ffffffffc0633f92
[36330.177740] ---[ end trace 5d67fb9cae2ae6de ]---

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 196685] System becomes unresponsive after intel_iommu kernel error

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=196685

Jeff Cook (jeff@jeffcook.io) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jeff@jeffcook.io

--- Comment #9 from Jeff Cook (jeff@jeffcook.io) ---
I've been getting a very similar crash on 4.12.x, also making heavy use of PCI
passthrough. Came across this bug, as I've been searching regularly looking for
others encountering the same troubles.

A snippet of the log observed on 4.12.{3/5}:

kernel: BUG: unable to handle kernel paging request at ffffffffc0955f96
kernel: IP: report_bug+0x94/0x120
kernel: PGD a67a0c067 
kernel: P4D a67a0c067 
kernel: PUD a67a0e067 
kernel: PMD 495886067 
kernel: PTE 800000049bbdd161

Last night I experienced a hard lock with 4.11.9, which has heretofore been
stable. The system did not log any information about that crash. This may mean
that the issue existed prior to 4.12, but was just much harder to trigger.

See also my related bug report at https://bugs.archlinux.org/task/55141.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 196685] System becomes unresponsive after intel_iommu kernel error

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=196685

--- Comment #10 from Jeff Cook (jeff@jeffcook.io) ---
Created attachment 258019
  --> https://bugzilla.kernel.org/attachment.cgi?id=258019&action=edit
oops trace from logs

On this particular error condition, the passthrough'd VM locks, but the host
system and other VMs initially remain responsive via SSH. They too fail after
issuing a few commands. Messages indicating a stall are emitted regularly until
the machine is rebooted:

kernel: INFO: rcu_preempt detected stalls on CPUs/tasks:
kernel: Tasks blocked on level-1 rcu_node (CPUs 0-15): P2837
kernel: (detected by 23, t=990092 jiffies, g=2933189, c=2933188, q=9740206)

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 196685] System becomes unresponsive after intel_iommu kernel error

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=196685

--- Comment #11 from Jeff Cook (jeff@jeffcook.io) ---
Created attachment 258021
  --> https://bugzilla.kernel.org/attachment.cgi?id=258021&action=edit
lspci -vt | Aug 19 2017 (on 4.11.9)

Attaching my lspci -vt for comparison, from 4.11.9. Of note, I am also using a
GeForce GTX 1070 for VGA passthrough.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 196685] System becomes unresponsive after intel_iommu kernel error

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=196685

--- Comment #12 from Jason Hicks (jaszhix@gmail.com) ---
Created attachment 258029
  --> https://bugzilla.kernel.org/attachment.cgi?id=258029&action=edit
dmesg aug 18-aug 20

This is an updated dmesg log from boot to kernel oops errors. This shows the
system being on for a couple days with the VM running with no issues, but
finally when left on over night, and then using the VM this morning (August
20), it flooded the log with call stacks for a while before my computer
completely froze. In addition, hardware differences between this and my last
complete dmesg log include removed USB devices - a 8GB usb drive, and a Sony
controller was removed.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 196685] System becomes unresponsive after intel_iommu kernel error

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=196685

Jason Hicks (jaszhix@gmail.com) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
     Kernel Version|4.12.7                      |4.12.8

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 196685] System becomes unresponsive after intel_iommu kernel error

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=196685

--- Comment #13 from Jeff Cook (jeff@jeffcook.io) ---
For the record. I have been running "Linux kvm_master 4.13.0-rc5-g58d4e450a490
#1 SMP PREEMPT", a git build from a commit made on Friday (8/19/2017), for the
last 36 hours or so. On this commit, it appears that I received the warning
message discussed here, but nothing crashed and the system continues to
function in what appears to be a normal manner.

I will attach full dmesg for reference. If this is indeed the same condition
and the trap is successful, the crasher may have been fixed somewhere in the
4.13 cycle.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 196685] System becomes unresponsive after intel_iommu kernel error

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=196685

--- Comment #14 from Jeff Cook (jeff@jeffcook.io) ---
Created attachment 258033
  --> https://bugzilla.kernel.org/attachment.cgi?id=258033&action=edit
dmesg WARN vmx_deliver_posted_interrupt

Full output of dmesg on Linux 4.13.0-rc5-g58d4e450a490, boot to WARN and
onward. There have been no apparent negative effects since the error was logged
approximately 20 hours ago (caveat: have not used heavily today).

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 196685] System becomes unresponsive after intel_iommu kernel error

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=196685

--- Comment #15 from changbin.du@hotmail.com ---
(In reply to Jeff Cook from comment #13)
> For the record. I have been running "Linux kvm_master
> 4.13.0-rc5-g58d4e450a490 #1 SMP PREEMPT", a git build from a commit made on
> Friday (8/19/2017), for the last 36 hours or so. On this commit, it appears
> that I received the warning message discussed here, but nothing crashed and
> the system continues to function in what appears to be a normal manner.
> 
> I will attach full dmesg for reference. If this is indeed the same condition
> and the trap is successful, the crasher may have been fixed somewhere in the
> 4.13 cycle.

The oops has been fixed by https://py3.patchwork.dja.id.au/patch/41163/. This
patch should be backported to stable branches.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 196685] System becomes unresponsive after intel_iommu kernel error

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=196685

--- Comment #16 from changbin.du@hotmail.com ---
I have send a request to stable queue. Suppose it will be fixed in next 4.11.x
and 4.12.x version. This issue is introduced from v4.11-rc3.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 196685] System becomes unresponsive after intel_iommu kernel error

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=196685

--- Comment #17 from Jeff Cook (jeff@jeffcook.io) ---
That's great news! It sounds like that solves the first half of the problem
(the crash), and that we still need to pin down the warning? Or is the warning
reflective of a bad hardware condition?

Separately, I've also been experiencing
https://bugzilla.kernel.org/show_bug.cgi?id=196717. Thought I'd mention it here
since there may be a cross-section of subscribers here experiencing it too.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 196685] System becomes unresponsive after intel_iommu kernel error

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=196685

Jason Hicks (jaszhix@gmail.com) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Regression|No                          |Yes

-- 
You are receiving this mail because:
You are watching the assignee of the bug.