[Qemu-devel] Bug: E1000 kernel panic when transferring files over private network

[Qemu-devel] Bug: E1000 kernel panic when transferring files over private network

[Owen Tuz]

[-- Attachment #1: Type: text/plain, Size: 4132 bytes --]

We are seeing a reliably reproducible panic when using the E1000 card over
a private network. This crash occurs inside the guest kernel of a VM with
some (we believe) unrelated network connectivity issues, running on
qemu-kvm 1.1.1.

We have seen this on all kernel versions we tested, the earliest version
being 2.6.32-5 on a Debian test VM. I can provide error messages from other
kernel versions or distributions for comparison if it would be useful.

To reproduce:

Add two machines to a VLAN where 192.168.0.1 is serving some file
('test.bin'), and 192.168.0.2 is using the Intel E1000 card.

>From 192.168.0.2:

wget -O /dev/null http://192.168.0.1/test.bin

On Arch Linux running 3.5.1-1 this produces a kernel panic as follows,
within a few seconds:

[ 151.310977] BUG: unable to handle kernel paging request at 51c39a07
[ 151.314106] IP: [<c01f35a8>] put_page+0x8/0x40
[ 151.314106] *pde = 00000000
[ 151.314106] Oops: 0000 [#1] PREEMPT SMP
[ 151.314106] Modules linked in: cirrus syscopyarea sysfillrect sysimgblt
drm_kms_helper ttm i2c_piix4 drm
serio_raw intel_agp microcode intel_gtt agpgart kvm_amd e1000 pcspkr
i2c_core_processor joydev button psmouse evdev
kvm ext4 crc16 jbd2 mbcache hid_generic usbhid sd_mod pata_acpi ata_generic
ata_piix uhci_hcd usbcore usb_common
libata scsi_mod floppy
[ 151.314106]
[ 151.314106] Pid: 387, comm: wget Not tainted 3.5.1-1-ARCH #1 Bochs Bochs
[ 151.314106] EIP: 0060:[<c01f35a8>] EFLAGS: 00210202 CPU: 0
[ 151.314106] EIP is at put_page+0x8/0x40
[ 151.314106] EAX: 51c39a07 EBX: 00000001 ECX: c07acdb8 EDX: ce350d40
[ 151.314106] ESI: cea51480 EDI: 000005a8 EBP: ce133da4 ESP: ce133da4
[ 151.314106]  DS: 007b ES: 007b FS:00d8 GS: 00e0 SS: 0068
[ 151.314106] CR0: 80050033 CR2: 51c39a07 CR3: 0ec54000 CR4: 000006d0
[ 151.314106] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[ 151.314106] DR6: ffff0ff0 DR7: 00000400
[ 151.314106] Process wget (pid: 387, ti=ce132000 task=cea92a80
task.ti=ce132000)
[ 151.314106] Stack:
[ 151.314106]  ce133db4 c03deb8c cea51480 ce03c000 ce133dc0 c03debe7
cea51480 ce133e28
[ 151.314106]  c0424bbc 00000000 ce1b5d20 ce133e38 ce133e08 c0334661
00000001 00000010
[ 151.314106]  6b5f2c4b cea92a80 00000000 00000001 ce133e9c 00000000
00000001 000005a8
[ 151.314106] Call Trace:
[ 151.314106]  [<c03deb8c>] skb_release_data+0x6c/0xb0
[ 151.314106]  [<c03debe7>] __kfree_skb+0x17/0x90
[ 151.314106]  [<c0424bbc>] tcp_recvmsg+0x92c/0xb40
[ 151.314106]  [<c0334661>] ? soft_cursor+0x191/0x200
[ 151.314106]  [<c0442eab>] inet+recvmsg+0x6b/0xb0
[ 151.314106]  [<c03d6584>] sock_aio_read+0x114/0x150
[ 151.314106]  [<c0234237>] do_sync_read+0xb7/0xf0
[ 151.314106]  [<c012a5dc>] ? pvclock_clocksource_read+0xec/0x150
[ 151.314106]  [<c02a7ce4>] ? security_file_permission+0x94/0xb0
[ 151.314106]  [<c02348e3>] ? rw_verify_area+0x63/0x110
[ 151.314106]  [<c0234e7d>] vfs_read+0x13d/0x160
[ 151.314106]  [<c0234edd>] sys_read+0x3d/0x80
[ 151.314106]  [<c04caf5f>] sysenter_do_call+0x12/0x28
[ 151.314106] Code: 2c 1f c0 8d 4d fc c7 45 fc 00 00 00 00 e8 e1 fe ff ff
8b 45 fc 64 01 05 64 28 6c c0 c9 c3 90 8d
74 26 00 55 89 e5 3e 8d 74 26 00 <f7> 00 00 c0 00 00 75 17 3e ff 48 10 0f
94 c2 84 d2 75 05 5d c3
[ 151.314106] EIP: [<c01f35a8>] put_page+0x8/0x40 SS:ESP 0068:ce133da4
[ 151.314106] CR2: 0000000051c39a07
2012 Aug 15 16:01:58 localhost [ 151.314106] Process wget (pid: 387,
ti=ce132000 task=cea92a80 task.ti=ce132000)
[ 151.314106] ---[end trace d015761e7dbf4ff2]---
2012 Aug 15 16:01:58 localhost [ 151.314106] Stack:
2012 Aug 15 16:01:58 localhost [ 151.314106] Call Trace:
2012 Aug 15 16:01:58 localhost [ 151.314106] Code: 2c 1f c0 8d 4d fc c7 45
fc 00 00 00 00 e8 e1 fe ff ff 8b 45 fc
64 01 05 64 28 6c c0 c9 c3 90 8d 74 26 00 55 89 e5 3e 8d 74 26 00 <f7> 00
00 c0 00 00 75 17 3e ff 48 10 0f 94 c2 84
d2 75 05 5d c3
2012 Aug 15 16:01:58 localhost [ 151.314106] EIP: [<c01f35a8>]
put_page+0x8/0x40 SS:ESP 0068:ce133da4

The host machines are running qemu-kvm 1.1.1, kernel 3.5.1. Hardware is 2 x
8-core AMD Opteron 6128 and 128GB RAM.

Please let me know if I can provide any more useful information, and as
always, thanks in advance.

[-- Attachment #2: Type: text/html, Size: 5175 bytes --]

Re: [Qemu-devel] Bug: E1000 kernel panic when transferring files over private network

[Richard Davies]

Just to be clear - we are sure that it is the virtualization host networking
problems which are triggering this VM OS nic driver crash, and we haven't
described these here anything like well enough to reproduce the situation -
they are complex.

However, in the same situation, a VM with rtl8139 or virtio network cards
doesn't crash, so I believe that the e1000 crash is evidence of some kind of
bug in the e1000 driver which is triggered in our specific situation.

Hopefully the backtrace gives enough information to identify what that e1000
driver bug might be?

Richard.