* RFH: Kernel OOPS in xen_netbk_rx_action / xenvif_gop_skb
@ 2014-06-06 10:26 Philipp Hahn
2014-06-06 10:58 ` Wei Liu
0 siblings, 1 reply; 18+ messages in thread
From: Philipp Hahn @ 2014-06-06 10:26 UTC (permalink / raw)
To: Wei Liu, Ian Campbell, Zoltan Kiss; +Cc: xen-devel, Erik Damrose
[-- Attachment #1: Type: text/plain, Size: 7912 bytes --]
Hello,
on one of our hosts (Xen-4.1.3 with Linux-3.10.26 + Debian patches)
running 16 Linux VMs (linux-3.2.39 and others) netback crashes during
the night when one of the VMs is rebooted by a cron-job:
> [38551.549615] Oops: 0000 [#1] SMP
> [38551.549665] Modules linked in: tun xt_physdev xen_blkback xen_netback ip6_tables
> iptable_filter ip_tables ebtable_nat ebtables x_tables xen_gntdev nfsv3 nfsv4
> rpcsec_gss_krb5 nfsd nfs_acl auth_rpcgss oid_registry nfs fscache dns_resolver lockd
> sunrpc fuse loop xen_blkfront xen_evtchn blktap quota_v2 quota_tree xenfs xen_privcmd
> coretemp crc32c_intel ghash_clmulni_intel aesni_intel ablk_helper cryptd lrw gf128mul
> glue_helper aes_x86_64 snd_pcm snd_timer snd soundcore snd_page_alloc tpm_tis tpm lpc_ich
> tpm_bios i7core_edac i2c_i801 psmouse microcode edac_core serio_raw pcspkr mperf ioatdma
> mfd_core processor evdev thermal_sys ext4 jbd2 crc16 bonding bridge stp llc dm_snapshot
> dm_mirror dm_region_hash dm_log dm_mod sd_mod crc_t10dif ehci_pci uhci_hcd ehci_hcd mptsas
> mptscsih mptbase scsi_transport_sas usbcore usb_common igb dca i2c_algo_bit i2c_core ptp
> pps_core button
> [38551.550601] CPU: 0 PID: 12587 Comm: netback/0 Not tainted 3.10.0-ucs58-amd64 #1 Debian
> 3.10.11-1.58.201405060908
> [38551.550693] Hardware name: FUJITSU PRIMERGY BX620 S6/D3051, BIOS 080015 Rev.3C78.3051
> 07/22/2011
> [38551.550781] task: ffff880004b067c0 ti: ffff8800561ec000 task.ti: ffff8800561ec000
> [38551.550865] RIP: e030:[<ffffffffa04147dc>] [<ffffffffa04147dc>]
> xen_netbk_rx_action+0x18b/0x6f0 [xen_netback]
> [38551.550959] RSP: e02b:ffff8800561edce8 EFLAGS: 00010202
> [38551.551009] RAX: ffffc900104adac0 RBX: ffff8800541e95c0 RCX: ffffc90010864000
> [38551.551064] RDX: 000000000000003b RSI: 0000000000000000 RDI: ffff880040014380
> [38551.551120] RBP: ffff8800570e6800 R08: 0000000000000000 R09: ffff880004799800
> [38551.551175] R10: ffffffff813ca115 R11: ffff88005e4fdb08 R12: ffff880054e6f800
> [38551.551231] R13: ffff8800561edd58 R14: ffffc900104a1000 R15: 0000000000000000
> [38551.551289] FS: 00007f19a54a8700(0000) GS:ffff88005da00000(0000)
> knlGS:0000000000000000
> [38551.551374] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> [38551.551425] CR2: ffffc900108641d8 CR3: 0000000054cb3000 CR4: 0000000000002660
> [38551.551481] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [38551.551537] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [38551.551592] Stack:
> [38551.551630] ffff880004b06ba0 0000000000000000 ffff88005da13ec0 ffff88005da13ec0
> [38551.551726] 0000000004b067c0 ffffc900104a8ac0 ffffc900104a1020 000000005da13ec0
> [38551.551823] 0000000000000000 0000000000000001 ffffc900104a8ac0 ffffc900104adac0
> [38551.551920] Call Trace:
> [38551.551966] [<ffffffff813ca32d>] ? _raw_spin_lock_irqsave+0x11/0x2f
> [38551.552021] [<ffffffffa0416033>] ? xen_netbk_kthread+0x174/0x841 [xen_netback]
> [38551.552106] [<ffffffff8105d373>] ? wake_up_bit+0x20/0x20
> [38551.560239] [<ffffffffa0415ebf>] ? xen_netbk_tx_build_gops+0xce8/0xce8 [xen_netback]
> [38551.560325] [<ffffffff8105cd73>] ? kthread_freezable_should_stop+0x56/0x56
> [38551.560381] [<ffffffffa0415ebf>] ? xen_netbk_tx_build_gops+0xce8/0xce8 [xen_netback]
> [38551.560466] [<ffffffff8105ce1e>] ? kthread+0xab/0xb3
> [38551.560518] [<ffffffff81003638>] ? xen_end_context_switch+0xe/0x1c
> [38551.560572] [<ffffffff8105cd73>] ? kthread_freezable_should_stop+0x56/0x56
> [38551.560628] [<ffffffff813cfbfc>] ? ret_from_fork+0x7c/0xb0
> [38551.560680] [<ffffffff8105cd73>] ? kthread_freezable_should_stop+0x56/0x56
> [38551.560734] Code: 8b b3 d0 00 00 00 48 8b bb d8 00 00 00 0f b7 74 37 02 89 70 08 eb 07
> c7 40 08 00 00 00 00 89 d2 c7 40 04 00 00 00 00 48 83 c2 08 <0f> b7 34 d1 89 30 c7 44 24
> 60 00 00 00 00 8b 44 d1 04 89 44 24
> [38551.561151] RIP [<ffffffffa04147dc>] xen_netbk_rx_action+0x18b/0x6f0 [xen_netback]
> [38551.561238] RSP <ffff8800561edce8>
> [38551.561283] CR2: ffffc900108641d8
> [38551.561624] ---[ end trace 8c260c6af259c4aa ]---
The host itself is still alive and reachable by network, but all VMs are
no longer reachable.
The crash does not happen on every reboot: The VM was running fine for
1½ week after a dom0 kernel update, but now crashed the following past
two nights.
I'm yet unable to reproduce this on demand, but would like to prepared
next time it happens again.
@Ian: I found your mail "Re: [Xen-devel] Kernel 3.7.0-pre-rc1 kernel BUG
at drivers/net/xen-netback/netback.c:405 RIP: e030:[<ffffffff814714f9>]
[<ffffffff814714f9>] netbk_gop_frag_copy+0x379/0x380" from 2012-10-09,
which describes a crash in the same function, but at a complete
different (later) location. You hinted that a difference in hardware
might explain, why I'm unable to reproduce it, as my test environment
has different HW (no "igb", but "e1000e").
Running "objdump -Sl xen-netback.ko" shows the OOPs to happen here:
> /root/linux-3.10.11/drivers/net/xen-netback/netback.c:606
> meta->gso_size = skb_shinfo(skb)->gso_size;
> 7b1: 8b b3 d0 00 00 00 mov 0xd0(%rbx),%esi
> 7b7: 48 8b bb d8 00 00 00 mov 0xd8(%rbx),%rdi
> 7be: 0f b7 74 37 02 movzwl 0x2(%rdi,%rsi,1),%esi
> 7c3: 89 70 08 mov %esi,0x8(%rax)
> 7c6: eb 07 jmp 7cf <xen_netbk_rx_action+0x17e>
> /root/linux-3.10.11/drivers/net/xen-netback/netback.c:608
> else
> meta->gso_size = 0;
> 7c8: c7 40 08 00 00 00 00 movl $0x0,0x8(%rax)
> /root/linux-3.10.11/drivers/net/xen-netback/netback.c:611
>
> meta->size = 0;
> meta->id = req->id;
> 7cf: 89 d2 mov %edx,%edx
> /root/linux-3.10.11/drivers/net/xen-netback/netback.c:610
> if (!vif->gso_prefix)
> meta->gso_size = skb_shinfo(skb)->gso_size;
> else
> meta->gso_size = 0;
>
> meta->size = 0;
> 7d1: c7 40 04 00 00 00 00 movl $0x0,0x4(%rax)
> /root/linux-3.10.11/drivers/net/xen-netback/netback.c:611
> meta->id = req->id;
> 7d8: 48 83 c2 08 add $0x8,%rdx
> 7dc: 0f b7 34 d1 movzwl (%rcx,%rdx,8),%esi
0x651 + 0x18B = 0x7DC
> 7e0: 89 30 mov %esi,(%rax)
> /root/linux-3.10.11/drivers/net/xen-netback/netback.c:612
> npo->copy_off = 0;
> 7e2: c7 44 24 60 00 00 00 movl $0x0,0x60(%rsp)
> 7e9: 00
> /root/linux-3.10.11/drivers/net/xen-netback/netback.c:613
> npo->copy_gref = req->gref;
> 7ea: 8b 44 d1 04 mov 0x4(%rcx,%rdx,8),%eax
> 7ee: 89 44 24 64 mov %eax,0x64(%rsp)
Ignoring the name change from {netbk -> xenvif}_gop_skb() and the
addition of GSO for IPv6 the function looks unchanged compared to
current GIT, so to me it looks like it might still be a problem with the
current implementation.
I tried to review the GIT commits myself, but I didn't see anything
obvious, but with all the recent additional changes to netback I'm
unsure of how to best proceed:
1. Is this a known bug and has someone observed it, too?
2. If yes, is there a fix in newer Linux kernels?
3. If no, What data should I collect in addition?
Xen-Hypervisor is 4.1.3 from Debian, but as this is a kernel crash, I
don't expect a newer version of Xen to fix it (correct me if I'm wrong).
Thanks in advance.
Philipp
PS: I'm not afraid of getting my hands dirty doing Linux coding, but
currently I'm out of ideas of how to best proceed.
--
Philipp Hahn
Open Source Software Engineer
Univention GmbH
be open.
Mary-Somerville-Str. 1
D-28359 Bremen
Tel.: +49 421 22232-0
Fax : +49 421 22232-99
hahn@univention.de
http://www.univention.de/
Geschäftsführer: Peter H. Ganten
HRB 20755 Amtsgericht Bremen
Steuer-Nr.: 71-597-02876
[-- Attachment #2: Attached Message --]
[-- Type: message/rfc822, Size: 45458 bytes --]
From: Ian Campbell <Ian.Campbell@citrix.com>
To: Sander Eikelenboom <linux@eikelenboom.it>
Cc: Konrad Rzeszutek Wilk <konrad@kernel.org>, "xen-devel@lists.xensource.com" <xen-devel@lists.xensource.com>, konrad <konrad.wilk@oracle.com>
Subject: Re: [Xen-devel] Kernel 3.7.0-pre-rc1 kernel BUG at drivers/net/xen-netback/netback.c:405 RIP: e030:[<ffffffff814714f9>] [<ffffffff814714f9>] netbk_gop_frag_copy+0x379/0x380
Date: Tue, 9 Oct 2012 14:16:58 +0100
Message-ID: <1349788618.21847.170.camel@zakaz.uk.xensource.com>
On Tue, 2012-10-09 at 12:07 +0100, Sander Eikelenboom wrote:
> [ 199.342570] netbk_gop_frag_copy: size 5a8 offset 7102
> [ 199.342570] => 76aa > 1000
> [ 199.354626] netbk_gop_frag_copy failed: skb frag 0 page
> [ 199.360930] copying from offset 7102, len 5a8
OK, this is now at least a real error. Making that last change
(belt&braces) you made shouldn't really have changed anything though :-(
> [ 199.366887] page:ffffea0000b0aa00 count:3 mapcount:0 mapping: (null) index:0x7f40fec00
> [ 199.373008] page flags: 0x40000000004000(head)
The final 0x4000 is indeed PG_head as described, which makes this is a
compound page. This could arise either from the use of transparent huge
pages or via explicit __GFP_comp. It seems that the core networking
stuff can generate these after
69b08f62e174 "net: use bigger pages in __netdev_alloc_frag".
It's not clear to me that the r8169 driver uses those interfaces though,
seems like only tg3 does currently.
In any case it's not obvious how this interacts with bridging and
forwarding, since even if a receiving driver can handle this sort of
thing there's no guarantee that the resending driver can do so (e.g.
netback can't!).
This is one for netdev@ I think. I'll post there and CC you guys.
> [ 199.379252] ------------[ cut here ]------------
> [ 199.385247] kernel BUG at drivers/net/xen-netback/netback.c:548!
> [ 199.391334] invalid opcode: 0000 [#1] PREEMPT SMP
> [ 199.397446] Modules linked in:
> [ 199.403450] CPU 4
> [ 199.403500] Pid: 1183, comm: netback/4 Not tainted 3.6.0pre-rc1-20121008bisect #1 MSI MS-7640/890FXA-GD70 (MS-7640)
> [ 199.415401] RIP: e030:[<ffffffff8147463a>] [<ffffffff8147463a>] xen_netbk_rx_action+0x89a/0x910
> [ 199.421690] RSP: e02b:ffff88003792bc20 EFLAGS: 00010282
> [ 199.428048] RAX: 0000000000000001 RBX: ffff88003197c600 RCX: 0000000000000000
> [ 199.434358] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8800379202b0
> [ 199.440582] RBP: ffff88003792bd50 R08: 0000000000000002 R09: 0000000000000000
> [ 199.446740] R10: 0000000000000001 R11: ffff88003a26c000 R12: 0000000000000030
> [ 199.452965] R13: 0000000000000000 R14: ffff88002c2ae900 R15: 0000000000000001
> [ 199.459203] FS: 00007fcec7740700(0000) GS:ffff88003f900000(0000) knlGS:0000000000000000
> [ 199.465527] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 199.471735] CR2: 00007fff5f59c000 CR3: 0000000001c0b000 CR4: 0000000000000660
> [ 199.477961] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 199.484102] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 199.490274] Process netback/4 (pid: 1183, threadinfo ffff88003792a000, task ffff880037cec140)
> [ 199.496631] Stack:
> [ 199.502834] ffff88003792bd1c ffff880037cec7f0 ffff88003792bd00 ffff88003792bc80
> [ 199.509198] ffffffff00000001 00000000000005ea ffffc90010851a98 ffffc9001084cf30
> [ 199.515579] 0000000001080083 ffffc9001084cee0 0000000000000000 ffff880032b449c0
> [ 199.521944] Call Trace:
> [ 199.528243] [<ffffffff810acf3d>] ? trace_hardirqs_on+0xd/0x10
> [ 199.534566] [<ffffffff8147569a>] xen_netbk_kthread+0xba/0xa90
> [ 199.540826] [<ffffffff810957e6>] ? try_to_wake_up+0x1b6/0x310
> [ 199.547193] [<ffffffff81086810>] ? wake_up_bit+0x40/0x40
> [ 199.553450] [<ffffffff814755e0>] ? xen_netbk_tx_build_gops+0xa70/0xa70
> [ 199.559683] [<ffffffff810861a6>] kthread+0xd6/0xe0
> [ 199.565827] [<ffffffff8174e664>] kernel_thread_helper+0x4/0x10
> [ 199.572086] [<ffffffff8174cb37>] ? retint_restore_args+0x13/0x13
> [ 199.578268] [<ffffffff8174e660>] ? gs_change+0x13/0x13
> [ 199.584344] Code: 00 00 00 42 8b 54 30 3c 41 8b 74 04 08 31 c0 e8 e5 37 2d 00 8b 83 c4 00 00 00 4c 03 b3 c8 00 00 00 4a 8b 7c 30 30 e8 36 24 c8 ff <0f> 0b eb fe 48 8b b3 d0 00 00 00 48 c7 c2 c0 36 47 81 48 c7 c7
> [ 199.597406] RIP [<ffffffff8147463a>] xen_netbk_rx_action+0x89a/0x910
> [ 199.604013] RSP <ffff88003792bc20>
> [ 199.610610] ---[ end trace 03f82ac72747fb5a ]---
> [ 199.990340] device vif11.0 entered promiscuous mode
> [ 200.466710] xen-blkback:ring-ref 9, event-channel 10, protocol 1 (x86_64-abi)
> [ 200.476634] xen_bridge: port 11(vif11.0) entered forwarding state
> [ 200.483621] xen_bridge: port 11(vif11.0) entered forwarding state
> [ 200.653782] pciback 0000:03:06.0: enabling device (0000 -> 0001)
> [ 200.661499] xen: registering gsi 22 triggering 0 polarity 1
> [ 200.669003] Already setup the GSI :22
> [ 200.677345] pciback 0000:03:06.0: enabling bus mastering
> [ 201.267297] xen_bridge: port 9(vif9.0) entered forwarding state
> [ 205.151290] tty_init_dev: 2 callbacks suppressed
> [ 206.534137] device vif12.0 entered promiscuous mode
> [ 206.867366] xen-blkback:ring-ref 8, event-channel 9, protocol 1 (x86_64-abi)
> [ 206.877552] xen_bridge: port 12(vif12.0) entered forwarding state
> [ 206.884869] xen_bridge: port 12(vif12.0) entered forwarding state
> [ 208.574036] xen_bridge: port 10(vif10.0) entered forwarding state
> [ 209.979799] netbk_gop_frag_copy: size 1080 offset 0
> [ 209.979799] => 1080 > 1000
> [ 209.994252] netbk_gop_frag_copy failed: skb frag 0 page
> [ 210.001191] copying from offset 0, len 1080
> [ 210.008121] page:ffffea0000b0a800 count:3 mapcount:0 mapping: (null) index:0x7f40fec00
> [ 210.015124] page flags: 0x40000000004000(head)
> [ 210.022122] ------------[ cut here ]------------
> [ 210.029035] kernel BUG at drivers/net/xen-netback/netback.c:548!
> [ 210.035973] invalid opcode: 0000 [#2] PREEMPT SMP
> [ 210.042819] Modules linked in:
> [ 210.049467] CPU 0
> [ 210.049518] Pid: 1179, comm: netback/0 Tainted: G D 3.6.0pre-rc1-20121008bisect #1 MSI MS-7640/890FXA-GD70 (MS-7640)
> [ 210.062788] RIP: e030:[<ffffffff8147463a>] [<ffffffff8147463a>] xen_netbk_rx_action+0x89a/0x910
> [ 210.069740] RSP: e02b:ffff880037923c20 EFLAGS: 00010282
> [ 210.076711] RAX: 0000000000000001 RBX: ffff880031993ae0 RCX: 0000000000000000
> [ 210.083744] RDX: ffff8800398a61e0 RSI: 0000000000000001 RDI: ffff8800379202b0
> [ 210.090801] RBP: ffff880037923d50 R08: 0000000000000002 R09: 0000000000000000
> [ 210.097787] R10: 0000000000000001 R11: ffff88003a26b330 R12: 0000000000000030
> [ 210.104759] R13: 0000000000000000 R14: ffff88002b4d8800 R15: 0000000000000001
> [ 210.111611] FS: 00007f695df80700(0000) GS:ffff88003f800000(0000) knlGS:0000000000000000
> [ 210.118570] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 210.125586] CR2: 00007f695402e000 CR3: 0000000032a8f000 CR4: 0000000000000660
> [ 210.132677] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 210.139560] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 210.146350] Process netback/0 (pid: 1179, threadinfo ffff880037922000, task ffff8800398a61e0)
> [ 210.153213] Stack:
> [ 210.159974] ffff880037923d1c ffff880037922010 ffff880037923d00 ffff880037923c80
> [ 210.166905] ffffffff810800b5 0000000000000662 ffffc90010824bb8 ffffc90010820050
> [ 210.173802] 0000000001080083 ffffc90010820000 0000000000000000 ffff8800375849c0
> [ 210.180780] Call Trace:
> [ 210.187656] [<ffffffff810800b5>] ? __alloc_workqueue_key+0x265/0x5d0
> [ 210.194674] [<ffffffff810acf3d>] ? trace_hardirqs_on+0xd/0x10
> [ 210.201690] [<ffffffff8147569a>] xen_netbk_kthread+0xba/0xa90
> [ 210.208659] [<ffffffff810957e6>] ? try_to_wake_up+0x1b6/0x310
> [ 210.215688] [<ffffffff81086810>] ? wake_up_bit+0x40/0x40
> [ 210.222665] [<ffffffff814755e0>] ? xen_netbk_tx_build_gops+0xa70/0xa70
> [ 210.229651] [<ffffffff810861a6>] kthread+0xd6/0xe0
> [ 210.236455] [<ffffffff8174e664>] kernel_thread_helper+0x4/0x10
> [ 210.243111] [<ffffffff8174cb37>] ? retint_restore_args+0x13/0x13
> [ 210.249687] [<ffffffff8174e660>] ? gs_change+0x13/0x13
> [ 210.256195] Code: 00 00 00 42 8b 54 30 3c 41 8b 74 04 08 31 c0 e8 e5 37 2d 00 8b 83 c4 00 00 00 4c 03 b3 c8 00 00 00 4a 8b 7c 30 30 e8 36 24 c8 ff <0f> 0b eb fe 48 8b b3 d0 00 00 00 48 c7 c2 c0 36 47 81 48 c7 c7
> [ 210.270166] RIP [<ffffffff8147463a>] xen_netbk_rx_action+0x89a/0x910
> [ 210.276925] RSP <ffff880037923c20>
> [ 210.284112] ---[ end trace 03f82ac72747fb5b ]---
> [ 213.634083] device vif13.0 entered promiscuous mode
> [ 213.911267] xen-blkback:ring-ref 8, event-channel 9, protocol 1 (x86_64-abi)
> [ 213.920749] vpn_bridge: port 1(vif13.0) entered forwarding state
> [ 213.927480] vpn_bridge: port 1(vif13.0) entered forwarding state
> [ 215.509632] xen_bridge: port 11(vif11.0) entered forwarding state
> [ 215.825483] netbk_gop_frag_copy: size 2c1 offset 12d6
> [ 215.825483] => 1597 > 1000
> [ 215.838666] netbk_gop_frag_copy failed: skb frag 0 page
> [ 215.845265] copying from offset 12d6, len 2c1
> [ 215.851790] page:ffffea0000b0a800 count:6 mapcount:0 mapping: (null) index:0x7f40fec00
> [ 215.858389] page flags: 0x40000000004000(head)
> [ 215.864925] ------------[ cut here ]------------
> [ 215.871426] kernel BUG at drivers/net/xen-netback/netback.c:548!
> [ 215.878069] invalid opcode: 0000 [#3] PREEMPT SMP
> [ 215.884696] Modules linked in:
> [ 215.891258] CPU 3
> [ 215.891308] Pid: 1182, comm: netback/3 Tainted: G D 3.6.0pre-rc1-20121008bisect #1 MSI MS-7640/890FXA-GD70 (MS-7640)
> [ 215.904613] RIP: e030:[<ffffffff8147463a>] [<ffffffff8147463a>] xen_netbk_rx_action+0x89a/0x910
> [ 215.911538] RSP: e02b:ffff880037929c20 EFLAGS: 00010282
> [ 215.918336] RAX: 0000000000000001 RBX: ffff88002c361ee0 RCX: 0000000000000000
> [ 215.925236] RDX: ffff880037ced190 RSI: 0000000000000001 RDI: ffff8800379202b0
> [ 215.932144] RBP: ffff880037929d50 R08: 0000000000000002 R09: 0000000000000000
> [ 215.938988] R10: 0000000000000001 R11: ffff88003a26aca0 R12: 0000000000000030
> [ 215.945835] R13: 0000000000000000 R14: ffff88002b49b400 R15: 0000000000000001
> [ 215.952652] FS: 00007f695c355700(0000) GS:ffff88003f8c0000(0000) knlGS:0000000000000000
> [ 215.959476] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 215.966165] CR2: 00007faa79583000 CR3: 0000000032a8f000 CR4: 0000000000000660
> [ 215.972789] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 215.979339] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 215.985844] Process netback/3 (pid: 1182, threadinfo ffff880037928000, task ffff880037ced190)
> [ 215.992486] Stack:
> [ 215.999085] ffff880037929d1c ffff880037928010 ffff880037929d00 ffff880037929c80
> [ 216.005896] ffffffff810800b5 00000000000000ba ffffc900108466e0 ffffc90010841b78
> [ 216.012651] 0000000101080083 ffffc90010841b28 0000000100000000 ffff880031a869c0
> [ 216.019386] Call Trace:
> [ 216.026026] [<ffffffff810800b5>] ? __alloc_workqueue_key+0x265/0x5d0
> [ 216.032830] [<ffffffff810acf3d>] ? trace_hardirqs_on+0xd/0x10
> [ 216.039668] [<ffffffff8147569a>] xen_netbk_kthread+0xba/0xa90
> [ 216.046435] [<ffffffff810957e6>] ? try_to_wake_up+0x1b6/0x310
> [ 216.053094] [<ffffffff81086810>] ? wake_up_bit+0x40/0x40
> [ 216.059670] [<ffffffff814755e0>] ? xen_netbk_tx_build_gops+0xa70/0xa70
> [ 216.066279] [<ffffffff810861a6>] kthread+0xd6/0xe0
> [ 216.072817] [<ffffffff8174e664>] kernel_thread_helper+0x4/0x10
> [ 216.079308] [<ffffffff8174cb37>] ? retint_restore_args+0x13/0x13
> [ 216.085783] [<ffffffff8174e660>] ? gs_change+0x13/0x13
> [ 216.092234] Code: 00 00 00 42 8b 54 30 3c 41 8b 74 04 08 31 c0 e8 e5 37 2d 00 8b 83 c4 00 00 00 4c 03 b3 c8 00 00 00 4a 8b 7c 30 30 e8 36 24 c8 ff <0f> 0b eb fe 48 8b b3 d0 00 00 00 48 c7 c2 c0 36 47 81 48 c7 c7
> [ 216.106108] RIP [<ffffffff8147463a>] xen_netbk_rx_action+0x89a/0x910
> [ 216.113118] RSP <ffff880037929c20>
> [ 216.120011] ---[ end trace 03f82ac72747fb5c ]---
> [ 219.765094] device vif14.0 entered promiscuous mode
> [ 220.062152] xen-blkback:ring-ref 8, event-channel 9, protocol 1 (x86_64-abi)
> [ 220.072238] xen_bridge: port 13(vif14.0) entered forwarding state
> [ 220.079416] xen_bridge: port 13(vif14.0) entered forwarding state
> [ 221.912781] xen_bridge: port 12(vif12.0) entered forwarding state
> [ 222.876167] netbk_gop_frag_copy: size 2c1 offset 1858
> [ 222.876167] => 1b19 > 1000
> [ 222.889279] netbk_gop_frag_copy failed: skb frag 0 page
> [ 222.895959] copying from offset 1858, len 2c1
> [ 222.902484] page:ffffea0000b0a800 count:8 mapcount:0 mapping: (null) index:0x7f40fec00
> [ 222.909119] page flags: 0x40000000004000(head)
> [ 222.915711] ------------[ cut here ]------------
> [ 222.922307] kernel BUG at drivers/net/xen-netback/netback.c:548!
> [ 222.928950] invalid opcode: 0000 [#4] PREEMPT SMP
> [ 222.935546] Modules linked in:
> [ 222.942110] CPU 5
> [ 222.942161] Pid: 1184, comm: netback/5 Tainted: G D 3.6.0pre-rc1-20121008bisect #1 MSI MS-7640/890FXA-GD70 (MS-7640)
> [ 222.955415] RIP: e030:[<ffffffff8147463a>] [<ffffffff8147463a>] xen_netbk_rx_action+0x89a/0x910
> [ 222.962350] RSP: e02b:ffff88003792dc20 EFLAGS: 00010282
> [ 222.969198] RAX: 0000000000000001 RBX: ffff88002b4f4ce0 RCX: 0000000000000000
> [ 222.976119] RDX: ffff880037ceb0f0 RSI: 0000000000000001 RDI: ffff8800379202b0
> [ 222.982987] RBP: ffff88003792dd50 R08: 0000000000000002 R09: 0000000000000000
> [ 222.989869] R10: 0000000000000001 R11: ffff88003a26b380 R12: 0000000000000030
> [ 222.996658] R13: 0000000000000000 R14: ffff88002b5a7800 R15: 0000000000000001
> [ 223.003490] FS: 00007f71c6ce2740(0000) GS:ffff88003f940000(0000) knlGS:0000000000000000
> [ 223.010257] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 223.016868] CR2: 00007f71c66b4d15 CR3: 0000000031f46000 CR4: 0000000000000660
> [ 223.023470] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 223.029999] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 223.036478] Process netback/5 (pid: 1184, threadinfo ffff88003792c000, task ffff880037ceb0f0)
> [ 223.043095] Stack:
> [ 223.049616] ffff88003792dd1c ffff88003792c010 ffff88003792dd00 ffff88003792dc80
> [ 223.056404] ffffffff810800b5 00000000000000ba ffffc9001085ce50 ffffc900108582e8
> [ 223.063150] 0000000101080083 ffffc90010858298 0000000100000000 ffff88002c38d9c0
> [ 223.069955] Call Trace:
> [ 223.076591] [<ffffffff810800b5>] ? __alloc_workqueue_key+0x265/0x5d0
> [ 223.083426] [<ffffffff810acf3d>] ? trace_hardirqs_on+0xd/0x10
> [ 223.090261] [<ffffffff8147569a>] xen_netbk_kthread+0xba/0xa90
> [ 223.096990] [<ffffffff810957e6>] ? try_to_wake_up+0x1b6/0x310
> [ 223.103620] [<ffffffff81086810>] ? wake_up_bit+0x40/0x40
> [ 223.110195] [<ffffffff814755e0>] ? xen_netbk_tx_build_gops+0xa70/0xa70
> [ 223.116768] [<ffffffff810861a6>] kthread+0xd6/0xe0
> [ 223.123312] [<ffffffff8174e664>] kernel_thread_helper+0x4/0x10
> [ 223.129794] [<ffffffff8174cb37>] ? retint_restore_args+0x13/0x13
> [ 223.136217] [<ffffffff8174e660>] ? gs_change+0x13/0x13
> [ 223.142658] Code: 00 00 00 42 8b 54 30 3c 41 8b 74 04 08 31 c0 e8 e5 37 2d 00 8b 83 c4 00 00 00 4c 03 b3 c8 00 00 00 4a 8b 7c 30 30 e8 36 24 c8 ff <0f> 0b eb fe 48 8b b3 d0 00 00 00 48 c7 c2 c0 36 47 81 48 c7 c7
> [ 223.156486] RIP [<ffffffff8147463a>] xen_netbk_rx_action+0x89a/0x910
> [ 223.163337] RSP <ffff88003792dc20>
> [ 223.170212] ---[ end trace 03f82ac72747fb5d ]---
> [ 228.705439] device vif15.0 entered promiscuous mode
> [ 228.880399] device vif15.0-emu entered promiscuous mode
> [ 228.889286] xen_bridge: port 15(vif15.0-emu) entered forwarding state
> [ 228.895546] xen_bridge: port 15(vif15.0-emu) entered forwarding state
> [ 228.956267] vpn_bridge: port 1(vif13.0) entered forwarding state
> [ 229.119709] pciback 0000:06:00.0: restoring config space at offset 0x3c (was 0x100, writing 0x10a)
> [ 229.126644] pciback 0000:06:00.0: restoring config space at offset 0x10 (was 0x4, writing 0xf9a00004)
> [ 229.133434] pciback 0000:06:00.0: restoring config space at offset 0xc (was 0x0, writing 0x10)
> [ 234.170536] tty_init_dev: 15 callbacks suppressed
> [ 235.092664] xen_bridge: port 13(vif14.0) entered forwarding state
> [ 235.684229] device vif16.0 entered promiscuous mode
> [ 235.805155] device vif16.0-emu entered promiscuous mode
> [ 235.813948] xen_bridge: port 17(vif16.0-emu) entered forwarding state
> [ 235.820242] xen_bridge: port 17(vif16.0-emu) entered forwarding state
> [ 239.632852] xen_bridge: port 15(vif15.0-emu) entered disabled state
> [ 239.641629] xen_bridge: port 15(vif15.0-emu) entered disabled state
> [ 239.650288] device vif15.0-emu left promiscuous mode
> [ 239.658618] xen_bridge: port 15(vif15.0-emu) entered disabled state
> [ 240.982436] tty_init_dev: 15 callbacks suppressed
> [ 241.386562] xen-blkback:ring-ref 8, event-channel 25, protocol 1 (x86_64-abi)
> [ 241.400247] xen-blkback:ring-ref 9, event-channel 26, protocol 1 (x86_64-abi)
> [ 241.454701] xen_bridge: port 14(vif15.0) entered forwarding state
> [ 241.463330] xen_bridge: port 14(vif15.0) entered forwarding state
> [ 246.690393] xen_bridge: port 17(vif16.0-emu) entered disabled state
> [ 246.699042] xen_bridge: port 17(vif16.0-emu) entered disabled state
> [ 246.708731] device vif16.0-emu left promiscuous mode
> [ 246.717465] xen_bridge: port 17(vif16.0-emu) entered disabled state
> [ 249.449321] xen-blkback:ring-ref 8, event-channel 25, protocol 1 (x86_64-abi)
> [ 249.619531] xen_bridge: port 16(vif16.0) entered forwarding state
> [ 249.628307] xen_bridge: port 16(vif16.0) entered forwarding state
> [ 256.489967] xen_bridge: port 14(vif15.0) entered forwarding state
> [ 264.654183] xen_bridge: port 16(vif16.0) entered forwarding state
> [ 414.296535] tty_init_dev: 16 callbacks suppressed
> [ 458.898093] netbk_gop_frag_copy: size 5a8 offset 3602
> [ 458.898093] => 3baa > 1000
> [ 458.920252] netbk_gop_frag_copy failed: skb frag 0 page
> [ 458.928746] copying from offset 3602, len 5a8
> [ 458.937114] page:ffffea0000ada800 count:32749 mapcount:0 mapping: (null) index:0xffff88002b6a6100
> [ 458.945813] page flags: 0x40000000004000(head)
> [ 458.954314] ------------[ cut here ]------------
> [ 458.962655] kernel BUG at drivers/net/xen-netback/netback.c:548!
> [ 458.970929] invalid opcode: 0000 [#5] PREEMPT SMP
> [ 458.979113] Modules linked in:
> [ 458.987128] CPU 1
> [ 458.987178] Pid: 1180, comm: netback/1 Tainted: G D 3.6.0pre-rc1-20121008bisect #1 MSI MS-7640/890FXA-GD70 (MS-7640)
> [ 459.003052] RIP: e030:[<ffffffff8147463a>] [<ffffffff8147463a>] xen_netbk_rx_action+0x89a/0x910
> [ 459.011121] RSP: e02b:ffff880037925c20 EFLAGS: 00010282
> [ 459.019135] RAX: 0000000000000001 RBX: ffff88002ab0bf00 RCX: 0000000000000000
> [ 459.027199] RDX: ffff8800398a30f0 RSI: 0000000000000001 RDI: ffff8800379202b0
> [ 459.035081] RBP: ffff880037925d50 R08: 0000000000000002 R09: 0000000000000000
> [ 459.042816] R10: 0000000000000001 R11: ffff88003a26bdb0 R12: 0000000000000030
> [ 459.050308] R13: 0000000000000000 R14: ffff88002b6a2e00 R15: 0000000000000001
> [ 459.057725] FS: 00007f8e25af5760(0000) GS:ffff88003f840000(0000) knlGS:0000000000000000
> [ 459.065052] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 459.072248] CR2: 00007fe6b4d12fb0 CR3: 000000002c2f6000 CR4: 0000000000000660
> [ 459.079480] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 459.086512] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 459.093386] Process netback/1 (pid: 1180, threadinfo ffff880037924000, task ffff8800398a30f0)
> [ 459.100357] Stack:
> [ 459.107071] ffff880037925d1c ffff880037924010 ffff880037925d00 ffff880037925c80
> [ 459.113808] ffffffff810800b5 000000000000042a ffffc9001082ff70 ffffc9001082b408
> [ 459.120494] 0000000001080083 ffffc9001082b3b8 0000000000000000 ffff8800329249c0
> [ 459.127129] Call Trace:
> [ 459.133509] [<ffffffff810800b5>] ? __alloc_workqueue_key+0x265/0x5d0
> [ 459.140118] [<ffffffff810acf3d>] ? trace_hardirqs_on+0xd/0x10
> [ 459.146604] [<ffffffff8147569a>] xen_netbk_kthread+0xba/0xa90
> [ 459.153504] [<ffffffff810957e6>] ? try_to_wake_up+0x1b6/0x310
> [ 459.159949] [<ffffffff81086810>] ? wake_up_bit+0x40/0x40
> [ 459.166431] [<ffffffff814755e0>] ? xen_netbk_tx_build_gops+0xa70/0xa70
> [ 459.172778] [<ffffffff810861a6>] kthread+0xd6/0xe0
> [ 459.179018] [<ffffffff8174e664>] kernel_thread_helper+0x4/0x10
> [ 459.185291] [<ffffffff8174cb37>] ? retint_restore_args+0x13/0x13
> [ 459.191523] [<ffffffff8174e660>] ? gs_change+0x13/0x13
> [ 459.197862] Code: 00 00 00 42 8b 54 30 3c 41 8b 74 04 08 31 c0 e8 e5 37 2d 00 8b 83 c4 00 00 00 4c 03 b3 c8 00 00 00 4a 8b 7c 30 30 e8 36 24 c8 ff <0f> 0b eb fe 48 8b b3 d0 00 00 00 48 c7 c2 c0 36 47 81 48 c7 c7
> [ 459.211184] RIP [<ffffffff8147463a>] xen_netbk_rx_action+0x89a/0x910
> [ 459.217785] RSP <ffff880037925c20>
> [ 459.224501] ---[ end trace 03f82ac72747fb5e ]---
>
>
>
>
> > This made me notice that offset and len in the caller are variously
> > unsigned int, u16 or u32 while gop_frag_copy takes them as unsigned
> > longs. None of the numbers involved here are anywhere big enough to
> > cause any sort of overflow related error though.
>
> >> [ 197.892781] page:ffffea0000b18400 count:3 mapcount:0 mapping: (null) index:0x0
> >> [ 197.900778] page flags: 0x40000000004000(head)
> >> [ 197.907074] ------------[ cut here ]------------
> >> [ 197.913345] kernel BUG at drivers/net/xen-netback/netback.c:546!
> >> [ 197.919626] invalid opcode: 0000 [#1] PREEMPT SMP
> >> [ 197.921573] xen_bridge: port 10(vif10.0) entered forwarding state
> >> [ 197.932106] Modules linked in:
> >> [ 197.938370] CPU 0
> >> [ 197.938420] Pid: 1180, comm: netback/0 Not tainted 3.6.0pre-rc1-20121008bisect #1 MSI MS-7640/890FXA-GD70 (MS-7640)
> >> [ 197.951203] RIP: e030:[<ffffffff8147462a>] [<ffffffff8147462a>] xen_netbk_rx_action+0x89a/0x910
> >> [ 197.957775] RSP: e02b:ffff880037911c20 EFLAGS: 00010282
> >> [ 197.964290] RAX: 0000000000000001 RBX: ffff880036862ee0 RCX: 0000000000000000
> >> [ 197.970956] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8800379102b0
> >> [ 197.977679] RBP: ffff880037911d50 R08: 0000000000000002 R09: 0000000000000000
> >> [ 197.984361] R10: 0000000000000001 R11: ffff880039925e40 R12: 0000000000000030
> >> [ 197.990958] R13: 0000000000000000 R14: ffff880031e71800 R15: 0000000000000001
> >> [ 197.997459] FS: 00007fb5dfcf7700(0000) GS:ffff88003f800000(0000) knlGS:0000000000000000
> >> [ 198.004123] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> >> [ 198.010827] CR2: 00007fb5d802d000 CR3: 0000000031fd3000 CR4: 0000000000000660
> >> [ 198.017534] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> >> [ 198.024168] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> >> [ 198.030717] Process netback/0 (pid: 1180, threadinfo ffff880037910000, task ffff88003997d190)
> >> [ 198.037326] Stack:
> >> [ 198.043817] ffff880037911d1c ffff88003997d840 ffff880037911d00 ffff880037911c80
> >> [ 198.050573] ffffffff00000001 0000000000000662 ffffc90010824bb8 ffffc90010820050
> >> [ 198.057413] 0000000001080083 ffffc90010820000 0000000000000000 ffff880031cf09c0
> >> [ 198.064228] Call Trace:
> >> [ 198.070887] [<ffffffff810acf3d>] ? trace_hardirqs_on+0xd/0x10
> >> [ 198.077604] [<ffffffff8147568a>] xen_netbk_kthread+0xba/0xa90
> >> [ 198.084394] [<ffffffff810957e6>] ? try_to_wake_up+0x1b6/0x310
> >> [ 198.091109] [<ffffffff81086810>] ? wake_up_bit+0x40/0x40
> >> [ 198.097726] [<ffffffff814755d0>] ? xen_netbk_tx_build_gops+0xa70/0xa70
> >> [ 198.104343] [<ffffffff810861a6>] kthread+0xd6/0xe0
> >> [ 198.111001] [<ffffffff8174e664>] kernel_thread_helper+0x4/0x10
> >> [ 198.117737] [<ffffffff8174cb37>] ? retint_restore_args+0x13/0x13
> >> [ 198.124425] [<ffffffff8174e660>] ? gs_change+0x13/0x13
> >> [ 198.131008] Code: 00 00 00 42 8b 54 30 3c 41 8b 74 04 08 31 c0 e8 e5 37 2d 00 8b 83 c4 00 00 00 4c 03 b3 c8 00 00 00 4a 8b 7c 30 30 e8 46 24 c8 ff <0f> 0b eb fe 48 8b b3 d0 00 00 00 48 c7 c2 c0 36 47 81 48 c7 c7
> >> [ 198.145094] RIP [<ffffffff8147462a>] xen_netbk_rx_action+0x89a/0x910
> >> [ 198.152192] RSP <ffff880037911c20>
> >> [ 198.159344] ---[ end trace cbdd0e4e80268fa8 ]---
> >> [ 199.703539] tty_init_dev: 2 callbacks suppressed
> >> [ 200.712098] device vif12.0 entered promiscuous mode
> >> [ 201.010433] xen-blkback:ring-ref 8, event-channel 9, protocol 1 (x86_64-abi)
> >> [ 201.020644] xen_bridge: port 12(vif12.0) entered forwarding state
> >> [ 201.027833] xen_bridge: port 12(vif12.0) entered forwarding state
> >> [ 206.774576] netbk_gop_frag_copy failed: skb frag 0 page
> >> [ 206.777945] device vif13.0 entered promiscuous mode
> >> [ 206.788845] copying from offset 1ba4, len 2c1
> >> [ 206.795791] page:ffffea0000b18400 count:6 mapcount:0 mapping: (null) index:0x0
> >> [ 206.802771] page flags: 0x40000000004000(head)
> >> [ 206.809619] ------------[ cut here ]------------
> >> [ 206.816498] kernel BUG at drivers/net/xen-netback/netback.c:546!
> >> [ 206.823465] invalid opcode: 0000 [#2] PREEMPT SMP
> >> [ 206.830354] Modules linked in:
> >> [ 206.837176] CPU 3
> >> [ 206.837234] Pid: 1183, comm: netback/3 Tainted: G D 3.6.0pre-rc1-20121008bisect #1 MSI MS-7640/890FXA-GD70 (MS-7640)
> >> [ 206.850881] RIP: e030:[<ffffffff8147462a>] [<ffffffff8147462a>] xen_netbk_rx_action+0x89a/0x910
> >> [ 206.857935] RSP: e02b:ffff880037917c20 EFLAGS: 00010282
> >> [ 206.864972] RAX: 0000000000000001 RBX: ffff880003313ae0 RCX: 0000000000000000
> >> [ 206.872049] RDX: ffff88003997b0f0 RSI: 0000000000000001 RDI: ffff8800379102b0
> >> [ 206.879147] RBP: ffff880037917d50 R08: 0000000000000002 R09: 0000000000000000
> >> [ 206.886242] R10: 0000000000000001 R11: ffff880039925640 R12: 0000000000000030
> >> [ 206.893163] R13: 0000000000000000 R14: ffff88002c7c4400 R15: 0000000000000001
> >> [ 206.900041] FS: 00007f800341a700(0000) GS:ffff88003f8c0000(0000) knlGS:0000000000000000
> >> [ 206.907145] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> >> [ 206.914126] CR2: 00007f8002b31fb0 CR3: 0000000001c0b000 CR4: 0000000000000660
> >> [ 206.921181] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> >> [ 206.927996] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> >> [ 206.934711] Process netback/3 (pid: 1183, threadinfo ffff880037916000, task ffff88003997b0f0)
> >> [ 206.941494] Stack:
> >> [ 206.948105] ffff880037917d1c ffff880037916010 ffff880037917d00 ffff880037917c80
> >> [ 206.955062] ffffffff810800b5 00000000000000ba ffffc900108466e0 ffffc90010841b78
> >> [ 206.962007] 0000000101080083 ffffc90010841b28 0000000100000000 ffff88002c5bb9c0
> >> [ 206.968967] Call Trace:
> >> [ 206.975830] [<ffffffff810800b5>] ? __alloc_workqueue_key+0x265/0x5d0
> >> [ 206.982789] [<ffffffff810acf3d>] ? trace_hardirqs_on+0xd/0x10
> >> [ 206.989662] [<ffffffff8147568a>] xen_netbk_kthread+0xba/0xa90
> >> [ 206.996570] [<ffffffff810957e6>] ? try_to_wake_up+0x1b6/0x310
> >> [ 207.003523] [<ffffffff81086810>] ? wake_up_bit+0x40/0x40
> >> [ 207.010333] [<ffffffff814755d0>] ? xen_netbk_tx_build_gops+0xa70/0xa70
> >> [ 207.017171] [<ffffffff810861a6>] kthread+0xd6/0xe0
> >> [ 207.023890] [<ffffffff8174e664>] kernel_thread_helper+0x4/0x10
> >> [ 207.030540] [<ffffffff8174cb37>] ? retint_restore_args+0x13/0x13
> >> [ 207.037275] [<ffffffff8174e660>] ? gs_change+0x13/0x13
> >> [ 207.043890] Code: 00 00 00 42 8b 54 30 3c 41 8b 74 04 08 31 c0 e8 e5 37 2d 00 8b 83 c4 00 00 00 4c 03 b3 c8 00 00 00 4a 8b 7c 30 30 e8 46 24 c8 ff <0f> 0b eb fe 48 8b b3 d0 00 00 00 48 c7 c2 c0 36 47 81 48 c7 c7
> >> [ 207.057976] RIP [<ffffffff8147462a>] xen_netbk_rx_action+0x89a/0x910
> >> [ 207.065064] RSP <ffff880037917c20>
> >> [ 207.072056] ---[ end trace cbdd0e4e80268fa9 ]---
> >> [ 207.079366] xen-blkback:ring-ref 8, event-channel 9, protocol 1 (x86_64-abi)
> >> [ 207.090256] vpn_bridge: port 1(vif13.0) entered forwarding state
> >> [ 207.097403] vpn_bridge: port 1(vif13.0) entered forwarding state
> >> [ 208.636257] xen_bridge: port 11(vif11.0) entered forwarding state
> >> [ 211.515779] netbk_gop_frag_copy failed: skb frag 0 page
> >> [ 211.522711] copying from offset 2126, len 2c1
> >> [ 211.529403] page:ffffea0000b18400 count:8 mapcount:0 mapping: (null) index:0x0
> >> [ 211.536142] page flags: 0x40000000004000(head)
> >> [ 211.542942] ------------[ cut here ]------------
> >> [ 211.549664] kernel BUG at drivers/net/xen-netback/netback.c:546!
> >> [ 211.556408] invalid opcode: 0000 [#3] PREEMPT SMP
> >> [ 211.563168] Modules linked in:
> >> [ 211.569739] CPU 4
> >> [ 211.569789] Pid: 1184, comm: netback/4 Tainted: G D 3.6.0pre-rc1-20121008bisect #1 MSI MS-7640/890FXA-GD70 (MS-7640)
> >> [ 211.583126] RIP: e030:[<ffffffff8147462a>] [<ffffffff8147462a>] xen_netbk_rx_action+0x89a/0x910
> >> [ 211.590041] RSP: e02b:ffff880037921c20 EFLAGS: 00010282
> >> [ 211.596868] RAX: 0000000000000001 RBX: ffff8800375bc4e0 RCX: 0000000000000000
> >> [ 211.603890] RDX: ffff88003997a0a0 RSI: 0000000000000001 RDI: ffff8800379202b0
> >> [ 211.610792] RBP: ffff880037921d50 R08: 0000000000000002 R09: 0000000000000000
> >> [ 211.617608] R10: 0000000000000001 R11: ffff8800399249e0 R12: 0000000000000030
> >> [ 211.624537] R13: 0000000000000000 R14: ffff88002b98d400 R15: 0000000000000001
> >> [ 211.631302] FS: 00007f332d735740(0000) GS:ffff88003f900000(0000) knlGS:0000000000000000
> >> [ 211.638090] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> >> [ 211.644965] CR2: 00007f1023d22000 CR3: 0000000031fba000 CR4: 0000000000000660
> >> [ 211.651894] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> >> [ 211.658652] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> >> [ 211.665288] Process netback/4 (pid: 1184, threadinfo ffff880037920000, task ffff88003997a0a0)
> >> [ 211.671884] Stack:
> >> [ 211.678376] ffff880037921d1c ffff880037920010 ffff880037921d00 ffff880037921c80
> >> [ 211.685145] ffffffff810800b5 00000000000000ba ffffc90010851a98 ffffc9001084cf30
> >> [ 211.691837] 0000000101080083 ffffc9001084cee0 0000000100000000 ffff88002c5bd9c0
> >> [ 211.698581] Call Trace:
> >> [ 211.705349] [<ffffffff810800b5>] ? __alloc_workqueue_key+0x265/0x5d0
> >> [ 211.712156] [<ffffffff810acf3d>] ? trace_hardirqs_on+0xd/0x10
> >> [ 211.718907] [<ffffffff8147568a>] xen_netbk_kthread+0xba/0xa90
> >> [ 211.725654] [<ffffffff810957e6>] ? try_to_wake_up+0x1b6/0x310
> >> [ 211.732369] [<ffffffff81086810>] ? wake_up_bit+0x40/0x40
> >> [ 211.739111] [<ffffffff814755d0>] ? xen_netbk_tx_build_gops+0xa70/0xa70
> >> [ 211.745858] [<ffffffff810861a6>] kthread+0xd6/0xe0
> >> [ 211.752449] [<ffffffff8174e664>] kernel_thread_helper+0x4/0x10
> >> [ 211.758975] [<ffffffff8174cb37>] ? retint_restore_args+0x13/0x13
> >> [ 211.765575] [<ffffffff8174e660>] ? gs_change+0x13/0x13
> >> [ 211.772016] Code: 00 00 00 42 8b 54 30 3c 41 8b 74 04 08 31 c0 e8 e5 37 2d 00 8b 83 c4 00 00 00 4c 03 b3 c8 00 00 00 4a 8b 7c 30 30 e8 46 24 c8 ff <0f> 0b eb fe 48 8b b3 d0 00 00 00 48 c7 c2 c0 36 47 81 48 c7 c7
> >> [ 211.785816] RIP [<ffffffff8147462a>] xen_netbk_rx_action+0x89a/0x910
> >> [ 211.792586] RSP <ffff880037921c20>
> >> [ 211.799394] ---[ end trace cbdd0e4e80268faa ]---
> >> [ 212.852714] device vif14.0 entered promiscuous mode
> >> [ 213.234995] xen-blkback:ring-ref 8, event-channel 9, protocol 1 (x86_64-abi)
> >> [ 213.245054] xen_bridge: port 13(vif14.0) entered forwarding state
> >> [ 213.252087] xen_bridge: port 13(vif14.0) entered forwarding state
> >> [ 214.691532] netbk_gop_frag_copy failed: skb frag 0 page
> >> [ 214.698515] copying from offset 26a8, len 2c1
> >> [ 214.705472] page:ffffea0000b18400 count:10 mapcount:0 mapping: (null) index:0x0
> >> [ 214.712415] page flags: 0x40000000004000(head)
> >> [ 214.719170] ------------[ cut here ]------------
> >> [ 214.725887] kernel BUG at drivers/net/xen-netback/netback.c:546!
> >> [ 214.732563] invalid opcode: 0000 [#4] PREEMPT SMP
> >> [ 214.739221] Modules linked in:
> >> [ 214.745808] CPU 5
> >> [ 214.745859] Pid: 1185, comm: netback/5 Tainted: G D 3.6.0pre-rc1-20121008bisect #1 MSI MS-7640/890FXA-GD70 (MS-7640)
> >> [ 214.759156] RIP: e030:[<ffffffff8147462a>] [<ffffffff8147462a>] xen_netbk_rx_action+0x89a/0x910
> >> [ 214.766127] RSP: e02b:ffff880037923c20 EFLAGS: 00010282
> >> [ 214.773012] RAX: 0000000000000001 RBX: ffff8800379172e0 RCX: 0000000000000000
> >> [ 214.780010] RDX: ffff880039ac8000 RSI: 0000000000000001 RDI: ffff8800379202b0
> >> [ 214.786988] RBP: ffff880037923d50 R08: 0000000000000002 R09: 0000000000000000
> >> [ 214.793870] R10: 0000000000000001 R11: ffff880039924460 R12: 0000000000000030
> >> [ 214.800812] R13: 0000000000000000 R14: ffff88002b8b4800 R15: 0000000000000001
> >> [ 214.807668] FS: 00007f236d331700(0000) GS:ffff88003f940000(0000) knlGS:0000000000000000
> >> [ 214.814545] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> >> [ 214.821415] CR2: 00007f236c42b6b0 CR3: 0000000039275000 CR4: 0000000000000660
> >> [ 214.828435] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> >> [ 214.835337] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> >> [ 214.841963] Process netback/5 (pid: 1185, threadinfo ffff880037922000, task ffff880039ac8000)
> >> [ 214.848655] Stack:
> >> [ 214.855220] ffff880037923d1c ffff880037922010 ffff880037923d00 ffff880037923c80
> >> [ 214.861945] ffffffff810800b5 00000000000000ba ffffc9001085ce50 ffffc900108582e8
> >> [ 214.868699] 0000000101080083 ffffc90010858298 0000000100000000 ffff880031e939c0
> >> [ 214.875477] Call Trace:
> >> [ 214.882247] [<ffffffff810800b5>] ? __alloc_workqueue_key+0x265/0x5d0
> >> [ 214.889083] [<ffffffff810acf3d>] ? trace_hardirqs_on+0xd/0x10
> >> [ 214.895851] [<ffffffff8147568a>] xen_netbk_kthread+0xba/0xa90
> >> [ 214.902612] [<ffffffff810957e6>] ? try_to_wake_up+0x1b6/0x310
> >> [ 214.909343] [<ffffffff81086810>] ? wake_up_bit+0x40/0x40
> >> [ 214.916115] [<ffffffff814755d0>] ? xen_netbk_tx_build_gops+0xa70/0xa70
> >> [ 214.922856] [<ffffffff810861a6>] kthread+0xd6/0xe0
> >> [ 214.929527] [<ffffffff8174e664>] kernel_thread_helper+0x4/0x10
> >> [ 214.936178] [<ffffffff8174cb37>] ? retint_restore_args+0x13/0x13
> >> [ 214.942781] [<ffffffff8174e660>] ? gs_change+0x13/0x13
> >> [ 214.949279] Code: 00 00 00 42 8b 54 30 3c 41 8b 74 04 08 31 c0 e8 e5 37 2d 00 8b 83 c4 00 00 00 4c 03 b3 c8 00 00 00 4a 8b 7c 30 30 e8 46 24 c8 ff <0f> 0b eb fe 48 8b b3 d0 00 00 00 48 c7 c2 c0 36 47 81 48 c7 c7
> >> [ 214.963107] RIP [<ffffffff8147462a>] xen_netbk_rx_action+0x89a/0x910
> >> [ 214.969952] RSP <ffff880037923c20>
> >> [ 214.976802] ---[ end trace cbdd0e4e80268fab ]---
> >> [ 216.045946] xen_bridge: port 12(vif12.0) entered forwarding state
> >> [ 220.405869] device vif15.0 entered promiscuous mode
> >> [ 220.607946] device vif15.0-emu entered promiscuous mode
> >> [ 220.625075] xen_bridge: port 15(vif15.0-emu) entered forwarding state
> >> [ 220.633333] xen_bridge: port 15(vif15.0-emu) entered forwarding state
> >> [ 220.890237] pciback 0000:06:00.0: restoring config space at offset 0x3c (was 0x100, writing 0x10a)
> >> [ 220.898814] pciback 0000:06:00.0: restoring config space at offset 0x10 (was 0x4, writing 0xf9a00004)
> >> [ 220.907406] pciback 0000:06:00.0: restoring config space at offset 0xc (was 0x0, writing 0x10)
> >> [ 222.122750] vpn_bridge: port 1(vif13.0) entered forwarding state
> >> [ 225.943971] tty_init_dev: 14 callbacks suppressed
> >> [ 226.654618] device vif16.0 entered promiscuous mode
> >> [ 226.775073] device vif16.0-emu entered promiscuous mode
> >> [ 226.784025] xen_bridge: port 17(vif16.0-emu) entered forwarding state
> >> [ 226.790188] xen_bridge: port 17(vif16.0-emu) entered forwarding state
> >> [ 228.253024] xen_bridge: port 13(vif14.0) entered forwarding state
> >> [ 229.788197] xen_bridge: port 15(vif15.0-emu) entered disabled state
> >> [ 229.796826] xen_bridge: port 15(vif15.0-emu) entered disabled state
> >> [ 229.805243] device vif15.0-emu left promiscuous mode
> >> [ 229.813385] xen_bridge: port 15(vif15.0-emu) entered disabled state
> >> [ 231.558329] xen-blkback:ring-ref 8, event-channel 25, protocol 1 (x86_64-abi)
> >> [ 231.569080] xen-blkback:ring-ref 9, event-channel 26, protocol 1 (x86_64-abi)
> >> [ 231.609663] xen_bridge: port 14(vif15.0) entered forwarding state
> >> [ 231.617943] xen_bridge: port 14(vif15.0) entered forwarding state
> >> [ 231.934347] tty_init_dev: 25 callbacks suppressed
> >>
> >>
> >>
> >>
> >>
> >>
> >> > Ian.
> >>
> >> > diff --git a/drivers/net/xen-netback/netback.c b/drivers/net/xen-netback/netback.c
> >> > index 05593d8..ca4c47d 100644
> >> > --- a/drivers/net/xen-netback/netback.c
> >> > +++ b/drivers/net/xen-netback/netback.c
> >> > @@ -386,7 +386,7 @@ static struct netbk_rx_meta *get_next_rx_buffer(struct xenvif *vif,
> >> > * Set up the grant operations for this fragment. If it's a flipping
> >> > * interface, we also set up the unmap request from here.
> >> > */
> >> > -static void netbk_gop_frag_copy(struct xenvif *vif, struct sk_buff *skb,
> >> > +static int netbk_gop_frag_copy(struct xenvif *vif, struct sk_buff *skb,
> >> > struct netrx_pending_operations *npo,
> >> > struct page *page, unsigned long size,
> >> > unsigned long offset, int *head)
> >> > @@ -402,7 +402,8 @@ static void netbk_gop_frag_copy(struct xenvif *vif, struct sk_buff *skb,
> >> > unsigned long bytes;
> >> >
> >> > /* Data must not cross a page boundary. */
> >> > - BUG_ON(size + offset > PAGE_SIZE);
> >> > + if (size + offset > PAGE_SIZE)
> >> > + return -1;
> >> >
> >> > meta = npo->meta + npo->meta_prod - 1;
> >> >
> >> > @@ -459,6 +460,7 @@ static void netbk_gop_frag_copy(struct xenvif *vif, struct sk_buff *skb,
> >> > *head = 0; /* There must be something in this buffer now. */
> >> >
> >> > }
> >> > + return 0;
> >> > }
> >> >
> >> > /*
> >> > @@ -517,17 +519,31 @@ static int netbk_gop_skb(struct sk_buff *skb,
> >> > if (data + len > skb_tail_pointer(skb))
> >> > len = skb_tail_pointer(skb) - data;
> >> >
> >> > - netbk_gop_frag_copy(vif, skb, npo,
> >> > - virt_to_page(data), len, offset, &head);
> >> > + if (netbk_gop_frag_copy(vif, skb, npo,
> >> > + virt_to_page(data), len, offset, &head) < 0) {
> >> > +printk(KERN_CRIT "netbk_gop_frag_copy failed: skb head %p-%p\n",
> >> + skb->>data, skb_tail_pointer);
> >> > +printk(KERN_CRIT "copying from %p-%p, offset %x, len %x\n",
> >> > + data, data+len, offset, len);
> >> > +dump_page(virt_to_page(data));
> >> > +BUG();
> >> > + }
> >> > data += len;
> >> > }
> >> >
> >> > for (i = 0; i < nr_frags; i++) {
> >> > - netbk_gop_frag_copy(vif, skb, npo,
> >> > + if (netbk_gop_frag_copy(vif, skb, npo,
> >> > skb_frag_page(&skb_shinfo(skb)->frags[i]),
> >> > skb_frag_size(&skb_shinfo(skb)->frags[i]),
> >> > skb_shinfo(skb)->frags[i].page_offset,
> >> > - &head);
> >> > + &head) < 0) {
> >> > +printk(KERN_CRIT "netbk_gop_frag_copy failed: skb frag %d page\n", i);
> >> > +printk(KERN_CRIT "copying from offset %x, len %x\n",
> >> > + skb_shinfo(skb)->frags[i].page_offset,
> >> > + skb_frag_size(&skb_shinfo(skb)->frags[i]));
> >> > +dump_page(skb_frag_page(&skb_shinfo(skb)->frags[i]));
> >> > +BUG();
> >> > + }
> >> > }
> >> >
> >> > return npo->meta_prod - old_meta_prod;
> >>
> >>
> >>
> >>
>
>
>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
[-- Attachment #3: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: RFH: Kernel OOPS in xen_netbk_rx_action / xenvif_gop_skb
2014-06-06 10:26 RFH: Kernel OOPS in xen_netbk_rx_action / xenvif_gop_skb Philipp Hahn
@ 2014-06-06 10:58 ` Wei Liu
2014-06-06 22:12 ` Philipp Hahn
0 siblings, 1 reply; 18+ messages in thread
From: Wei Liu @ 2014-06-06 10:58 UTC (permalink / raw)
To: Philipp Hahn; +Cc: xen-devel, Erik Damrose, Wei Liu, Ian Campbell, Zoltan Kiss
On Fri, Jun 06, 2014 at 12:26:55PM +0200, Philipp Hahn wrote:
> Hello,
>
> on one of our hosts (Xen-4.1.3 with Linux-3.10.26 + Debian patches)
> running 16 Linux VMs (linux-3.2.39 and others) netback crashes during
> the night when one of the VMs is rebooted by a cron-job:
> > [38551.549615] Oops: 0000 [#1] SMP
Is there any more output above this line? Is it a NULL pointer
dereference or something else?
> > [38551.549665] Modules linked in: tun xt_physdev xen_blkback xen_netback ip6_tables
> > iptable_filter ip_tables ebtable_nat ebtables x_tables xen_gntdev nfsv3 nfsv4
> > rpcsec_gss_krb5 nfsd nfs_acl auth_rpcgss oid_registry nfs fscache dns_resolver lockd
> > sunrpc fuse loop xen_blkfront xen_evtchn blktap quota_v2 quota_tree xenfs xen_privcmd
> > coretemp crc32c_intel ghash_clmulni_intel aesni_intel ablk_helper cryptd lrw gf128mul
> > glue_helper aes_x86_64 snd_pcm snd_timer snd soundcore snd_page_alloc tpm_tis tpm lpc_ich
> > tpm_bios i7core_edac i2c_i801 psmouse microcode edac_core serio_raw pcspkr mperf ioatdma
> > mfd_core processor evdev thermal_sys ext4 jbd2 crc16 bonding bridge stp llc dm_snapshot
> > dm_mirror dm_region_hash dm_log dm_mod sd_mod crc_t10dif ehci_pci uhci_hcd ehci_hcd mptsas
> > mptscsih mptbase scsi_transport_sas usbcore usb_common igb dca i2c_algo_bit i2c_core ptp
> > pps_core button
> > [38551.550601] CPU: 0 PID: 12587 Comm: netback/0 Not tainted 3.10.0-ucs58-amd64 #1 Debian
> > 3.10.11-1.58.201405060908
> > [38551.550693] Hardware name: FUJITSU PRIMERGY BX620 S6/D3051, BIOS 080015 Rev.3C78.3051
> > 07/22/2011
> > [38551.550781] task: ffff880004b067c0 ti: ffff8800561ec000 task.ti: ffff8800561ec000
> > [38551.550865] RIP: e030:[<ffffffffa04147dc>] [<ffffffffa04147dc>]
> > xen_netbk_rx_action+0x18b/0x6f0 [xen_netback]
Try addr2line?
> > [38551.550959] RSP: e02b:ffff8800561edce8 EFLAGS: 00010202
> > [38551.551009] RAX: ffffc900104adac0 RBX: ffff8800541e95c0 RCX: ffffc90010864000
> > [38551.551064] RDX: 000000000000003b RSI: 0000000000000000 RDI: ffff880040014380
> > [38551.551120] RBP: ffff8800570e6800 R08: 0000000000000000 R09: ffff880004799800
> > [38551.551175] R10: ffffffff813ca115 R11: ffff88005e4fdb08 R12: ffff880054e6f800
> > [38551.551231] R13: ffff8800561edd58 R14: ffffc900104a1000 R15: 0000000000000000
> > [38551.551289] FS: 00007f19a54a8700(0000) GS:ffff88005da00000(0000)
> > knlGS:0000000000000000
> > [38551.551374] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> > [38551.551425] CR2: ffffc900108641d8 CR3: 0000000054cb3000 CR4: 0000000000002660
> > [38551.551481] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > [38551.551537] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > [38551.551592] Stack:
> > [38551.551630] ffff880004b06ba0 0000000000000000 ffff88005da13ec0 ffff88005da13ec0
> > [38551.551726] 0000000004b067c0 ffffc900104a8ac0 ffffc900104a1020 000000005da13ec0
> > [38551.551823] 0000000000000000 0000000000000001 ffffc900104a8ac0 ffffc900104adac0
> > [38551.551920] Call Trace:
> > [38551.551966] [<ffffffff813ca32d>] ? _raw_spin_lock_irqsave+0x11/0x2f
> > [38551.552021] [<ffffffffa0416033>] ? xen_netbk_kthread+0x174/0x841 [xen_netback]
> > [38551.552106] [<ffffffff8105d373>] ? wake_up_bit+0x20/0x20
> > [38551.560239] [<ffffffffa0415ebf>] ? xen_netbk_tx_build_gops+0xce8/0xce8 [xen_netback]
> > [38551.560325] [<ffffffff8105cd73>] ? kthread_freezable_should_stop+0x56/0x56
> > [38551.560381] [<ffffffffa0415ebf>] ? xen_netbk_tx_build_gops+0xce8/0xce8 [xen_netback]
> > [38551.560466] [<ffffffff8105ce1e>] ? kthread+0xab/0xb3
> > [38551.560518] [<ffffffff81003638>] ? xen_end_context_switch+0xe/0x1c
> > [38551.560572] [<ffffffff8105cd73>] ? kthread_freezable_should_stop+0x56/0x56
> > [38551.560628] [<ffffffff813cfbfc>] ? ret_from_fork+0x7c/0xb0
> > [38551.560680] [<ffffffff8105cd73>] ? kthread_freezable_should_stop+0x56/0x56
> > [38551.560734] Code: 8b b3 d0 00 00 00 48 8b bb d8 00 00 00 0f b7 74 37 02 89 70 08 eb 07
> > c7 40 08 00 00 00 00 89 d2 c7 40 04 00 00 00 00 48 83 c2 08 <0f> b7 34 d1 89 30 c7 44 24
> > 60 00 00 00 00 8b 44 d1 04 89 44 24
> > [38551.561151] RIP [<ffffffffa04147dc>] xen_netbk_rx_action+0x18b/0x6f0 [xen_netback]
> > [38551.561238] RSP <ffff8800561edce8>
> > [38551.561283] CR2: ffffc900108641d8
> > [38551.561624] ---[ end trace 8c260c6af259c4aa ]---
>
> The host itself is still alive and reachable by network, but all VMs are
> no longer reachable.
> The crash does not happen on every reboot: The VM was running fine for
> 1½ week after a dom0 kernel update, but now crashed the following past
> two nights.
>
What's the Dom0 kernel version before upgrading? That would help us
narrow down the range of changesets.
The oops happens in guest receive path. Unfortunately that's a very
complex function, it's hard to identify the problem by looking at the
code.
And as you seem to be using a distro kernel, have your reported to
Debian yet? I don't quite understand which Debian release has 3.10
kernel though.
> I'm yet unable to reproduce this on demand, but would like to prepared
> next time it happens again.
>
> @Ian: I found your mail "Re: [Xen-devel] Kernel 3.7.0-pre-rc1 kernel BUG
> at drivers/net/xen-netback/netback.c:405 RIP: e030:[<ffffffff814714f9>]
> [<ffffffff814714f9>] netbk_gop_frag_copy+0x379/0x380" from 2012-10-09,
> which describes a crash in the same function, but at a complete
> different (later) location. You hinted that a difference in hardware
> might explain, why I'm unable to reproduce it, as my test environment
> has different HW (no "igb", but "e1000e").
>
3.7.0 is too old. There has been lots of changes since then.
> Running "objdump -Sl xen-netback.ko" shows the OOPs to happen here:
> > /root/linux-3.10.11/drivers/net/xen-netback/netback.c:606
> > meta->gso_size = skb_shinfo(skb)->gso_size;
> > 7b1: 8b b3 d0 00 00 00 mov 0xd0(%rbx),%esi
> > 7b7: 48 8b bb d8 00 00 00 mov 0xd8(%rbx),%rdi
> > 7be: 0f b7 74 37 02 movzwl 0x2(%rdi,%rsi,1),%esi
> > 7c3: 89 70 08 mov %esi,0x8(%rax)
> > 7c6: eb 07 jmp 7cf <xen_netbk_rx_action+0x17e>
> > /root/linux-3.10.11/drivers/net/xen-netback/netback.c:608
You mentioned 3.10.26 at the beginning but now it's 3.10.11? I'm
confused.
If it's dereferencing NULL pointer, skb_shinfo(skb) == NULL?
> > else
> > meta->gso_size = 0;
> > 7c8: c7 40 08 00 00 00 00 movl $0x0,0x8(%rax)
> > /root/linux-3.10.11/drivers/net/xen-netback/netback.c:611
> >
> > meta->size = 0;
> > meta->id = req->id;
> > 7cf: 89 d2 mov %edx,%edx
> > /root/linux-3.10.11/drivers/net/xen-netback/netback.c:610
> > if (!vif->gso_prefix)
> > meta->gso_size = skb_shinfo(skb)->gso_size;
> > else
> > meta->gso_size = 0;
> >
> > meta->size = 0;
> > 7d1: c7 40 04 00 00 00 00 movl $0x0,0x4(%rax)
> > /root/linux-3.10.11/drivers/net/xen-netback/netback.c:611
> > meta->id = req->id;
> > 7d8: 48 83 c2 08 add $0x8,%rdx
> > 7dc: 0f b7 34 d1 movzwl (%rcx,%rdx,8),%esi
> 0x651 + 0x18B = 0x7DC
>
> > 7e0: 89 30 mov %esi,(%rax)
> > /root/linux-3.10.11/drivers/net/xen-netback/netback.c:612
> > npo->copy_off = 0;
> > 7e2: c7 44 24 60 00 00 00 movl $0x0,0x60(%rsp)
> > 7e9: 00
> > /root/linux-3.10.11/drivers/net/xen-netback/netback.c:613
> > npo->copy_gref = req->gref;
> > 7ea: 8b 44 d1 04 mov 0x4(%rcx,%rdx,8),%eax
> > 7ee: 89 44 24 64 mov %eax,0x64(%rsp)
>
> Ignoring the name change from {netbk -> xenvif}_gop_skb() and the
> addition of GSO for IPv6 the function looks unchanged compared to
> current GIT, so to me it looks like it might still be a problem with the
> current implementation.
> I tried to review the GIT commits myself, but I didn't see anything
> obvious, but with all the recent additional changes to netback I'm
> unsure of how to best proceed:
> 1. Is this a known bug and has someone observed it, too?
Not that I know of.
> 2. If yes, is there a fix in newer Linux kernels?
> 3. If no, What data should I collect in addition?
>
There's one more patch that you can pick up from 3.10.y tree. I doubt it
will make much difference though.
I think the first thing to do is to identify which line of code is
causing the problem. If it is actually the line you're referring to in
your analyse then we need to figure out why skb_shinfo(skb) is NULL...
> Xen-Hypervisor is 4.1.3 from Debian, but as this is a kernel crash, I
> don't expect a newer version of Xen to fix it (correct me if I'm wrong).
>
You're correct. Upgrading hypervisor won't help.
Wei.
> Thanks in advance.
>
> Philipp
>
> PS: I'm not afraid of getting my hands dirty doing Linux coding, but
> currently I'm out of ideas of how to best proceed.
> --
> Philipp Hahn
> Open Source Software Engineer
>
> Univention GmbH
> be open.
> Mary-Somerville-Str. 1
> D-28359 Bremen
> Tel.: +49 421 22232-0
> Fax : +49 421 22232-99
> hahn@univention.de
>
> http://www.univention.de/
> Geschäftsführer: Peter H. Ganten
> HRB 20755 Amtsgericht Bremen
> Steuer-Nr.: 71-597-02876
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: RFH: Kernel OOPS in xen_netbk_rx_action / xenvif_gop_skb
2014-06-06 10:58 ` Wei Liu
@ 2014-06-06 22:12 ` Philipp Hahn
2014-06-18 16:48 ` Philipp Hahn
0 siblings, 1 reply; 18+ messages in thread
From: Philipp Hahn @ 2014-06-06 22:12 UTC (permalink / raw)
To: Wei Liu; +Cc: xen-devel, Erik Damrose, Ian Campbell, Zoltan Kiss
Hello,
On 06.06.2014 12:58, Wei Liu wrote:
> On Fri, Jun 06, 2014 at 12:26:55PM +0200, Philipp Hahn wrote:
>> on one of our hosts (Xen-4.1.3 with Linux-3.10.26 + Debian patches)
>> running 16 Linux VMs (linux-3.2.39 and others) netback crashes during
>> the night when one of the VMs is rebooted by a cron-job:
>>> [38551.549615] Oops: 0000 [#1] SMP
>
> Is there any more output above this line? Is it a NULL pointer
> dereference or something else?
Sorry, those lines got lost somehow during copy&paste:
[38551.547728] XXXlan0: port 9(vif26.0) entered disabled state
[38551.549365] BUG: unable to handle kernel paging request at
ffffc900108641d8
[38551.549461] IP: [<ffffffffa04147dc>] xen_netbk_rx_action+0x18b/0x6f0
[xen_netback]
[38551.549551] PGD 57e20067 PUD 57e21067 PMD 571a7067 PTE 0
[38551.549615] Oops: 0000 [#1] SMP
>>> [38551.550865] RIP: e030:[<ffffffffa04147dc>] [<ffffffffa04147dc>]
>>> xen_netbk_rx_action+0x18b/0x6f0 [xen_netback]
>
> Try addr2line?
Good to know, but since that host is already rebooted, I no longer know
the module load address, which seems to render addr2line useless.
>> The host itself is still alive and reachable by network, but all VMs are
>> no longer reachable.
>> The crash does not happen on every reboot: The VM was running fine for
>> 1½ week after a dom0 kernel update, but now crashed the following past
>> two nights.
>>
>
> What's the Dom0 kernel version before upgrading? That would help us
> narrow down the range of changesets.
The previous kernel was 3.10.15. The update was performed to get another
bug fixed, which went into the Debian update between .11 and .26:
commit 0ff773f59ff375c42af2238457bda98ed4ddcd25
Author: David Vrabel <david.vrabel@citrix.com>
Date: Wed Sep 11 14:52:48 2013 +0100
xen-netback: count number required slots for an skb more carefully
[ Upstream commit 6e43fc04a6bc357d260583b8440882f28069207f ]
> The oops happens in guest receive path. Unfortunately that's a very
> complex function, it's hard to identify the problem by looking at the
> code.
>
> And as you seem to be using a distro kernel, have your reported to
> Debian yet? I don't quite understand which Debian release has 3.10
> kernel though.
Actually this is "Univention Corporate Server" (UCS), which is
Debian-Squeeze based but with a newer Xen-4.1.3 and newer Linux-3.10 kernel.
> 3.7.0 is too old. There has been lots of changes since then.
Probably so, but thanks for the confirmation.
>> Running "objdump -Sl xen-netback.ko" shows the OOPs to happen here:
>>> /root/linux-3.10.11/drivers/net/xen-netback/netback.c:606
...
> You mentioned 3.10.26 at the beginning but now it's 3.10.11? I'm
> confused.
This has something to do with Debian patch policy: The first 3.10 kernel
was 3.10.11, so that number stays, even when it up-patched to 3.10.26.
> If it's dereferencing NULL pointer, skb_shinfo(skb) == NULL?
If I got the math right, it looks like it's crashing here:
>>> /root/linux-3.10.11/drivers/net/xen-netback/netback.c:611
>>> meta->id = req->id;
>>> 7d8: 48 83 c2 08 add $0x8,%rdx
>>> 7dc: 0f b7 34 d1 movzwl (%rcx,%rdx,8),%esi
>> 0x651 + 0x18B = 0x7DC
0x651 is the start of xen_netbk_rx_action() from objdump.
...
> There's one more patch that you can pick up from 3.10.y tree. I doubt it
> will make much difference though.
>
> I think the first thing to do is to identify which line of code is
> causing the problem. If it is actually the line you're referring to in
> your analyse then we need to figure out why skb_shinfo(skb) is NULL...
I'll try to add some debug output to yell if skb_shinfo() is NULL, but
it might take some time until the bug manifests again.
> Wei.
Thank you for your feedback.
Philipp
--
Philipp Hahn
Open Source Software Engineer
Univention GmbH
be open.
Mary-Somerville-Str. 1
D-28359 Bremen
Tel.: +49 421 22232-0
Fax : +49 421 22232-99
hahn@univention.de
http://www.univention.de/
Geschäftsführer: Peter H. Ganten
HRB 20755 Amtsgericht Bremen
Steuer-Nr.: 71-597-02876
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: RFH: Kernel OOPS in xen_netbk_rx_action / xenvif_gop_skb
2014-06-06 22:12 ` Philipp Hahn
@ 2014-06-18 16:48 ` Philipp Hahn
2014-06-19 14:12 ` Wei Liu
0 siblings, 1 reply; 18+ messages in thread
From: Philipp Hahn @ 2014-06-18 16:48 UTC (permalink / raw)
To: Wei Liu; +Cc: xen-devel, Erik Damrose, Ian Campbell, Zoltan Kiss
[-- Attachment #1: Type: text/plain, Size: 5763 bytes --]
Hello,
We are now more or less able to reproduce the OOPS within one hour by
constantly shutting down the vm and rebooting it:
> [32918.795695] XXXlan0: port 3(vif18.0) entered disabled state
> [32918.798732] BUG: unable to handle kernel paging request at ffffc90010da2188
> [32918.798823] IP: [<ffffffffa04287dc>] xen_netbk_rx_action+0x18b/0x6f0 [xen_netback]
> [32918.798911] PGD 95822067 PUD 95823067 PMD 94f47067 PTE 0
> [32918.798974] Oops: 0000 [#1] SMP
> [32918.799023] Modules linked in: xt_physdev xen_blkback xen_netback ip6_tables iptable_filter ip_tables ebtable_nat ebtables x_tables xen_gntdev nfsv3 nfsv4 rpcsec_gss_krb5 nfsd nfs_acl auth_rpcgss oid_registry nfs fscache dns_resolver lockd sunrpc fuse loop xen_blkfront xen_evtchn blktap quota_v2 quota_tree xenfs xen_privcmd coretemp crc32c_intel ghash_clmulni_intel aesni_intel ablk_helper cryptd lrw snd_pcm gf128mul snd_timer glue_helper snd aes_x86_64 soundcore snd_page_alloc microcode tpm_tis tpm tpm_bios pcspkr lpc_ich mfd_core acpi_power_meter i7core_edac mperf serio_raw i2c_i801 evdev edac_core processor ioatdma thermal_sys ext4 jbd2 crc16 bonding bridge stp llc dm_snapshot dm_mirror dm_region_hash dm_log dm_mod sd_mod crc_t10dif hid_generic usbhid hid mptsas mptscsih mptbase scs
i_transport_sas ehci_pci button uhci_hcd ehci_hcd usbcore usb_common igb dca i2c_algo_bit i2c_core ptp pps_core
> [32918.799958] CPU: 0 PID: 6450 Comm: netback/0 Not tainted 3.10.0-ucs58-amd64 #1 Debian 3.10.11-1.58.201405060908
> [32918.800050] Hardware name: FUJITSU PRIMERGY BX920 S2/D3030, BIOS 080015 Rev.3D94.3030 10/09/2012
> [32918.800137] task: ffff880093864880 ti: ffff88009266c000 task.ti: ffff88009266c000
> [32918.800220] RIP: e030:[<ffffffffa04287dc>] [<ffffffffa04287dc>] xen_netbk_rx_action+0x18b/0x6f0 [xen_netback]
> [32918.800314] RSP: e02b:ffff88009266dce8 EFLAGS: 00010212
> [32918.800364] RAX: ffffc9001082dac0 RBX: ffff880004d86ac0 RCX: ffffc90010da2000
> [32918.800419] RDX: 0000000000000031 RSI: 0000000000000000 RDI: ffff880004bdd280
> [32918.800474] RBP: ffff8800932db800 R08: 0000000000000000 R09: ffff8800952f3800
> [32918.800529] R10: 0000000000007ff0 R11: ffff88009c611380 R12: ffff8800932db800
> [32918.800584] R13: ffff88009266dd58 R14: ffffc90010821000 R15: 0000000000000000
> [32918.800642] FS: 00007f2f8fdcd700(0000) GS:ffff88009c600000(0000) knlGS:0000000000000000
> [32918.800728] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> [32918.800778] CR2: ffffc90010da2188 CR3: 0000000093eb0000 CR4: 0000000000002660
> [32918.800834] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [32918.800889] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [32918.800943] Stack:
> [32918.800981] ffff880093864c60 000000008106d2af ffff88009c613ec0 ffff88009c613ec0
> [32918.801077] 0000000093864880 ffffc90010828ac0 ffffc90010821020 000000009c613ec0
> [32918.801173] 0000000000000000 0000000000000001 ffffc90010828ac0 ffffc9001082dac0
> [32918.801269] Call Trace:
> [32918.801314] [<ffffffff813ca32d>] ? _raw_spin_lock_irqsave+0x11/0x2f
> [32918.801368] [<ffffffffa042a033>] ? xen_netbk_kthread+0x174/0x841 [xen_netback]
> [32918.801454] [<ffffffff8105d373>] ? wake_up_bit+0x20/0x20
> [32918.801504] [<ffffffffa0429ebf>] ? xen_netbk_tx_build_gops+0xce8/0xce8 [xen_netback]
> [32918.801590] [<ffffffff8105cd73>] ? kthread_freezable_should_stop+0x56/0x56
> [32918.801645] [<ffffffffa0429ebf>] ? xen_netbk_tx_build_gops+0xce8/0xce8 [xen_netback]
> [32918.801730] [<ffffffff8105ce1e>] ? kthread+0xab/0xb3
> [32918.801781] [<ffffffff81003638>] ? xen_end_context_switch+0xe/0x1c
> [32918.801834] [<ffffffff8105cd73>] ? kthread_freezable_should_stop+0x56/0x56
> [32918.801890] [<ffffffff813cfbfc>] ? ret_from_fork+0x7c/0xb0
> [32918.801941] [<ffffffff8105cd73>] ? kthread_freezable_should_stop+0x56/0x56
> [32918.801995] Code: 8b b3 d0 00 00 00 48 8b bb d8 00 00 00 0f b7 74 37 02 89 70 08 eb 07 c7 40 08 00 00 00 00 89 d2 c7 40 04 00 00 00 00 48 83 c2 08 <0f> b7 34 d1 89 30 c7 44 24 60 00 00 00 00 8b 44 d1 04 89 44 24
> [32918.802400] RIP [<ffffffffa04287dc>] xen_netbk_rx_action+0x18b/0x6f0 [xen_netback]
> [32918.802486] RSP <ffff88009266dce8>
> [32918.802529] CR2: ffffc90010da2188
> [32918.802859] ---[ end trace baf81e34c52eb41c ]---
(gdb) list *(xen_netbk_rx_action+0x18b)
0xffffffffa04287dc is in xen_netbk_rx_action
(/var/build/temp/tmp.hW3dNilayw/pbuilder/linux-3.10.11/drivers/net/xen-netback/netback
.c:611).
606 meta->gso_size = skb_shinfo(skb)->gso_size;
607 else
608 meta->gso_size = 0;
609
610 meta->size = 0;
611 meta->id = req->id;
612 npo->copy_off = 0;
613 npo->copy_gref = req->gref;
614
615 data = skb->data;
After more debugging today I think something like this happens:
1. The VM is receiving packets through bonding + bridge + netback +
netfront.
2. For some unknown reason at least one packet remains in the rx queue
and is not delivered to the domU immediately by netback.
3. The VM finishes shutting down.
4. The shared ring between dom0 and domU is freed.
5. then xen-netback continues processing the pending requests and tries
to put the packet into the now already released shared ring.
>From reading the attached disassembly I guess, that
AX = &meta
CX = &rx->string
DX =~ rx.req_cons
CR2 = &req->id
where
CX + DX * sizeof(union struct xen_netif_rx_{request,response})=8 = CR2
Any additional ideas or insight is appreciated.
FYI: The host has only a single CPU and is running >=2 VMs so far.
>> There's one more patch that you can pick up from 3.10.y tree. I doubt it
>> will make much difference though.
Which patch are you referring to?
Sincerely
Philipp
[-- Attachment #2: xen-netback.s --]
[-- Type: text/plain, Size: 5282 bytes --]
drivers/net/xen-netback/netback.c:582
* frontend-side LRO).
*/
static int netbk_gop_skb(struct sk_buff *skb,
struct netrx_pending_operations *npo)
{
struct xenvif *vif = netdev_priv(skb->dev);
721: 48 81 c5 00 08 00 00 add $0x800,%rbp
drivers/net/xen-netback/netback.c:594
int old_meta_prod;
old_meta_prod = npo->meta_prod;
/* Set up a GSO prefix descriptor, if necessary */
if (skb_shinfo(skb)->gso_size && vif->gso_prefix) {
728: 66 83 7c 02 02 00 cmpw $0x0,0x2(%rdx,%rax,1)
72e: 74 53 je 783 <xen_netbk_rx_action+0x132>
730: f6 45 60 04 testb $0x4,0x60(%rbp)
734: 74 4d je 783 <xen_netbk_rx_action+0x132>
drivers/net/xen-netback/netback.c:595
req = RING_GET_REQUEST(&vif->rx, vif->rx.req_cons++);
736: 8b 55 4c mov 0x4c(%rbp),%edx
739: 48 8b 75 58 mov 0x58(%rbp),%rsi
73d: 8b 7d 50 mov 0x50(%rbp),%edi
740: 8d 42 01 lea 0x1(%rdx),%eax
743: ff cf dec %edi
745: 89 45 4c mov %eax,0x4c(%rbp)
drivers/net/xen-netback/netback.c:596
meta = npo->meta + npo->meta_prod++;
748: 8b 4c 24 48 mov 0x48(%rsp),%ecx
drivers/net/xen-netback/netback.c:599
meta->gso_size = skb_shinfo(skb)->gso_size;
meta->size = 0;
meta->id = req->id;
74c: 21 fa and %edi,%edx
drivers/net/xen-netback/netback.c:596
old_meta_prod = npo->meta_prod;
/* Set up a GSO prefix descriptor, if necessary */
if (skb_shinfo(skb)->gso_size && vif->gso_prefix) {
req = RING_GET_REQUEST(&vif->rx, vif->rx.req_cons++);
meta = npo->meta + npo->meta_prod++;
74e: 89 c8 mov %ecx,%eax
750: ff c1 inc %ecx
752: 89 4c 24 48 mov %ecx,0x48(%rsp)
drivers/net/xen-netback/netback.c:597
meta->gso_size = skb_shinfo(skb)->gso_size;
756: 8b 8b d0 00 00 00 mov 0xd0(%rbx),%ecx
75c: 4c 8b 83 d8 00 00 00 mov 0xd8(%rbx),%r8
drivers/net/xen-netback/netback.c:596
old_meta_prod = npo->meta_prod;
/* Set up a GSO prefix descriptor, if necessary */
if (skb_shinfo(skb)->gso_size && vif->gso_prefix) {
req = RING_GET_REQUEST(&vif->rx, vif->rx.req_cons++);
meta = npo->meta + npo->meta_prod++;
763: 48 6b c0 0c imul $0xc,%rax,%rax
767: 48 03 44 24 58 add 0x58(%rsp),%rax
drivers/net/xen-netback/netback.c:597
meta->gso_size = skb_shinfo(skb)->gso_size;
76c: 41 0f b7 4c 08 02 movzwl 0x2(%r8,%rcx,1),%ecx
drivers/net/xen-netback/netback.c:598
meta->size = 0;
772: c7 40 04 00 00 00 00 movl $0x0,0x4(%rax)
drivers/net/xen-netback/netback.c:597
/* Set up a GSO prefix descriptor, if necessary */
if (skb_shinfo(skb)->gso_size && vif->gso_prefix) {
req = RING_GET_REQUEST(&vif->rx, vif->rx.req_cons++);
meta = npo->meta + npo->meta_prod++;
meta->gso_size = skb_shinfo(skb)->gso_size;
779: 89 48 08 mov %ecx,0x8(%rax)
drivers/net/xen-netback/netback.c:599
meta->size = 0;
meta->id = req->id;
77c: 0f b7 54 d6 40 movzwl 0x40(%rsi,%rdx,8),%edx
781: 89 10 mov %edx,(%rax)
drivers/net/xen-netback/netback.c:602
}
req = RING_GET_REQUEST(&vif->rx, vif->rx.req_cons++);
783: 8b 55 50 mov 0x50(%rbp),%edx
786: 8b 45 4c mov 0x4c(%rbp),%eax
789: 48 8b 4d 58 mov 0x58(%rbp),%rcx
78d: ff ca dec %edx
78f: 21 c2 and %eax,%edx
791: ff c0 inc %eax
793: 89 45 4c mov %eax,0x4c(%rbp)
drivers/net/xen-netback/netback.c:603
meta = npo->meta + npo->meta_prod++;
796: 8b 74 24 48 mov 0x48(%rsp),%esi
79a: 89 f0 mov %esi,%eax
79c: ff c6 inc %esi
79e: 48 6b c0 0c imul $0xc,%rax,%rax
7a2: 89 74 24 48 mov %esi,0x48(%rsp)
7a6: 48 03 44 24 58 add 0x58(%rsp),%rax
drivers/net/xen-netback/netback.c:605
if (!vif->gso_prefix)
7ab: f6 45 60 04 testb $0x4,0x60(%rbp)
7af: 75 17 jne 7c8 <xen_netbk_rx_action+0x177>
drivers/net/xen-netback/netback.c:606
meta->gso_size = skb_shinfo(skb)->gso_size;
7b1: 8b b3 d0 00 00 00 mov 0xd0(%rbx),%esi
7b7: 48 8b bb d8 00 00 00 mov 0xd8(%rbx),%rdi
7be: 0f b7 74 37 02 movzwl 0x2(%rdi,%rsi,1),%esi
7c3: 89 70 08 mov %esi,0x8(%rax)
7c6: eb 07 jmp 7cf <xen_netbk_rx_action+0x17e>
drivers/net/xen-netback/netback.c:608
else
meta->gso_size = 0;
7c8: c7 40 08 00 00 00 00 movl $0x0,0x8(%rax)
drivers/net/xen-netback/netback.c:611
meta->size = 0;
meta->id = req->id;
7cf: 89 d2 mov %edx,%edx
drivers/net/xen-netback/netback.c:610
if (!vif->gso_prefix)
meta->gso_size = skb_shinfo(skb)->gso_size;
else
meta->gso_size = 0;
meta->size = 0;
7d1: c7 40 04 00 00 00 00 movl $0x0,0x4(%rax)
drivers/net/xen-netback/netback.c:611
meta->id = req->id;
7d8: 48 83 c2 08 add $0x8,%rdx
7dc: 0f b7 34 d1 movzwl (%rcx,%rdx,8),%esi
7e0: 89 30 mov %esi,(%rax)
[-- Attachment #3: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: RFH: Kernel OOPS in xen_netbk_rx_action / xenvif_gop_skb
2014-06-18 16:48 ` Philipp Hahn
@ 2014-06-19 14:12 ` Wei Liu
2014-06-19 14:35 ` David Vrabel
2014-06-23 14:56 ` Philipp Hahn
0 siblings, 2 replies; 18+ messages in thread
From: Wei Liu @ 2014-06-19 14:12 UTC (permalink / raw)
To: Philipp Hahn; +Cc: xen-devel, Erik Damrose, Wei Liu, Ian Campbell, Zoltan Kiss
On Wed, Jun 18, 2014 at 06:48:31PM +0200, Philipp Hahn wrote:
[...]
>
> (gdb) list *(xen_netbk_rx_action+0x18b)
> 0xffffffffa04287dc is in xen_netbk_rx_action
> (/var/build/temp/tmp.hW3dNilayw/pbuilder/linux-3.10.11/drivers/net/xen-netback/netback
> .c:611).
> 606 meta->gso_size = skb_shinfo(skb)->gso_size;
> 607 else
> 608 meta->gso_size = 0;
> 609
> 610 meta->size = 0;
> 611 meta->id = req->id;
> 612 npo->copy_off = 0;
> 613 npo->copy_gref = req->gref;
> 614
> 615 data = skb->data;
>
>
> After more debugging today I think something like this happens:
>
> 1. The VM is receiving packets through bonding + bridge + netback +
> netfront.
>
> 2. For some unknown reason at least one packet remains in the rx queue
> and is not delivered to the domU immediately by netback.
>
> 3. The VM finishes shutting down.
>
> 4. The shared ring between dom0 and domU is freed.
>
> 5. then xen-netback continues processing the pending requests and tries
> to put the packet into the now already released shared ring.
>
>
> >From reading the attached disassembly I guess, that
> AX = &meta
> CX = &rx->string
> DX =~ rx.req_cons
> CR2 = &req->id
> where
> CX + DX * sizeof(union struct xen_netif_rx_{request,response})=8 = CR2
>
>
> Any additional ideas or insight is appreciated.
>
I think your analysis makes sense. Netback does have it's internal queue
and kthread can certainly be scheduled away. There doesn't seem to be a
synchronisation point between a vif getting disconnet and internal queue
gets processed. I attach a quick hack. If it does work to a degree then
we can try to work out a proper fix.
> FYI: The host has only a single CPU and is running >=2 VMs so far.
>
> >> There's one more patch that you can pick up from 3.10.y tree. I doubt it
> >> will make much difference though.
>
> Which patch are you referring to?
>
You can have a look at 3.10.y tree for all the patches between your
current version and the latest stable version.
Wei.
---8<---
>From d2f428a93e6e296bc5f55e16f44ac1ad63a951a8 Mon Sep 17 00:00:00 2001
From: Wei Liu <wei.liu2@citrix.com>
Date: Thu, 19 Jun 2014 15:07:47 +0100
Subject: [PATCH] quick hack
---
drivers/net/xen-netback/common.h | 1 +
drivers/net/xen-netback/interface.c | 1 +
drivers/net/xen-netback/netback.c | 8 ++++++++
3 files changed, 10 insertions(+)
diff --git a/drivers/net/xen-netback/common.h b/drivers/net/xen-netback/common.h
index f2faa77..9239824 100644
--- a/drivers/net/xen-netback/common.h
+++ b/drivers/net/xen-netback/common.h
@@ -66,6 +66,7 @@ struct xenvif {
/* The shared rings and indexes. */
struct xen_netif_tx_back_ring tx;
struct xen_netif_rx_back_ring rx;
+ bool mapped;
/* Frontend feature information. */
u8 can_sg:1;
diff --git a/drivers/net/xen-netback/interface.c b/drivers/net/xen-netback/interface.c
index 540a796..5f11763 100644
--- a/drivers/net/xen-netback/interface.c
+++ b/drivers/net/xen-netback/interface.c
@@ -271,6 +271,7 @@ struct xenvif *xenvif_alloc(struct device *parent, domid_t domid,
vif->dev = dev;
INIT_LIST_HEAD(&vif->schedule_list);
INIT_LIST_HEAD(&vif->notify_list);
+ vif->mapped = false;
vif->credit_bytes = vif->remaining_credit = ~0UL;
vif->credit_usec = 0UL;
diff --git a/drivers/net/xen-netback/netback.c b/drivers/net/xen-netback/netback.c
index 36efb41..f4f3693 100644
--- a/drivers/net/xen-netback/netback.c
+++ b/drivers/net/xen-netback/netback.c
@@ -720,6 +720,11 @@ static void xen_netbk_rx_action(struct xen_netbk *netbk)
vif = netdev_priv(skb->dev);
nr_frags = skb_shinfo(skb)->nr_frags;
+ if (!vif->mapped) {
+ dev_kfree_skb(skb);
+ continue;
+ }
+
sco = (struct skb_cb_overlay *)skb->cb;
sco->meta_slots_used = netbk_gop_skb(skb, &npo);
@@ -1864,6 +1869,8 @@ static int xen_netbk_kthread(void *data)
void xen_netbk_unmap_frontend_rings(struct xenvif *vif)
{
+ vif->mapped = false;
+
if (vif->tx.sring)
xenbus_unmap_ring_vfree(xenvif_to_xenbus_device(vif),
vif->tx.sring);
@@ -1899,6 +1906,7 @@ int xen_netbk_map_frontend_rings(struct xenvif *vif,
BACK_RING_INIT(&vif->rx, rxs, PAGE_SIZE);
vif->rx_req_cons_peek = 0;
+ vif->mapped = true;
return 0;
--
1.7.10.4
^ permalink raw reply related [flat|nested] 18+ messages in thread
* Re: RFH: Kernel OOPS in xen_netbk_rx_action / xenvif_gop_skb
2014-06-19 14:12 ` Wei Liu
@ 2014-06-19 14:35 ` David Vrabel
2014-06-19 14:41 ` Wei Liu
2014-06-23 14:56 ` Philipp Hahn
1 sibling, 1 reply; 18+ messages in thread
From: David Vrabel @ 2014-06-19 14:35 UTC (permalink / raw)
To: Wei Liu, Philipp Hahn; +Cc: xen-devel, Erik Damrose, Ian Campbell, Zoltan Kiss
On 19/06/14 15:12, Wei Liu wrote:
> On Wed, Jun 18, 2014 at 06:48:31PM +0200, Philipp Hahn wrote:
> [...]
>>
>> (gdb) list *(xen_netbk_rx_action+0x18b)
>> 0xffffffffa04287dc is in xen_netbk_rx_action
>> (/var/build/temp/tmp.hW3dNilayw/pbuilder/linux-3.10.11/drivers/net/xen-netback/netback
>> .c:611).
>> 606 meta->gso_size = skb_shinfo(skb)->gso_size;
>> 607 else
>> 608 meta->gso_size = 0;
>> 609
>> 610 meta->size = 0;
>> 611 meta->id = req->id;
>> 612 npo->copy_off = 0;
>> 613 npo->copy_gref = req->gref;
>> 614
>> 615 data = skb->data;
>>
>>
>> After more debugging today I think something like this happens:
>>
>> 1. The VM is receiving packets through bonding + bridge + netback +
>> netfront.
>>
>> 2. For some unknown reason at least one packet remains in the rx queue
>> and is not delivered to the domU immediately by netback.
>>
>> 3. The VM finishes shutting down.
>>
>> 4. The shared ring between dom0 and domU is freed.
>>
>> 5. then xen-netback continues processing the pending requests and tries
>> to put the packet into the now already released shared ring.
>>
>>
>> >From reading the attached disassembly I guess, that
>> AX = &meta
>> CX = &rx->string
>> DX =~ rx.req_cons
>> CR2 = &req->id
>> where
>> CX + DX * sizeof(union struct xen_netif_rx_{request,response})=8 = CR2
>>
>>
>> Any additional ideas or insight is appreciated.
>>
>
> I think your analysis makes sense. Netback does have it's internal queue
> and kthread can certainly be scheduled away. There doesn't seem to be a
> synchronisation point between a vif getting disconnet and internal queue
> gets processed. I attach a quick hack. If it does work to a degree then
> we can try to work out a proper fix.
The kthread_stop() in xenvif_disconnect() waits for the kthread to exit
so I don't see how Philipp's analysis can be right.
David
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: RFH: Kernel OOPS in xen_netbk_rx_action / xenvif_gop_skb
2014-06-19 14:35 ` David Vrabel
@ 2014-06-19 14:41 ` Wei Liu
0 siblings, 0 replies; 18+ messages in thread
From: Wei Liu @ 2014-06-19 14:41 UTC (permalink / raw)
To: David Vrabel
Cc: Wei Liu, Ian Campbell, Philipp Hahn, Zoltan Kiss, xen-devel,
Erik Damrose
On Thu, Jun 19, 2014 at 03:35:11PM +0100, David Vrabel wrote:
> On 19/06/14 15:12, Wei Liu wrote:
> > On Wed, Jun 18, 2014 at 06:48:31PM +0200, Philipp Hahn wrote:
> > [...]
> >>
> >> (gdb) list *(xen_netbk_rx_action+0x18b)
> >> 0xffffffffa04287dc is in xen_netbk_rx_action
> >> (/var/build/temp/tmp.hW3dNilayw/pbuilder/linux-3.10.11/drivers/net/xen-netback/netback
> >> .c:611).
> >> 606 meta->gso_size = skb_shinfo(skb)->gso_size;
> >> 607 else
> >> 608 meta->gso_size = 0;
> >> 609
> >> 610 meta->size = 0;
> >> 611 meta->id = req->id;
> >> 612 npo->copy_off = 0;
> >> 613 npo->copy_gref = req->gref;
> >> 614
> >> 615 data = skb->data;
> >>
> >>
> >> After more debugging today I think something like this happens:
> >>
> >> 1. The VM is receiving packets through bonding + bridge + netback +
> >> netfront.
> >>
> >> 2. For some unknown reason at least one packet remains in the rx queue
> >> and is not delivered to the domU immediately by netback.
> >>
> >> 3. The VM finishes shutting down.
> >>
> >> 4. The shared ring between dom0 and domU is freed.
> >>
> >> 5. then xen-netback continues processing the pending requests and tries
> >> to put the packet into the now already released shared ring.
> >>
> >>
> >> >From reading the attached disassembly I guess, that
> >> AX = &meta
> >> CX = &rx->string
> >> DX =~ rx.req_cons
> >> CR2 = &req->id
> >> where
> >> CX + DX * sizeof(union struct xen_netif_rx_{request,response})=8 = CR2
> >>
> >>
> >> Any additional ideas or insight is appreciated.
> >>
> >
> > I think your analysis makes sense. Netback does have it's internal queue
> > and kthread can certainly be scheduled away. There doesn't seem to be a
> > synchronisation point between a vif getting disconnet and internal queue
> > gets processed. I attach a quick hack. If it does work to a degree then
> > we can try to work out a proper fix.
>
> The kthread_stop() in xenvif_disconnect() waits for the kthread to exit
> so I don't see how Philipp's analysis can be right.
>
He's using 3.10 kernel. One kthread serves many vifs. The kthread won't
stop.
Wei.
> David
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: RFH: Kernel OOPS in xen_netbk_rx_action / xenvif_gop_skb
2014-06-19 14:12 ` Wei Liu
2014-06-19 14:35 ` David Vrabel
@ 2014-06-23 14:56 ` Philipp Hahn
2014-06-27 8:42 ` Philipp Hahn
1 sibling, 1 reply; 18+ messages in thread
From: Philipp Hahn @ 2014-06-23 14:56 UTC (permalink / raw)
To: Wei Liu; +Cc: xen-devel, Erik Damrose, Ian Campbell, Zoltan Kiss
Hello Wei Liu,
On 19.06.2014 16:12, Wei Liu wrote:
> On Wed, Jun 18, 2014 at 06:48:31PM +0200, Philipp Hahn wrote:
...
>> 5. then xen-netback continues processing the pending requests and tries
...
> I think your analysis makes sense. Netback does have it's internal queue
> and kthread can certainly be scheduled away. There doesn't seem to be a
> synchronisation point between a vif getting disconnet and internal queue
> gets processed. I attach a quick hack. If it does work to a degree then
> we can try to work out a proper fix.
Your quick hack seems to have solved the problem: The network survived
the week-end, but we had to change the VMs as one of them was required
last weekend. We're currently re-checking that the bug still occurs with
the old kernel but the new set of VMs.
>>>> There's one more patch that you can pick up from 3.10.y tree. I doubt it
>>>> will make much difference though.
>>
>> Which patch are you referring to?
>
> You can have a look at 3.10.y tree for all the patches between your
> current version and the latest stable version.
Thanks, I'll have a look.
> ---8<---
> From d2f428a93e6e296bc5f55e16f44ac1ad63a951a8 Mon Sep 17 00:00:00 2001
> From: Wei Liu <wei.liu2@citrix.com>
> Date: Thu, 19 Jun 2014 15:07:47 +0100
> Subject: [PATCH] quick hack
>
> ---
> drivers/net/xen-netback/common.h | 1 +
> drivers/net/xen-netback/interface.c | 1 +
> drivers/net/xen-netback/netback.c | 8 ++++++++
> 3 files changed, 10 insertions(+)
>
> diff --git a/drivers/net/xen-netback/common.h b/drivers/net/xen-netback/common.h
> index f2faa77..9239824 100644
> --- a/drivers/net/xen-netback/common.h
> +++ b/drivers/net/xen-netback/common.h
> @@ -66,6 +66,7 @@ struct xenvif {
> /* The shared rings and indexes. */
> struct xen_netif_tx_back_ring tx;
> struct xen_netif_rx_back_ring rx;
> + bool mapped;
>
> /* Frontend feature information. */
> u8 can_sg:1;
> diff --git a/drivers/net/xen-netback/interface.c b/drivers/net/xen-netback/interface.c
> index 540a796..5f11763 100644
> --- a/drivers/net/xen-netback/interface.c
> +++ b/drivers/net/xen-netback/interface.c
> @@ -271,6 +271,7 @@ struct xenvif *xenvif_alloc(struct device *parent, domid_t domid,
> vif->dev = dev;
> INIT_LIST_HEAD(&vif->schedule_list);
> INIT_LIST_HEAD(&vif->notify_list);
> + vif->mapped = false;
>
> vif->credit_bytes = vif->remaining_credit = ~0UL;
> vif->credit_usec = 0UL;
> diff --git a/drivers/net/xen-netback/netback.c b/drivers/net/xen-netback/netback.c
> index 36efb41..f4f3693 100644
> --- a/drivers/net/xen-netback/netback.c
> +++ b/drivers/net/xen-netback/netback.c
> @@ -720,6 +720,11 @@ static void xen_netbk_rx_action(struct xen_netbk *netbk)
> vif = netdev_priv(skb->dev);
> nr_frags = skb_shinfo(skb)->nr_frags;
>
> + if (!vif->mapped) {
> + dev_kfree_skb(skb);
> + continue;
> + }
> +
> sco = (struct skb_cb_overlay *)skb->cb;
> sco->meta_slots_used = netbk_gop_skb(skb, &npo);
>
> @@ -1864,6 +1869,8 @@ static int xen_netbk_kthread(void *data)
>
> void xen_netbk_unmap_frontend_rings(struct xenvif *vif)
> {
> + vif->mapped = false;
> +
> if (vif->tx.sring)
> xenbus_unmap_ring_vfree(xenvif_to_xenbus_device(vif),
> vif->tx.sring);
> @@ -1899,6 +1906,7 @@ int xen_netbk_map_frontend_rings(struct xenvif *vif,
> BACK_RING_INIT(&vif->rx, rxs, PAGE_SIZE);
>
> vif->rx_req_cons_peek = 0;
> + vif->mapped = true;
>
> return 0;
>
>
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: RFH: Kernel OOPS in xen_netbk_rx_action / xenvif_gop_skb
2014-06-23 14:56 ` Philipp Hahn
@ 2014-06-27 8:42 ` Philipp Hahn
2014-06-27 17:48 ` Philipp Hahn
0 siblings, 1 reply; 18+ messages in thread
From: Philipp Hahn @ 2014-06-27 8:42 UTC (permalink / raw)
To: Wei Liu; +Cc: xen-devel, Erik Damrose, Ian Campbell, Zoltan Kiss
Hello Wei Liu,
On 23.06.2014 16:56, Philipp Hahn wrote:
> On 19.06.2014 16:12, Wei Liu wrote:
>> On Wed, Jun 18, 2014 at 06:48:31PM +0200, Philipp Hahn wrote:
> ...
>>> 5. then xen-netback continues processing the pending requests and tries
> ...
>> I think your analysis makes sense. Netback does have it's internal queue
>> and kthread can certainly be scheduled away. There doesn't seem to be a
>> synchronisation point between a vif getting disconnet and internal queue
>> gets processed. I attach a quick hack. If it does work to a degree then
>> we can try to work out a proper fix.
>
> Your quick hack seems to have solved the problem: The network survived
> the week-end, but we had to change the VMs as one of them was required
> last weekend. We're currently re-checking that the bug still occurs with
> the old kernel but the ne
We added some debug output (UniDEBUG) as we observed another OOPS in one
test run, but I think that was a mis-compiled kernel as the size of
function was the same as previous, but it should be 0x712 according to
"objdump -S":
> [ 6196.712232] BUG: unable to handle kernel paging request at ffffc90010d94678
> [ 6196.712322] IP: [<ffffffffa04147dc>] xen_netbk_rx_action+0x18b/0x6f0 [xen_netback]
> [ 6196.712410] PGD 95822067 PUD 95823067 PMD 94721067 PTE 0
> [ 6196.712473] Oops: 0000 [#1] SMP
...
> [ 6196.713434] CPU: 0 PID: 11743 Comm: netback/0 Not tainted 3.10.0-ucs58-amd64 #1
> Univention 3.10.11-1.58.201405060908a~xenXXX
...
> [ 6196.713618] task: ffff8800917f7840 ti: ffff880004bde000 task.ti: ffff880004bde000
> [ 6196.713701] RIP: e030:[<ffffffffa04147dc>] [<ffffffffa04147dc>]
> xen_netbk_rx_action+0x18b/0x6f0 [xen_netback]
With the modified patch we now get the following hang:
> [ 84.833333] device eth2 entered promiscuous mode
> [ 248.191165] UniDEBUG vif->mapped is set to false (xenvif_alloc)
> [ 248.442727] device vif1.0 entered promiscuous mode
> [ 250.721054] UniDEBUG vif->mapped is true (xen_netbk_map_frontend_rings)
> [ 250.721099] XXXlan0: port 2(vif1.0) entered forwarding state
> [ 250.721103] XXXlan0: port 2(vif1.0) entered forwarding state
> [ 253.473859] UniDEBUG vif->mapped is set to false (xenvif_alloc)
> [ 253.737812] device vif2.0 entered promiscuous mode
> [ 255.639021] UniDEBUG vif->mapped is true (xen_netbk_map_frontend_rings)
> [ 255.639067] XXXlan0: port 3(vif2.0) entered forwarding state
> [ 255.639072] XXXlan0: port 3(vif2.0) entered forwarding state
> [ 592.867375] UniDEBUG vif->mapped is set to false(xen_netbk_unmap_frontend_rings)
> [ 592.868147] XXXlan0: port 3(vif2.0) entered disabled state
> [ 593.499258] XXXlan0: port 3(vif2.0) entered disabled state
> [ 593.499293] device vif2.0 left promiscuous mode
> [ 593.499295] XXXlan0: port 3(vif2.0) entered disabled state
> [ 595.386548] UniDEBUG vif->mapped is set to false (xenvif_alloc)
> [ 595.633665] device vif3.0 entered promiscuous mode
> [ 597.390410] UniDEBUG vif->mapped is true (xen_netbk_map_frontend_rings)
> [ 597.390458] XXXlan0: port 3(vif3.0) entered forwarding state
> [ 597.390462] XXXlan0: port 3(vif3.0) entered forwarding state
> [ 936.549840] UniDEBUG vif->mapped is set to false(xen_netbk_unmap_frontend_rings)
> [ 936.549869] XXXlan0: port 3(vif3.0) entered disabled state
> [ 936.553024] UniDEBUG vif->mapped is false
here it would oops previously.
> [ 937.459565] device vif3.0 left promiscuous mode
> [ 937.459570] XXXlan0: port 3(vif3.0) entered disabled state
> [ 1115.250900] INFO: task xenwatch:14 blocked for more than 120 seconds.
> [ 1115.250902] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 1115.250904] xenwatch D ffff8800952d9080 0 14 2 0x00000000
> [ 1115.250907] ffff8800952d9080 0000000000000246 ffff880094510880 0000000000013ec0
> [ 1115.250909] ffff88009530ffd8 0000000000013ec0 ffff88009530ffd8 0000000000013ec0
> [ 1115.250911] ffff8800952d9080 0000000000013ec0 0000000000013ec0 ffff88009530e010
> [ 1115.250913] Call Trace:
> [ 1115.250921] [<ffffffff813ca32d>] ? _raw_spin_lock_irqsave+0x11/0x2f
> [ 1115.250925] [<ffffffffa040a396>] ? xenvif_free+0x7a/0xb6 [xen_netback]
> [ 1115.250930] [<ffffffff8105d373>] ? wake_up_bit+0x20/0x20
> [ 1115.250934] [<ffffffff812697c2>] ? xenbus_rm+0x44/0x4f
> [ 1115.250937] [<ffffffffa0409ead>] ? netback_remove+0x5d/0x7e [xen_netback]
> [ 1115.250940] [<ffffffff8126a72f>] ? xenbus_dev_remove+0x29/0x4e
> [ 1115.250943] [<ffffffff812a4ff0>] ? __device_release_driver+0x7f/0xd5
> [ 1115.250946] [<ffffffff812a50fc>] ? device_release_driver+0x1d/0x29
> [ 1115.250948] [<ffffffff812a4519>] ? bus_remove_device+0xee/0x103
> [ 1115.250950] [<ffffffff812a2b49>] ? device_del+0x112/0x182
> [ 1115.250952] [<ffffffff812a2bc2>] ? device_unregister+0x9/0x12
> [ 1115.250955] [<ffffffff81268e20>] ? xenwatch_thread+0x122/0x15f
> [ 1115.250957] [<ffffffff8105d373>] ? wake_up_bit+0x20/0x20
> [ 1115.250959] [<ffffffff81268cfe>] ? xs_watch+0x57/0x57
> [ 1115.250962] [<ffffffff8105cd73>] ? kthread_freezable_should_stop+0x56/0x56
> [ 1115.250964] [<ffffffff81268cfe>] ? xs_watch+0x57/0x57
> [ 1115.250966] [<ffffffff8105ce1e>] ? kthread+0xab/0xb3
> [ 1115.250969] [<ffffffff81003638>] ? xen_end_context_switch+0xe/0x1c
> [ 1115.250972] [<ffffffff8105cd73>] ? kthread_freezable_should_stop+0x56/0x56
> [ 1115.250975] [<ffffffff813cfbfc>] ? ret_from_fork+0x7c/0xb0
> [ 1115.250977] [<ffffffff8105cd73>] ? kthread_freezable_should_stop+0x56/0x56
Any idea?
Sincerely
Philipp
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: RFH: Kernel OOPS in xen_netbk_rx_action / xenvif_gop_skb
2014-06-27 8:42 ` Philipp Hahn
@ 2014-06-27 17:48 ` Philipp Hahn
2014-06-27 18:24 ` Philipp Hahn
0 siblings, 1 reply; 18+ messages in thread
From: Philipp Hahn @ 2014-06-27 17:48 UTC (permalink / raw)
To: Wei Liu; +Cc: xen-devel, Erik Damrose, Ian Campbell, Zoltan Kiss
Hello Wei Liu,
On 27.06.2014 10:42, Philipp Hahn wrote:
> With the modified patch we now get the following hang:
...
>> [ 1115.250900] INFO: task xenwatch:14 blocked for more than 120 seconds.
>> [ 1115.250902] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> [ 1115.250904] xenwatch D ffff8800952d9080 0 14 2 0x00000000
>> [ 1115.250907] ffff8800952d9080 0000000000000246 ffff880094510880 0000000000013ec0
>> [ 1115.250909] ffff88009530ffd8 0000000000013ec0 ffff88009530ffd8 0000000000013ec0
>> [ 1115.250911] ffff8800952d9080 0000000000013ec0 0000000000013ec0 ffff88009530e010
>> [ 1115.250913] Call Trace:
>> [ 1115.250921] [<ffffffff813ca32d>] ? _raw_spin_lock_irqsave+0x11/0x2f
>> [ 1115.250925] [<ffffffffa040a396>] ? xenvif_free+0x7a/0xb6 [xen_netback]
...
> Any idea?
I guess we found the problem ourselves: For thus removed skb's the
reference counter on the associated vif was not decremented, as it is
normally done in two locations at the end of the function
xen_netbk_rx_action():
> Subject: [PATCH] quick hack
...
> diff --git a/drivers/net/xen-netback/netback.c b/drivers/net/xen-netback/netback.c
> index 36efb41..f4f3693 100644
> --- a/drivers/net/xen-netback/netback.c
> +++ b/drivers/net/xen-netback/netback.c
> @@ -720,6 +720,11 @@ static void xen_netbk_rx_action(struct xen_netbk *netbk)
> vif = netdev_priv(skb->dev);
> nr_frags = skb_shinfo(skb)->nr_frags;
>
> + if (!vif->mapped) {
> + dev_kfree_skb(skb);
xenvif_put(vif);
> + continue;
> + }
> +
> sco = (struct skb_cb_overlay *)skb->cb;
> sco->meta_slots_used = netbk_gop_skb(skb, &npo);
>
The test is currently running again for the weekend and on Monday we
will hopefully know more.
Thanks again.
Philipp
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: RFH: Kernel OOPS in xen_netbk_rx_action / xenvif_gop_skb
2014-06-27 17:48 ` Philipp Hahn
@ 2014-06-27 18:24 ` Philipp Hahn
2014-07-02 7:45 ` [PATCH] " Philipp Hahn
0 siblings, 1 reply; 18+ messages in thread
From: Philipp Hahn @ 2014-06-27 18:24 UTC (permalink / raw)
To: Wei Liu; +Cc: xen-devel, Erik Damrose, Ian Campbell, Zoltan Kiss
Hello Wei Liu,
On 27.06.2014 19:48, Philipp Hahn wrote:> I guess we found the problem
ourselves: For thus removed skb's the
> reference counter on the associated vif was not decremented, as it is
> normally done in two locations at the end of the function
> xen_netbk_rx_action():
...
> The test is currently running again for the weekend and on Monday we
> will hopefully know more.
FYI: The test VM survived the first reboot without locking up:
Jun 27 19:43:43 xenmbint05b01 kernel: [ 1716.359537] UniDEBUG
vif->mapped is set to false (xenvif_alloc)
Jun 27 19:43:45 xenmbint05b01 kernel: [ 1718.558540] UniDEBUG
vif->mapped is true (xen_netbk_map_frontend_rings)
Jun 27 19:49:23 xenmbint05b01 kernel: [ 2055.895207] UniDEBUG
vif->mapped is set to false(xen_netbk_unmap_frontend_rings)
Jun 27 19:49:23 xenmbint05b01 kernel: [ 2055.898349] UniDEBUG
vif->mapped is false
Jun 27 19:49:23 xenmbint05b01 kernel: [ 2055.898352] UniDEBUG
vif->mapped is false
Jun 27 19:49:25 xenmbint05b01 kernel: [ 2058.529454] UniDEBUG
vif->mapped is set to false (xenvif_alloc)
BYtE
Philipp
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH] Kernel OOPS in xen_netbk_rx_action / xenvif_gop_skb
2014-06-27 18:24 ` Philipp Hahn
@ 2014-07-02 7:45 ` Philipp Hahn
2014-07-10 12:41 ` Wei Liu
[not found] ` <20140710124122.GA2381@zion.uk.xensource.com>
0 siblings, 2 replies; 18+ messages in thread
From: Philipp Hahn @ 2014-07-02 7:45 UTC (permalink / raw)
To: Wei Liu; +Cc: xen-devel, stable, Erik Damrose, Ian Campbell, Zoltan Kiss
[-- Attachment #1: Type: text/plain, Size: 1380 bytes --]
Hello Wei Liu,
On 27.06.2014 20:24, Philipp Hahn wrote:
> On 27.06.2014 19:48, Philipp Hahn wrote:> I guess we found the problem
> ourselves: For thus removed skb's the
>> reference counter on the associated vif was not decremented, as it is
>> normally done in two locations at the end of the function
>> xen_netbk_rx_action():
> ...
>> The test is currently running again for the weekend and on Monday we
>> will hopefully know more.
>
> FYI: The test VM survived the first reboot without locking up:
...
> Jun 27 19:49:23 xenmbint05b01 kernel: [ 2055.898349] UniDEBUG
> vif->mapped is false
The host survived the weekend with the problematic VM rebooting every 5
minutes; the log shows the shared ring being accessed unmapped, where
the kernel crashed previously.
So the attached patch fixes the bug (or at least prevents the OOPS).
@Wei Liu: You said that the patch is only a quick hack to detect, if my
analysis is correct and a proper fix would be needed. For us the
attached patch works, as the problem does not happen that often and is
hard to reproduce anyway, so spending more time on that issue is
probably not worth it. And that flag doesn't look that ugly.
@stable: at least 3.10 has the bug, but other long-term-stable kernels
have it too. The code in current is different as multi-queue was added,
so the patch wouldn't be in current.
Sincerely
Philipp Hahn
[-- Attachment #2: 0001-xen-netback-skip-pending-packets-in-unmapped-ring.patch --]
[-- Type: text/x-patch, Size: 7249 bytes --]
>From 45fd183c8faa5b7213aa4663cf224b414aacfc4b Mon Sep 17 00:00:00 2001
Message-Id: <45fd183c8faa5b7213aa4663cf224b414aacfc4b.1404286695.git.hahn@univention.de>
From: Wei Liu <wei.liu2@citrix.com>
Date: Wed, 2 Jul 2014 09:14:22 +0200
Subject: [PATCH] xen-netback: skip pending packets in unmapped ring
Organization: Univention GmbH, Bremen, Germany
1. The VM is receiving packets through bonding + bridge + netback +
netfront.
2. For some unknown reason at least one packet remains in the rx queue
and is not delivered to the domU immediately by netback.
3. The VM finishes shutting down.
4. The shared ring between dom0 and domU is freed.
5. then xen-netback continues processing the pending requests and tries
to put the packet into the now already released shared ring.
> [38551.547728] XXXlan0: port 9(vif26.0) entered disabled state
> [38551.549365] BUG: unable to handle kernel paging request at ffffc900108641d8
> [38551.549461] IP: [<ffffffffa04147dc>] xen_netbk_rx_action+0x18b/0x6f0
> [xen_netback]
> [38551.549551] PGD 57e20067 PUD 57e21067 PMD 571a7067 PTE 0
> [38551.549615] Oops: 0000 [#1] SMP
> [38551.549665] Modules linked in: tun xt_physdev xen_blkback xen_netback ip6_tables
> iptable_filter ip_tables ebtable_nat ebtables x_tables xen_gntdev nfsv3 nfsv4
> rpcsec_gss_krb5 nfsd nfs_acl auth_rpcgss oid_registry nfs fscache dns_resolver lockd
> sunrpc fuse loop xen_blkfront xen_evtchn blktap quota_v2 quota_tree xenfs xen_privcmd
> coretemp crc32c_intel ghash_clmulni_intel aesni_intel ablk_helper cryptd lrw gf128mul
> glue_helper aes_x86_64 snd_pcm snd_timer snd soundcore snd_page_alloc tpm_tis tpm lpc_ich
> tpm_bios i7core_edac i2c_i801 psmouse microcode edac_core serio_raw pcspkr mperf ioatdma
> mfd_core processor evdev thermal_sys ext4 jbd2 crc16 bonding bridge stp llc dm_snapshot
> dm_mirror dm_region_hash dm_log dm_mod sd_mod crc_t10dif ehci_pci uhci_hcd ehci_hcd mptsas
> mptscsih mptbase scsi_transport_sas usbcore usb_common igb dca i2c_algo_bit i2c_core ptp
> pps_core button
> [38551.550601] CPU: 0 PID: 12587 Comm: netback/0 Not tainted 3.10.0-ucs58-amd64 #1 Debian
> 3.10.11-1.58.201405060908
> [38551.550693] Hardware name: FUJITSU PRIMERGY BX620 S6/D3051, BIOS 080015 Rev.3C78.3051
> 07/22/2011
> [38551.550781] task: ffff880004b067c0 ti: ffff8800561ec000 task.ti: ffff8800561ec000
> [38551.550865] RIP: e030:[<ffffffffa04147dc>] [<ffffffffa04147dc>]
> xen_netbk_rx_action+0x18b/0x6f0 [xen_netback]
> [38551.550959] RSP: e02b:ffff8800561edce8 EFLAGS: 00010202
> [38551.551009] RAX: ffffc900104adac0 RBX: ffff8800541e95c0 RCX: ffffc90010864000
> [38551.551064] RDX: 000000000000003b RSI: 0000000000000000 RDI: ffff880040014380
> [38551.551120] RBP: ffff8800570e6800 R08: 0000000000000000 R09: ffff880004799800
> [38551.551175] R10: ffffffff813ca115 R11: ffff88005e4fdb08 R12: ffff880054e6f800
> [38551.551231] R13: ffff8800561edd58 R14: ffffc900104a1000 R15: 0000000000000000
> [38551.551289] FS: 00007f19a54a8700(0000) GS:ffff88005da00000(0000)
> knlGS:0000000000000000
> [38551.551374] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> [38551.551425] CR2: ffffc900108641d8 CR3: 0000000054cb3000 CR4: 0000000000002660
> [38551.551481] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [38551.551537] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [38551.551592] Stack:
> [38551.551630] ffff880004b06ba0 0000000000000000 ffff88005da13ec0 ffff88005da13ec0
> [38551.551726] 0000000004b067c0 ffffc900104a8ac0 ffffc900104a1020 000000005da13ec0
> [38551.551823] 0000000000000000 0000000000000001 ffffc900104a8ac0 ffffc900104adac0
> [38551.551920] Call Trace:
> [38551.551966] [<ffffffff813ca32d>] ? _raw_spin_lock_irqsave+0x11/0x2f
> [38551.552021] [<ffffffffa0416033>] ? xen_netbk_kthread+0x174/0x841 [xen_netback]
> [38551.552106] [<ffffffff8105d373>] ? wake_up_bit+0x20/0x20
> [38551.560239] [<ffffffffa0415ebf>] ? xen_netbk_tx_build_gops+0xce8/0xce8 [xen_netback]
> [38551.560325] [<ffffffff8105cd73>] ? kthread_freezable_should_stop+0x56/0x56
> [38551.560381] [<ffffffffa0415ebf>] ? xen_netbk_tx_build_gops+0xce8/0xce8 [xen_netback]
> [38551.560466] [<ffffffff8105ce1e>] ? kthread+0xab/0xb3
> [38551.560518] [<ffffffff81003638>] ? xen_end_context_switch+0xe/0x1c
> [38551.560572] [<ffffffff8105cd73>] ? kthread_freezable_should_stop+0x56/0x56
> [38551.560628] [<ffffffff813cfbfc>] ? ret_from_fork+0x7c/0xb0
> [38551.560680] [<ffffffff8105cd73>] ? kthread_freezable_should_stop+0x56/0x56
> [38551.560734] Code: 8b b3 d0 00 00 00 48 8b bb d8 00 00 00 0f b7 74 37 02 89 70 08 eb 07
> c7 40 08 00 00 00 00 89 d2 c7 40 04 00 00 00 00 48 83 c2 08 <0f> b7 34 d1 89 30 c7 44 24
> 60 00 00 00 00 8b 44 d1 04 89 44 24
> [38551.561151] RIP [<ffffffffa04147dc>] xen_netbk_rx_action+0x18b/0x6f0 [xen_netback]
> [38551.561238] RSP <ffff8800561edce8>
> [38551.561283] CR2: ffffc900108641d8
> [38551.561624] ---[ end trace 8c260c6af259c4aa ]---
Track the shared ring buffer being unmapped and drop those packets.
Signed-off-by: Wei Liu <wei.liu2@citrix.com>
Signed-off-by: Philipp Hahn <hahn@univention.de>
Tested-by: Philipp Hahn <hahn@univention.de>
---
drivers/net/xen-netback/common.h | 1 +
drivers/net/xen-netback/interface.c | 1 +
drivers/net/xen-netback/netback.c | 10 ++++++++++
3 files changed, 12 insertions(+)
diff --git a/drivers/net/xen-netback/common.h b/drivers/net/xen-netback/common.h
index f2faa77..9239824 100644
--- a/drivers/net/xen-netback/common.h
+++ b/drivers/net/xen-netback/common.h
@@ -66,6 +66,7 @@ struct xenvif {
/* The shared rings and indexes. */
struct xen_netif_tx_back_ring tx;
struct xen_netif_rx_back_ring rx;
+ bool mapped;
/* Frontend feature information. */
u8 can_sg:1;
diff --git a/drivers/net/xen-netback/interface.c b/drivers/net/xen-netback/interface.c
index 540a796..5f11763 100644
--- a/drivers/net/xen-netback/interface.c
+++ b/drivers/net/xen-netback/interface.c
@@ -271,6 +271,7 @@ struct xenvif *xenvif_alloc(struct device *parent, domid_t domid,
vif->dev = dev;
INIT_LIST_HEAD(&vif->schedule_list);
INIT_LIST_HEAD(&vif->notify_list);
+ vif->mapped = false;
vif->credit_bytes = vif->remaining_credit = ~0UL;
vif->credit_usec = 0UL;
diff --git a/drivers/net/xen-netback/netback.c b/drivers/net/xen-netback/netback.c
index 70b830f..bab7043 100644
--- a/drivers/net/xen-netback/netback.c
+++ b/drivers/net/xen-netback/netback.c
@@ -720,6 +720,13 @@ static void xen_netbk_rx_action(struct xen_netbk *netbk)
vif = netdev_priv(skb->dev);
nr_frags = skb_shinfo(skb)->nr_frags;
+ if (!vif->mapped) {
+ netdev_err(vif->dev, "Shared ring is unmapped\n");
+ dev_kfree_skb(skb);
+ xenvif_put(vif);
+ continue;
+ }
+
sco = (struct skb_cb_overlay *)skb->cb;
sco->meta_slots_used = netbk_gop_skb(skb, &npo);
@@ -1864,6 +1871,8 @@ static int xen_netbk_kthread(void *data)
void xen_netbk_unmap_frontend_rings(struct xenvif *vif)
{
+ vif->mapped = false;
+
if (vif->tx.sring)
xenbus_unmap_ring_vfree(xenvif_to_xenbus_device(vif),
vif->tx.sring);
@@ -1899,6 +1908,7 @@ int xen_netbk_map_frontend_rings(struct xenvif *vif,
BACK_RING_INIT(&vif->rx, rxs, PAGE_SIZE);
vif->rx_req_cons_peek = 0;
+ vif->mapped = true;
return 0;
--
1.9.1
[-- Attachment #3: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 18+ messages in thread
* Re: [PATCH] Kernel OOPS in xen_netbk_rx_action / xenvif_gop_skb
2014-07-02 7:45 ` [PATCH] " Philipp Hahn
@ 2014-07-10 12:41 ` Wei Liu
[not found] ` <20140710124122.GA2381@zion.uk.xensource.com>
1 sibling, 0 replies; 18+ messages in thread
From: Wei Liu @ 2014-07-10 12:41 UTC (permalink / raw)
To: Philipp Hahn
Cc: Wei Liu, Ian Campbell, stable, Zoltan Kiss, xen-devel,
Erik Damrose
On Wed, Jul 02, 2014 at 09:45:44AM +0200, Philipp Hahn wrote:
> Hello Wei Liu,
>
> On 27.06.2014 20:24, Philipp Hahn wrote:
> > On 27.06.2014 19:48, Philipp Hahn wrote:> I guess we found the problem
> > ourselves: For thus removed skb's the
> >> reference counter on the associated vif was not decremented, as it is
> >> normally done in two locations at the end of the function
> >> xen_netbk_rx_action():
> > ...
> >> The test is currently running again for the weekend and on Monday we
> >> will hopefully know more.
> >
> > FYI: The test VM survived the first reboot without locking up:
> ...
> > Jun 27 19:49:23 xenmbint05b01 kernel: [ 2055.898349] UniDEBUG
> > vif->mapped is false
>
> The host survived the weekend with the problematic VM rebooting every 5
> minutes; the log shows the shared ring being accessed unmapped, where
> the kernel crashed previously.
>
> So the attached patch fixes the bug (or at least prevents the OOPS).
>
> @Wei Liu: You said that the patch is only a quick hack to detect, if my
> analysis is correct and a proper fix would be needed. For us the
> attached patch works, as the problem does not happen that often and is
> hard to reproduce anyway, so spending more time on that issue is
> probably not worth it. And that flag doesn't look that ugly.
>
Sorry for the late reply. I was away for two weeks.
I agree that we would like to avoid spending too much time on this
issue.
Since the problem is confirmed, I think a proper fix will be to
reference count vif and prevent it from unmapping the ring before all
queued SKBs are consumed. But it might require much more work than that
quick hack. Would you up for writing a patch? I won't be able to write
one in the near future. Further more, you're the only party now can
verify a fix.
> @stable: at least 3.10 has the bug, but other long-term-stable kernels
> have it too. The code in current is different as multi-queue was added,
> so the patch wouldn't be in current.
>
FWIW this bug doesn't exist in kernel >=3.12.
Wei.
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH] Kernel OOPS in xen_netbk_rx_action / xenvif_gop_skb
[not found] ` <20140710124122.GA2381@zion.uk.xensource.com>
@ 2014-07-11 9:41 ` Philipp Hahn
[not found] ` <53BFB142.7050201@univention.de>
1 sibling, 0 replies; 18+ messages in thread
From: Philipp Hahn @ 2014-07-11 9:41 UTC (permalink / raw)
To: Wei Liu; +Cc: xen-devel, stable, Erik Damrose, Ian Campbell, Zoltan Kiss
[-- Attachment #1: Type: text/plain, Size: 1880 bytes --]
Hello Wei Liu,
On 10.07.2014 14:41, Wei Liu wrote:
> On Wed, Jul 02, 2014 at 09:45:44AM +0200, Philipp Hahn wrote:
>> @Wei Liu: You said that the patch is only a quick hack to detect, if my
>> analysis is correct and a proper fix would be needed. For us the
>> attached patch works, as the problem does not happen that often and is
>> hard to reproduce anyway, so spending more time on that issue is
>> probably not worth it. And that flag doesn't look that ugly.
...
> I agree that we would like to avoid spending too much time on this
> issue.
This is also what I'm thinking.
> Since the problem is confirmed, I think a proper fix will be to
> reference count vif and prevent it from unmapping the ring before all
> queued SKBs are consumed.
vif is already ref-counted; see attached *untested* patch for a start.
What I don't like is xenbus_unmap_ring_vfree() potentially printing an
error on double-free when called from xen_netbk_unmap_frontend_rings().
> But it might require much more work than that quick hack.
I'm no network driver/Xen expert, but that sounds like more work for no
gain: We would still copy packets for a guest, which is already dead. If
the ring get full, we would probably need to go to sleep and wait for an
answer we will never get anymore.
> FWIW this bug doesn't exist in kernel >=3.12.
That is even one more point for the hack, as there the problem is
properly fixed and the problem is very obscure to trigger.
> Would you up for writing a patch? I won't be able to write
> one in the near future.
> Further more, you're the only party now can verify a fix.
I had a look and created the attached patch, which is untested, as I
currently can't access the faulting system and have been unable to
reproduce it in my development environment.
The quick hack now runs on that system for several weeks now without
problems.
Sincerely
Philipp
[-- Attachment #2: 0001-xen-netback-unmap-only-empty-shared-ring.patch --]
[-- Type: text/x-patch, Size: 6221 bytes --]
>From b2cb74f337e4af1bd1081249db6825ac3208d09f Mon Sep 17 00:00:00 2001
Message-Id: <b2cb74f337e4af1bd1081249db6825ac3208d09f.1405071570.git.hahn@univention.de>
From: Wei Liu <wei.liu2@citrix.com>
Date: Wed, 2 Jul 2014 09:14:22 +0200
Subject: [PATCHv2] xen-netback: unmap only empty shared ring
Organization: Univention GmbH, Bremen, Germany
1. The VM is receiving packets through bonding + bridge + netback +
netfront.
2. For some unknown reason at least one packet remains in the rx queue
and is not delivered to the domU immediately by netback.
3. The VM finishes shutting down.
4. The shared ring between dom0 and domU is freed.
5. then xen-netback continues processing the pending requests and tries
to put the packet into the now already released shared ring.
> [38551.547728] XXXlan0: port 9(vif26.0) entered disabled state
> [38551.549365] BUG: unable to handle kernel paging request at ffffc900108641d8
> [38551.549461] IP: [<ffffffffa04147dc>] xen_netbk_rx_action+0x18b/0x6f0
> [xen_netback]
> [38551.549551] PGD 57e20067 PUD 57e21067 PMD 571a7067 PTE 0
> [38551.549615] Oops: 0000 [#1] SMP
> [38551.549665] Modules linked in: tun xt_physdev xen_blkback xen_netback ip6_tables
> iptable_filter ip_tables ebtable_nat ebtables x_tables xen_gntdev nfsv3 nfsv4
> rpcsec_gss_krb5 nfsd nfs_acl auth_rpcgss oid_registry nfs fscache dns_resolver lockd
> sunrpc fuse loop xen_blkfront xen_evtchn blktap quota_v2 quota_tree xenfs xen_privcmd
> coretemp crc32c_intel ghash_clmulni_intel aesni_intel ablk_helper cryptd lrw gf128mul
> glue_helper aes_x86_64 snd_pcm snd_timer snd soundcore snd_page_alloc tpm_tis tpm lpc_ich
> tpm_bios i7core_edac i2c_i801 psmouse microcode edac_core serio_raw pcspkr mperf ioatdma
> mfd_core processor evdev thermal_sys ext4 jbd2 crc16 bonding bridge stp llc dm_snapshot
> dm_mirror dm_region_hash dm_log dm_mod sd_mod crc_t10dif ehci_pci uhci_hcd ehci_hcd mptsas
> mptscsih mptbase scsi_transport_sas usbcore usb_common igb dca i2c_algo_bit i2c_core ptp
> pps_core button
> [38551.550601] CPU: 0 PID: 12587 Comm: netback/0 Not tainted 3.10.0-ucs58-amd64 #1 Debian
> 3.10.11-1.58.201405060908
> [38551.550693] Hardware name: FUJITSU PRIMERGY BX620 S6/D3051, BIOS 080015 Rev.3C78.3051
> 07/22/2011
> [38551.550781] task: ffff880004b067c0 ti: ffff8800561ec000 task.ti: ffff8800561ec000
> [38551.550865] RIP: e030:[<ffffffffa04147dc>] [<ffffffffa04147dc>]
> xen_netbk_rx_action+0x18b/0x6f0 [xen_netback]
> [38551.550959] RSP: e02b:ffff8800561edce8 EFLAGS: 00010202
> [38551.551009] RAX: ffffc900104adac0 RBX: ffff8800541e95c0 RCX: ffffc90010864000
> [38551.551064] RDX: 000000000000003b RSI: 0000000000000000 RDI: ffff880040014380
> [38551.551120] RBP: ffff8800570e6800 R08: 0000000000000000 R09: ffff880004799800
> [38551.551175] R10: ffffffff813ca115 R11: ffff88005e4fdb08 R12: ffff880054e6f800
> [38551.551231] R13: ffff8800561edd58 R14: ffffc900104a1000 R15: 0000000000000000
> [38551.551289] FS: 00007f19a54a8700(0000) GS:ffff88005da00000(0000)
> knlGS:0000000000000000
> [38551.551374] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> [38551.551425] CR2: ffffc900108641d8 CR3: 0000000054cb3000 CR4: 0000000000002660
> [38551.551481] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [38551.551537] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [38551.551592] Stack:
> [38551.551630] ffff880004b06ba0 0000000000000000 ffff88005da13ec0 ffff88005da13ec0
> [38551.551726] 0000000004b067c0 ffffc900104a8ac0 ffffc900104a1020 000000005da13ec0
> [38551.551823] 0000000000000000 0000000000000001 ffffc900104a8ac0 ffffc900104adac0
> [38551.551920] Call Trace:
> [38551.551966] [<ffffffff813ca32d>] ? _raw_spin_lock_irqsave+0x11/0x2f
> [38551.552021] [<ffffffffa0416033>] ? xen_netbk_kthread+0x174/0x841 [xen_netback]
> [38551.552106] [<ffffffff8105d373>] ? wake_up_bit+0x20/0x20
> [38551.560239] [<ffffffffa0415ebf>] ? xen_netbk_tx_build_gops+0xce8/0xce8 [xen_netback]
> [38551.560325] [<ffffffff8105cd73>] ? kthread_freezable_should_stop+0x56/0x56
> [38551.560381] [<ffffffffa0415ebf>] ? xen_netbk_tx_build_gops+0xce8/0xce8 [xen_netback]
> [38551.560466] [<ffffffff8105ce1e>] ? kthread+0xab/0xb3
> [38551.560518] [<ffffffff81003638>] ? xen_end_context_switch+0xe/0x1c
> [38551.560572] [<ffffffff8105cd73>] ? kthread_freezable_should_stop+0x56/0x56
> [38551.560628] [<ffffffff813cfbfc>] ? ret_from_fork+0x7c/0xb0
> [38551.560680] [<ffffffff8105cd73>] ? kthread_freezable_should_stop+0x56/0x56
> [38551.560734] Code: 8b b3 d0 00 00 00 48 8b bb d8 00 00 00 0f b7 74 37 02 89 70 08 eb 07
> c7 40 08 00 00 00 00 89 d2 c7 40 04 00 00 00 00 48 83 c2 08 <0f> b7 34 d1 89 30 c7 44 24
> 60 00 00 00 00 8b 44 d1 04 89 44 24
> [38551.561151] RIP [<ffffffffa04147dc>] xen_netbk_rx_action+0x18b/0x6f0 [xen_netback]
> [38551.561238] RSP <ffff8800561edce8>
> [38551.561283] CR2: ffffc900108641d8
> [38551.561624] ---[ end trace 8c260c6af259c4aa ]---
Only unmap the ring when no pending packets exist which still reference
the vif.
Signed-off-by: Wei Liu <wei.liu2@citrix.com>
Signed-off-by: Philipp Hahn <hahn@univention.de>
---
v2: Use reference counting instead of boolean flag.
---
drivers/net/xen-netback/interface.c | 1 +
drivers/net/xen-netback/netback.c | 3 +++
2 files changed, 4 insertions(+)
diff --git a/drivers/net/xen-netback/interface.c b/drivers/net/xen-netback/interface.c
index 540a796..16a1d46 100644
--- a/drivers/net/xen-netback/interface.c
+++ b/drivers/net/xen-netback/interface.c
@@ -378,6 +378,7 @@ void xenvif_free(struct xenvif *vif)
atomic_dec(&vif->refcnt);
wait_event(vif->waiting_to_free, atomic_read(&vif->refcnt) == 0);
+ xen_netbk_unmap_frontend_rings(vif);
unregister_netdev(vif->dev);
free_netdev(vif->dev);
diff --git a/drivers/net/xen-netback/netback.c b/drivers/net/xen-netback/netback.c
index 70b830f..104094d 100644
--- a/drivers/net/xen-netback/netback.c
+++ b/drivers/net/xen-netback/netback.c
@@ -1864,6 +1864,9 @@ static int xen_netbk_kthread(void *data)
void xen_netbk_unmap_frontend_rings(struct xenvif *vif)
{
+ if (atomic_read(&vif->refcnt) != 0)
+ return;
+
if (vif->tx.sring)
xenbus_unmap_ring_vfree(xenvif_to_xenbus_device(vif),
vif->tx.sring);
--
1.9.1
[-- Attachment #3: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 18+ messages in thread
* Re: [PATCH] Kernel OOPS in xen_netbk_rx_action / xenvif_gop_skb
[not found] ` <53BFB142.7050201@univention.de>
@ 2014-07-11 9:53 ` Wei Liu
2014-07-11 10:32 ` Wei Liu
[not found] ` <20140711103236.GB12584@zion.uk.xensource.com>
2 siblings, 0 replies; 18+ messages in thread
From: Wei Liu @ 2014-07-11 9:53 UTC (permalink / raw)
To: Philipp Hahn
Cc: Wei Liu, Ian Campbell, stable, Zoltan Kiss, xen-devel,
Erik Damrose
On Fri, Jul 11, 2014 at 11:41:22AM +0200, Philipp Hahn wrote:
> Hello Wei Liu,
>
> On 10.07.2014 14:41, Wei Liu wrote:
> > On Wed, Jul 02, 2014 at 09:45:44AM +0200, Philipp Hahn wrote:
> >> @Wei Liu: You said that the patch is only a quick hack to detect, if my
> >> analysis is correct and a proper fix would be needed. For us the
> >> attached patch works, as the problem does not happen that often and is
> >> hard to reproduce anyway, so spending more time on that issue is
> >> probably not worth it. And that flag doesn't look that ugly.
> ...
> > I agree that we would like to avoid spending too much time on this
> > issue.
>
> This is also what I'm thinking.
>
> > Since the problem is confirmed, I think a proper fix will be to
> > reference count vif and prevent it from unmapping the ring before all
> > queued SKBs are consumed.
>
> vif is already ref-counted; see attached *untested* patch for a start.
>
What I meant was that, in xenvif_free, we will wait for refcnt to become
0 (or 1?) before freeing vif structure. I think we should do similar
thing before unmapping the ring.
> What I don't like is xenbus_unmap_ring_vfree() potentially printing an
> error on double-free when called from xen_netbk_unmap_frontend_rings().
>
How can it be called twice? Does it happen before or after applying my
patch?
> > But it might require much more work than that quick hack.
>
> I'm no network driver/Xen expert, but that sounds like more work for no
> gain: We would still copy packets for a guest, which is already dead. If
> the ring get full, we would probably need to go to sleep and wait for an
> answer we will never get anymore.
>
That's true, so dropping those packets seems to be a good solution.
> > FWIW this bug doesn't exist in kernel >=3.12.
>
> That is even one more point for the hack, as there the problem is
> properly fixed and the problem is very obscure to trigger.
>
> > Would you up for writing a patch? I won't be able to write
> > one in the near future.
> > Further more, you're the only party now can verify a fix.
>
> I had a look and created the attached patch, which is untested, as I
> currently can't access the faulting system and have been unable to
> reproduce it in my development environment.
>
> The quick hack now runs on that system for several weeks now without
> problems.
>
Thanks for testing.
I will need to think further if this hack is really a proper solution. I
will keep you posted if there's any update on this. In the mean time you
can carry that patch in your own tree for a while.
Wei.
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH] Kernel OOPS in xen_netbk_rx_action / xenvif_gop_skb
[not found] ` <53BFB142.7050201@univention.de>
2014-07-11 9:53 ` Wei Liu
@ 2014-07-11 10:32 ` Wei Liu
[not found] ` <20140711103236.GB12584@zion.uk.xensource.com>
2 siblings, 0 replies; 18+ messages in thread
From: Wei Liu @ 2014-07-11 10:32 UTC (permalink / raw)
To: Philipp Hahn
Cc: Wei Liu, Ian Campbell, stable, Zoltan Kiss, xen-devel,
Erik Damrose
On Fri, Jul 11, 2014 at 11:41:22AM +0200, Philipp Hahn wrote:
> Hello Wei Liu,
>
> On 10.07.2014 14:41, Wei Liu wrote:
> > On Wed, Jul 02, 2014 at 09:45:44AM +0200, Philipp Hahn wrote:
> >> @Wei Liu: You said that the patch is only a quick hack to detect, if my
> >> analysis is correct and a proper fix would be needed. For us the
> >> attached patch works, as the problem does not happen that often and is
> >> hard to reproduce anyway, so spending more time on that issue is
> >> probably not worth it. And that flag doesn't look that ugly.
> ...
> > I agree that we would like to avoid spending too much time on this
> > issue.
>
> This is also what I'm thinking.
>
> > Since the problem is confirmed, I think a proper fix will be to
> > reference count vif and prevent it from unmapping the ring before all
> > queued SKBs are consumed.
>
> vif is already ref-counted; see attached *untested* patch for a start.
>
[...]
>
> The quick hack now runs on that system for several weeks now without
> problems.
>
I'm a bit confused, you said that patch was *untested*, then you said
the hack runs fine.
I guess what you meant is: the approach is generally working but the
patch you submitted is not the one running in your system? Is there
significant difference between the patch you submitted and the one
you're using in production?
Wei.
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH] Kernel OOPS in xen_netbk_rx_action / xenvif_gop_skb
[not found] ` <20140711103236.GB12584@zion.uk.xensource.com>
@ 2014-07-11 11:02 ` Philipp Hahn
[not found] ` <53BFC43A.4080709@univention.de>
1 sibling, 0 replies; 18+ messages in thread
From: Philipp Hahn @ 2014-07-11 11:02 UTC (permalink / raw)
To: Wei Liu; +Cc: xen-devel, Zoltan Kiss, Erik Damrose, Ian Campbell, stable
Hello Wei Liu,
On 11.07.2014 12:32, Wei Liu wrote:
> On Fri, Jul 11, 2014 at 11:41:22AM +0200, Philipp Hahn wrote:
>> On 10.07.2014 14:41, Wei Liu wrote:
>>> On Wed, Jul 02, 2014 at 09:45:44AM +0200, Philipp Hahn wrote:
>>>> @Wei Liu: You said that the patch is only a quick hack to
>>>> detect, if my analysis is correct and a proper fix would be
>>>> needed. For us the attached patch works, as the problem does
>>>> not happen that often and is hard to reproduce anyway, so
>>>> spending more time on that issue is probably not worth it. And
>>>> that flag doesn't look that ugly.
>> ...
>>> I agree that we would like to avoid spending too much time on
>>> this issue.
>>
>> This is also what I'm thinking.
>>
>>> Since the problem is confirmed, I think a proper fix will be to
>>> reference count vif and prevent it from unmapping the ring
>>> before all queued SKBs are consumed.
>>
>> vif is already ref-counted; see attached *untested* patch for a
>> start.
v2 = using refcnt
>> The quick hack now runs on that system for several weeks now
>> without problems.
v1 = Your quick hack patch + my additional dev_kfree_skb(skb);
> I'm a bit confused, you said that patch was *untested*, then you said
> the hack runs fine.
>
> I guess what you meant is: the approach is generally working but the
> patch you submitted is not the one running in your system? Is there
> significant difference between the patch you submitted and the one
> you're using in production?
The PATCH form 02.07.2014 09:45 using your "vif->mapped" is running fine
on the system for ~2 weeks now.
The PATCHv2 using refcnt I posted today is untested and for RFC.
On 11.07.2014 11:53, Wei Liu wrote:
>> What I don't like is xenbus_unmap_ring_vfree() potentially
>> printing an error on double-free when called from
>> xen_netbk_unmap_frontend_rings().
>
> How can it be called twice? Does it happen before or after applying
> my patch?
I've not seen it called twice, but while studying the code I asked
myself if it's safe to be called twice. I've not gone down to road to
proof that it never is. Just being cautious.
Worst thing which could happen is an error message:
| xenbus_dev_error(dev, -ENOENT,
| "can't find mapped virtual address %p", vaddr);
> I will need to think further if this hack is really a proper
> solution. I will keep you posted if there's any update on this.
Thank you for your feedback and support.
> In the mean time you can carry that patch in your own tree for a
> while.
We will keep using v1 for now and I will keep you updated on this as well.
Sincerely
Philipp
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH] Kernel OOPS in xen_netbk_rx_action / xenvif_gop_skb
[not found] ` <53BFC43A.4080709@univention.de>
@ 2014-07-11 11:16 ` Wei Liu
0 siblings, 0 replies; 18+ messages in thread
From: Wei Liu @ 2014-07-11 11:16 UTC (permalink / raw)
To: Philipp Hahn
Cc: Wei Liu, Ian Campbell, stable, Zoltan Kiss, xen-devel,
Erik Damrose
On Fri, Jul 11, 2014 at 01:02:18PM +0200, Philipp Hahn wrote:
> Hello Wei Liu,
>
> On 11.07.2014 12:32, Wei Liu wrote:
> > On Fri, Jul 11, 2014 at 11:41:22AM +0200, Philipp Hahn wrote:
> >> On 10.07.2014 14:41, Wei Liu wrote:
> >>> On Wed, Jul 02, 2014 at 09:45:44AM +0200, Philipp Hahn wrote:
> >>>> @Wei Liu: You said that the patch is only a quick hack to
> >>>> detect, if my analysis is correct and a proper fix would be
> >>>> needed. For us the attached patch works, as the problem does
> >>>> not happen that often and is hard to reproduce anyway, so
> >>>> spending more time on that issue is probably not worth it. And
> >>>> that flag doesn't look that ugly.
> >> ...
> >>> I agree that we would like to avoid spending too much time on
> >>> this issue.
> >>
> >> This is also what I'm thinking.
> >>
> >>> Since the problem is confirmed, I think a proper fix will be to
> >>> reference count vif and prevent it from unmapping the ring
> >>> before all queued SKBs are consumed.
> >>
> >> vif is already ref-counted; see attached *untested* patch for a
> >> start.
>
> v2 = using refcnt
>
Oh I see, I misread your email. You actually sent me two patches, sorry.
I looked at your first one and somehow missed second one.
Unfortunately your patch with refcnt won't work.
> >> The quick hack now runs on that system for several weeks now
> >> without problems.
>
> v1 = Your quick hack patch + my additional dev_kfree_skb(skb);
I guess you mean "my additional xenvif_put". :-)
I will see what I can do.
Wei.
^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2014-07-11 11:16 UTC | newest]
Thread overview: 18+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-06-06 10:26 RFH: Kernel OOPS in xen_netbk_rx_action / xenvif_gop_skb Philipp Hahn
2014-06-06 10:58 ` Wei Liu
2014-06-06 22:12 ` Philipp Hahn
2014-06-18 16:48 ` Philipp Hahn
2014-06-19 14:12 ` Wei Liu
2014-06-19 14:35 ` David Vrabel
2014-06-19 14:41 ` Wei Liu
2014-06-23 14:56 ` Philipp Hahn
2014-06-27 8:42 ` Philipp Hahn
2014-06-27 17:48 ` Philipp Hahn
2014-06-27 18:24 ` Philipp Hahn
2014-07-02 7:45 ` [PATCH] " Philipp Hahn
2014-07-10 12:41 ` Wei Liu
[not found] ` <20140710124122.GA2381@zion.uk.xensource.com>
2014-07-11 9:41 ` Philipp Hahn
[not found] ` <53BFB142.7050201@univention.de>
2014-07-11 9:53 ` Wei Liu
2014-07-11 10:32 ` Wei Liu
[not found] ` <20140711103236.GB12584@zion.uk.xensource.com>
2014-07-11 11:02 ` Philipp Hahn
[not found] ` <53BFC43A.4080709@univention.de>
2014-07-11 11:16 ` Wei Liu
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).