* Fw: [Bug 32772] New: PROBLEM: kernel BUG at net/ipv4/inetpeer.c:386
@ 2011-04-06 15:18 Stephen Hemminger
2011-04-06 16:05 ` Eric Dumazet
0 siblings, 1 reply; 9+ messages in thread
From: Stephen Hemminger @ 2011-04-06 15:18 UTC (permalink / raw)
To: netdev
Begin forwarded message:
Date: Wed, 6 Apr 2011 07:39:54 GMT
From: bugzilla-daemon@bugzilla.kernel.org
To: shemminger@linux-foundation.org
Subject: [Bug 32772] New: PROBLEM: kernel BUG at net/ipv4/inetpeer.c:386
https://bugzilla.kernel.org/show_bug.cgi?id=32772
Summary: PROBLEM: kernel BUG at net/ipv4/inetpeer.c:386
Product: Networking
Version: 2.5
Kernel Version: 2.6.38
Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: IPV4
AssignedTo: shemminger@linux-foundation.org
ReportedBy: dimetrios@gmail.com
Regression: No
Kernel oopses periodically with 'kernel BUG at net/ipv4/inetpeer.c:386'
message. Machine is used as BGP router and runs Quagga. Nonordinary kernel
config option set: CONFIG_IP_FIB_TRIE=y.
Two traces:
--------------------trace begin--------------
[625279.329241] kernel BUG at net/ipv4/inetpeer.c:386!
[625279.329241] invalid opcode: 0000 [#1] SMP
[625279.329241] last sysfs file: /sys/module/ip_tables/initstate
[625279.329241] Modules linked in: nf_nat_pptp nf_nat_proto_gre
nf_conntrack_pptp nf_conntrack_proto_gre nf_nat_ftp nf_conntrack_ftp ipt_REJECT
xt_state xt_tcpudp xt_multiport ip_set iptable_filter iptable_mangle
iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables
x_tables act_police cls_u32 sch_ingress sch_tbf 8021q garp bridge ipv6 stp llc
loop i2c_i801 intel_agp parport_pc i2c_core intel_gtt rng_core agpgart
processor parport button evdev pcspkr thermal_sys serio_raw tpm_tis tpm
tpm_bios ext3 jbd mbcache sd_mod crc_t10dif ata_generic ata_piix libata
scsi_mod uhci_hcd ide_pci_generic e1000e ehci_hcd r8169 ide_core igb dca mii
usbcore nls_base [last unloaded: scsi_wait_scan]
[625279.329241]
[625279.329241] Pid: 0, comm: kworker/0:0 Not tainted 2.6.38-demyan-1.1demyan
#1 Gigabyte Technology Co., Ltd. G41MT-ES2L/G41MT-ES2L
[625279.329241] EIP: 0060:[<c11e0caa>] EFLAGS: 00010283 CPU: 1
[625279.329241] EIP is at unlink_from_pool+0x85/0x14a
[625279.329241] EAX: c125ff04 EBX: ed21cd40 ECX: c351ce70 EDX: e8db5b40
[625279.329241] ESI: c1333338 EDI: f4c91ca0 EBP: c351b55e ESP: f4c91c48
[625279.329241] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
[625279.329241] Process kworker/0:0 (pid: 0, ti=f4c90000 task=f4c6a400
task.ti=f4c8c000)
[625279.329241] Stack:
[625279.329241] f1be9b00 00000001 c351ce70 c133333c c1333338 ed21a684 f11a3f84
f0146f00
[625279.329241] ed2dca80 ed21a900 f0146644 ec4c2f40 f0146280 ec701dc0 f0467fc0
eea79604
[625279.329241] f16c12c0 ef727900 ec865784 e721a3c0 ee859cc4 e8db5b40 00000640
00000014
[625279.329241] Call Trace:
[625279.329241] [<c11ea34a>] ? tcp_tso_segment+0x24d/0x25c
[625279.329241] [<f820048a>] ? tcp_packet+0xb8e/0xbb8 [nf_conntrack]
[625279.329241] [<c11e0de9>] ? cleanup_once+0x7a/0x7f
[625279.329241] [<c11e0fa9>] ? inet_getpeer+0x1bb/0x1dc
[625279.329241] [<c11d0001>] ? store_xps_map+0xa1/0x2b8
[625279.329241] [<c11c3477>] ? dev_hard_start_xmit+0x36f/0x454
[625279.329241] [<c1021ef1>] ? get_nohz_timer_target+0x47/0x64
[625279.329241] [<c11e1cb0>] ? ip4_frag_init+0x66/0x71
[625279.329241] [<c120bb54>] ? inet_frag_find+0x80/0x18d
[625279.329241] [<c11e1dec>] ? ip_defrag+0x131/0x955
[625279.329241] [<f81be0b1>] ? ipv4_conntrack_defrag+0xb0/0xd3
[nf_defrag_ipv4]
[625279.329241] [<c11dc036>] ? nf_iterate+0x32/0x5d
[625279.329241] [<c11e10e0>] ? ip_rcv_finish+0x0/0x31f
[625279.329241] [<c11dc13d>] ? nf_hook_slow+0x40/0xb5
[625279.329241] [<c11e10e0>] ? ip_rcv_finish+0x0/0x31f
[625279.329241] [<c11e164c>] ? ip_rcv+0x24d/0x293
[625279.329241] [<c11e10e0>] ? ip_rcv_finish+0x0/0x31f
[625279.329241] [<c11c1b3c>] ? __netif_receive_skb+0x405/0x42c
[625279.329241] [<c11c1a63>] ? __netif_receive_skb+0x32c/0x42c
[625279.329241] [<c1047585>] ? ktime_get_real+0x10/0x2d
[625279.329241] [<c11c2547>] ? netif_receive_skb+0x5a/0x5f
[625279.329241] [<c11c25ff>] ? napi_skb_finish+0x1b/0x30
[625279.329241] [<f80a9723>] ? igb_poll+0x649/0x94a [igb]
[625279.329241] [<c1007765>] ? sched_clock+0x9/0xd
[625279.329241] [<c1030582>] ? do_exit+0x2e/0x60c
[625279.329241] [<c104438f>] ? sched_clock_local+0x17/0x13d
[625279.329241] [<c11c2b7b>] ? net_rx_action+0x90/0x150
[625279.329241] [<c1031f12>] ? __do_softirq+0x75/0x10e
[625279.329241] [<c1031e9d>] ? __do_softirq+0x0/0x10e
[625279.329241] <IRQ>
[625279.329241] [<c1031df3>] ? irq_exit+0x31/0x64
[625279.329241] [<c1004397>] ? do_IRQ+0x73/0x84
[625279.329241] [<c1003429>] ? common_interrupt+0x29/0x30
[625279.329241] [<c10089b4>] ? mwait_idle+0x4f/0x59
[625279.329241] [<c10021ef>] ? cpu_idle+0x46/0x63
[625279.329241] Code: 24 08 39 cd 75 09 42 3b 54 24 04 7c e9 eb 18 3b 6c 24 08
8d 50 04 0f 42 d0 89 17 83 c7 04 8b 02 3d 04 ff 25 c1 75 bb 39 d8 74 04 <0f> 0b
eb fe 8d 6f fc 81 3b 04 ff 25 c1 89 6c 24 08 75 0d 8b 47
[625279.329241] EIP: [<c11e0caa>] unlink_from_pool+0x85/0x14a SS:ESP
0068:f4c91c48
[625280.416294] ---[ end trace b75ce593ad6cbee7 ]---
[625280.430422] Kernel panic - not syncing: Fatal exception in interrupt
[625280.449739] Pid: 0, comm: kworker/0:0 Tainted: G D
2.6.38-demyan-1.1demyan #1
[625280.473762] Call Trace:
[625280.481380] [<c1231f71>] ? panic+0x4d/0x137
[625280.494457] [<c1005722>] ? oops_end+0x8e/0x99
[625280.508054] [<c1003a0e>] ? do_invalid_op+0x0/0x75
[625280.522693] [<c1003a7a>] ? do_invalid_op+0x6c/0x75
[625280.537588] [<c11e0caa>] ? unlink_from_pool+0x85/0x14a
[625280.553527] [<c11e0bbd>] ? inet_putpeer+0x15/0x47
[625280.568165] [<c11e0d64>] ? unlink_from_pool+0x13f/0x14a
[625280.584367] [<f80a9ed6>] ? igb_xmit_frame_ring_adv+0x4b2/0x795 [igb]
[625280.603941] [<c1007765>] ? sched_clock+0x9/0xd
[625280.617797] [<c123464e>] ? error_code+0x5a/0x60
[625280.631913] [<c1003a0e>] ? do_invalid_op+0x0/0x75
[625280.646552] [<c11e0caa>] ? unlink_from_pool+0x85/0x14a
[625280.662490] [<c11ea34a>] ? tcp_tso_segment+0x24d/0x25c
[625280.678427] [<f820048a>] ? tcp_packet+0xb8e/0xbb8 [nf_conntrack]
[625280.696964] [<c11e0de9>] ? cleanup_once+0x7a/0x7f
[625280.711600] [<c11e0fa9>] ? inet_getpeer+0x1bb/0x1dc
[625280.726758] [<c11d0001>] ? store_xps_map+0xa1/0x2b8
[625280.741916] [<c11c3477>] ? dev_hard_start_xmit+0x36f/0x454
[625280.758894] [<c1021ef1>] ? get_nohz_timer_target+0x47/0x64
[625280.775870] [<c11e1cb0>] ? ip4_frag_init+0x66/0x71
[625280.790768] [<c120bb54>] ? inet_frag_find+0x80/0x18d
[625280.806184] [<c11e1dec>] ? ip_defrag+0x131/0x955
[625280.820562] [<f81be0b1>] ? ipv4_conntrack_defrag+0xb0/0xd3
[nf_defrag_ipv4]
[625280.841961] [<c11dc036>] ? nf_iterate+0x32/0x5d
[625280.856078] [<c11e10e0>] ? ip_rcv_finish+0x0/0x31f
[625280.870975] [<c11dc13d>] ? nf_hook_slow+0x40/0xb5
[625280.885612] [<c11e10e0>] ? ip_rcv_finish+0x0/0x31f
[625280.900510] [<c11e164c>] ? ip_rcv+0x24d/0x293
[625280.914107] [<c11e10e0>] ? ip_rcv_finish+0x0/0x31f
[625280.929005] [<c11c1b3c>] ? __netif_receive_skb+0x405/0x42c
[625280.945982] [<c11c1a63>] ? __netif_receive_skb+0x32c/0x42c
[625280.962960] [<c1047585>] ? ktime_get_real+0x10/0x2d
[625280.978121] [<c11c2547>] ? netif_receive_skb+0x5a/0x5f
[625280.994055] [<c11c25ff>] ? napi_skb_finish+0x1b/0x30
[625281.009473] [<f80a9723>] ? igb_poll+0x649/0x94a [igb]
[625281.025150] [<c1007765>] ? sched_clock+0x9/0xd
[625281.039005] [<c1030582>] ? do_exit+0x2e/0x60c
[625281.052603] [<c104438f>] ? sched_clock_local+0x17/0x13d
[625281.068800] [<c11c2b7b>] ? net_rx_action+0x90/0x150
[625281.083958] [<c1031f12>] ? __do_softirq+0x75/0x10e
[625281.098857] [<c1031e9d>] ? __do_softirq+0x0/0x10e
[625281.113493] <IRQ> [<c1031df3>] ? irq_exit+0x31/0x64
[625281.128963] [<c1004397>] ? do_IRQ+0x73/0x84
[625281.142040] [<c1003429>] ? common_interrupt+0x29/0x30
[625281.157718] [<c10089b4>] ? mwait_idle+0x4f/0x59
[625281.171836] [<c10021ef>] ? cpu_idle+0x46/0x63
[625281.185435] Rebooting in 5 seconds..
--------------------trace end--------------
--------------------trace begin--------------
[237684.673906] kernel BUG at net/ipv4/inetpeer.c:386!
[237684.673906] invalid opcode: 0000 [#1] SMP
[237684.673906] last sysfs file: /sys/module/nf_conntrack_pptp/initstate
[237684.673906] Modules linked in: nf_nat_pptp nf_nat_proto_gre
nf_conntrack_pptp nf_conntrack_proto_gre nf_nat_ftp nf_conntrack_ftp ipt_REJECT
xt_state xt_tcpudp xt_multiport ip_set iptable_filter iptable_mangle
iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables
x_tables act_police cls_u32 sch_ingress sch_tbf 8021q garp bridge ipv6 stp llc
loop i2c_i801 rng_core intel_agp intel_gtt agpgart i2c_core tpm_tis evdev
pcspkr parport_pc processor parport button tpm thermal_sys tpm_bios serio_raw
ext3 jbd mbcache sd_mod crc_t10dif ata_generic ata_piix libata scsi_mod
uhci_hcd ide_pci_generic r8169 ehci_hcd e1000e mii igb dca ide_core usbcore
nls_base [last unloaded: scsi_wait_scan]
[237684.673906]
[237684.673906] Pid: 0, comm: swapper Not tainted 2.6.38-demyan-1.1demyan #1
Gigabyte Technology Co., Ltd. G41MT-ES2L/G41MT-ES2L
[237684.673906] EIP: 0060:[<c11e0caa>] EFLAGS: 00010287 CPU: 0
[237684.673906] EIP is at unlink_from_pool+0x85/0x14a
[237684.673906] EAX: c125ff04 EBX: ed76d180 ECX: 75c219bc EDX: e8de9444
[237684.673906] ESI: c1333338 EDI: f4c0bbfc EBP: 75c25152 ESP: f4c0bba8
[237684.673906] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
[237684.673906] Process swapper (pid: 0, ti=f4c0a000 task=c1315f20
task.ti=c1300000)
[237684.673906] Stack:
[237684.673906] ef42e49c 00000001 75c219bc c133333c c1333338 f4780d80 ed76d744
f19e3744
[237684.673906] ed71f980 f452a404 f183fc04 f452afc4 f4780ac0 f18ca680 f474fe84
f1871180
[237684.673906] ee2a9884 edef3844 f1cf3e04 edf15284 e8de9444 f4c0bcb4 ef42e49c
f4c0bc78
[237684.673906] Call Trace:
[237684.673906] [<c120f068>] ? fib4_rule_action+0x40/0x4d
[237684.673906] [<c11d1be3>] ? fib_rules_lookup+0x8d/0xe4
[237684.673906] [<c109bf68>] ? cache_alloc_refill+0x75/0x3dc
[237684.673906] [<c11e0de9>] ? cleanup_once+0x7a/0x7f
[237684.673906] [<c11e0fa9>] ? inet_getpeer+0x1bb/0x1dc
[237684.673906] [<c11dc073>] ? nf_ct_attach+0x12/0x13
[237684.673906] [<c1202404>] ? icmp_glue_bits+0x65/0x6a
[237684.673906] [<c11e4109>] ? ip_append_data+0x595/0x850
[237684.673906] [<c11e025d>] ? rt_bind_peer+0x1d/0x3d
[237684.673906] [<c11e029f>] ? __ip_select_ident+0x22/0xa6
[237684.673906] [<c11e4f60>] ? ip_push_pending_frames+0x206/0x2cb
[237684.673906] [<c120301b>] ? icmp_send+0x4fe/0x523
[237684.673906] [<f8270b09>] ? ____nf_conntrack_find+0xfa/0x142 [nf_conntrack]
[237684.673906] [<f8272069>] ? nf_conntrack_in+0x4f3/0x5e3 [nf_conntrack]
[237684.673906] [<f81ef536>] ? ipt_do_table+0x4bc/0x4eb [ip_tables]
[237684.673906] [<c11e2949>] ? ip_forward+0x2ef/0x316
[237684.673906] [<c11e13da>] ? ip_rcv_finish+0x2fa/0x31f
[237684.673906] [<c11c1b3c>] ? __netif_receive_skb+0x405/0x42c
[237684.673906] [<c11c1a63>] ? __netif_receive_skb+0x32c/0x42c
[237684.673906] [<c1047585>] ? ktime_get_real+0x10/0x2d
[237684.673906] [<c11c2547>] ? netif_receive_skb+0x5a/0x5f
[237684.673906] [<c11c25ff>] ? napi_skb_finish+0x1b/0x30
[237684.673906] [<f80e1723>] ? igb_poll+0x649/0x94a [igb]
[237684.673906] [<c1007765>] ? sched_clock+0x9/0xd
[237684.673906] [<c1030091>] ? wait_consider_task+0x974/0xa91
[237684.673906] [<c104438f>] ? sched_clock_local+0x17/0x13d
[237684.673906] [<c11c2b7b>] ? net_rx_action+0x90/0x150
[237684.673906] [<c1031f12>] ? __do_softirq+0x75/0x10e
[237684.673906] [<c1031e9d>] ? __do_softirq+0x0/0x10e
[237684.673906] <IRQ>
[237684.673906] [<c1031df3>] ? irq_exit+0x31/0x64
[237684.673906] [<c1004397>] ? do_IRQ+0x73/0x84
[237684.673906] [<c1003429>] ? common_interrupt+0x29/0x30
[237684.673906] [<c10089b4>] ? mwait_idle+0x4f/0x59
[237684.673906] [<c10021ef>] ? cpu_idle+0x46/0x63
[237684.673906] [<c133b85c>] ? start_kernel+0x2e2/0x2e5
[237684.673906] Code: 24 08 39 cd 75 09 42 3b 54 24 04 7c e9 eb 18 3b 6c 24 08
8d 50 04 0f 42 d0 89 17 83 c7 04 8b 02 3d 04 ff 25 c1 75 bb 39 d8 74 04 <0f> 0b
eb fe 8d 6f fc 81 3b 04 ff 25 c1 89 6c 24 08 75 0d 8b 47
[237684.673906] EIP: [<c11e0caa>] unlink_from_pool+0x85/0x14a SS:ESP
0068:f4c0bba8
[237685.787747] ---[ end trace e3c73323a4e3b283 ]---
[237685.801876] Kernel panic - not syncing: Fatal exception in interrupt
[237685.821194] Pid: 0, comm: swapper Tainted: G D
2.6.38-demyan-1.1demyan #1
[237685.844177] Call Trace:
[237685.851797] [<c1231f71>] ? panic+0x4d/0x137
[237685.864874] [<c1005722>] ? oops_end+0x8e/0x99
[237685.878471] [<c1003a0e>] ? do_invalid_op+0x0/0x75
[237685.893109] [<c1003a7a>] ? do_invalid_op+0x6c/0x75
[237685.908005] [<c11e0caa>] ? unlink_from_pool+0x85/0x14a
[237685.923942] [<c1007765>] ? sched_clock+0x9/0xd
[237685.937801] [<c1007765>] ? sched_clock+0x9/0xd
[237685.951658] [<c104438f>] ? sched_clock_local+0x17/0x13d
[237685.967856] [<c123464e>] ? error_code+0x5a/0x60
[237685.981973] [<c1003a0e>] ? do_invalid_op+0x0/0x75
[237685.996610] [<c11e0caa>] ? unlink_from_pool+0x85/0x14a
[237686.012548] [<c120f068>] ? fib4_rule_action+0x40/0x4d
[237686.028225] [<c11d1be3>] ? fib_rules_lookup+0x8d/0xe4
[237686.043902] [<c109bf68>] ? cache_alloc_refill+0x75/0x3dc
[237686.060359] [<c11e0de9>] ? cleanup_once+0x7a/0x7f
[237686.074997] [<c11e0fa9>] ? inet_getpeer+0x1bb/0x1dc
[237686.090156] [<c11dc073>] ? nf_ct_attach+0x12/0x13
[237686.104792] [<c1202404>] ? icmp_glue_bits+0x65/0x6a
[237686.119949] [<c11e4109>] ? ip_append_data+0x595/0x850
[237686.135626] [<c11e025d>] ? rt_bind_peer+0x1d/0x3d
[237686.150264] [<c11e029f>] ? __ip_select_ident+0x22/0xa6
[237686.166202] [<c11e4f60>] ? ip_push_pending_frames+0x206/0x2cb
[237686.183959] [<c120301b>] ? icmp_send+0x4fe/0x523
[237686.198338] [<f8270b09>] ? ____nf_conntrack_find+0xfa/0x142 [nf_conntrack]
[237686.219474] [<f8272069>] ? nf_conntrack_in+0x4f3/0x5e3 [nf_conntrack]
[237686.239311] [<f81ef536>] ? ipt_do_table+0x4bc/0x4eb [ip_tables]
[237686.257589] [<c11e2949>] ? ip_forward+0x2ef/0x316
[237686.272227] [<c11e13da>] ? ip_rcv_finish+0x2fa/0x31f
[237686.287643] [<c11c1b3c>] ? __netif_receive_skb+0x405/0x42c
[237686.304620] [<c11c1a63>] ? __netif_receive_skb+0x32c/0x42c
[237686.321599] [<c1047585>] ? ktime_get_real+0x10/0x2d
[237686.336760] [<c11c2547>] ? netif_receive_skb+0x5a/0x5f
[237686.352692] [<c11c25ff>] ? napi_skb_finish+0x1b/0x30
[237686.368111] [<f80e1723>] ? igb_poll+0x649/0x94a [igb]
[237686.383788] [<c1007765>] ? sched_clock+0x9/0xd
[237686.397645] [<c1030091>] ? wait_consider_task+0x974/0xa91
[237686.414362] [<c104438f>] ? sched_clock_local+0x17/0x13d
[237686.430559] [<c11c2b7b>] ? net_rx_action+0x90/0x150
[237686.445718] [<c1031f12>] ? __do_softirq+0x75/0x10e
[237686.460614] [<c1031e9d>] ? __do_softirq+0x0/0x10e
[237686.475251] <IRQ> [<c1031df3>] ? irq_exit+0x31/0x64
[237686.490722] [<c1004397>] ? do_IRQ+0x73/0x84
[237686.503799] [<c1003429>] ? common_interrupt+0x29/0x30
[237686.519476] [<c10089b4>] ? mwait_idle+0x4f/0x59
[237686.533593] [<c10021ef>] ? cpu_idle+0x46/0x63
[237686.547191] [<c133b85c>] ? start_kernel+0x2e2/0x2e5
[237686.562350] Rebooting in 5 seconds..
--------------------trace end--------------
--
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
--
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: Fw: [Bug 32772] New: PROBLEM: kernel BUG at net/ipv4/inetpeer.c:386 2011-04-06 15:18 Fw: [Bug 32772] New: PROBLEM: kernel BUG at net/ipv4/inetpeer.c:386 Stephen Hemminger @ 2011-04-06 16:05 ` Eric Dumazet 2011-04-06 16:25 ` Eric Dumazet 0 siblings, 1 reply; 9+ messages in thread From: Eric Dumazet @ 2011-04-06 16:05 UTC (permalink / raw) To: Stephen Hemminger; +Cc: netdev, dimetrios, David Miller Le mercredi 06 avril 2011 à 08:18 -0700, Stephen Hemminger a écrit : > > Begin forwarded message: > > Date: Wed, 6 Apr 2011 07:39:54 GMT > From: bugzilla-daemon@bugzilla.kernel.org > To: shemminger@linux-foundation.org > Subject: [Bug 32772] New: PROBLEM: kernel BUG at net/ipv4/inetpeer.c:386 > > > https://bugzilla.kernel.org/show_bug.cgi?id=32772 > > Summary: PROBLEM: kernel BUG at net/ipv4/inetpeer.c:386 > Product: Networking > Version: 2.5 > Kernel Version: 2.6.38 > Platform: All > OS/Version: Linux > Tree: Mainline > Status: NEW > Severity: normal > Priority: P1 > Component: IPV4 > AssignedTo: shemminger@linux-foundation.org > ReportedBy: dimetrios@gmail.com > Regression: No > > > Kernel oopses periodically with 'kernel BUG at net/ipv4/inetpeer.c:386' > message. Machine is used as BGP router and runs Quagga. Nonordinary kernel > config option set: CONFIG_IP_FIB_TRIE=y. > Two traces: > --------------------trace begin-------------- > [625279.329241] kernel BUG at net/ipv4/inetpeer.c:386! Hmm... if (atomic_cmpxchg(&p->refcnt, 1, -1) == 1) { struct inet_peer __rcu **stack[PEER_MAXDEPTH]; struct inet_peer __rcu ***stackptr, ***delp; if (lookup(&p->daddr, stack, base) != p) BUG(); So we cant find a peer in AVL tree, while we really should at this stage. This reminds me a possible memory corruption (from another layer) Could Dmitry try to boot with boot parameter "slub_nomerge" , to make sure inetpeer layer doesnt share its kmem_cache with a corrupter ? ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Fw: [Bug 32772] New: PROBLEM: kernel BUG at net/ipv4/inetpeer.c:386 2011-04-06 16:05 ` Eric Dumazet @ 2011-04-06 16:25 ` Eric Dumazet 2011-04-06 18:15 ` Dmitry Novikov 0 siblings, 1 reply; 9+ messages in thread From: Eric Dumazet @ 2011-04-06 16:25 UTC (permalink / raw) To: Stephen Hemminger; +Cc: netdev, dimetrios, David Miller Le mercredi 06 avril 2011 à 18:05 +0200, Eric Dumazet a écrit : > > This reminds me a possible memory corruption (from another layer) > I found the reference of a past bug report, where slub_nomerge was used too. http://www.spinics.net/lists/netdev/msg154206.html ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Fw: [Bug 32772] New: PROBLEM: kernel BUG at net/ipv4/inetpeer.c:386 2011-04-06 16:25 ` Eric Dumazet @ 2011-04-06 18:15 ` Dmitry Novikov 2011-04-06 18:16 ` David Miller 0 siblings, 1 reply; 9+ messages in thread From: Dmitry Novikov @ 2011-04-06 18:15 UTC (permalink / raw) To: Eric Dumazet; +Cc: Stephen Hemminger, netdev, David Miller 2011/4/6 Eric Dumazet <eric.dumazet@gmail.com> > > Le mercredi 06 avril 2011 à 18:05 +0200, Eric Dumazet a écrit : > > > > > This reminds me a possible memory corruption (from another layer) > > > > I found the reference of a past bug report, where slub_nomerge was used > too. > > http://www.spinics.net/lists/netdev/msg154206.html > > > I will schedule reboot in 6 hours so that slub_nomerge parmeter been applied ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Bug 32772] New: PROBLEM: kernel BUG at net/ipv4/inetpeer.c:386 2011-04-06 18:15 ` Dmitry Novikov @ 2011-04-06 18:16 ` David Miller 2011-04-13 20:14 ` Dmitry Novikov 0 siblings, 1 reply; 9+ messages in thread From: David Miller @ 2011-04-06 18:16 UTC (permalink / raw) To: dimetrios; +Cc: eric.dumazet, shemminger, netdev From: Dmitry Novikov <dimetrios@gmail.com> Date: Wed, 6 Apr 2011 21:15:17 +0300 > 2011/4/6 Eric Dumazet <eric.dumazet@gmail.com> >> >> Le mercredi 06 avril 2011 à 18:05 +0200, Eric Dumazet a écrit : >> >> > >> > This reminds me a possible memory corruption (from another layer) >> > >> >> I found the reference of a past bug report, where slub_nomerge was used >> too. >> >> http://www.spinics.net/lists/netdev/msg154206.html >> >> >> > I will schedule reboot in 6 hours so that slub_nomerge parmeter been applied Thanks for helping us track this down. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Bug 32772] New: PROBLEM: kernel BUG at net/ipv4/inetpeer.c:386 2011-04-06 18:16 ` David Miller @ 2011-04-13 20:14 ` Dmitry Novikov 2011-04-13 20:24 ` David Miller 0 siblings, 1 reply; 9+ messages in thread From: Dmitry Novikov @ 2011-04-13 20:14 UTC (permalink / raw) To: David Miller; +Cc: eric.dumazet, shemminger, netdev Hello. Crash again after 7 days of uptime. slub_nomerge is set [559353.216526] ------------[ cut here ]------------ [559353.217494] kernel BUG at net/ipv4/inetpeer.c:386! [559353.217494] invalid opcode: 0000 [#1] SMP [559353.217494] last sysfs file: /sys/module/nf_conntrack_pptp/initstate [559353.217494] Modules linked in: nf_nat_pptp nf_nat_proto_gre nf_conntrack_pptp nf_conntrack_proto_gre nf_nat_ftp nf_conntrack_ftp ipt_REJECT xt_state xt_tcpudp xt_multiport ip_set iptable_filter iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables act_police cls_u32 sch_ingress sch_tbf 8021q garp bridge ipv6 stp llc loop intel_agp intel_gtt agpgart rng_core pcspkr i2c_i801 i2c_core processor thermal_sys parport_pc evdev parport serio_raw tpm_tis tpm button tpm_bios ext3 jbd mbcache sd_mod crc_t10dif ata_generic ata_piix libata scsi_mod uhci_hcd ide_pci_generic e1000e ehci_hcd igb r8169 ide_core dca mii usbcore nls_base [last unloaded: scsi_wait_scan] [559353.217494] [559353.217494] Pid: 0, comm: kworker/0:0 Not tainted 2.6.38-demyan-1.1demyan #1 Gigabyte Technology Co., Ltd. G41MT-ES2L/G41MT-ES2L [559353.217494] EIP: 0060:[<c11e0caa>] EFLAGS: 00010287 CPU: 1 [559353.217494] EIP is at unlink_from_pool+0x85/0x14a [559353.217494] EAX: c125ff04 EBX: efcb09c0 ECX: abfd6970 EDX: ee6d77c4 [559353.217494] ESI: c1333338 EDI: f4c91bfc EBP: abfea42e ESP: f4c91ba8 [559353.217494] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 [559353.217494] Process kworker/0:0 (pid: 0, ti=f4c90000 task=f4c6a400 task.ti=f4c8c000) [559353.217494] Stack: [559353.217494] f351c790 00000001 abfd6970 c133333c c1333338 efc6b384 efe2af80 efcd1c04 [559353.217494] f3cc2784 ef3f62c4 f251da80 ef054b40 efcdc300 f0373dc0 f429a144 ef254a80 [559353.217494] ed4e6340 f0705f40 efcdb580 f05261c0 ee6d77c4 f4c91cb4 f351c790 f4c91c78 [559353.217494] Call Trace: [559353.217494] [<c120f068>] ? fib4_rule_action+0x40/0x4d [559353.217494] [<c11d1be3>] ? fib_rules_lookup+0x8d/0xe4 [559353.217494] [<c11e0de9>] ? cleanup_once+0x7a/0x7f [559353.217494] [<c11e0fa9>] ? inet_getpeer+0x1bb/0x1dc [559353.217494] [<c11dc073>] ? nf_ct_attach+0x12/0x13 [559353.217494] [<c1202404>] ? icmp_glue_bits+0x65/0x6a [559353.217494] [<c11e4109>] ? ip_append_data+0x595/0x850 [559353.217494] [<c11e025d>] ? rt_bind_peer+0x1d/0x3d [559353.217494] [<c11e029f>] ? __ip_select_ident+0x22/0xa6 [559353.217494] [<c11e4f60>] ? ip_push_pending_frames+0x206/0x2cb [559353.217494] [<c120301b>] ? icmp_send+0x4fe/0x523 [559353.217494] [<f81a6b09>] ? ____nf_conntrack_find+0xfa/0x142 [nf_conntrack] [559353.217494] [<f81a8069>] ? nf_conntrack_in+0x4f3/0x5e3 [nf_conntrack] [559353.217494] [<f815c536>] ? ipt_do_table+0x4bc/0x4eb [ip_tables] [559353.217494] [<c11e2949>] ? ip_forward+0x2ef/0x316 [559353.217494] [<c11e13da>] ? ip_rcv_finish+0x2fa/0x31f [559353.217494] [<c11c1b3c>] ? __netif_receive_skb+0x405/0x42c [559353.217494] [<c11c1a63>] ? __netif_receive_skb+0x32c/0x42c [559353.217494] [<c1047585>] ? ktime_get_real+0x10/0x2d [559353.217494] [<c11c2547>] ? netif_receive_skb+0x5a/0x5f [559353.217494] [<c11c25ff>] ? napi_skb_finish+0x1b/0x30 [559353.217494] [<f8104723>] ? igb_poll+0x649/0x94a [igb] [559353.217494] [<c1007765>] ? sched_clock+0x9/0xd [559353.217494] [<c1030094>] ? wait_consider_task+0x977/0xa91 [559353.217494] [<c104438f>] ? sched_clock_local+0x17/0x13d [559353.217494] [<c11c2b7b>] ? net_rx_action+0x90/0x150 [559353.217494] [<c1031f12>] ? __do_softirq+0x75/0x10e [559353.217494] [<c1031e9d>] ? __do_softirq+0x0/0x10e [559353.217494] <IRQ> [559353.217494] [<c1031df3>] ? irq_exit+0x31/0x64 [559353.217494] [<c1004397>] ? do_IRQ+0x73/0x84 [559353.217494] [<c1003429>] ? common_interrupt+0x29/0x30 [559353.217494] [<c10089b4>] ? mwait_idle+0x4f/0x59 [559353.217494] [<c10021ef>] ? cpu_idle+0x46/0x63 [559353.217494] Code: 24 08 39 cd 75 09 42 3b 54 24 04 7c e9 eb 18 3b 6c 24 08 8d 50 04 0f 42 d0 89 17 83 c7 04 8b 02 3d 04 ff 25 c1 75 bb 39 d8 74 04 <0f> 0b eb fe 8d 6f fc 81 3b 04 ff 25 c1 89 6c 24 08 75 0d 8b 47 [559353.217494] EIP: [<c11e0caa>] unlink_from_pool+0x85/0x14a SS:ESP 0068:f4c91ba8 [559354.302112] ---[ end trace 55cdab910854890a ]--- [559354.316239] Kernel panic - not syncing: Fatal exception in interrupt [559354.335557] Pid: 0, comm: kworker/0:0 Tainted: G D 2.6.38-demyan-1.1demyan #1 [559354.359578] Call Trace: [559354.367198] [<c1231f71>] ? panic+0x4d/0x137 [559354.380274] [<c1005722>] ? oops_end+0x8e/0x99 [559354.393871] [<c1003a0e>] ? do_invalid_op+0x0/0x75 [559354.408509] [<c1003a7a>] ? do_invalid_op+0x6c/0x75 [559354.423407] [<c11e0caa>] ? unlink_from_pool+0x85/0x14a [559354.439345] [<c120f068>] ? fib4_rule_action+0x40/0x4d [559354.455022] [<c11d1be3>] ? fib_rules_lookup+0x8d/0xe4 [559354.470700] [<c120f122>] ? fib_lookup+0x31/0x3f [559354.484818] [<c11ca4f1>] ? neigh_lookup+0x8e/0x96 [559354.499454] [<c123464e>] ? error_code+0x5a/0x60 [559354.513571] [<c1003a0e>] ? do_invalid_op+0x0/0x75 [559354.528208] [<c11e0caa>] ? unlink_from_pool+0x85/0x14a [559354.544146] [<c120f068>] ? fib4_rule_action+0x40/0x4d [559354.559823] [<c11d1be3>] ? fib_rules_lookup+0x8d/0xe4 [559354.575500] [<c11e0de9>] ? cleanup_once+0x7a/0x7f [559354.590137] [<c11e0fa9>] ? inet_getpeer+0x1bb/0x1dc [559354.605297] [<c11dc073>] ? nf_ct_attach+0x12/0x13 [559354.619934] [<c1202404>] ? icmp_glue_bits+0x65/0x6a [559354.635090] [<c11e4109>] ? ip_append_data+0x595/0x850 [559354.650767] [<c11e025d>] ? rt_bind_peer+0x1d/0x3d [559354.665405] [<c11e029f>] ? __ip_select_ident+0x22/0xa6 [559354.681344] [<c11e4f60>] ? ip_push_pending_frames+0x206/0x2cb [559354.699099] [<c120301b>] ? icmp_send+0x4fe/0x523 [559354.713479] [<f81a6b09>] ? ____nf_conntrack_find+0xfa/0x142 [nf_conntrack] [559354.734615] [<f81a8069>] ? nf_conntrack_in+0x4f3/0x5e3 [nf_conntrack] [559354.754452] [<f815c536>] ? ipt_do_table+0x4bc/0x4eb [ip_tables] [559354.772731] [<c11e2949>] ? ip_forward+0x2ef/0x316 [559354.787366] [<c11e13da>] ? ip_rcv_finish+0x2fa/0x31f [559354.802785] [<c11c1b3c>] ? __netif_receive_skb+0x405/0x42c [559354.819762] [<c11c1a63>] ? __netif_receive_skb+0x32c/0x42c [559354.836738] [<c1047585>] ? ktime_get_real+0x10/0x2d [559354.851901] [<c11c2547>] ? netif_receive_skb+0x5a/0x5f [559354.867835] [<c11c25ff>] ? napi_skb_finish+0x1b/0x30 [559354.883254] [<f8104723>] ? igb_poll+0x649/0x94a [igb] [559354.898930] [<c1007765>] ? sched_clock+0x9/0xd [559354.912786] [<c1030094>] ? wait_consider_task+0x977/0xa91 [559354.929502] [<c104438f>] ? sched_clock_local+0x17/0x13d [559354.945701] [<c11c2b7b>] ? net_rx_action+0x90/0x150 [559354.960857] [<c1031f12>] ? __do_softirq+0x75/0x10e [559354.975756] [<c1031e9d>] ? __do_softirq+0x0/0x10e [559354.990393] <IRQ> [<c1031df3>] ? irq_exit+0x31/0x64 [559355.005862] [<c1004397>] ? do_IRQ+0x73/0x84 [559355.018941] [<c1003429>] ? common_interrupt+0x29/0x30 [559355.034618] [<c10089b4>] ? mwait_idle+0x4f/0x59 [559355.048734] [<c10021ef>] ? cpu_idle+0x46/0x63 [559355.062333] Rebooting in 5 seconds.. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Bug 32772] New: PROBLEM: kernel BUG at net/ipv4/inetpeer.c:386 2011-04-13 20:14 ` Dmitry Novikov @ 2011-04-13 20:24 ` David Miller 2011-04-14 13:02 ` Dmitry Novikov 0 siblings, 1 reply; 9+ messages in thread From: David Miller @ 2011-04-13 20:24 UTC (permalink / raw) To: dimetrios; +Cc: eric.dumazet, shemminger, netdev From: Dmitry Novikov <dimetrios@gmail.com> Date: Wed, 13 Apr 2011 23:14:03 +0300 > Crash again after 7 days of uptime. slub_nomerge is set Looks like too deep stack, try this patch which is in net-2.6: -------------------- inetpeer: reduce stack usage On 64bit arches, we use 752 bytes of stack when cleanup_once() is called from inet_getpeer(). Lets share the avl stack to save ~376 bytes. Before patch : # objdump -d net/ipv4/inetpeer.o | scripts/checkstack.pl 0x000006c3 unlink_from_pool [inetpeer.o]: 376 0x00000721 unlink_from_pool [inetpeer.o]: 376 0x00000cb1 inet_getpeer [inetpeer.o]: 376 0x00000e6d inet_getpeer [inetpeer.o]: 376 0x0004 inet_initpeers [inetpeer.o]: 112 # size net/ipv4/inetpeer.o text data bss dec hex filename 5320 432 21 5773 168d net/ipv4/inetpeer.o After patch : objdump -d net/ipv4/inetpeer.o | scripts/checkstack.pl 0x00000c11 inet_getpeer [inetpeer.o]: 376 0x00000dcd inet_getpeer [inetpeer.o]: 376 0x00000ab9 peer_check_expire [inetpeer.o]: 328 0x00000b7f peer_check_expire [inetpeer.o]: 328 0x0004 inet_initpeers [inetpeer.o]: 112 # size net/ipv4/inetpeer.o text data bss dec hex filename 5163 432 21 5616 15f0 net/ipv4/inetpeer.o Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com> Cc: Scot Doyle <lkml@scotdoyle.com> Cc: Stephen Hemminger <shemminger@vyatta.com> Cc: Hiroaki SHIMODA <shimoda.hiroaki@gmail.com> Reviewed-by: Hiroaki SHIMODA <shimoda.hiroaki@gmail.com> Signed-off-by: David S. Miller <davem@davemloft.net> --- net/ipv4/inetpeer.c | 13 +++++++------ 1 files changed, 7 insertions(+), 6 deletions(-) diff --git a/net/ipv4/inetpeer.c b/net/ipv4/inetpeer.c index dd1b20e..9df4e63 100644 --- a/net/ipv4/inetpeer.c +++ b/net/ipv4/inetpeer.c @@ -354,7 +354,8 @@ static void inetpeer_free_rcu(struct rcu_head *head) } /* May be called with local BH enabled. */ -static void unlink_from_pool(struct inet_peer *p, struct inet_peer_base *base) +static void unlink_from_pool(struct inet_peer *p, struct inet_peer_base *base, + struct inet_peer __rcu **stack[PEER_MAXDEPTH]) { int do_free; @@ -368,7 +369,6 @@ static void unlink_from_pool(struct inet_peer *p, struct inet_peer_base *base) * We use refcnt=-1 to alert lockless readers this entry is deleted. */ if (atomic_cmpxchg(&p->refcnt, 1, -1) == 1) { - struct inet_peer __rcu **stack[PEER_MAXDEPTH]; struct inet_peer __rcu ***stackptr, ***delp; if (lookup(&p->daddr, stack, base) != p) BUG(); @@ -422,7 +422,7 @@ static struct inet_peer_base *peer_to_base(struct inet_peer *p) } /* May be called with local BH enabled. */ -static int cleanup_once(unsigned long ttl) +static int cleanup_once(unsigned long ttl, struct inet_peer __rcu **stack[PEER_MAXDEPTH]) { struct inet_peer *p = NULL; @@ -454,7 +454,7 @@ static int cleanup_once(unsigned long ttl) * happen because of entry limits in route cache. */ return -1; - unlink_from_pool(p, peer_to_base(p)); + unlink_from_pool(p, peer_to_base(p), stack); return 0; } @@ -524,7 +524,7 @@ struct inet_peer *inet_getpeer(struct inetpeer_addr *daddr, int create) if (base->total >= inet_peer_threshold) /* Remove one less-recently-used entry. */ - cleanup_once(0); + cleanup_once(0, stack); return p; } @@ -540,6 +540,7 @@ static void peer_check_expire(unsigned long dummy) { unsigned long now = jiffies; int ttl, total; + struct inet_peer __rcu **stack[PEER_MAXDEPTH]; total = compute_total(); if (total >= inet_peer_threshold) @@ -548,7 +549,7 @@ static void peer_check_expire(unsigned long dummy) ttl = inet_peer_maxttl - (inet_peer_maxttl - inet_peer_minttl) / HZ * total / inet_peer_threshold * HZ; - while (!cleanup_once(ttl)) { + while (!cleanup_once(ttl, stack)) { if (jiffies != now) break; } -- 1.7.4.3 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [Bug 32772] New: PROBLEM: kernel BUG at net/ipv4/inetpeer.c:386 2011-04-13 20:24 ` David Miller @ 2011-04-14 13:02 ` Dmitry Novikov 2011-04-14 13:32 ` Eric Dumazet 0 siblings, 1 reply; 9+ messages in thread From: Dmitry Novikov @ 2011-04-14 13:02 UTC (permalink / raw) To: David Miller; +Cc: eric.dumazet, shemminger, netdev Thanks. Patch applied . Will wait results 2011/4/13 David Miller <davem@davemloft.net>: > From: Dmitry Novikov <dimetrios@gmail.com> > Date: Wed, 13 Apr 2011 23:14:03 +0300 > >> Crash again after 7 days of uptime. slub_nomerge is set > > Looks like too deep stack, try this patch which is in net-2.6: > > -------------------- > inetpeer: reduce stack usage > > On 64bit arches, we use 752 bytes of stack when cleanup_once() is called > from inet_getpeer(). > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Bug 32772] New: PROBLEM: kernel BUG at net/ipv4/inetpeer.c:386 2011-04-14 13:02 ` Dmitry Novikov @ 2011-04-14 13:32 ` Eric Dumazet 0 siblings, 0 replies; 9+ messages in thread From: Eric Dumazet @ 2011-04-14 13:32 UTC (permalink / raw) To: Dmitry Novikov; +Cc: David Miller, shemminger, netdev Le jeudi 14 avril 2011 à 16:02 +0300, Dmitry Novikov a écrit : > Thanks. Patch applied . Will wait results Thanks to you Dmitry, for your patience ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2011-04-14 13:32 UTC | newest] Thread overview: 9+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2011-04-06 15:18 Fw: [Bug 32772] New: PROBLEM: kernel BUG at net/ipv4/inetpeer.c:386 Stephen Hemminger 2011-04-06 16:05 ` Eric Dumazet 2011-04-06 16:25 ` Eric Dumazet 2011-04-06 18:15 ` Dmitry Novikov 2011-04-06 18:16 ` David Miller 2011-04-13 20:14 ` Dmitry Novikov 2011-04-13 20:24 ` David Miller 2011-04-14 13:02 ` Dmitry Novikov 2011-04-14 13:32 ` Eric Dumazet
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).