* kernel panic when using netns+bridges+tc(netem)
@ 2010-05-06 1:01 Martín Ferrari
2010-05-06 6:40 ` Eric Dumazet
0 siblings, 1 reply; 7+ messages in thread
From: Martín Ferrari @ 2010-05-06 1:01 UTC (permalink / raw)
To: netdev; +Cc: Mathieu Lacage
Hi there,
While working on my project that uses netns, I found another bug. This
one causes a "Kernel panic - not syncing: Fatal exception in
interrupt", and I can reproduce it in 2.6.33 and 2.6.34-rc5, but not
in 2.6.32. It dies during a call to __free_skb.
I tested this on my x86_64 laptop (2 cores) and on qemu. In qemu it
was not triggered until I asked it to emulate 2 cpus instead of one,
so it is probably a SMP-only issue.
Scenario:
I set up a number of network namespaces, each with two veths to netns
1. In the main namespace I take those veths and bridge them in pairs,
to configure a linear topology; also I configure the netem qdisc to
simulate link delay.
Once the network is set up, I run a client/server program to send UDP
packets from one end of the topology to the other. After a few seconds
of sending packets (not really deterministic) it panics.
Note that I didn't experience this problem when using only 2
namespaces (so, no routing)
below the dumps. These all come from the qemu, as I couldn't use
netconsole in the network at work, but I checked and the backtraces
were essentially the same
First, two panics with 2.6.34, each one with a slightly different backtrace
[ 65.272131] ------------[ cut here ]------------
[ 65.272135] kernel BUG at mm/slub.c:2846!
[ 65.272135] invalid opcode: 0000 [#1] SMP
[ 65.272135] last sysfs file: /sys/devices/virtual/net/lo/operstate
[ 65.272135] CPU 1
[ 65.272135] Modules linked in: sch_netem veth bridge stp netconsole
configfs loop parport_pc parport evdev tpm_tis tpm snd_pcm tpm_bios
snd_timer snd soundcore snd_page_alloc pcspkr psmouse serio_raw
i2c_piix4 button i2c_core processor ext3 jbd mbcache ide_cd_mod cdrom
ide_gd_mod ata_generic ata_piix libata 8139too scsi_mod floppy piix
8139cp mii ide_core thermal thermal_sys [last unloaded: configfs]
[ 65.272135]
[ 65.272135] Pid: 1518, comm: udp-perf Not tainted 2.6.34-rc5 #1 /
[ 65.272135] RIP: 0010:[<ffffffff810e0d6b>] [<ffffffff810e0d6b>]
kfree+0x55/0xc6
[ 65.272135] RSP: 0018:ffff880001a23d90 EFLAGS: 00010246
[ 65.272135] RAX: 0100000000000000 RBX: ffff88007d6bc600 RCX: 0000000000012850
[ 65.272135] RDX: ffff88007d6bc600 RSI: 000000000000000e RDI: ffffea0001b6f610
[ 65.272135] RBP: ffff88007d6ae200 R08: ffff88007d6bc600 R09: ffffffffa0280690
[ 65.272135] R10: 0000000000000002 R11: ffff88007d6bc500 R12: ffffffff8123a77f
[ 65.272135] R13: 000000000000002b R14: ffff88007d39b600 R15: ffff88007d6bc600
[ 65.272135] FS: 00007f637c9dd6f0(0000) GS:ffff880001a20000(0000)
knlGS:0000000000000000
[ 65.272135] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 65.272135] CR2: 00000000009deaa0 CR3: 000000007d82d000 CR4: 00000000000006e0
[ 65.272135] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 65.272135] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 65.272135] Process udp-perf (pid: 1518, threadinfo
ffff88007d92a000, task ffff88007e7ff810)
[ 65.272135] Stack:
[ 65.272135] ffff88007d6bc600 ffff88007d6bc600 0000000000000246
ffffffff8123a77f
[ 65.272135] <0> ffff880001a32860 ffffffff81241e01 ffffe8ffff623280
ffffe8ffff83ffc0
[ 65.272135] <0> ffff88007d6bc600 ffffffffa028057d 000000027d4c06c8
ffff88007d4c0600
[ 65.272135] Call Trace:
[ 65.272135] <IRQ>
[ 65.272135] [<ffffffff8123a77f>] ? __kfree_skb+0x11/0x7d
[ 65.272135] [<ffffffff81241e01>] ? netif_rx+0xe2/0xee
[ 65.272135] [<ffffffffa028057d>] ? veth_xmit+0x6e/0xad [veth]
[ 65.272135] [<ffffffff8124301f>] ? dev_hard_start_xmit+0x221/0x301
[ 65.272135] [<ffffffff81256d9a>] ? sch_direct_xmit+0x5b/0x15d
[ 65.272135] [<ffffffff81256f55>] ? __qdisc_run+0xb9/0xd8
[ 65.272135] [<ffffffff81240511>] ? net_tx_action+0xd6/0x149
[ 65.272135] [<ffffffff8104ba02>] ? __do_softirq+0xdd/0x19f
[ 65.272135] [<ffffffff8101e515>] ? lapic_next_event+0x18/0x1d
[ 65.272135] [<ffffffff810099dc>] ? call_softirq+0x1c/0x30
[ 65.272135] [<ffffffff8100b863>] ? do_softirq+0x3f/0x79
[ 65.272135] [<ffffffff8104b88e>] ? irq_exit+0x36/0x76
[ 65.272135] [<ffffffff8101f000>] ? smp_apic_timer_interrupt+0x86/0x94
[ 65.272135] [<ffffffff81009493>] ? apic_timer_interrupt+0x13/0x20
[ 65.272135] <EOI>
[ 65.272135] [<ffffffff8114c1ea>] ? cap_sk_getsecid+0x0/0x1
[ 65.272135] [<ffffffff812a0b0f>] ? __xfrm_lookup+0x2/0xb04
[ 65.272135] [<ffffffff81264b8d>] ? ip_route_output_flow+0x77/0x1cc
[ 65.272135] [<ffffffff812887c6>] ? udp_sendmsg+0x32d/0x5f3
[ 65.272135] [<ffffffff81009400>] ? irq_entries_start+0x3c0/0x400
[ 65.272135] [<ffffffff8128e112>] ? inet_sendmsg+0x53/0x58
[ 65.272135] [<ffffffff8123388d>] ? sock_sendmsg+0x83/0x9b
[ 65.272135] [<ffffffff8103a042>] ? pick_next_task_fair+0xca/0xd6
[ 65.272135] [<ffffffff812ef74a>] ? schedule+0x52b/0x593
[ 65.272135] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
[ 65.272135] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
[ 65.272135] [<ffffffff81185d4b>] ? _copy_from_user+0x1b/0x30
[ 65.272135] [<ffffffff8100765f>] ? __switch_to+0x1b5/0x2a6
[ 65.272135] [<ffffffff81232227>] ? copy_from_user+0x13/0x25
[ 65.272135] [<ffffffff812353cb>] ? sys_sendto+0xd7/0x117
[ 65.272135] [<ffffffff8103f7b1>] ? finish_task_switch+0x34/0xa1
[ 65.272135] [<ffffffff812ef74a>] ? schedule+0x52b/0x593
[ 65.272135] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
[ 65.272135] [<ffffffff81008ac2>] ? system_call_fastpath+0x16/0x1b
[ 65.272135] Code: 83 c3 08 48 83 3b 00 eb ec 48 83 fd 10 0f 86 84
00 00 00 48 89 ef e8 f4 e2 ff ff 48 89 c7 48 8b 00 84 c0 78 13 66 a9
00 c0 75 04 <0f> 0b eb fe 5b 5d 41 5c e9 fb 57 fd ff 48 8b 4c 24 18 4c
8b 47
[ 65.272135] RIP [<ffffffff810e0d6b>] kfree+0x55/0xc6
[ 65.272135] RSP <ffff880001a23d90>
[ 65.385803] ---[ end trace 42d2fb5b94980ab5 ]---
[ 65.386337] Kernel panic - not syncing: Fatal exception in interrupt
[ 65.386943] Pid: 1518, comm: udp-perf Tainted: G D 2.6.34-rc5 #1
[ 65.387557] Call Trace:
[ 65.388011] <IRQ> [<ffffffff812eef9b>] ? panic+0x77/0xf7
[ 65.388729] [<ffffffff81046b88>] ? kmsg_dump+0xa6/0x13e
[ 65.389292] [<ffffffff8100c8c2>] ? oops_end+0xa7/0xb4
[ 65.389871] [<ffffffff8123a77f>] ? __kfree_skb+0x11/0x7d
[ 65.390441] [<ffffffff8100a695>] ? do_invalid_op+0x8b/0x95
[ 65.391032] [<ffffffff810e0d6b>] ? kfree+0x55/0xc6
[ 65.391587] [<ffffffffa026f301>] ?
br_nf_pre_routing_finish+0x0/0x25e [bridge]
[ 65.392512] [<ffffffffa026f301>] ?
br_nf_pre_routing_finish+0x0/0x25e [bridge]
[ 65.393399] [<ffffffff8100975b>] ? invalid_op+0x1b/0x20
[ 65.393978] [<ffffffff8123a77f>] ? __kfree_skb+0x11/0x7d
[ 65.394547] [<ffffffff810e0d6b>] ? kfree+0x55/0xc6
[ 65.395091] [<ffffffff8123a77f>] ? __kfree_skb+0x11/0x7d
[ 65.395668] [<ffffffff81241e01>] ? netif_rx+0xe2/0xee
[ 65.396438] [<ffffffffa028057d>] ? veth_xmit+0x6e/0xad [veth]
[ 65.397026] [<ffffffff8124301f>] ? dev_hard_start_xmit+0x221/0x301
[ 65.397640] [<ffffffff81256d9a>] ? sch_direct_xmit+0x5b/0x15d
[ 65.398222] [<ffffffff81256f55>] ? __qdisc_run+0xb9/0xd8
[ 65.398788] [<ffffffff81240511>] ? net_tx_action+0xd6/0x149
[ 65.399365] [<ffffffff8104ba02>] ? __do_softirq+0xdd/0x19f
[ 65.399935] [<ffffffff8101e515>] ? lapic_next_event+0x18/0x1d
[ 65.400560] [<ffffffff810099dc>] ? call_softirq+0x1c/0x30
[ 65.401125] [<ffffffff8100b863>] ? do_softirq+0x3f/0x79
[ 65.401708] [<ffffffff8104b88e>] ? irq_exit+0x36/0x76
[ 65.402267] [<ffffffff8101f000>] ? smp_apic_timer_interrupt+0x86/0x94
[ 65.402869] [<ffffffff81009493>] ? apic_timer_interrupt+0x13/0x20
[ 65.403463] <EOI> [<ffffffff8114c1ea>] ? cap_sk_getsecid+0x0/0x1
[ 65.404210] [<ffffffff812a0b0f>] ? __xfrm_lookup+0x2/0xb04
[ 65.404779] [<ffffffff81264b8d>] ? ip_route_output_flow+0x77/0x1cc
[ 65.405375] [<ffffffff812887c6>] ? udp_sendmsg+0x32d/0x5f3
[ 65.405945] [<ffffffff81009400>] ? irq_entries_start+0x3c0/0x400
[ 65.406534] [<ffffffff8128e112>] ? inet_sendmsg+0x53/0x58
[ 65.407102] [<ffffffff8123388d>] ? sock_sendmsg+0x83/0x9b
[ 65.407674] [<ffffffff8103a042>] ? pick_next_task_fair+0xca/0xd6
[ 65.408296] [<ffffffff812ef74a>] ? schedule+0x52b/0x593
[ 65.408854] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
[ 65.409468] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
[ 65.410084] [<ffffffff81185d4b>] ? _copy_from_user+0x1b/0x30
[ 65.410667] [<ffffffff8100765f>] ? __switch_to+0x1b5/0x2a6
[ 65.510031] [<ffffffff81232227>] ? copy_from_user+0x13/0x25
[ 65.510818] [<ffffffff812353cb>] ? sys_sendto+0xd7/0x117
[ 65.511608] [<ffffffff8103f7b1>] ? finish_task_switch+0x34/0xa1
[ 65.512465] [<ffffffff812ef74a>] ? schedule+0x52b/0x593
[ 65.513229] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
[ 65.514040] [<ffffffff81008ac2>] ? system_call_fastpath+0x16/0x1b
[ 1438.042568] ------------[ cut here ]------------
[ 1438.043170] kernel BUG at mm/slub.c:2846!
[ 1438.043711] invalid opcode: 0000 [#1] SMP
[ 1438.044531] last sysfs file: /sys/devices/virtual/net/lo/operstate
[ 1438.045148] CPU 1
[ 1438.045350] Modules linked in: sch_netem veth bridge stp netconsole
configfs loop parport_pc tpm_tis tpm snd_pcm psmouse parport tpm_bios
snd_timer snd soundcore snd_page_alloc evdev pcspkr serio_raw
i2c_piix4 i2c_core button processor ext3 jbd mbcache ide_cd_mod cdrom
ide_gd_mod ata_generic ata_piix 8139too libata scsi_mod floppy 8139cp
mii thermal thermal_sys piix ide_core [last unloaded: scsi_wait_scan]
[ 1438.046215]
[ 1438.046215] Pid: 1476, comm: udp-perf Not tainted 2.6.34-rc5 #1 /
[ 1438.046215] RIP: 0010:[<ffffffff810e0d6b>] [<ffffffff810e0d6b>]
kfree+0x55/0xc6
[ 1438.046215] RSP: 0018:ffff880001a23d90 EFLAGS: 00010246
[ 1438.046215] RAX: 0100000000000000 RBX: ffff88007d882200 RCX: 0000000000012850
[ 1438.046215] RDX: ffff88007d882200 RSI: 000000000000000e RDI: ffffea0001b972a0
[ 1438.046215] RBP: ffff88007e20c000 R08: ffff88007d882200 R09: ffffffffa026c690
[ 1438.046215] R10: 0000000000000002 R11: ffff88007d882100 R12: ffffffff8123a77f
[ 1438.046215] R13: 0000000000000032 R14: ffff8800378e0700 R15: ffff88007d882200
[ 1438.046215] FS: 00007f1c1d07f6f0(0000) GS:ffff880001a20000(0000)
knlGS:0000000000000000
[ 1438.046215] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1438.046215] CR2: 00007f6ea9e07310 CR3: 0000000037834000 CR4: 00000000000006e0
[ 1438.046215] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1438.046215] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1438.046215] Process udp-perf (pid: 1476, threadinfo
ffff88007db0e000, task ffff8800379a2350)
[ 1438.046215] Stack:
[ 1438.046215] ffff88007d882200 ffff88007d882200 0000000000000246
ffffffff8123a77f
[ 1438.046215] <0> ffff880001a32860 ffffffff81241e01 ffffe8ffffa3f430
ffffe8ffffa3d180
[ 1438.046215] <0> ffff88007d882200 ffffffffa026c57d 0000000201a30500
ffff88007d171200
[ 1438.046215] Call Trace:
[ 1438.046215] <IRQ>
[ 1438.046215] [<ffffffff8123a77f>] ? __kfree_skb+0x11/0x7d
[ 1438.046215] [<ffffffff81241e01>] ? netif_rx+0xe2/0xee
[ 1438.046215] [<ffffffffa026c57d>] ? veth_xmit+0x6e/0xad [veth]
[ 1438.046215] [<ffffffff8124301f>] ? dev_hard_start_xmit+0x221/0x301
[ 1438.046215] [<ffffffff81256d9a>] ? sch_direct_xmit+0x5b/0x15d
[ 1438.046215] [<ffffffff81256f55>] ? __qdisc_run+0xb9/0xd8
[ 1438.046215] [<ffffffff81240511>] ? net_tx_action+0xd6/0x149
[ 1438.046215] [<ffffffff8104ba02>] ? __do_softirq+0xdd/0x19f
[ 1438.046215] [<ffffffff8101e515>] ? lapic_next_event+0x18/0x1d
[ 1438.046215] [<ffffffff810099dc>] ? call_softirq+0x1c/0x30
[ 1438.046215] [<ffffffff8100b863>] ? do_softirq+0x3f/0x79
[ 1438.046215] [<ffffffff8104b88e>] ? irq_exit+0x36/0x76
[ 1438.046215] [<ffffffff8101f000>] ? smp_apic_timer_interrupt+0x86/0x94
[ 1438.046215] [<ffffffff81009493>] ? apic_timer_interrupt+0x13/0x20
[ 1438.046215] <EOI>
[ 1438.046215] [<ffffffff81288499>] ? udp_sendmsg+0x0/0x5f3
[ 1438.046215] [<ffffffff8128850f>] ? udp_sendmsg+0x76/0x5f3
[ 1438.046215] [<ffffffff812889a7>] ? udp_sendmsg+0x50e/0x5f3
[ 1438.046215] [<ffffffff8123388d>] ? sock_sendmsg+0x83/0x9b
[ 1438.046215] [<ffffffff8103a042>] ? pick_next_task_fair+0xca/0xd6
[ 1438.046215] [<ffffffff812ef74a>] ? schedule+0x52b/0x593
[ 1438.046215] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
[ 1438.046215] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
[ 1438.046215] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
[ 1438.046215] [<ffffffff810e93ed>] ? fget_light+0x0/0xa1
[ 1438.046215] [<ffffffff81185e40>] ? copy_user_generic_string+0x30/0x40
[ 1438.046215] [<ffffffff81232227>] ? copy_from_user+0x13/0x25
[ 1438.046215] [<ffffffff812353cb>] ? sys_sendto+0xd7/0x117
[ 1438.046215] [<ffffffff8103f7b1>] ? finish_task_switch+0x34/0xa1
[ 1438.046215] [<ffffffff812ef74a>] ? schedule+0x52b/0x593
[ 1438.046215] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
[ 1438.046215] [<ffffffff81008ac2>] ? system_call_fastpath+0x16/0x1b
[ 1438.046215] Code: 83 c3 08 48 83 3b 00 eb ec 48 83 fd 10 0f 86 84
00 00 00 48 89 ef e8 f4 e2 ff ff 48 89 c7 48 8b 00 84 c0 78 13 66 a9
00 c0 75 04 <0f> 0b eb fe 5b 5d 41 5c e9 fb 57 fd ff 48 8b 4c 24 18 4c
8b 47
[ 1438.046215] RIP [<ffffffff810e0d6b>] kfree+0x55/0xc6
[ 1438.046215] RSP <ffff880001a23d90>
[ 1438.102706] ---[ end trace ab36062dcf233d6a ]---
[ 1438.103251] Kernel panic - not syncing: Fatal exception in interrupt
[ 1438.103912] Pid: 1476, comm: udp-perf Tainted: G D 2.6.34-rc5 #1
[ 1438.104563] Call Trace:
[ 1438.105017] <IRQ> [<ffffffff812eef9b>] ? panic+0x77/0xf7
[ 1438.105718] [<ffffffff81046b88>] ? kmsg_dump+0xa6/0x13e
[ 1438.106293] [<ffffffff8100c8c2>] ? oops_end+0xa7/0xb4
[ 1438.106866] [<ffffffff8123a77f>] ? __kfree_skb+0x11/0x7d
[ 1438.107451] [<ffffffff8100a695>] ? do_invalid_op+0x8b/0x95
[ 1438.108036] [<ffffffff810e0d6b>] ? kfree+0x55/0xc6
[ 1438.108799] [<ffffffffa025b301>] ?
br_nf_pre_routing_finish+0x0/0x25e [bridge]
[ 1438.109699] [<ffffffffa025b301>] ?
br_nf_pre_routing_finish+0x0/0x25e [bridge]
[ 1438.110604] [<ffffffff8100975b>] ? invalid_op+0x1b/0x20
[ 1438.111172] [<ffffffff8123a77f>] ? __kfree_skb+0x11/0x7d
[ 1438.111753] [<ffffffff810e0d6b>] ? kfree+0x55/0xc6
[ 1438.112359] [<ffffffff8123a77f>] ? __kfree_skb+0x11/0x7d
[ 1438.112952] [<ffffffff81241e01>] ? netif_rx+0xe2/0xee
[ 1438.113527] [<ffffffffa026c57d>] ? veth_xmit+0x6e/0xad [veth]
[ 1438.114123] [<ffffffff8124301f>] ? dev_hard_start_xmit+0x221/0x301
[ 1438.114734] [<ffffffff81256d9a>] ? sch_direct_xmit+0x5b/0x15d
[ 1438.115334] [<ffffffff81256f55>] ? __qdisc_run+0xb9/0xd8
[ 1438.115912] [<ffffffff81240511>] ? net_tx_action+0xd6/0x149
[ 1438.116544] [<ffffffff8104ba02>] ? __do_softirq+0xdd/0x19f
[ 1438.117128] [<ffffffff8101e515>] ? lapic_next_event+0x18/0x1d
[ 1438.117732] [<ffffffff810099dc>] ? call_softirq+0x1c/0x30
[ 1438.118420] [<ffffffff8100b863>] ? do_softirq+0x3f/0x79
[ 1438.119066] [<ffffffff8104b88e>] ? irq_exit+0x36/0x76
[ 1438.147879] [<ffffffff8101f000>] ? smp_apic_timer_interrupt+0x86/0x94
[ 1438.148615] [<ffffffff81009493>] ? apic_timer_interrupt+0x13/0x20
[ 1438.149294] <EOI> [<ffffffff81288499>] ? udp_sendmsg+0x0/0x5f3
[ 1438.150101] [<ffffffff8128850f>] ? udp_sendmsg+0x76/0x5f3
[ 1438.150681] [<ffffffff812889a7>] ? udp_sendmsg+0x50e/0x5f3
[ 1438.151276] [<ffffffff8123388d>] ? sock_sendmsg+0x83/0x9b
[ 1438.151853] [<ffffffff8103a042>] ? pick_next_task_fair+0xca/0xd6
[ 1438.152508] [<ffffffff812ef74a>] ? schedule+0x52b/0x593
[ 1438.153078] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
[ 1438.153687] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
[ 1438.154295] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
[ 1438.154902] [<ffffffff810e93ed>] ? fget_light+0x0/0xa1
[ 1438.155477] [<ffffffff81185e40>] ? copy_user_generic_string+0x30/0x40
[ 1438.156097] [<ffffffff81232227>] ? copy_from_user+0x13/0x25
[ 1438.156720] [<ffffffff812353cb>] ? sys_sendto+0xd7/0x117
[ 1438.157376] [<ffffffff8103f7b1>] ? finish_task_switch+0x34/0xa1
[ 1438.157970] [<ffffffff812ef74a>] ? schedule+0x52b/0x593
[ 1438.158557] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
[ 1438.159153] [<ffffffff81008ac2>] ? system_call_fastpath+0x16/0x1b
------------
Finally, a panic in 2.6.33. Note that the line in wioch BUG is
triggered is different
[ 102.442815] ------------[ cut here ]------------
[ 102.443433] kernel BUG at
/build/mattems-linux-2.6_2.6.33-1~experimental.4-amd64-ieqSsa/linux-2.6-2.6.33-1~experimental.4/debian/build/source_amd64_none/mm/slub.c:2969!
[ 102.444874] invalid opcode: 0000 [#1] SMP
[ 102.444958] last sysfs file: /sys/devices/virtual/net/lo/operstate
[ 102.444958] CPU 0
[ 102.444958] Pid: 4, comm: ksoftirqd/0 Not tainted 2.6.33-2-amd64 #1 /
[ 102.444958] RIP: 0010:[<ffffffff810e1e2c>] [<ffffffff810e1e2c>]
kfree+0x55/0xcb
[ 102.444958] RSP: 0018:ffff880001a03df8 EFLAGS: 00010246
[ 102.444958] RAX: 0100000000000000 RBX: ffff88007e439000 RCX: 0000000000012d70
[ 102.444958] RDX: 000000000000006a RSI: ffffea0001b76a70 RDI: ffffea0000c1a328
[ 102.444958] RBP: ffff880037533c00 R08: ffff88007e21e500 R09: ffffffff8162bbe0
[ 102.444958] R10: 000000037e439e00 R11: ffff88007e439e00 R12: ffffffff81239cf2
[ 102.444958] R13: 000000000000006a R14: ffff88007efb3000 R15: ffff88007e21e500
[ 102.444958] FS: 0000000000000000(0000) GS:ffff880001a00000(0000)
knlGS:0000000000000000
[ 102.444958] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 102.444958] CR2: 00007fc8e25cd0f0 CR3: 000000007d088000 CR4: 00000000000006f0
[ 102.444958] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 102.444958] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 102.444958] Process ksoftirqd/0 (pid: 4, threadinfo
ffff88007fb78000, task ffff88007fb61500)
[ 102.444958] Stack:
[ 102.444958] ffff88007e439000 ffff88007e439000 0000000000000246
ffffffff81239cf2
[ 102.444958] <0> ffff880001a12d80 ffffffff8124125d ffffe8ffff818190
ffffe8ffff815ee0
[ 102.444958] <0> ffff88007e439000 ffffffffa025254b ffff880001a10010
ffff88007ef95200
[ 102.444958] Call Trace:
[ 102.444958] <IRQ>
[ 102.444958] [<ffffffff81239cf2>] ? __kfree_skb+0x11/0x7d
[ 102.444958] [<ffffffff8124125d>] ? netif_rx+0xe2/0xee
[ 102.444958] [<ffffffffa025254b>] ? veth_xmit+0x6e/0xad [veth]
[ 102.444958] [<ffffffff8124243a>] ? dev_hard_start_xmit+0x221/0x2dc
[ 102.444958] [<ffffffff81255514>] ? sch_direct_xmit+0x5b/0x15d
[ 102.444958] [<ffffffff812556cf>] ? __qdisc_run+0xb9/0xda
[ 102.444958] [<ffffffff8123fa99>] ? net_tx_action+0xd6/0x149
[ 102.444958] [<ffffffff8104c918>] ? __do_softirq+0xdd/0x1a1
[ 102.444958] [<ffffffff810099dc>] ? call_softirq+0x1c/0x30
[ 102.444958] <EOI>
[ 102.444958] [<ffffffff8100b85f>] ? do_softirq+0x3f/0x79
[ 102.444958] [<ffffffff8104c504>] ? run_ksoftirqd+0x6a/0x118
[ 102.444958] [<ffffffff8104c49a>] ? run_ksoftirqd+0x0/0x118
[ 102.444958] [<ffffffff8105ee99>] ? kthread+0x79/0x81
[ 102.444958] [<ffffffff810098e4>] ? kernel_thread_helper+0x4/0x10
[ 102.444958] [<ffffffff8105ee20>] ? kthread+0x0/0x81
[ 102.444958] [<ffffffff810098e0>] ? kernel_thread_helper+0x0/0x10
[ 102.444958] Code: 83 c3 08 48 83 3b 00 eb ec 48 83 fd 10 0f 86 89
00 00 00 48 89 ef e8 f0 e6 ff ff 48 89 c7 48 8b 00 84 c0 78 13 66 a9
00 c0 75 04 <0f> 0b eb fe 5b 5d 41 5c e9 45 5a fd ff 48 8b 4c 24 18 4c
8b 4f
[ 102.444958] RIP [<ffffffff810e1e2c>] kfree+0x55/0xcb
[ 102.444958] RSP <ffff880001a03df8>
[ 102.484000] ---[ end trace b1157390d40df1cb ]---
[ 102.485018] Kernel panic - not syncing: Fatal exception in interrupt
[ 102.485647] Pid: 4, comm: ksoftirqd/0 Tainted: G D 2.6.33-2-amd64 #1
[ 102.486630] Call Trace:
[ 102.487112] <IRQ> [<ffffffff812ec605>] ? panic+0x86/0x14b
[ 102.487870] [<ffffffff8104c799>] ? irq_exit+0x48/0x76
[ 102.488474] [<ffffffff812ee893>] ? ret_from_intr+0x0/0x11
[ 102.489068] [<ffffffff810478c0>] ? kmsg_dump+0xa6/0x13e
[ 102.489661] [<ffffffff8100c89a>] ? oops_end+0xa7/0xb4
[ 102.490245] [<ffffffff81239cf2>] ? __kfree_skb+0x11/0x7d
[ 102.490836] [<ffffffff8100a690>] ? do_invalid_op+0x8b/0x95
[ 102.491436] [<ffffffff810e1e2c>] ? kfree+0x55/0xcb
[ 102.492020] [<ffffffffa0244aee>] ?
br_nf_pre_routing_finish+0x284/0x2a6 [bridge]
[ 102.492942] [<ffffffffa024486a>] ?
br_nf_pre_routing_finish+0x0/0x2a6 [bridge]
[ 102.493857] [<ffffffff8125f204>] ? nf_hook_slow+0x62/0xc3
[ 102.523082] [<ffffffffa024486a>] ?
br_nf_pre_routing_finish+0x0/0x2a6 [bridge]
[ 102.524008] [<ffffffff8100975b>] ? invalid_op+0x1b/0x20
[ 102.524631] [<ffffffff81239cf2>] ? __kfree_skb+0x11/0x7d
[ 102.525225] [<ffffffff810e1e2c>] ? kfree+0x55/0xcb
[ 102.525795] [<ffffffff810e1e1c>] ? kfree+0x45/0xcb
[ 102.526402] [<ffffffff81239cf2>] ? __kfree_skb+0x11/0x7d
[ 102.526992] [<ffffffff8124125d>] ? netif_rx+0xe2/0xee
[ 102.527576] [<ffffffffa025254b>] ? veth_xmit+0x6e/0xad [veth]
[ 102.528199] [<ffffffff8124243a>] ? dev_hard_start_xmit+0x221/0x2dc
[ 102.528821] [<ffffffff81255514>] ? sch_direct_xmit+0x5b/0x15d
[ 102.529429] [<ffffffff812556cf>] ? __qdisc_run+0xb9/0xda
[ 102.530018] [<ffffffff8123fa99>] ? net_tx_action+0xd6/0x149
[ 102.530618] [<ffffffff8104c918>] ? __do_softirq+0xdd/0x1a1
[ 102.531214] [<ffffffff810099dc>] ? call_softirq+0x1c/0x30
[ 102.531804] <EOI> [<ffffffff8100b85f>] ? do_softirq+0x3f/0x79
[ 102.532572] [<ffffffff8104c504>] ? run_ksoftirqd+0x6a/0x118
[ 102.533169] [<ffffffff8104c49a>] ? run_ksoftirqd+0x0/0x118
[ 102.533764] [<ffffffff8105ee99>] ? kthread+0x79/0x81
[ 102.534340] [<ffffffff810098e4>] ? kernel_thread_helper+0x4/0x10
[ 102.534954] [<ffffffff8105ee20>] ? kthread+0x0/0x81
[ 102.535526] [<ffffffff810098e0>] ? kernel_thread_helper+0x0/0x10
--
Martín Ferrari
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: kernel panic when using netns+bridges+tc(netem)
2010-05-06 1:01 kernel panic when using netns+bridges+tc(netem) Martín Ferrari
@ 2010-05-06 6:40 ` Eric Dumazet
2010-05-06 6:47 ` Eric Dumazet
0 siblings, 1 reply; 7+ messages in thread
From: Eric Dumazet @ 2010-05-06 6:40 UTC (permalink / raw)
To: Martín Ferrari; +Cc: netdev, Mathieu Lacage, David Miller
Le jeudi 06 mai 2010 à 03:01 +0200, Martín Ferrari a écrit :
> Hi there,
>
> While working on my project that uses netns, I found another bug. This
> one causes a "Kernel panic - not syncing: Fatal exception in
> interrupt", and I can reproduce it in 2.6.33 and 2.6.34-rc5, but not
> in 2.6.32. It dies during a call to __free_skb.
> I tested this on my x86_64 laptop (2 cores) and on qemu. In qemu it
> was not triggered until I asked it to emulate 2 cpus instead of one,
> so it is probably a SMP-only issue.
>
> Scenario:
>
> I set up a number of network namespaces, each with two veths to netns
> 1. In the main namespace I take those veths and bridge them in pairs,
> to configure a linear topology; also I configure the netem qdisc to
> simulate link delay.
>
> Once the network is set up, I run a client/server program to send UDP
> packets from one end of the topology to the other. After a few seconds
> of sending packets (not really deterministic) it panics.
>
> Note that I didn't experience this problem when using only 2
> namespaces (so, no routing)
>
> below the dumps. These all come from the qemu, as I couldn't use
> netconsole in the network at work, but I checked and the backtraces
> were essentially the same
>
> First, two panics with 2.6.34, each one with a slightly different backtrace
>
> [ 65.272131] ------------[ cut here ]------------
> [ 65.272135] kernel BUG at mm/slub.c:2846!
> [ 65.272135] invalid opcode: 0000 [#1] SMP
> [ 65.272135] last sysfs file: /sys/devices/virtual/net/lo/operstate
> [ 65.272135] CPU 1
> [ 65.272135] Modules linked in: sch_netem veth bridge stp netconsole
> configfs loop parport_pc parport evdev tpm_tis tpm snd_pcm tpm_bios
> snd_timer snd soundcore snd_page_alloc pcspkr psmouse serio_raw
> i2c_piix4 button i2c_core processor ext3 jbd mbcache ide_cd_mod cdrom
> ide_gd_mod ata_generic ata_piix libata 8139too scsi_mod floppy piix
> 8139cp mii ide_core thermal thermal_sys [last unloaded: configfs]
> [ 65.272135]
> [ 65.272135] Pid: 1518, comm: udp-perf Not tainted 2.6.34-rc5 #1 /
> [ 65.272135] RIP: 0010:[<ffffffff810e0d6b>] [<ffffffff810e0d6b>]
> kfree+0x55/0xc6
> [ 65.272135] RSP: 0018:ffff880001a23d90 EFLAGS: 00010246
> [ 65.272135] RAX: 0100000000000000 RBX: ffff88007d6bc600 RCX: 0000000000012850
> [ 65.272135] RDX: ffff88007d6bc600 RSI: 000000000000000e RDI: ffffea0001b6f610
> [ 65.272135] RBP: ffff88007d6ae200 R08: ffff88007d6bc600 R09: ffffffffa0280690
> [ 65.272135] R10: 0000000000000002 R11: ffff88007d6bc500 R12: ffffffff8123a77f
> [ 65.272135] R13: 000000000000002b R14: ffff88007d39b600 R15: ffff88007d6bc600
> [ 65.272135] FS: 00007f637c9dd6f0(0000) GS:ffff880001a20000(0000)
> knlGS:0000000000000000
> [ 65.272135] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 65.272135] CR2: 00000000009deaa0 CR3: 000000007d82d000 CR4: 00000000000006e0
> [ 65.272135] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 65.272135] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 65.272135] Process udp-perf (pid: 1518, threadinfo
> ffff88007d92a000, task ffff88007e7ff810)
> [ 65.272135] Stack:
> [ 65.272135] ffff88007d6bc600 ffff88007d6bc600 0000000000000246
> ffffffff8123a77f
> [ 65.272135] <0> ffff880001a32860 ffffffff81241e01 ffffe8ffff623280
> ffffe8ffff83ffc0
> [ 65.272135] <0> ffff88007d6bc600 ffffffffa028057d 000000027d4c06c8
> ffff88007d4c0600
> [ 65.272135] Call Trace:
> [ 65.272135] <IRQ>
> [ 65.272135] [<ffffffff8123a77f>] ? __kfree_skb+0x11/0x7d
> [ 65.272135] [<ffffffff81241e01>] ? netif_rx+0xe2/0xee
> [ 65.272135] [<ffffffffa028057d>] ? veth_xmit+0x6e/0xad [veth]
> [ 65.272135] [<ffffffff8124301f>] ? dev_hard_start_xmit+0x221/0x301
> [ 65.272135] [<ffffffff81256d9a>] ? sch_direct_xmit+0x5b/0x15d
> [ 65.272135] [<ffffffff81256f55>] ? __qdisc_run+0xb9/0xd8
> [ 65.272135] [<ffffffff81240511>] ? net_tx_action+0xd6/0x149
> [ 65.272135] [<ffffffff8104ba02>] ? __do_softirq+0xdd/0x19f
> [ 65.272135] [<ffffffff8101e515>] ? lapic_next_event+0x18/0x1d
> [ 65.272135] [<ffffffff810099dc>] ? call_softirq+0x1c/0x30
> [ 65.272135] [<ffffffff8100b863>] ? do_softirq+0x3f/0x79
> [ 65.272135] [<ffffffff8104b88e>] ? irq_exit+0x36/0x76
> [ 65.272135] [<ffffffff8101f000>] ? smp_apic_timer_interrupt+0x86/0x94
> [ 65.272135] [<ffffffff81009493>] ? apic_timer_interrupt+0x13/0x20
> [ 65.272135] <EOI>
> [ 65.272135] [<ffffffff8114c1ea>] ? cap_sk_getsecid+0x0/0x1
> [ 65.272135] [<ffffffff812a0b0f>] ? __xfrm_lookup+0x2/0xb04
> [ 65.272135] [<ffffffff81264b8d>] ? ip_route_output_flow+0x77/0x1cc
> [ 65.272135] [<ffffffff812887c6>] ? udp_sendmsg+0x32d/0x5f3
> [ 65.272135] [<ffffffff81009400>] ? irq_entries_start+0x3c0/0x400
> [ 65.272135] [<ffffffff8128e112>] ? inet_sendmsg+0x53/0x58
> [ 65.272135] [<ffffffff8123388d>] ? sock_sendmsg+0x83/0x9b
> [ 65.272135] [<ffffffff8103a042>] ? pick_next_task_fair+0xca/0xd6
> [ 65.272135] [<ffffffff812ef74a>] ? schedule+0x52b/0x593
> [ 65.272135] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
> [ 65.272135] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
> [ 65.272135] [<ffffffff81185d4b>] ? _copy_from_user+0x1b/0x30
> [ 65.272135] [<ffffffff8100765f>] ? __switch_to+0x1b5/0x2a6
> [ 65.272135] [<ffffffff81232227>] ? copy_from_user+0x13/0x25
> [ 65.272135] [<ffffffff812353cb>] ? sys_sendto+0xd7/0x117
> [ 65.272135] [<ffffffff8103f7b1>] ? finish_task_switch+0x34/0xa1
> [ 65.272135] [<ffffffff812ef74a>] ? schedule+0x52b/0x593
> [ 65.272135] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
> [ 65.272135] [<ffffffff81008ac2>] ? system_call_fastpath+0x16/0x1b
> [ 65.272135] Code: 83 c3 08 48 83 3b 00 eb ec 48 83 fd 10 0f 86 84
> 00 00 00 48 89 ef e8 f4 e2 ff ff 48 89 c7 48 8b 00 84 c0 78 13 66 a9
> 00 c0 75 04 <0f> 0b eb fe 5b 5d 41 5c e9 fb 57 fd ff 48 8b 4c 24 18 4c
> 8b 47
> [ 65.272135] RIP [<ffffffff810e0d6b>] kfree+0x55/0xc6
> [ 65.272135] RSP <ffff880001a23d90>
> [ 65.385803] ---[ end trace 42d2fb5b94980ab5 ]---
> [ 65.386337] Kernel panic - not syncing: Fatal exception in interrupt
> [ 65.386943] Pid: 1518, comm: udp-perf Tainted: G D 2.6.34-rc5 #1
> [ 65.387557] Call Trace:
> [ 65.388011] <IRQ> [<ffffffff812eef9b>] ? panic+0x77/0xf7
> [ 65.388729] [<ffffffff81046b88>] ? kmsg_dump+0xa6/0x13e
> [ 65.389292] [<ffffffff8100c8c2>] ? oops_end+0xa7/0xb4
> [ 65.389871] [<ffffffff8123a77f>] ? __kfree_skb+0x11/0x7d
> [ 65.390441] [<ffffffff8100a695>] ? do_invalid_op+0x8b/0x95
> [ 65.391032] [<ffffffff810e0d6b>] ? kfree+0x55/0xc6
> [ 65.391587] [<ffffffffa026f301>] ?
> br_nf_pre_routing_finish+0x0/0x25e [bridge]
> [ 65.392512] [<ffffffffa026f301>] ?
> br_nf_pre_routing_finish+0x0/0x25e [bridge]
> [ 65.393399] [<ffffffff8100975b>] ? invalid_op+0x1b/0x20
> [ 65.393978] [<ffffffff8123a77f>] ? __kfree_skb+0x11/0x7d
> [ 65.394547] [<ffffffff810e0d6b>] ? kfree+0x55/0xc6
> [ 65.395091] [<ffffffff8123a77f>] ? __kfree_skb+0x11/0x7d
> [ 65.395668] [<ffffffff81241e01>] ? netif_rx+0xe2/0xee
> [ 65.396438] [<ffffffffa028057d>] ? veth_xmit+0x6e/0xad [veth]
> [ 65.397026] [<ffffffff8124301f>] ? dev_hard_start_xmit+0x221/0x301
> [ 65.397640] [<ffffffff81256d9a>] ? sch_direct_xmit+0x5b/0x15d
> [ 65.398222] [<ffffffff81256f55>] ? __qdisc_run+0xb9/0xd8
> [ 65.398788] [<ffffffff81240511>] ? net_tx_action+0xd6/0x149
> [ 65.399365] [<ffffffff8104ba02>] ? __do_softirq+0xdd/0x19f
> [ 65.399935] [<ffffffff8101e515>] ? lapic_next_event+0x18/0x1d
> [ 65.400560] [<ffffffff810099dc>] ? call_softirq+0x1c/0x30
> [ 65.401125] [<ffffffff8100b863>] ? do_softirq+0x3f/0x79
> [ 65.401708] [<ffffffff8104b88e>] ? irq_exit+0x36/0x76
> [ 65.402267] [<ffffffff8101f000>] ? smp_apic_timer_interrupt+0x86/0x94
> [ 65.402869] [<ffffffff81009493>] ? apic_timer_interrupt+0x13/0x20
> [ 65.403463] <EOI> [<ffffffff8114c1ea>] ? cap_sk_getsecid+0x0/0x1
> [ 65.404210] [<ffffffff812a0b0f>] ? __xfrm_lookup+0x2/0xb04
> [ 65.404779] [<ffffffff81264b8d>] ? ip_route_output_flow+0x77/0x1cc
> [ 65.405375] [<ffffffff812887c6>] ? udp_sendmsg+0x32d/0x5f3
> [ 65.405945] [<ffffffff81009400>] ? irq_entries_start+0x3c0/0x400
> [ 65.406534] [<ffffffff8128e112>] ? inet_sendmsg+0x53/0x58
> [ 65.407102] [<ffffffff8123388d>] ? sock_sendmsg+0x83/0x9b
> [ 65.407674] [<ffffffff8103a042>] ? pick_next_task_fair+0xca/0xd6
> [ 65.408296] [<ffffffff812ef74a>] ? schedule+0x52b/0x593
> [ 65.408854] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
> [ 65.409468] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
> [ 65.410084] [<ffffffff81185d4b>] ? _copy_from_user+0x1b/0x30
> [ 65.410667] [<ffffffff8100765f>] ? __switch_to+0x1b5/0x2a6
> [ 65.510031] [<ffffffff81232227>] ? copy_from_user+0x13/0x25
> [ 65.510818] [<ffffffff812353cb>] ? sys_sendto+0xd7/0x117
> [ 65.511608] [<ffffffff8103f7b1>] ? finish_task_switch+0x34/0xa1
> [ 65.512465] [<ffffffff812ef74a>] ? schedule+0x52b/0x593
> [ 65.513229] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
> [ 65.514040] [<ffffffff81008ac2>] ? system_call_fastpath+0x16/0x1b
>
> [ 1438.042568] ------------[ cut here ]------------
> [ 1438.043170] kernel BUG at mm/slub.c:2846!
> [ 1438.043711] invalid opcode: 0000 [#1] SMP
> [ 1438.044531] last sysfs file: /sys/devices/virtual/net/lo/operstate
> [ 1438.045148] CPU 1
> [ 1438.045350] Modules linked in: sch_netem veth bridge stp netconsole
> configfs loop parport_pc tpm_tis tpm snd_pcm psmouse parport tpm_bios
> snd_timer snd soundcore snd_page_alloc evdev pcspkr serio_raw
> i2c_piix4 i2c_core button processor ext3 jbd mbcache ide_cd_mod cdrom
> ide_gd_mod ata_generic ata_piix 8139too libata scsi_mod floppy 8139cp
> mii thermal thermal_sys piix ide_core [last unloaded: scsi_wait_scan]
> [ 1438.046215]
> [ 1438.046215] Pid: 1476, comm: udp-perf Not tainted 2.6.34-rc5 #1 /
> [ 1438.046215] RIP: 0010:[<ffffffff810e0d6b>] [<ffffffff810e0d6b>]
> kfree+0x55/0xc6
> [ 1438.046215] RSP: 0018:ffff880001a23d90 EFLAGS: 00010246
> [ 1438.046215] RAX: 0100000000000000 RBX: ffff88007d882200 RCX: 0000000000012850
> [ 1438.046215] RDX: ffff88007d882200 RSI: 000000000000000e RDI: ffffea0001b972a0
> [ 1438.046215] RBP: ffff88007e20c000 R08: ffff88007d882200 R09: ffffffffa026c690
> [ 1438.046215] R10: 0000000000000002 R11: ffff88007d882100 R12: ffffffff8123a77f
> [ 1438.046215] R13: 0000000000000032 R14: ffff8800378e0700 R15: ffff88007d882200
> [ 1438.046215] FS: 00007f1c1d07f6f0(0000) GS:ffff880001a20000(0000)
> knlGS:0000000000000000
> [ 1438.046215] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 1438.046215] CR2: 00007f6ea9e07310 CR3: 0000000037834000 CR4: 00000000000006e0
> [ 1438.046215] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 1438.046215] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 1438.046215] Process udp-perf (pid: 1476, threadinfo
> ffff88007db0e000, task ffff8800379a2350)
> [ 1438.046215] Stack:
> [ 1438.046215] ffff88007d882200 ffff88007d882200 0000000000000246
> ffffffff8123a77f
> [ 1438.046215] <0> ffff880001a32860 ffffffff81241e01 ffffe8ffffa3f430
> ffffe8ffffa3d180
> [ 1438.046215] <0> ffff88007d882200 ffffffffa026c57d 0000000201a30500
> ffff88007d171200
> [ 1438.046215] Call Trace:
> [ 1438.046215] <IRQ>
> [ 1438.046215] [<ffffffff8123a77f>] ? __kfree_skb+0x11/0x7d
> [ 1438.046215] [<ffffffff81241e01>] ? netif_rx+0xe2/0xee
> [ 1438.046215] [<ffffffffa026c57d>] ? veth_xmit+0x6e/0xad [veth]
> [ 1438.046215] [<ffffffff8124301f>] ? dev_hard_start_xmit+0x221/0x301
> [ 1438.046215] [<ffffffff81256d9a>] ? sch_direct_xmit+0x5b/0x15d
> [ 1438.046215] [<ffffffff81256f55>] ? __qdisc_run+0xb9/0xd8
> [ 1438.046215] [<ffffffff81240511>] ? net_tx_action+0xd6/0x149
> [ 1438.046215] [<ffffffff8104ba02>] ? __do_softirq+0xdd/0x19f
> [ 1438.046215] [<ffffffff8101e515>] ? lapic_next_event+0x18/0x1d
> [ 1438.046215] [<ffffffff810099dc>] ? call_softirq+0x1c/0x30
> [ 1438.046215] [<ffffffff8100b863>] ? do_softirq+0x3f/0x79
> [ 1438.046215] [<ffffffff8104b88e>] ? irq_exit+0x36/0x76
> [ 1438.046215] [<ffffffff8101f000>] ? smp_apic_timer_interrupt+0x86/0x94
> [ 1438.046215] [<ffffffff81009493>] ? apic_timer_interrupt+0x13/0x20
> [ 1438.046215] <EOI>
> [ 1438.046215] [<ffffffff81288499>] ? udp_sendmsg+0x0/0x5f3
> [ 1438.046215] [<ffffffff8128850f>] ? udp_sendmsg+0x76/0x5f3
> [ 1438.046215] [<ffffffff812889a7>] ? udp_sendmsg+0x50e/0x5f3
> [ 1438.046215] [<ffffffff8123388d>] ? sock_sendmsg+0x83/0x9b
> [ 1438.046215] [<ffffffff8103a042>] ? pick_next_task_fair+0xca/0xd6
> [ 1438.046215] [<ffffffff812ef74a>] ? schedule+0x52b/0x593
> [ 1438.046215] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
> [ 1438.046215] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
> [ 1438.046215] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
> [ 1438.046215] [<ffffffff810e93ed>] ? fget_light+0x0/0xa1
> [ 1438.046215] [<ffffffff81185e40>] ? copy_user_generic_string+0x30/0x40
> [ 1438.046215] [<ffffffff81232227>] ? copy_from_user+0x13/0x25
> [ 1438.046215] [<ffffffff812353cb>] ? sys_sendto+0xd7/0x117
> [ 1438.046215] [<ffffffff8103f7b1>] ? finish_task_switch+0x34/0xa1
> [ 1438.046215] [<ffffffff812ef74a>] ? schedule+0x52b/0x593
> [ 1438.046215] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
> [ 1438.046215] [<ffffffff81008ac2>] ? system_call_fastpath+0x16/0x1b
> [ 1438.046215] Code: 83 c3 08 48 83 3b 00 eb ec 48 83 fd 10 0f 86 84
> 00 00 00 48 89 ef e8 f4 e2 ff ff 48 89 c7 48 8b 00 84 c0 78 13 66 a9
> 00 c0 75 04 <0f> 0b eb fe 5b 5d 41 5c e9 fb 57 fd ff 48 8b 4c 24 18 4c
> 8b 47
> [ 1438.046215] RIP [<ffffffff810e0d6b>] kfree+0x55/0xc6
> [ 1438.046215] RSP <ffff880001a23d90>
> [ 1438.102706] ---[ end trace ab36062dcf233d6a ]---
> [ 1438.103251] Kernel panic - not syncing: Fatal exception in interrupt
> [ 1438.103912] Pid: 1476, comm: udp-perf Tainted: G D 2.6.34-rc5 #1
> [ 1438.104563] Call Trace:
> [ 1438.105017] <IRQ> [<ffffffff812eef9b>] ? panic+0x77/0xf7
> [ 1438.105718] [<ffffffff81046b88>] ? kmsg_dump+0xa6/0x13e
> [ 1438.106293] [<ffffffff8100c8c2>] ? oops_end+0xa7/0xb4
> [ 1438.106866] [<ffffffff8123a77f>] ? __kfree_skb+0x11/0x7d
> [ 1438.107451] [<ffffffff8100a695>] ? do_invalid_op+0x8b/0x95
> [ 1438.108036] [<ffffffff810e0d6b>] ? kfree+0x55/0xc6
> [ 1438.108799] [<ffffffffa025b301>] ?
> br_nf_pre_routing_finish+0x0/0x25e [bridge]
> [ 1438.109699] [<ffffffffa025b301>] ?
> br_nf_pre_routing_finish+0x0/0x25e [bridge]
> [ 1438.110604] [<ffffffff8100975b>] ? invalid_op+0x1b/0x20
> [ 1438.111172] [<ffffffff8123a77f>] ? __kfree_skb+0x11/0x7d
> [ 1438.111753] [<ffffffff810e0d6b>] ? kfree+0x55/0xc6
> [ 1438.112359] [<ffffffff8123a77f>] ? __kfree_skb+0x11/0x7d
> [ 1438.112952] [<ffffffff81241e01>] ? netif_rx+0xe2/0xee
> [ 1438.113527] [<ffffffffa026c57d>] ? veth_xmit+0x6e/0xad [veth]
> [ 1438.114123] [<ffffffff8124301f>] ? dev_hard_start_xmit+0x221/0x301
> [ 1438.114734] [<ffffffff81256d9a>] ? sch_direct_xmit+0x5b/0x15d
> [ 1438.115334] [<ffffffff81256f55>] ? __qdisc_run+0xb9/0xd8
> [ 1438.115912] [<ffffffff81240511>] ? net_tx_action+0xd6/0x149
> [ 1438.116544] [<ffffffff8104ba02>] ? __do_softirq+0xdd/0x19f
> [ 1438.117128] [<ffffffff8101e515>] ? lapic_next_event+0x18/0x1d
> [ 1438.117732] [<ffffffff810099dc>] ? call_softirq+0x1c/0x30
> [ 1438.118420] [<ffffffff8100b863>] ? do_softirq+0x3f/0x79
> [ 1438.119066] [<ffffffff8104b88e>] ? irq_exit+0x36/0x76
> [ 1438.147879] [<ffffffff8101f000>] ? smp_apic_timer_interrupt+0x86/0x94
> [ 1438.148615] [<ffffffff81009493>] ? apic_timer_interrupt+0x13/0x20
> [ 1438.149294] <EOI> [<ffffffff81288499>] ? udp_sendmsg+0x0/0x5f3
> [ 1438.150101] [<ffffffff8128850f>] ? udp_sendmsg+0x76/0x5f3
> [ 1438.150681] [<ffffffff812889a7>] ? udp_sendmsg+0x50e/0x5f3
> [ 1438.151276] [<ffffffff8123388d>] ? sock_sendmsg+0x83/0x9b
> [ 1438.151853] [<ffffffff8103a042>] ? pick_next_task_fair+0xca/0xd6
> [ 1438.152508] [<ffffffff812ef74a>] ? schedule+0x52b/0x593
> [ 1438.153078] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
> [ 1438.153687] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
> [ 1438.154295] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
> [ 1438.154902] [<ffffffff810e93ed>] ? fget_light+0x0/0xa1
> [ 1438.155477] [<ffffffff81185e40>] ? copy_user_generic_string+0x30/0x40
> [ 1438.156097] [<ffffffff81232227>] ? copy_from_user+0x13/0x25
> [ 1438.156720] [<ffffffff812353cb>] ? sys_sendto+0xd7/0x117
> [ 1438.157376] [<ffffffff8103f7b1>] ? finish_task_switch+0x34/0xa1
> [ 1438.157970] [<ffffffff812ef74a>] ? schedule+0x52b/0x593
> [ 1438.158557] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
> [ 1438.159153] [<ffffffff81008ac2>] ? system_call_fastpath+0x16/0x1b
>
> ------------
>
> Finally, a panic in 2.6.33. Note that the line in wioch BUG is
> triggered is different
>
> [ 102.442815] ------------[ cut here ]------------
> [ 102.443433] kernel BUG at
> /build/mattems-linux-2.6_2.6.33-1~experimental.4-amd64-ieqSsa/linux-2.6-2.6.33-1~experimental.4/debian/build/source_amd64_none/mm/slub.c:2969!
> [ 102.444874] invalid opcode: 0000 [#1] SMP
> [ 102.444958] last sysfs file: /sys/devices/virtual/net/lo/operstate
> [ 102.444958] CPU 0
> [ 102.444958] Pid: 4, comm: ksoftirqd/0 Not tainted 2.6.33-2-amd64 #1 /
> [ 102.444958] RIP: 0010:[<ffffffff810e1e2c>] [<ffffffff810e1e2c>]
> kfree+0x55/0xcb
> [ 102.444958] RSP: 0018:ffff880001a03df8 EFLAGS: 00010246
> [ 102.444958] RAX: 0100000000000000 RBX: ffff88007e439000 RCX: 0000000000012d70
> [ 102.444958] RDX: 000000000000006a RSI: ffffea0001b76a70 RDI: ffffea0000c1a328
> [ 102.444958] RBP: ffff880037533c00 R08: ffff88007e21e500 R09: ffffffff8162bbe0
> [ 102.444958] R10: 000000037e439e00 R11: ffff88007e439e00 R12: ffffffff81239cf2
> [ 102.444958] R13: 000000000000006a R14: ffff88007efb3000 R15: ffff88007e21e500
> [ 102.444958] FS: 0000000000000000(0000) GS:ffff880001a00000(0000)
> knlGS:0000000000000000
> [ 102.444958] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 102.444958] CR2: 00007fc8e25cd0f0 CR3: 000000007d088000 CR4: 00000000000006f0
> [ 102.444958] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 102.444958] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 102.444958] Process ksoftirqd/0 (pid: 4, threadinfo
> ffff88007fb78000, task ffff88007fb61500)
> [ 102.444958] Stack:
> [ 102.444958] ffff88007e439000 ffff88007e439000 0000000000000246
> ffffffff81239cf2
> [ 102.444958] <0> ffff880001a12d80 ffffffff8124125d ffffe8ffff818190
> ffffe8ffff815ee0
> [ 102.444958] <0> ffff88007e439000 ffffffffa025254b ffff880001a10010
> ffff88007ef95200
> [ 102.444958] Call Trace:
> [ 102.444958] <IRQ>
> [ 102.444958] [<ffffffff81239cf2>] ? __kfree_skb+0x11/0x7d
> [ 102.444958] [<ffffffff8124125d>] ? netif_rx+0xe2/0xee
> [ 102.444958] [<ffffffffa025254b>] ? veth_xmit+0x6e/0xad [veth]
> [ 102.444958] [<ffffffff8124243a>] ? dev_hard_start_xmit+0x221/0x2dc
> [ 102.444958] [<ffffffff81255514>] ? sch_direct_xmit+0x5b/0x15d
> [ 102.444958] [<ffffffff812556cf>] ? __qdisc_run+0xb9/0xda
> [ 102.444958] [<ffffffff8123fa99>] ? net_tx_action+0xd6/0x149
> [ 102.444958] [<ffffffff8104c918>] ? __do_softirq+0xdd/0x1a1
> [ 102.444958] [<ffffffff810099dc>] ? call_softirq+0x1c/0x30
> [ 102.444958] <EOI>
> [ 102.444958] [<ffffffff8100b85f>] ? do_softirq+0x3f/0x79
> [ 102.444958] [<ffffffff8104c504>] ? run_ksoftirqd+0x6a/0x118
> [ 102.444958] [<ffffffff8104c49a>] ? run_ksoftirqd+0x0/0x118
> [ 102.444958] [<ffffffff8105ee99>] ? kthread+0x79/0x81
> [ 102.444958] [<ffffffff810098e4>] ? kernel_thread_helper+0x4/0x10
> [ 102.444958] [<ffffffff8105ee20>] ? kthread+0x0/0x81
> [ 102.444958] [<ffffffff810098e0>] ? kernel_thread_helper+0x0/0x10
> [ 102.444958] Code: 83 c3 08 48 83 3b 00 eb ec 48 83 fd 10 0f 86 89
> 00 00 00 48 89 ef e8 f0 e6 ff ff 48 89 c7 48 8b 00 84 c0 78 13 66 a9
> 00 c0 75 04 <0f> 0b eb fe 5b 5d 41 5c e9 45 5a fd ff 48 8b 4c 24 18 4c
> 8b 4f
> [ 102.444958] RIP [<ffffffff810e1e2c>] kfree+0x55/0xcb
> [ 102.444958] RSP <ffff880001a03df8>
> [ 102.484000] ---[ end trace b1157390d40df1cb ]---
> [ 102.485018] Kernel panic - not syncing: Fatal exception in interrupt
> [ 102.485647] Pid: 4, comm: ksoftirqd/0 Tainted: G D 2.6.33-2-amd64 #1
> [ 102.486630] Call Trace:
> [ 102.487112] <IRQ> [<ffffffff812ec605>] ? panic+0x86/0x14b
> [ 102.487870] [<ffffffff8104c799>] ? irq_exit+0x48/0x76
> [ 102.488474] [<ffffffff812ee893>] ? ret_from_intr+0x0/0x11
> [ 102.489068] [<ffffffff810478c0>] ? kmsg_dump+0xa6/0x13e
> [ 102.489661] [<ffffffff8100c89a>] ? oops_end+0xa7/0xb4
> [ 102.490245] [<ffffffff81239cf2>] ? __kfree_skb+0x11/0x7d
> [ 102.490836] [<ffffffff8100a690>] ? do_invalid_op+0x8b/0x95
> [ 102.491436] [<ffffffff810e1e2c>] ? kfree+0x55/0xcb
> [ 102.492020] [<ffffffffa0244aee>] ?
> br_nf_pre_routing_finish+0x284/0x2a6 [bridge]
> [ 102.492942] [<ffffffffa024486a>] ?
> br_nf_pre_routing_finish+0x0/0x2a6 [bridge]
> [ 102.493857] [<ffffffff8125f204>] ? nf_hook_slow+0x62/0xc3
> [ 102.523082] [<ffffffffa024486a>] ?
> br_nf_pre_routing_finish+0x0/0x2a6 [bridge]
> [ 102.524008] [<ffffffff8100975b>] ? invalid_op+0x1b/0x20
> [ 102.524631] [<ffffffff81239cf2>] ? __kfree_skb+0x11/0x7d
> [ 102.525225] [<ffffffff810e1e2c>] ? kfree+0x55/0xcb
> [ 102.525795] [<ffffffff810e1e1c>] ? kfree+0x45/0xcb
> [ 102.526402] [<ffffffff81239cf2>] ? __kfree_skb+0x11/0x7d
> [ 102.526992] [<ffffffff8124125d>] ? netif_rx+0xe2/0xee
> [ 102.527576] [<ffffffffa025254b>] ? veth_xmit+0x6e/0xad [veth]
> [ 102.528199] [<ffffffff8124243a>] ? dev_hard_start_xmit+0x221/0x2dc
> [ 102.528821] [<ffffffff81255514>] ? sch_direct_xmit+0x5b/0x15d
> [ 102.529429] [<ffffffff812556cf>] ? __qdisc_run+0xb9/0xda
> [ 102.530018] [<ffffffff8123fa99>] ? net_tx_action+0xd6/0x149
> [ 102.530618] [<ffffffff8104c918>] ? __do_softirq+0xdd/0x1a1
> [ 102.531214] [<ffffffff810099dc>] ? call_softirq+0x1c/0x30
> [ 102.531804] <EOI> [<ffffffff8100b85f>] ? do_softirq+0x3f/0x79
> [ 102.532572] [<ffffffff8104c504>] ? run_ksoftirqd+0x6a/0x118
> [ 102.533169] [<ffffffff8104c49a>] ? run_ksoftirqd+0x0/0x118
> [ 102.533764] [<ffffffff8105ee99>] ? kthread+0x79/0x81
> [ 102.534340] [<ffffffff810098e4>] ? kernel_thread_helper+0x4/0x10
> [ 102.534954] [<ffffffff8105ee20>] ? kthread+0x0/0x81
> [ 102.535526] [<ffffffff810098e0>] ? kernel_thread_helper+0x0/0x10
>
Could you please try following patch ?
Thanks
[PATCH] veth: Dont kfree_skb() after dev_forward_skb()
In case of congestion, dev_forward_skb() already free the skb
Reported-by: Martín Ferrari <martin.ferrari@gmail.com>
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
---
diff --git a/drivers/net/veth.c b/drivers/net/veth.c
index f9f0730..5ec542d 100644
--- a/drivers/net/veth.c
+++ b/drivers/net/veth.c
@@ -187,7 +187,6 @@ tx_drop:
return NETDEV_TX_OK;
rx_drop:
- kfree_skb(skb);
rcv_stats->rx_dropped++;
return NETDEV_TX_OK;
}
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: kernel panic when using netns+bridges+tc(netem)
2010-05-06 6:40 ` Eric Dumazet
@ 2010-05-06 6:47 ` Eric Dumazet
2010-05-06 6:59 ` Eric Dumazet
0 siblings, 1 reply; 7+ messages in thread
From: Eric Dumazet @ 2010-05-06 6:47 UTC (permalink / raw)
To: Martín Ferrari; +Cc: netdev, Mathieu Lacage, David Miller
Le jeudi 06 mai 2010 à 08:40 +0200, Eric Dumazet a écrit :
> Le jeudi 06 mai 2010 à 03:01 +0200, Martín Ferrari a écrit :
> > Hi there,
> >
> > While working on my project that uses netns, I found another bug. This
> > one causes a "Kernel panic - not syncing: Fatal exception in
> > interrupt", and I can reproduce it in 2.6.33 and 2.6.34-rc5, but not
> > in 2.6.32. It dies during a call to __free_skb.
> > I tested this on my x86_64 laptop (2 cores) and on qemu. In qemu it
> > was not triggered until I asked it to emulate 2 cpus instead of one,
> > so it is probably a SMP-only issue.
> >
> > Scenario:
> >
> > I set up a number of network namespaces, each with two veths to netns
> > 1. In the main namespace I take those veths and bridge them in pairs,
> > to configure a linear topology; also I configure the netem qdisc to
> > simulate link delay.
> >
> > Once the network is set up, I run a client/server program to send UDP
> > packets from one end of the topology to the other. After a few seconds
> > of sending packets (not really deterministic) it panics.
> >
> > Note that I didn't experience this problem when using only 2
> > namespaces (so, no routing)
> >
> > below the dumps. These all come from the qemu, as I couldn't use
> > netconsole in the network at work, but I checked and the backtraces
> > were essentially the same
> >
> > First, two panics with 2.6.34, each one with a slightly different backtrace
> >
> > [ 65.272131] ------------[ cut here ]------------
> > [ 65.272135] kernel BUG at mm/slub.c:2846!
> > [ 65.272135] invalid opcode: 0000 [#1] SMP
> > [ 65.272135] last sysfs file: /sys/devices/virtual/net/lo/operstate
> > [ 65.272135] CPU 1
> > [ 65.272135] Modules linked in: sch_netem veth bridge stp netconsole
> > configfs loop parport_pc parport evdev tpm_tis tpm snd_pcm tpm_bios
> > snd_timer snd soundcore snd_page_alloc pcspkr psmouse serio_raw
> > i2c_piix4 button i2c_core processor ext3 jbd mbcache ide_cd_mod cdrom
> > ide_gd_mod ata_generic ata_piix libata 8139too scsi_mod floppy piix
> > 8139cp mii ide_core thermal thermal_sys [last unloaded: configfs]
> > [ 65.272135]
> > [ 65.272135] Pid: 1518, comm: udp-perf Not tainted 2.6.34-rc5 #1 /
> > [ 65.272135] RIP: 0010:[<ffffffff810e0d6b>] [<ffffffff810e0d6b>]
> > kfree+0x55/0xc6
> > [ 65.272135] RSP: 0018:ffff880001a23d90 EFLAGS: 00010246
> > [ 65.272135] RAX: 0100000000000000 RBX: ffff88007d6bc600 RCX: 0000000000012850
> > [ 65.272135] RDX: ffff88007d6bc600 RSI: 000000000000000e RDI: ffffea0001b6f610
> > [ 65.272135] RBP: ffff88007d6ae200 R08: ffff88007d6bc600 R09: ffffffffa0280690
> > [ 65.272135] R10: 0000000000000002 R11: ffff88007d6bc500 R12: ffffffff8123a77f
> > [ 65.272135] R13: 000000000000002b R14: ffff88007d39b600 R15: ffff88007d6bc600
> > [ 65.272135] FS: 00007f637c9dd6f0(0000) GS:ffff880001a20000(0000)
> > knlGS:0000000000000000
> > [ 65.272135] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > [ 65.272135] CR2: 00000000009deaa0 CR3: 000000007d82d000 CR4: 00000000000006e0
> > [ 65.272135] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > [ 65.272135] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > [ 65.272135] Process udp-perf (pid: 1518, threadinfo
> > ffff88007d92a000, task ffff88007e7ff810)
> > [ 65.272135] Stack:
> > [ 65.272135] ffff88007d6bc600 ffff88007d6bc600 0000000000000246
> > ffffffff8123a77f
> > [ 65.272135] <0> ffff880001a32860 ffffffff81241e01 ffffe8ffff623280
> > ffffe8ffff83ffc0
> > [ 65.272135] <0> ffff88007d6bc600 ffffffffa028057d 000000027d4c06c8
> > ffff88007d4c0600
> > [ 65.272135] Call Trace:
> > [ 65.272135] <IRQ>
> > [ 65.272135] [<ffffffff8123a77f>] ? __kfree_skb+0x11/0x7d
> > [ 65.272135] [<ffffffff81241e01>] ? netif_rx+0xe2/0xee
> > [ 65.272135] [<ffffffffa028057d>] ? veth_xmit+0x6e/0xad [veth]
> > [ 65.272135] [<ffffffff8124301f>] ? dev_hard_start_xmit+0x221/0x301
> > [ 65.272135] [<ffffffff81256d9a>] ? sch_direct_xmit+0x5b/0x15d
> > [ 65.272135] [<ffffffff81256f55>] ? __qdisc_run+0xb9/0xd8
> > [ 65.272135] [<ffffffff81240511>] ? net_tx_action+0xd6/0x149
> > [ 65.272135] [<ffffffff8104ba02>] ? __do_softirq+0xdd/0x19f
> > [ 65.272135] [<ffffffff8101e515>] ? lapic_next_event+0x18/0x1d
> > [ 65.272135] [<ffffffff810099dc>] ? call_softirq+0x1c/0x30
> > [ 65.272135] [<ffffffff8100b863>] ? do_softirq+0x3f/0x79
> > [ 65.272135] [<ffffffff8104b88e>] ? irq_exit+0x36/0x76
> > [ 65.272135] [<ffffffff8101f000>] ? smp_apic_timer_interrupt+0x86/0x94
> > [ 65.272135] [<ffffffff81009493>] ? apic_timer_interrupt+0x13/0x20
> > [ 65.272135] <EOI>
> > [ 65.272135] [<ffffffff8114c1ea>] ? cap_sk_getsecid+0x0/0x1
> > [ 65.272135] [<ffffffff812a0b0f>] ? __xfrm_lookup+0x2/0xb04
> > [ 65.272135] [<ffffffff81264b8d>] ? ip_route_output_flow+0x77/0x1cc
> > [ 65.272135] [<ffffffff812887c6>] ? udp_sendmsg+0x32d/0x5f3
> > [ 65.272135] [<ffffffff81009400>] ? irq_entries_start+0x3c0/0x400
> > [ 65.272135] [<ffffffff8128e112>] ? inet_sendmsg+0x53/0x58
> > [ 65.272135] [<ffffffff8123388d>] ? sock_sendmsg+0x83/0x9b
> > [ 65.272135] [<ffffffff8103a042>] ? pick_next_task_fair+0xca/0xd6
> > [ 65.272135] [<ffffffff812ef74a>] ? schedule+0x52b/0x593
> > [ 65.272135] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
> > [ 65.272135] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
> > [ 65.272135] [<ffffffff81185d4b>] ? _copy_from_user+0x1b/0x30
> > [ 65.272135] [<ffffffff8100765f>] ? __switch_to+0x1b5/0x2a6
> > [ 65.272135] [<ffffffff81232227>] ? copy_from_user+0x13/0x25
> > [ 65.272135] [<ffffffff812353cb>] ? sys_sendto+0xd7/0x117
> > [ 65.272135] [<ffffffff8103f7b1>] ? finish_task_switch+0x34/0xa1
> > [ 65.272135] [<ffffffff812ef74a>] ? schedule+0x52b/0x593
> > [ 65.272135] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
> > [ 65.272135] [<ffffffff81008ac2>] ? system_call_fastpath+0x16/0x1b
> > [ 65.272135] Code: 83 c3 08 48 83 3b 00 eb ec 48 83 fd 10 0f 86 84
> > 00 00 00 48 89 ef e8 f4 e2 ff ff 48 89 c7 48 8b 00 84 c0 78 13 66 a9
> > 00 c0 75 04 <0f> 0b eb fe 5b 5d 41 5c e9 fb 57 fd ff 48 8b 4c 24 18 4c
> > 8b 47
> > [ 65.272135] RIP [<ffffffff810e0d6b>] kfree+0x55/0xc6
> > [ 65.272135] RSP <ffff880001a23d90>
> > [ 65.385803] ---[ end trace 42d2fb5b94980ab5 ]---
> > [ 65.386337] Kernel panic - not syncing: Fatal exception in interrupt
> > [ 65.386943] Pid: 1518, comm: udp-perf Tainted: G D 2.6.34-rc5 #1
> > [ 65.387557] Call Trace:
> > [ 65.388011] <IRQ> [<ffffffff812eef9b>] ? panic+0x77/0xf7
> > [ 65.388729] [<ffffffff81046b88>] ? kmsg_dump+0xa6/0x13e
> > [ 65.389292] [<ffffffff8100c8c2>] ? oops_end+0xa7/0xb4
> > [ 65.389871] [<ffffffff8123a77f>] ? __kfree_skb+0x11/0x7d
> > [ 65.390441] [<ffffffff8100a695>] ? do_invalid_op+0x8b/0x95
> > [ 65.391032] [<ffffffff810e0d6b>] ? kfree+0x55/0xc6
> > [ 65.391587] [<ffffffffa026f301>] ?
> > br_nf_pre_routing_finish+0x0/0x25e [bridge]
> > [ 65.392512] [<ffffffffa026f301>] ?
> > br_nf_pre_routing_finish+0x0/0x25e [bridge]
> > [ 65.393399] [<ffffffff8100975b>] ? invalid_op+0x1b/0x20
> > [ 65.393978] [<ffffffff8123a77f>] ? __kfree_skb+0x11/0x7d
> > [ 65.394547] [<ffffffff810e0d6b>] ? kfree+0x55/0xc6
> > [ 65.395091] [<ffffffff8123a77f>] ? __kfree_skb+0x11/0x7d
> > [ 65.395668] [<ffffffff81241e01>] ? netif_rx+0xe2/0xee
> > [ 65.396438] [<ffffffffa028057d>] ? veth_xmit+0x6e/0xad [veth]
> > [ 65.397026] [<ffffffff8124301f>] ? dev_hard_start_xmit+0x221/0x301
> > [ 65.397640] [<ffffffff81256d9a>] ? sch_direct_xmit+0x5b/0x15d
> > [ 65.398222] [<ffffffff81256f55>] ? __qdisc_run+0xb9/0xd8
> > [ 65.398788] [<ffffffff81240511>] ? net_tx_action+0xd6/0x149
> > [ 65.399365] [<ffffffff8104ba02>] ? __do_softirq+0xdd/0x19f
> > [ 65.399935] [<ffffffff8101e515>] ? lapic_next_event+0x18/0x1d
> > [ 65.400560] [<ffffffff810099dc>] ? call_softirq+0x1c/0x30
> > [ 65.401125] [<ffffffff8100b863>] ? do_softirq+0x3f/0x79
> > [ 65.401708] [<ffffffff8104b88e>] ? irq_exit+0x36/0x76
> > [ 65.402267] [<ffffffff8101f000>] ? smp_apic_timer_interrupt+0x86/0x94
> > [ 65.402869] [<ffffffff81009493>] ? apic_timer_interrupt+0x13/0x20
> > [ 65.403463] <EOI> [<ffffffff8114c1ea>] ? cap_sk_getsecid+0x0/0x1
> > [ 65.404210] [<ffffffff812a0b0f>] ? __xfrm_lookup+0x2/0xb04
> > [ 65.404779] [<ffffffff81264b8d>] ? ip_route_output_flow+0x77/0x1cc
> > [ 65.405375] [<ffffffff812887c6>] ? udp_sendmsg+0x32d/0x5f3
> > [ 65.405945] [<ffffffff81009400>] ? irq_entries_start+0x3c0/0x400
> > [ 65.406534] [<ffffffff8128e112>] ? inet_sendmsg+0x53/0x58
> > [ 65.407102] [<ffffffff8123388d>] ? sock_sendmsg+0x83/0x9b
> > [ 65.407674] [<ffffffff8103a042>] ? pick_next_task_fair+0xca/0xd6
> > [ 65.408296] [<ffffffff812ef74a>] ? schedule+0x52b/0x593
> > [ 65.408854] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
> > [ 65.409468] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
> > [ 65.410084] [<ffffffff81185d4b>] ? _copy_from_user+0x1b/0x30
> > [ 65.410667] [<ffffffff8100765f>] ? __switch_to+0x1b5/0x2a6
> > [ 65.510031] [<ffffffff81232227>] ? copy_from_user+0x13/0x25
> > [ 65.510818] [<ffffffff812353cb>] ? sys_sendto+0xd7/0x117
> > [ 65.511608] [<ffffffff8103f7b1>] ? finish_task_switch+0x34/0xa1
> > [ 65.512465] [<ffffffff812ef74a>] ? schedule+0x52b/0x593
> > [ 65.513229] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
> > [ 65.514040] [<ffffffff81008ac2>] ? system_call_fastpath+0x16/0x1b
> >
> > [ 1438.042568] ------------[ cut here ]------------
> > [ 1438.043170] kernel BUG at mm/slub.c:2846!
> > [ 1438.043711] invalid opcode: 0000 [#1] SMP
> > [ 1438.044531] last sysfs file: /sys/devices/virtual/net/lo/operstate
> > [ 1438.045148] CPU 1
> > [ 1438.045350] Modules linked in: sch_netem veth bridge stp netconsole
> > configfs loop parport_pc tpm_tis tpm snd_pcm psmouse parport tpm_bios
> > snd_timer snd soundcore snd_page_alloc evdev pcspkr serio_raw
> > i2c_piix4 i2c_core button processor ext3 jbd mbcache ide_cd_mod cdrom
> > ide_gd_mod ata_generic ata_piix 8139too libata scsi_mod floppy 8139cp
> > mii thermal thermal_sys piix ide_core [last unloaded: scsi_wait_scan]
> > [ 1438.046215]
> > [ 1438.046215] Pid: 1476, comm: udp-perf Not tainted 2.6.34-rc5 #1 /
> > [ 1438.046215] RIP: 0010:[<ffffffff810e0d6b>] [<ffffffff810e0d6b>]
> > kfree+0x55/0xc6
> > [ 1438.046215] RSP: 0018:ffff880001a23d90 EFLAGS: 00010246
> > [ 1438.046215] RAX: 0100000000000000 RBX: ffff88007d882200 RCX: 0000000000012850
> > [ 1438.046215] RDX: ffff88007d882200 RSI: 000000000000000e RDI: ffffea0001b972a0
> > [ 1438.046215] RBP: ffff88007e20c000 R08: ffff88007d882200 R09: ffffffffa026c690
> > [ 1438.046215] R10: 0000000000000002 R11: ffff88007d882100 R12: ffffffff8123a77f
> > [ 1438.046215] R13: 0000000000000032 R14: ffff8800378e0700 R15: ffff88007d882200
> > [ 1438.046215] FS: 00007f1c1d07f6f0(0000) GS:ffff880001a20000(0000)
> > knlGS:0000000000000000
> > [ 1438.046215] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > [ 1438.046215] CR2: 00007f6ea9e07310 CR3: 0000000037834000 CR4: 00000000000006e0
> > [ 1438.046215] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > [ 1438.046215] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > [ 1438.046215] Process udp-perf (pid: 1476, threadinfo
> > ffff88007db0e000, task ffff8800379a2350)
> > [ 1438.046215] Stack:
> > [ 1438.046215] ffff88007d882200 ffff88007d882200 0000000000000246
> > ffffffff8123a77f
> > [ 1438.046215] <0> ffff880001a32860 ffffffff81241e01 ffffe8ffffa3f430
> > ffffe8ffffa3d180
> > [ 1438.046215] <0> ffff88007d882200 ffffffffa026c57d 0000000201a30500
> > ffff88007d171200
> > [ 1438.046215] Call Trace:
> > [ 1438.046215] <IRQ>
> > [ 1438.046215] [<ffffffff8123a77f>] ? __kfree_skb+0x11/0x7d
> > [ 1438.046215] [<ffffffff81241e01>] ? netif_rx+0xe2/0xee
> > [ 1438.046215] [<ffffffffa026c57d>] ? veth_xmit+0x6e/0xad [veth]
> > [ 1438.046215] [<ffffffff8124301f>] ? dev_hard_start_xmit+0x221/0x301
> > [ 1438.046215] [<ffffffff81256d9a>] ? sch_direct_xmit+0x5b/0x15d
> > [ 1438.046215] [<ffffffff81256f55>] ? __qdisc_run+0xb9/0xd8
> > [ 1438.046215] [<ffffffff81240511>] ? net_tx_action+0xd6/0x149
> > [ 1438.046215] [<ffffffff8104ba02>] ? __do_softirq+0xdd/0x19f
> > [ 1438.046215] [<ffffffff8101e515>] ? lapic_next_event+0x18/0x1d
> > [ 1438.046215] [<ffffffff810099dc>] ? call_softirq+0x1c/0x30
> > [ 1438.046215] [<ffffffff8100b863>] ? do_softirq+0x3f/0x79
> > [ 1438.046215] [<ffffffff8104b88e>] ? irq_exit+0x36/0x76
> > [ 1438.046215] [<ffffffff8101f000>] ? smp_apic_timer_interrupt+0x86/0x94
> > [ 1438.046215] [<ffffffff81009493>] ? apic_timer_interrupt+0x13/0x20
> > [ 1438.046215] <EOI>
> > [ 1438.046215] [<ffffffff81288499>] ? udp_sendmsg+0x0/0x5f3
> > [ 1438.046215] [<ffffffff8128850f>] ? udp_sendmsg+0x76/0x5f3
> > [ 1438.046215] [<ffffffff812889a7>] ? udp_sendmsg+0x50e/0x5f3
> > [ 1438.046215] [<ffffffff8123388d>] ? sock_sendmsg+0x83/0x9b
> > [ 1438.046215] [<ffffffff8103a042>] ? pick_next_task_fair+0xca/0xd6
> > [ 1438.046215] [<ffffffff812ef74a>] ? schedule+0x52b/0x593
> > [ 1438.046215] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
> > [ 1438.046215] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
> > [ 1438.046215] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
> > [ 1438.046215] [<ffffffff810e93ed>] ? fget_light+0x0/0xa1
> > [ 1438.046215] [<ffffffff81185e40>] ? copy_user_generic_string+0x30/0x40
> > [ 1438.046215] [<ffffffff81232227>] ? copy_from_user+0x13/0x25
> > [ 1438.046215] [<ffffffff812353cb>] ? sys_sendto+0xd7/0x117
> > [ 1438.046215] [<ffffffff8103f7b1>] ? finish_task_switch+0x34/0xa1
> > [ 1438.046215] [<ffffffff812ef74a>] ? schedule+0x52b/0x593
> > [ 1438.046215] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
> > [ 1438.046215] [<ffffffff81008ac2>] ? system_call_fastpath+0x16/0x1b
> > [ 1438.046215] Code: 83 c3 08 48 83 3b 00 eb ec 48 83 fd 10 0f 86 84
> > 00 00 00 48 89 ef e8 f4 e2 ff ff 48 89 c7 48 8b 00 84 c0 78 13 66 a9
> > 00 c0 75 04 <0f> 0b eb fe 5b 5d 41 5c e9 fb 57 fd ff 48 8b 4c 24 18 4c
> > 8b 47
> > [ 1438.046215] RIP [<ffffffff810e0d6b>] kfree+0x55/0xc6
> > [ 1438.046215] RSP <ffff880001a23d90>
> > [ 1438.102706] ---[ end trace ab36062dcf233d6a ]---
> > [ 1438.103251] Kernel panic - not syncing: Fatal exception in interrupt
> > [ 1438.103912] Pid: 1476, comm: udp-perf Tainted: G D 2.6.34-rc5 #1
> > [ 1438.104563] Call Trace:
> > [ 1438.105017] <IRQ> [<ffffffff812eef9b>] ? panic+0x77/0xf7
> > [ 1438.105718] [<ffffffff81046b88>] ? kmsg_dump+0xa6/0x13e
> > [ 1438.106293] [<ffffffff8100c8c2>] ? oops_end+0xa7/0xb4
> > [ 1438.106866] [<ffffffff8123a77f>] ? __kfree_skb+0x11/0x7d
> > [ 1438.107451] [<ffffffff8100a695>] ? do_invalid_op+0x8b/0x95
> > [ 1438.108036] [<ffffffff810e0d6b>] ? kfree+0x55/0xc6
> > [ 1438.108799] [<ffffffffa025b301>] ?
> > br_nf_pre_routing_finish+0x0/0x25e [bridge]
> > [ 1438.109699] [<ffffffffa025b301>] ?
> > br_nf_pre_routing_finish+0x0/0x25e [bridge]
> > [ 1438.110604] [<ffffffff8100975b>] ? invalid_op+0x1b/0x20
> > [ 1438.111172] [<ffffffff8123a77f>] ? __kfree_skb+0x11/0x7d
> > [ 1438.111753] [<ffffffff810e0d6b>] ? kfree+0x55/0xc6
> > [ 1438.112359] [<ffffffff8123a77f>] ? __kfree_skb+0x11/0x7d
> > [ 1438.112952] [<ffffffff81241e01>] ? netif_rx+0xe2/0xee
> > [ 1438.113527] [<ffffffffa026c57d>] ? veth_xmit+0x6e/0xad [veth]
> > [ 1438.114123] [<ffffffff8124301f>] ? dev_hard_start_xmit+0x221/0x301
> > [ 1438.114734] [<ffffffff81256d9a>] ? sch_direct_xmit+0x5b/0x15d
> > [ 1438.115334] [<ffffffff81256f55>] ? __qdisc_run+0xb9/0xd8
> > [ 1438.115912] [<ffffffff81240511>] ? net_tx_action+0xd6/0x149
> > [ 1438.116544] [<ffffffff8104ba02>] ? __do_softirq+0xdd/0x19f
> > [ 1438.117128] [<ffffffff8101e515>] ? lapic_next_event+0x18/0x1d
> > [ 1438.117732] [<ffffffff810099dc>] ? call_softirq+0x1c/0x30
> > [ 1438.118420] [<ffffffff8100b863>] ? do_softirq+0x3f/0x79
> > [ 1438.119066] [<ffffffff8104b88e>] ? irq_exit+0x36/0x76
> > [ 1438.147879] [<ffffffff8101f000>] ? smp_apic_timer_interrupt+0x86/0x94
> > [ 1438.148615] [<ffffffff81009493>] ? apic_timer_interrupt+0x13/0x20
> > [ 1438.149294] <EOI> [<ffffffff81288499>] ? udp_sendmsg+0x0/0x5f3
> > [ 1438.150101] [<ffffffff8128850f>] ? udp_sendmsg+0x76/0x5f3
> > [ 1438.150681] [<ffffffff812889a7>] ? udp_sendmsg+0x50e/0x5f3
> > [ 1438.151276] [<ffffffff8123388d>] ? sock_sendmsg+0x83/0x9b
> > [ 1438.151853] [<ffffffff8103a042>] ? pick_next_task_fair+0xca/0xd6
> > [ 1438.152508] [<ffffffff812ef74a>] ? schedule+0x52b/0x593
> > [ 1438.153078] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
> > [ 1438.153687] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
> > [ 1438.154295] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
> > [ 1438.154902] [<ffffffff810e93ed>] ? fget_light+0x0/0xa1
> > [ 1438.155477] [<ffffffff81185e40>] ? copy_user_generic_string+0x30/0x40
> > [ 1438.156097] [<ffffffff81232227>] ? copy_from_user+0x13/0x25
> > [ 1438.156720] [<ffffffff812353cb>] ? sys_sendto+0xd7/0x117
> > [ 1438.157376] [<ffffffff8103f7b1>] ? finish_task_switch+0x34/0xa1
> > [ 1438.157970] [<ffffffff812ef74a>] ? schedule+0x52b/0x593
> > [ 1438.158557] [<ffffffff8100948e>] ? apic_timer_interrupt+0xe/0x20
> > [ 1438.159153] [<ffffffff81008ac2>] ? system_call_fastpath+0x16/0x1b
> >
> > ------------
> >
> > Finally, a panic in 2.6.33. Note that the line in wioch BUG is
> > triggered is different
> >
> > [ 102.442815] ------------[ cut here ]------------
> > [ 102.443433] kernel BUG at
> > /build/mattems-linux-2.6_2.6.33-1~experimental.4-amd64-ieqSsa/linux-2.6-2.6.33-1~experimental.4/debian/build/source_amd64_none/mm/slub.c:2969!
> > [ 102.444874] invalid opcode: 0000 [#1] SMP
> > [ 102.444958] last sysfs file: /sys/devices/virtual/net/lo/operstate
> > [ 102.444958] CPU 0
> > [ 102.444958] Pid: 4, comm: ksoftirqd/0 Not tainted 2.6.33-2-amd64 #1 /
> > [ 102.444958] RIP: 0010:[<ffffffff810e1e2c>] [<ffffffff810e1e2c>]
> > kfree+0x55/0xcb
> > [ 102.444958] RSP: 0018:ffff880001a03df8 EFLAGS: 00010246
> > [ 102.444958] RAX: 0100000000000000 RBX: ffff88007e439000 RCX: 0000000000012d70
> > [ 102.444958] RDX: 000000000000006a RSI: ffffea0001b76a70 RDI: ffffea0000c1a328
> > [ 102.444958] RBP: ffff880037533c00 R08: ffff88007e21e500 R09: ffffffff8162bbe0
> > [ 102.444958] R10: 000000037e439e00 R11: ffff88007e439e00 R12: ffffffff81239cf2
> > [ 102.444958] R13: 000000000000006a R14: ffff88007efb3000 R15: ffff88007e21e500
> > [ 102.444958] FS: 0000000000000000(0000) GS:ffff880001a00000(0000)
> > knlGS:0000000000000000
> > [ 102.444958] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > [ 102.444958] CR2: 00007fc8e25cd0f0 CR3: 000000007d088000 CR4: 00000000000006f0
> > [ 102.444958] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > [ 102.444958] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > [ 102.444958] Process ksoftirqd/0 (pid: 4, threadinfo
> > ffff88007fb78000, task ffff88007fb61500)
> > [ 102.444958] Stack:
> > [ 102.444958] ffff88007e439000 ffff88007e439000 0000000000000246
> > ffffffff81239cf2
> > [ 102.444958] <0> ffff880001a12d80 ffffffff8124125d ffffe8ffff818190
> > ffffe8ffff815ee0
> > [ 102.444958] <0> ffff88007e439000 ffffffffa025254b ffff880001a10010
> > ffff88007ef95200
> > [ 102.444958] Call Trace:
> > [ 102.444958] <IRQ>
> > [ 102.444958] [<ffffffff81239cf2>] ? __kfree_skb+0x11/0x7d
> > [ 102.444958] [<ffffffff8124125d>] ? netif_rx+0xe2/0xee
> > [ 102.444958] [<ffffffffa025254b>] ? veth_xmit+0x6e/0xad [veth]
> > [ 102.444958] [<ffffffff8124243a>] ? dev_hard_start_xmit+0x221/0x2dc
> > [ 102.444958] [<ffffffff81255514>] ? sch_direct_xmit+0x5b/0x15d
> > [ 102.444958] [<ffffffff812556cf>] ? __qdisc_run+0xb9/0xda
> > [ 102.444958] [<ffffffff8123fa99>] ? net_tx_action+0xd6/0x149
> > [ 102.444958] [<ffffffff8104c918>] ? __do_softirq+0xdd/0x1a1
> > [ 102.444958] [<ffffffff810099dc>] ? call_softirq+0x1c/0x30
> > [ 102.444958] <EOI>
> > [ 102.444958] [<ffffffff8100b85f>] ? do_softirq+0x3f/0x79
> > [ 102.444958] [<ffffffff8104c504>] ? run_ksoftirqd+0x6a/0x118
> > [ 102.444958] [<ffffffff8104c49a>] ? run_ksoftirqd+0x0/0x118
> > [ 102.444958] [<ffffffff8105ee99>] ? kthread+0x79/0x81
> > [ 102.444958] [<ffffffff810098e4>] ? kernel_thread_helper+0x4/0x10
> > [ 102.444958] [<ffffffff8105ee20>] ? kthread+0x0/0x81
> > [ 102.444958] [<ffffffff810098e0>] ? kernel_thread_helper+0x0/0x10
> > [ 102.444958] Code: 83 c3 08 48 83 3b 00 eb ec 48 83 fd 10 0f 86 89
> > 00 00 00 48 89 ef e8 f0 e6 ff ff 48 89 c7 48 8b 00 84 c0 78 13 66 a9
> > 00 c0 75 04 <0f> 0b eb fe 5b 5d 41 5c e9 45 5a fd ff 48 8b 4c 24 18 4c
> > 8b 4f
> > [ 102.444958] RIP [<ffffffff810e1e2c>] kfree+0x55/0xcb
> > [ 102.444958] RSP <ffff880001a03df8>
> > [ 102.484000] ---[ end trace b1157390d40df1cb ]---
> > [ 102.485018] Kernel panic - not syncing: Fatal exception in interrupt
> > [ 102.485647] Pid: 4, comm: ksoftirqd/0 Tainted: G D 2.6.33-2-amd64 #1
> > [ 102.486630] Call Trace:
> > [ 102.487112] <IRQ> [<ffffffff812ec605>] ? panic+0x86/0x14b
> > [ 102.487870] [<ffffffff8104c799>] ? irq_exit+0x48/0x76
> > [ 102.488474] [<ffffffff812ee893>] ? ret_from_intr+0x0/0x11
> > [ 102.489068] [<ffffffff810478c0>] ? kmsg_dump+0xa6/0x13e
> > [ 102.489661] [<ffffffff8100c89a>] ? oops_end+0xa7/0xb4
> > [ 102.490245] [<ffffffff81239cf2>] ? __kfree_skb+0x11/0x7d
> > [ 102.490836] [<ffffffff8100a690>] ? do_invalid_op+0x8b/0x95
> > [ 102.491436] [<ffffffff810e1e2c>] ? kfree+0x55/0xcb
> > [ 102.492020] [<ffffffffa0244aee>] ?
> > br_nf_pre_routing_finish+0x284/0x2a6 [bridge]
> > [ 102.492942] [<ffffffffa024486a>] ?
> > br_nf_pre_routing_finish+0x0/0x2a6 [bridge]
> > [ 102.493857] [<ffffffff8125f204>] ? nf_hook_slow+0x62/0xc3
> > [ 102.523082] [<ffffffffa024486a>] ?
> > br_nf_pre_routing_finish+0x0/0x2a6 [bridge]
> > [ 102.524008] [<ffffffff8100975b>] ? invalid_op+0x1b/0x20
> > [ 102.524631] [<ffffffff81239cf2>] ? __kfree_skb+0x11/0x7d
> > [ 102.525225] [<ffffffff810e1e2c>] ? kfree+0x55/0xcb
> > [ 102.525795] [<ffffffff810e1e1c>] ? kfree+0x45/0xcb
> > [ 102.526402] [<ffffffff81239cf2>] ? __kfree_skb+0x11/0x7d
> > [ 102.526992] [<ffffffff8124125d>] ? netif_rx+0xe2/0xee
> > [ 102.527576] [<ffffffffa025254b>] ? veth_xmit+0x6e/0xad [veth]
> > [ 102.528199] [<ffffffff8124243a>] ? dev_hard_start_xmit+0x221/0x2dc
> > [ 102.528821] [<ffffffff81255514>] ? sch_direct_xmit+0x5b/0x15d
> > [ 102.529429] [<ffffffff812556cf>] ? __qdisc_run+0xb9/0xda
> > [ 102.530018] [<ffffffff8123fa99>] ? net_tx_action+0xd6/0x149
> > [ 102.530618] [<ffffffff8104c918>] ? __do_softirq+0xdd/0x1a1
> > [ 102.531214] [<ffffffff810099dc>] ? call_softirq+0x1c/0x30
> > [ 102.531804] <EOI> [<ffffffff8100b85f>] ? do_softirq+0x3f/0x79
> > [ 102.532572] [<ffffffff8104c504>] ? run_ksoftirqd+0x6a/0x118
> > [ 102.533169] [<ffffffff8104c49a>] ? run_ksoftirqd+0x0/0x118
> > [ 102.533764] [<ffffffff8105ee99>] ? kthread+0x79/0x81
> > [ 102.534340] [<ffffffff810098e4>] ? kernel_thread_helper+0x4/0x10
> > [ 102.534954] [<ffffffff8105ee20>] ? kthread+0x0/0x81
> > [ 102.535526] [<ffffffff810098e0>] ? kernel_thread_helper+0x0/0x10
> >
>
> Could you please try following patch ?
>
> Thanks
>
> [PATCH] veth: Dont kfree_skb() after dev_forward_skb()
>
> In case of congestion, dev_forward_skb() already free the skb
>
> Reported-by: Martín Ferrari <martin.ferrari@gmail.com>
> Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
> ---
> diff --git a/drivers/net/veth.c b/drivers/net/veth.c
> index f9f0730..5ec542d 100644
> --- a/drivers/net/veth.c
> +++ b/drivers/net/veth.c
> @@ -187,7 +187,6 @@ tx_drop:
> return NETDEV_TX_OK;
>
> rx_drop:
> - kfree_skb(skb);
> rcv_stats->rx_dropped++;
> return NETDEV_TX_OK;
> }
>
Hmm, scratch that one, I'll resubmit a proper fix in few minutes
(We must change dev_forward_skb() too)
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: kernel panic when using netns+bridges+tc(netem)
2010-05-06 6:47 ` Eric Dumazet
@ 2010-05-06 6:59 ` Eric Dumazet
2010-05-06 7:54 ` David Miller
2010-05-06 11:23 ` Martín Ferrari
0 siblings, 2 replies; 7+ messages in thread
From: Eric Dumazet @ 2010-05-06 6:59 UTC (permalink / raw)
To: Martín Ferrari, Arnd Bergmann; +Cc: netdev, Mathieu Lacage, David Miller
Le jeudi 06 mai 2010 à 08:47 +0200, Eric Dumazet a écrit :
> Le jeudi 06 mai 2010 à 08:40 +0200, Eric Dumazet a écrit :
> > Le jeudi 06 mai 2010 à 03:01 +0200, Martín Ferrari a écrit :
> > > Hi there,
> > >
> > > While working on my project that uses netns, I found another bug. This
> > > one causes a "Kernel panic - not syncing: Fatal exception in
> > > interrupt", and I can reproduce it in 2.6.33 and 2.6.34-rc5, but not
> > > in 2.6.32. It dies during a call to __free_skb.
> > > I tested this on my x86_64 laptop (2 cores) and on qemu. In qemu it
> > > was not triggered until I asked it to emulate 2 cpus instead of one,
> > > so it is probably a SMP-only issue.
> > >
> > > Scenario:
> > >
> > > I set up a number of network namespaces, each with two veths to netns
> > > 1. In the main namespace I take those veths and bridge them in pairs,
> > > to configure a linear topology; also I configure the netem qdisc to
> > > simulate link delay.
> > >
> > > Once the network is set up, I run a client/server program to send UDP
> > > packets from one end of the topology to the other. After a few seconds
> > > of sending packets (not really deterministic) it panics.
> > >
> > > Note that I didn't experience this problem when using only 2
> > > namespaces (so, no routing)
> > >
> > > below the dumps. These all come from the qemu, as I couldn't use
> > > netconsole in the network at work, but I checked and the backtraces
> > > were essentially the same
> > >
...
> > Could you please try following patch ?
> >
> > Thanks
> >
> > [PATCH] veth: Dont kfree_skb() after dev_forward_skb()
> >
> > In case of congestion, dev_forward_skb() already free the skb
> >
> > Reported-by: Martín Ferrari <martin.ferrari@gmail.com>
> > Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
> > ---
> > diff --git a/drivers/net/veth.c b/drivers/net/veth.c
> > index f9f0730..5ec542d 100644
> > --- a/drivers/net/veth.c
> > +++ b/drivers/net/veth.c
> > @@ -187,7 +187,6 @@ tx_drop:
> > return NETDEV_TX_OK;
> >
> > rx_drop:
> > - kfree_skb(skb);
> > rcv_stats->rx_dropped++;
> > return NETDEV_TX_OK;
> > }
> >
>
> Hmm, scratch that one, I'll resubmit a proper fix in few minutes
>
> (We must change dev_forward_skb() too)
>
David, this is a stable candidate, once tested and acked, thanks !
[PATCH] veth: Dont kfree_skb() after dev_forward_skb()
In case of congestion, netif_rx() frees the skb, so we must assume
dev_forward_skb() also consume skb.
Bug introduced by commit 445409602c092
(veth: move loopback logic to common location)
We must change dev_forward_skb() to always consume skb, and veth to not
double free it.
Bug report : http://marc.info/?l=linux-netdev&m=127310770900442&w=3
Reported-by: Martín Ferrari <martin.ferrari@gmail.com>
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
---
drivers/net/veth.c | 1 -
net/core/dev.c | 11 +++++------
2 files changed, 5 insertions(+), 7 deletions(-)
diff --git a/drivers/net/veth.c b/drivers/net/veth.c
index f9f0730..5ec542d 100644
--- a/drivers/net/veth.c
+++ b/drivers/net/veth.c
@@ -187,7 +187,6 @@ tx_drop:
return NETDEV_TX_OK;
rx_drop:
- kfree_skb(skb);
rcv_stats->rx_dropped++;
return NETDEV_TX_OK;
}
diff --git a/net/core/dev.c b/net/core/dev.c
index f769098..264137f 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -1451,7 +1451,7 @@ static inline void net_timestamp(struct sk_buff *skb)
*
* return values:
* NET_RX_SUCCESS (no congestion)
- * NET_RX_DROP (packet was dropped)
+ * NET_RX_DROP (packet was dropped, but freed)
*
* dev_forward_skb can be used for injecting an skb from the
* start_xmit function of one device into the receive queue
@@ -1465,12 +1465,11 @@ int dev_forward_skb(struct net_device *dev, struct sk_buff *skb)
{
skb_orphan(skb);
- if (!(dev->flags & IFF_UP))
- return NET_RX_DROP;
-
- if (skb->len > (dev->mtu + dev->hard_header_len))
+ if (!(dev->flags & IFF_UP) ||
+ (skb->len > (dev->mtu + dev->hard_header_len))) {
+ kfree_skb(skb);
return NET_RX_DROP;
-
+ }
skb_set_dev(skb, dev);
skb->tstamp.tv64 = 0;
skb->pkt_type = PACKET_HOST;
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: kernel panic when using netns+bridges+tc(netem)
2010-05-06 6:59 ` Eric Dumazet
@ 2010-05-06 7:54 ` David Miller
2010-05-06 11:23 ` Martín Ferrari
1 sibling, 0 replies; 7+ messages in thread
From: David Miller @ 2010-05-06 7:54 UTC (permalink / raw)
To: eric.dumazet; +Cc: martin.ferrari, arnd, netdev, mathieu.lacage
From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Thu, 06 May 2010 08:59:47 +0200
> David, this is a stable candidate, once tested and acked, thanks !
>
> [PATCH] veth: Dont kfree_skb() after dev_forward_skb()
>
> In case of congestion, netif_rx() frees the skb, so we must assume
> dev_forward_skb() also consume skb.
>
> Bug introduced by commit 445409602c092
> (veth: move loopback logic to common location)
>
> We must change dev_forward_skb() to always consume skb, and veth to not
> double free it.
>
> Bug report : http://marc.info/?l=linux-netdev&m=127310770900442&w=3
>
> Reported-by: Martín Ferrari <martin.ferrari@gmail.com>
> Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Ugh, applied and queued up for -stable, thanks Eric!
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: kernel panic when using netns+bridges+tc(netem)
2010-05-06 6:59 ` Eric Dumazet
2010-05-06 7:54 ` David Miller
@ 2010-05-06 11:23 ` Martín Ferrari
2010-05-06 19:15 ` Eric Dumazet
1 sibling, 1 reply; 7+ messages in thread
From: Martín Ferrari @ 2010-05-06 11:23 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Arnd Bergmann, netdev, Mathieu Lacage, David Miller
Eric,
On Thu, May 6, 2010 at 08:59, Eric Dumazet <eric.dumazet@gmail.com> wrote:
>> > Could you please try following patch ?
> David, this is a stable candidate, once tested and acked, thanks !
>
> [PATCH] veth: Dont kfree_skb() after dev_forward_skb()
I have been running a bunch of benchmarks and it didn't cause any
trouble. Seems the fix is good
Thanks a lot!!
--
Martín Ferrari
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: kernel panic when using netns+bridges+tc(netem)
2010-05-06 11:23 ` Martín Ferrari
@ 2010-05-06 19:15 ` Eric Dumazet
0 siblings, 0 replies; 7+ messages in thread
From: Eric Dumazet @ 2010-05-06 19:15 UTC (permalink / raw)
To: Martín Ferrari; +Cc: Arnd Bergmann, netdev, Mathieu Lacage, David Miller
Le jeudi 06 mai 2010 à 13:23 +0200, Martín Ferrari a écrit :
> Eric,
>
> On Thu, May 6, 2010 at 08:59, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> >> > Could you please try following patch ?
> > David, this is a stable candidate, once tested and acked, thanks !
> >
> > [PATCH] veth: Dont kfree_skb() after dev_forward_skb()
>
> I have been running a bunch of benchmarks and it didn't cause any
> trouble. Seems the fix is good
>
> Thanks a lot!!
>
Thanks for testing and report !
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2010-05-06 19:15 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2010-05-06 1:01 kernel panic when using netns+bridges+tc(netem) Martín Ferrari
2010-05-06 6:40 ` Eric Dumazet
2010-05-06 6:47 ` Eric Dumazet
2010-05-06 6:59 ` Eric Dumazet
2010-05-06 7:54 ` David Miller
2010-05-06 11:23 ` Martín Ferrari
2010-05-06 19:15 ` Eric Dumazet
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox