* 2.6.38.2, kernel panic, probably related to framentation handling
@ 2011-05-04 11:36 Denys Fedoryshchenko
2011-05-04 17:03 ` Eric Dumazet
0 siblings, 1 reply; 6+ messages in thread
From: Denys Fedoryshchenko @ 2011-05-04 11:36 UTC (permalink / raw)
To: netdev
Seems once more, during trying to bring another type of tunnel (this
time userspace, working over tun device) and switching routes got one
more kernel panic
It is vanilla kernel, but many source routing rules, firewall, QoS and
etc, including this tunnel now also. Here is what i got on netconsole:
Any other info required?
netc [1192230.881002]
netc [1192230.881002] Pid: 0, comm: kworker/0:1 Not tainted
2.6.38.2-devel2 #2
netc
netc Dell Inc. PowerEdge 1950
netc /
netc 0D8635
netc
netc [1192230.881002] EIP: 0060:[<c03c0847>] EFLAGS: 00010206 CPU: 3
netc [1192230.881002] EIP is at icmp_send+0x39/0x396
netc [1192230.881002] EAX: 121a8aca EBX: d1d28600 ECX: 00000001 EDX:
c63b6600
netc [1192230.881002] ESI: d1d28600 EDI: c33438a0 EBP: f2a41840 ESP:
f64b5e8c
netc [1192230.881002] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
netc [1192230.881002] Process kworker/0:1 (pid: 0, ti=f64b4000
task=f64a4a80 task.ti=f64b0000)
netc [1192230.881002] Stack:
netc [1192230.881002] c0113ea1
netc 00000001
netc 0000000b
netc 00000000
netc efed48c7
netc 0000114a
netc 00000000
netc f6b01fa8
netc
netc [1192230.881002] e21be5e1
netc c0148bf3
netc e217d5d0
netc 00043c53
netc 00000001
netc f6b02d14
netc e21be5e1
netc 00043c53
netc
netc [1192230.881002] e21be5e1
netc 00043c53
netc c0148d2e
netc 00000000
netc 00000058
netc 00000000
netc c0140779
netc ce9f5aa9
netc
netc [1192230.881002] Call Trace:
netc [1192230.881002] [<c0113ea1>] ? lapic_next_event+0x13/0x16
netc [1192230.881002] [<c0148bf3>] ? tick_dev_program_event+0x26/0x116
netc [1192230.881002] [<c0148d2e>] ? tick_program_event+0x1b/0x1f
netc [1192230.881002] [<c0140779>] ? hrtimer_interrupt+0x10c/0x1ca
netc [1192230.881002] [<c0140e49>] ? hrtimer_start+0x20/0x25
netc [1192230.881002] [<c012f18e>] ? irq_exit+0x36/0x59
netc [1192230.881002] [<c0114933>] ?
smp_apic_timer_interrupt+0x71/0x7d
netc [1192230.881002] [<c03f2752>] ? apic_timer_interrupt+0x2a/0x30
netc [1192230.881002] [<c039f527>] ? ip_expire+0xf2/0x11b
netc [1192230.881002] [<c039f435>] ? ip_expire+0x0/0x11b
netc [1192230.881002] [<c0133421>] ? run_timer_softirq+0x140/0x1c7
netc [1192230.881002] [<c012f28f>] ? __do_softirq+0x6b/0x104
netc [1192230.881002] [<c012f224>] ? __do_softirq+0x0/0x104
netc [1192230.881002] [<c012f224>] ? __do_softirq+0x0/0x104
netc [1192230.881002] <IRQ>
netc
netc [1192230.881002] [<c012f17e>] ? irq_exit+0x26/0x59
netc [1192230.881002] [<c0103b3d>] ? do_IRQ+0x81/0x95
netc [1192230.881002] [<c0114933>] ?
smp_apic_timer_interrupt+0x71/0x7d
netc [1192230.881002] [<c0102ca9>] ? common_interrupt+0x29/0x30
netc [1192230.881002] [<c010807a>] ? mwait_idle+0x51/0x56
netc [1192230.881002] [<c0101a97>] ? cpu_idle+0x41/0x5e
netc [1192230.881002] Code:
netc 08
netc 89
netc c6
netc 89
netc 4c
netc 24
netc 04
netc 8b
netc 40
netc 48
netc 89
netc c2
netc 83
netc e2
netc fe
netc 0f
netc 84
netc 66
netc 03
netc 00
netc 00
netc 89
netc 94
netc 24
netc c0
netc 00
netc 00
netc 00
netc 8b
netc 42
netc 0c
netc 8b
netc be
netc 94
netc 00
netc 00
netc 00
netc 3b
netc be
netc a4
netc 00
netc 00
netc 00
May 4 11:17:12 217.151.224.119 unparseable log message: "<8b> "
netc 80
netc 80
netc 02
netc 00
netc 00
netc 89
netc 44
netc 24
netc 10
netc 0f
netc 82
netc 40
netc 03
netc 00
netc 00
netc 8d
netc 47
netc 14
netc 39
netc 86
netc
netc [1192230.881002] EIP: [<c03c0847>]
netc icmp_send+0x39/0x396
netc SS:ESP 0068:f64b5e8c
netc [1192230.881002] CR2: 00000000121a8d4a
netc [1192230.910072] ---[ end trace 42aae79d7fb08725 ]---
netc [1192230.910354] Kernel panic - not syncing: Fatal exception in
interrupt
netc [1192230.911062] Rebooting in 5 seconds..
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: 2.6.38.2, kernel panic, probably related to framentation handling 2011-05-04 11:36 2.6.38.2, kernel panic, probably related to framentation handling Denys Fedoryshchenko @ 2011-05-04 17:03 ` Eric Dumazet 2011-05-04 18:09 ` Denys Fedoryshchenko 2011-05-04 18:11 ` Eric Dumazet 0 siblings, 2 replies; 6+ messages in thread From: Eric Dumazet @ 2011-05-04 17:03 UTC (permalink / raw) To: Denys Fedoryshchenko; +Cc: netdev Le mercredi 04 mai 2011 à 14:36 +0300, Denys Fedoryshchenko a écrit : > Seems once more, during trying to bring another type of tunnel (this > time userspace, working over tun device) and switching routes got one > more kernel panic > It is vanilla kernel, but many source routing rules, firewall, QoS and > etc, including this tunnel now also. Here is what i got on netconsole: > Any other info required? > > netc [1192230.881002] > netc [1192230.881002] Pid: 0, comm: kworker/0:1 Not tainted > 2.6.38.2-devel2 #2 > netc > netc Dell Inc. PowerEdge 1950 > netc / > netc 0D8635 > netc > netc [1192230.881002] EIP: 0060:[<c03c0847>] EFLAGS: 00010206 CPU: 3 > netc [1192230.881002] EIP is at icmp_send+0x39/0x396 > netc [1192230.881002] EAX: 121a8aca EBX: d1d28600 ECX: 00000001 EDX: > c63b6600 > netc [1192230.881002] ESI: d1d28600 EDI: c33438a0 EBP: f2a41840 ESP: > f64b5e8c > netc [1192230.881002] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 > netc [1192230.881002] Process kworker/0:1 (pid: 0, ti=f64b4000 > task=f64a4a80 task.ti=f64b0000) > netc [1192230.881002] Stack: > netc [1192230.881002] c0113ea1 > netc 00000001 > netc 0000000b > netc 00000000 > netc efed48c7 > netc 0000114a > netc 00000000 > netc f6b01fa8 > netc > netc [1192230.881002] e21be5e1 > netc c0148bf3 > netc e217d5d0 > netc 00043c53 > netc 00000001 > netc f6b02d14 > netc e21be5e1 > netc 00043c53 > netc > netc [1192230.881002] e21be5e1 > netc 00043c53 > netc c0148d2e > netc 00000000 > netc 00000058 > netc 00000000 > netc c0140779 > netc ce9f5aa9 > netc > netc [1192230.881002] Call Trace: > netc [1192230.881002] [<c0113ea1>] ? lapic_next_event+0x13/0x16 > netc [1192230.881002] [<c0148bf3>] ? tick_dev_program_event+0x26/0x116 > netc [1192230.881002] [<c0148d2e>] ? tick_program_event+0x1b/0x1f > netc [1192230.881002] [<c0140779>] ? hrtimer_interrupt+0x10c/0x1ca > netc [1192230.881002] [<c0140e49>] ? hrtimer_start+0x20/0x25 > netc [1192230.881002] [<c012f18e>] ? irq_exit+0x36/0x59 > netc [1192230.881002] [<c0114933>] ? > smp_apic_timer_interrupt+0x71/0x7d > netc [1192230.881002] [<c03f2752>] ? apic_timer_interrupt+0x2a/0x30 > netc [1192230.881002] [<c039f527>] ? ip_expire+0xf2/0x11b > netc [1192230.881002] [<c039f435>] ? ip_expire+0x0/0x11b > netc [1192230.881002] [<c0133421>] ? run_timer_softirq+0x140/0x1c7 > netc [1192230.881002] [<c012f28f>] ? __do_softirq+0x6b/0x104 > netc [1192230.881002] [<c012f224>] ? __do_softirq+0x0/0x104 > netc [1192230.881002] [<c012f224>] ? __do_softirq+0x0/0x104 > netc [1192230.881002] <IRQ> > netc > netc [1192230.881002] [<c012f17e>] ? irq_exit+0x26/0x59 > netc [1192230.881002] [<c0103b3d>] ? do_IRQ+0x81/0x95 > netc [1192230.881002] [<c0114933>] ? > smp_apic_timer_interrupt+0x71/0x7d > netc [1192230.881002] [<c0102ca9>] ? common_interrupt+0x29/0x30 > netc [1192230.881002] [<c010807a>] ? mwait_idle+0x51/0x56 > netc [1192230.881002] [<c0101a97>] ? cpu_idle+0x41/0x5e > netc [1192230.881002] Code: > netc 08 > netc 89 > netc c6 > netc 89 > netc 4c > netc 24 > netc 04 > netc 8b > netc 40 > netc 48 > netc 89 > netc c2 > netc 83 > netc e2 > netc fe > netc 0f > netc 84 > netc 66 > netc 03 > netc 00 > netc 00 > netc 89 > netc 94 > netc 24 > netc c0 > netc 00 > netc 00 > netc 00 > netc 8b > netc 42 > netc 0c > netc 8b > netc be > netc 94 > netc 00 > netc 00 > netc 00 > netc 3b > netc be > netc a4 > netc 00 > netc 00 > netc 00 > May 4 11:17:12 217.151.224.119 unparseable log message: "<8b> " > netc 80 > netc 80 > netc 02 > netc 00 > netc 00 > netc 89 > netc 44 > netc 24 > netc 10 > netc 0f > netc 82 > netc 40 > netc 03 > netc 00 > netc 00 > netc 8d > netc 47 > netc 14 > netc 39 > netc 86 > netc > netc [1192230.881002] EIP: [<c03c0847>] > netc icmp_send+0x39/0x396 > netc SS:ESP 0068:f64b5e8c > netc [1192230.881002] CR2: 00000000121a8d4a > netc [1192230.910072] ---[ end trace 42aae79d7fb08725 ]--- > netc [1192230.910354] Kernel panic - not syncing: Fatal exception in > interrupt > netc [1192230.911062] Rebooting in 5 seconds.. > Hi Denys Is it reproductible, and possibly on latest kernel ? We fixed some bugs lately (assuming you also use a bridge ?) Could you send the disassembled code on your kernel of icmp_send() ? Thanks ! ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 2.6.38.2, kernel panic, probably related to framentation handling 2011-05-04 17:03 ` Eric Dumazet @ 2011-05-04 18:09 ` Denys Fedoryshchenko 2011-05-04 18:11 ` Eric Dumazet 1 sibling, 0 replies; 6+ messages in thread From: Denys Fedoryshchenko @ 2011-05-04 18:09 UTC (permalink / raw) To: Eric Dumazet; +Cc: netdev On Wed, 04 May 2011 19:03:01 +0200, Eric Dumazet wrote: > Le mercredi 04 mai 2011 à 14:36 +0300, Denys Fedoryshchenko a écrit : >> Seems once more, during trying to bring another type of tunnel (this >> time userspace, working over tun device) and switching routes got >> one >> more kernel panic >> It is vanilla kernel, but many source routing rules, firewall, QoS >> and >> etc, including this tunnel now also. Here is what i got on >> netconsole: >> Any other info required? >> >> netc [1192230.881002] >> netc [1192230.881002] Pid: 0, comm: kworker/0:1 Not tainted >> 2.6.38.2-devel2 #2 >> netc >> netc Dell Inc. PowerEdge 1950 >> netc / >> netc 0D8635 >> netc >> netc [1192230.881002] EIP: 0060:[<c03c0847>] EFLAGS: 00010206 CPU: >> 3 >> netc [1192230.881002] EIP is at icmp_send+0x39/0x396 >> netc [1192230.881002] EAX: 121a8aca EBX: d1d28600 ECX: 00000001 >> EDX: >> c63b6600 >> netc [1192230.881002] ESI: d1d28600 EDI: c33438a0 EBP: f2a41840 >> ESP: >> f64b5e8c >> netc [1192230.881002] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 >> netc [1192230.881002] Process kworker/0:1 (pid: 0, ti=f64b4000 >> task=f64a4a80 task.ti=f64b0000) >> netc [1192230.881002] Stack: >> netc [1192230.881002] c0113ea1 >> netc 00000001 >> netc 0000000b >> netc 00000000 >> netc efed48c7 >> netc 0000114a >> netc 00000000 >> netc f6b01fa8 >> netc >> netc [1192230.881002] e21be5e1 >> netc c0148bf3 >> netc e217d5d0 >> netc 00043c53 >> netc 00000001 >> netc f6b02d14 >> netc e21be5e1 >> netc 00043c53 >> netc >> netc [1192230.881002] e21be5e1 >> netc 00043c53 >> netc c0148d2e >> netc 00000000 >> netc 00000058 >> netc 00000000 >> netc c0140779 >> netc ce9f5aa9 >> netc >> netc [1192230.881002] Call Trace: >> netc [1192230.881002] [<c0113ea1>] ? lapic_next_event+0x13/0x16 >> netc [1192230.881002] [<c0148bf3>] ? >> tick_dev_program_event+0x26/0x116 >> netc [1192230.881002] [<c0148d2e>] ? tick_program_event+0x1b/0x1f >> netc [1192230.881002] [<c0140779>] ? hrtimer_interrupt+0x10c/0x1ca >> netc [1192230.881002] [<c0140e49>] ? hrtimer_start+0x20/0x25 >> netc [1192230.881002] [<c012f18e>] ? irq_exit+0x36/0x59 >> netc [1192230.881002] [<c0114933>] ? >> smp_apic_timer_interrupt+0x71/0x7d >> netc [1192230.881002] [<c03f2752>] ? >> apic_timer_interrupt+0x2a/0x30 >> netc [1192230.881002] [<c039f527>] ? ip_expire+0xf2/0x11b >> netc [1192230.881002] [<c039f435>] ? ip_expire+0x0/0x11b >> netc [1192230.881002] [<c0133421>] ? run_timer_softirq+0x140/0x1c7 >> netc [1192230.881002] [<c012f28f>] ? __do_softirq+0x6b/0x104 >> netc [1192230.881002] [<c012f224>] ? __do_softirq+0x0/0x104 >> netc [1192230.881002] [<c012f224>] ? __do_softirq+0x0/0x104 >> netc [1192230.881002] <IRQ> >> netc >> netc [1192230.881002] [<c012f17e>] ? irq_exit+0x26/0x59 >> netc [1192230.881002] [<c0103b3d>] ? do_IRQ+0x81/0x95 >> netc [1192230.881002] [<c0114933>] ? >> smp_apic_timer_interrupt+0x71/0x7d >> netc [1192230.881002] [<c0102ca9>] ? common_interrupt+0x29/0x30 >> netc [1192230.881002] [<c010807a>] ? mwait_idle+0x51/0x56 >> netc [1192230.881002] [<c0101a97>] ? cpu_idle+0x41/0x5e >> netc [1192230.881002] Code: >> netc 08 >> netc 89 >> netc c6 >> netc 89 >> netc 4c >> netc 24 >> netc 04 >> netc 8b >> netc 40 >> netc 48 >> netc 89 >> netc c2 >> netc 83 >> netc e2 >> netc fe >> netc 0f >> netc 84 >> netc 66 >> netc 03 >> netc 00 >> netc 00 >> netc 89 >> netc 94 >> netc 24 >> netc c0 >> netc 00 >> netc 00 >> netc 00 >> netc 8b >> netc 42 >> netc 0c >> netc 8b >> netc be >> netc 94 >> netc 00 >> netc 00 >> netc 00 >> netc 3b >> netc be >> netc a4 >> netc 00 >> netc 00 >> netc 00 >> May 4 11:17:12 217.151.224.119 unparseable log message: "<8b> " >> netc 80 >> netc 80 >> netc 02 >> netc 00 >> netc 00 >> netc 89 >> netc 44 >> netc 24 >> netc 10 >> netc 0f >> netc 82 >> netc 40 >> netc 03 >> netc 00 >> netc 00 >> netc 8d >> netc 47 >> netc 14 >> netc 39 >> netc 86 >> netc >> netc [1192230.881002] EIP: [<c03c0847>] >> netc icmp_send+0x39/0x396 >> netc SS:ESP 0068:f64b5e8c >> netc [1192230.881002] CR2: 00000000121a8d4a >> netc [1192230.910072] ---[ end trace 42aae79d7fb08725 ]--- >> netc [1192230.910354] Kernel panic - not syncing: Fatal exception >> in >> interrupt >> netc [1192230.911062] Rebooting in 5 seconds.. >> > > Hi Denys > > Is it reproductible, and possibly on latest kernel ? > > We fixed some bugs lately (assuming you also use a bridge ?) > > Could you send the disassembled code on your kernel of icmp_send() ? > > Thanks ! No bridge. Tunnel was used in TUN mode, not TAP. I will try to reproduce at night on latest kernel. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 2.6.38.2, kernel panic, probably related to framentation handling 2011-05-04 17:03 ` Eric Dumazet 2011-05-04 18:09 ` Denys Fedoryshchenko @ 2011-05-04 18:11 ` Eric Dumazet 2011-05-04 20:02 ` Eric Dumazet 1 sibling, 1 reply; 6+ messages in thread From: Eric Dumazet @ 2011-05-04 18:11 UTC (permalink / raw) To: Denys Fedoryshchenko; +Cc: netdev Le mercredi 04 mai 2011 à 19:03 +0200, Eric Dumazet a écrit : > Hi Denys > > Is it reproductible, and possibly on latest kernel ? > > We fixed some bugs lately (assuming you also use a bridge ?) > > Could you send the disassembled code on your kernel of icmp_send() ? Oh well, I think I found the problem, I am working on a patch and send it shortly. Thanks ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 2.6.38.2, kernel panic, probably related to framentation handling 2011-05-04 18:11 ` Eric Dumazet @ 2011-05-04 20:02 ` Eric Dumazet 2011-05-04 21:05 ` David Miller 0 siblings, 1 reply; 6+ messages in thread From: Eric Dumazet @ 2011-05-04 20:02 UTC (permalink / raw) To: Denys Fedoryshchenko, David Miller; +Cc: netdev Le mercredi 04 mai 2011 à 20:11 +0200, Eric Dumazet a écrit : > Le mercredi 04 mai 2011 à 19:03 +0200, Eric Dumazet a écrit : > > > Hi Denys > > > > Is it reproductible, and possibly on latest kernel ? > > > > We fixed some bugs lately (assuming you also use a bridge ?) > > > > Could you send the disassembled code on your kernel of icmp_send() ? > > Oh well, I think I found the problem, I am working on a patch and send > it shortly. > > Thanks > I believe bug is one year old (2.6.35), please try following patch. Thanks ! [PATCH] net: ip_expire() must revalidate route Commit 4a94445c9a5c (net: Use ip_route_input_noref() in input path) added a bug in IP defragmentation handling, in case timeout is fired. When a frame is defragmented, we use last skb dst field when building final skb. Its dst is valid, since we are in rcu read section. But if a timeout occurs, we take first queued fragment to build one ICMP TIME EXCEEDED message. Problem is all queued skb have weak dst pointers, since we escaped RCU critical section after their queueing. icmp_send() might dereference a now freed (and possibly reused) part of memory. Calling skb_dst_drop() and ip_route_input_noref() to revalidate route is the only possible choice. Reported-by: Denys Fedoryshchenko <denys@visp.net.lb> Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com> --- net/ipv4/ip_fragment.c | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-) diff --git a/net/ipv4/ip_fragment.c b/net/ipv4/ip_fragment.c index a1151b8..b1d282f 100644 --- a/net/ipv4/ip_fragment.c +++ b/net/ipv4/ip_fragment.c @@ -223,31 +223,30 @@ static void ip_expire(unsigned long arg) if ((qp->q.last_in & INET_FRAG_FIRST_IN) && qp->q.fragments != NULL) { struct sk_buff *head = qp->q.fragments; + const struct iphdr *iph; + int err; rcu_read_lock(); head->dev = dev_get_by_index_rcu(net, qp->iif); if (!head->dev) goto out_rcu_unlock; + /* skb dst is stale, drop it, and perform route lookup again */ + skb_dst_drop(head); + iph = ip_hdr(head); + err = ip_route_input_noref(head, iph->daddr, iph->saddr, + iph->tos, head->dev); + if (err) + goto out_rcu_unlock; + /* - * Only search router table for the head fragment, - * when defraging timeout at PRE_ROUTING HOOK. + * Only an end host needs to send an ICMP + * "Fragment Reassembly Timeout" message, per RFC792. */ - if (qp->user == IP_DEFRAG_CONNTRACK_IN && !skb_dst(head)) { - const struct iphdr *iph = ip_hdr(head); - int err = ip_route_input(head, iph->daddr, iph->saddr, - iph->tos, head->dev); - if (unlikely(err)) - goto out_rcu_unlock; - - /* - * Only an end host needs to send an ICMP - * "Fragment Reassembly Timeout" message, per RFC792. - */ - if (skb_rtable(head)->rt_type != RTN_LOCAL) - goto out_rcu_unlock; + if (qp->user == IP_DEFRAG_CONNTRACK_IN && + skb_rtable(head)->rt_type != RTN_LOCAL) + goto out_rcu_unlock; - } /* Send an ICMP "Fragment Reassembly Timeout" message. */ icmp_send(head, ICMP_TIME_EXCEEDED, ICMP_EXC_FRAGTIME, 0); ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: 2.6.38.2, kernel panic, probably related to framentation handling 2011-05-04 20:02 ` Eric Dumazet @ 2011-05-04 21:05 ` David Miller 0 siblings, 0 replies; 6+ messages in thread From: David Miller @ 2011-05-04 21:05 UTC (permalink / raw) To: eric.dumazet; +Cc: denys, netdev From: Eric Dumazet <eric.dumazet@gmail.com> Date: Wed, 04 May 2011 22:02:26 +0200 > [PATCH] net: ip_expire() must revalidate route > > Commit 4a94445c9a5c (net: Use ip_route_input_noref() in input path) > added a bug in IP defragmentation handling, in case timeout is fired. > > When a frame is defragmented, we use last skb dst field when building > final skb. Its dst is valid, since we are in rcu read section. > > But if a timeout occurs, we take first queued fragment to build one ICMP > TIME EXCEEDED message. Problem is all queued skb have weak dst pointers, > since we escaped RCU critical section after their queueing. icmp_send() > might dereference a now freed (and possibly reused) part of memory. > > Calling skb_dst_drop() and ip_route_input_noref() to revalidate route is > the only possible choice. > > Reported-by: Denys Fedoryshchenko <denys@visp.net.lb> > Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com> > --- Applied to net-2.6 and queued up for -stable, thanks Eric! ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2011-05-04 21:06 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2011-05-04 11:36 2.6.38.2, kernel panic, probably related to framentation handling Denys Fedoryshchenko 2011-05-04 17:03 ` Eric Dumazet 2011-05-04 18:09 ` Denys Fedoryshchenko 2011-05-04 18:11 ` Eric Dumazet 2011-05-04 20:02 ` Eric Dumazet 2011-05-04 21:05 ` David Miller
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).