* 2.6.30-rc1: e1000: list_add corruption
@ 2009-04-09 3:45 Alexander Beregalov
[not found] ` <a4423d670904082045q41274747ha08d28a0645ae378-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2009-04-09 8:08 ` David Miller
0 siblings, 2 replies; 9+ messages in thread
From: Alexander Beregalov @ 2009-04-09 3:45 UTC (permalink / raw)
To: Kernel Testers List, Netdev,
e1000-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f
WARNING: at lib/list_debug.c:30 __list_add+0x89/0x90()
Hardware name:
list_add corruption. prev->next should be next (c06ea834), but was
f70244c8. (prev=c06ea834).
Modules linked in: w83627hf hwmon_vid i2c_nforce2
Pid: 1436, comm: portageq Not tainted 2.6.30-rc1 #1
Call Trace:
[<c0129d73>] warn_slowpath+0x73/0xd0
[<c03c6008>] ? __kfree_skb+0x38/0x90
[<c03f9b06>] ? tcp_data_snd_check+0x26/0xe0
[<c03fd67f>] ? tcp_rcv_established+0x2bf/0x5e0
[<c040557a>] ? tcp_v4_rcv+0x47a/0x610
[<c014cebd>] ? print_lock_contention_bug+0x1d/0x110
[<c044a967>] ? _spin_unlock+0x27/0x50
[<c040564b>] ? tcp_v4_rcv+0x54b/0x610
[<c02d86f9>] __list_add+0x89/0x90
[<c03ccff9>] __napi_schedule+0x29/0x60
[<c036946d>] e1000_intr+0xbd/0x1a0
[<c015c5de>] handle_IRQ_event+0x3e/0x120
[<c015e190>] handle_fasteoi_irq+0x60/0xd0
[<c0104fd4>] handle_irq+0x34/0x60
[<c015f748>] ? rcu_irq_enter+0x8/0x40
[<c0104b29>] do_IRQ+0x39/0xa0
[<c03c592c>] ? skb_release_head_state+0x2c/0x60
[<c01034ee>] common_interrupt+0x2e/0x34
[<c02d8601>] ? list_del+0x21/0x90
[<c014e54b>] ? trace_hardirqs_on+0xb/0x10
[<c03cd4da>] __napi_complete+0x1a/0x30
[<c0381971>] nv_napi_poll+0xd1/0x5c0
[<c014e54b>] ? trace_hardirqs_on+0xb/0x10
[<c03cd5f6>] net_rx_action+0x106/0x1b0
[<c012e8df>] __do_softirq+0x6f/0x100
[<c044a967>] ? _spin_unlock+0x27/0x50
[<c015e1b8>] ? handle_fasteoi_irq+0x88/0xd0
[<c012e9cd>] do_softirq+0x5d/0x70
[<c012ebad>] irq_exit+0x7d/0xa0
[<c0104b32>] do_IRQ+0x42/0xa0
[<c012e9b7>] ? do_softirq+0x47/0x70
[<c01034ee>] common_interrupt+0x2e/0x34
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 2.6.30-rc1: e1000: list_add corruption
[not found] ` <a4423d670904082045q41274747ha08d28a0645ae378-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2009-04-09 3:48 ` Alexander Beregalov
[not found] ` <a4423d670904082048i26585b9boacf7ae449e004b8c-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
0 siblings, 1 reply; 9+ messages in thread
From: Alexander Beregalov @ 2009-04-09 3:48 UTC (permalink / raw)
To: Kernel Testers List, Netdev,
e1000-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f
2009/4/9 Alexander Beregalov <a.beregalov-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
> WARNING: at lib/list_debug.c:30 __list_add+0x89/0x90()
> Hardware name:
> list_add corruption. prev->next should be next (c06ea834), but was
> f70244c8. (prev=c06ea834).
> Modules linked in: w83627hf hwmon_vid i2c_nforce2
> Pid: 1436, comm: portageq Not tainted 2.6.30-rc1 #1
> Call Trace:
> [<c0129d73>] warn_slowpath+0x73/0xd0
> [<c03c6008>] ? __kfree_skb+0x38/0x90
> [<c03f9b06>] ? tcp_data_snd_check+0x26/0xe0
> [<c03fd67f>] ? tcp_rcv_established+0x2bf/0x5e0
> [<c040557a>] ? tcp_v4_rcv+0x47a/0x610
> [<c014cebd>] ? print_lock_contention_bug+0x1d/0x110
> [<c044a967>] ? _spin_unlock+0x27/0x50
> [<c040564b>] ? tcp_v4_rcv+0x54b/0x610
> [<c02d86f9>] __list_add+0x89/0x90
> [<c03ccff9>] __napi_schedule+0x29/0x60
> [<c036946d>] e1000_intr+0xbd/0x1a0
> [<c015c5de>] handle_IRQ_event+0x3e/0x120
> [<c015e190>] handle_fasteoi_irq+0x60/0xd0
> [<c0104fd4>] handle_irq+0x34/0x60
> [<c015f748>] ? rcu_irq_enter+0x8/0x40
> [<c0104b29>] do_IRQ+0x39/0xa0
> [<c03c592c>] ? skb_release_head_state+0x2c/0x60
> [<c01034ee>] common_interrupt+0x2e/0x34
> [<c02d8601>] ? list_del+0x21/0x90
> [<c014e54b>] ? trace_hardirqs_on+0xb/0x10
> [<c03cd4da>] __napi_complete+0x1a/0x30
> [<c0381971>] nv_napi_poll+0xd1/0x5c0
> [<c014e54b>] ? trace_hardirqs_on+0xb/0x10
> [<c03cd5f6>] net_rx_action+0x106/0x1b0
> [<c012e8df>] __do_softirq+0x6f/0x100
> [<c044a967>] ? _spin_unlock+0x27/0x50
> [<c015e1b8>] ? handle_fasteoi_irq+0x88/0xd0
> [<c012e9cd>] do_softirq+0x5d/0x70
> [<c012ebad>] irq_exit+0x7d/0xa0
> [<c0104b32>] do_IRQ+0x42/0xa0
> [<c012e9b7>] ? do_softirq+0x47/0x70
> [<c01034ee>] common_interrupt+0x2e/0x34
>
Addition:
WARNING: at net/sched/sch_generic.c:226 dev_watchdog+0x243/0x250()
Hardware name:
NETDEV WATCHDOG: eth1 (forcedeth): transmit timed out
Modules linked in: w83627hf hwmon_vid i2c_nforce2
Pid: 0, comm: swapper Tainted: G W 2.6.30-rc1 #1
Call Trace:
[<c0129d73>] warn_slowpath+0x73/0xd0
[<c014cebd>] ? print_lock_contention_bug+0x1d/0x110
[<c014cebd>] ? print_lock_contention_bug+0x1d/0x110
[<c014cebd>] ? print_lock_contention_bug+0x1d/0x110
[<c014cebd>] ? print_lock_contention_bug+0x1d/0x110
[<c01272f5>] ? enqueue_task_fair+0x1a5/0x330
[<c014cebd>] ? print_lock_contention_bug+0x1d/0x110
[<c014cebd>] ? print_lock_contention_bug+0x1d/0x110
[<c03df1ad>] ? dev_watchdog+0x5d/0x250
[<c02c699d>] ? strlcpy+0x1d/0x80
[<c03cd1c0>] ? netdev_drivername+0x40/0x50
[<c03df393>] dev_watchdog+0x243/0x250
[<c0132ee0>] ? run_timer_softirq+0xc0/0x230
[<c0132f86>] run_timer_softirq+0x166/0x230
[<c0132ee0>] ? run_timer_softirq+0xc0/0x230
[<c03df150>] ? dev_watchdog+0x0/0x250
[<c012e8df>] __do_softirq+0x6f/0x100
[<c0149537>] ? tick_program_event+0x17/0x20
[<c012e9cd>] do_softirq+0x5d/0x70
[<c012ebad>] irq_exit+0x7d/0xa0
[<c0114a0c>] smp_apic_timer_interrupt+0x4c/0x80
[<c02c97a4>] ? trace_hardirqs_off_thunk+0xc/0x18
[<c0103557>] apic_timer_interrupt+0x2f/0x34
[<c0101e73>] ? cpu_idle+0x43/0x80
[<c0109a6f>] ? default_idle+0x4f/0x60
[<c0101e79>] cpu_idle+0x49/0x80
[<c043dee6>] rest_init+0x66/0x70
[<c06fc9dd>] start_kernel+0x2a4/0x2e4
[<c06fc4fc>] ? unknown_bootoption+0x0/0x1ec
[<c06fc2f1>] i386_start_kernel+0x41/0x49
---[ end trace 6efabe98c03fc44a ]---
eth1: Got tx_timeout. irq: 00000032
eth1: Ring at 36b62000
eth1: Dumping tx registers
0: 00000032 00000000 00000003 c1f4ffff 00000032 00000000 00000003 c1f4ffff
20: 00000032 00000000 00000003 beb2ffff 00000032 00000000 00000003 beb2ffff
40: 00000032 00000000 00000003 bb72ffff 00000032 00000000 00000003 bb72ffff
60: 00000032 00000000 00000003 b830ffff 00000032 00000000 00000003 b831ffff
80: 003b0f3c 00000001 00000000 007f0020 0000061c 00000001 00000000 00007f8f
a0: 0015050f 00000016 f58d5000 00001611 005e0001 00000100 ffffffff 0000ffff
c0: 00000001 00000001 00000001 00000001 00000001 00000001 00000001 00000001
e0: 00000001 00000001 00000001 00000001 00000001 00000001 00000001 00000001
100: 36b63000 36b62000 01ff00ff 00008000 00010064 00000000 00000063 36b63588
120: 36b62340 2f2818ee 000105e6 366d98b0 8000061c 36b63590 36b62348 00200010
140: 00300010 00000008 00000000 00000000 00000000 00000000 00000000 00000000
160: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
180: 00000016 00000008 01947869 00008103 00000025 000041e1 0194786d 0000c1e3
1a0: 0000001e 00000008 0194786d 00008103 00000025 000041e1 0194786d 0000c1e3
1c0: 0000001e 00000008 0194786d 00008103 00000025 000041e1 0194786d 0000c1e3
1e0: 0000001e 00000008 0194786d 00008103 00000025 000041e1 0194786d 0000c1e3
200: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
220: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
240: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
260: 00000000 00000000 fe020001 00000100 00000000 00000000 fe020001 00000100
eth1: Dumping tx ring
000: 34a4c002 00010040 // 3609b3b2 00010040 // 3609ba42 00010040 //
3609bc72 00010040
004: 2f1f4002 00010040 // 2f1f4232 00010040 // 2f1f4462 00010040 //
2f1f48c2 00010040
008: 2f1f4af2 00010040 // 2f1f4d22 00010040 // 2f1f55e2 00010040 //
2f1f5812 00010040
00c: 2f1f5a42 00010040 // 205ac232 00010040 // 205ac462 00010040 //
205ac692 00010040
010: 205acaf2 00010040 // 205acd22 00010040 // 205acf52 00010040 //
205ad5e2 00010040
014: 205ad812 00010040 // 205ada42 00010040 // 260d0232 00010040 //
260d0462 00010040
018: 260d0692 00010040 // 260d0d22 00010040 // 260d0f52 00010040 //
260d1182 00010040
01c: 260d1a42 00010040 // 260d1c72 00010040 // 33794002 00010040 //
33794692 00010040
020: 337948c2 00010040 // 33794af2 00010040 // 337953b2 00010040 //
337955e2 00010040
024: 33795812 00010040 // 26202462 00010040 // 262028c2 00010040 //
26202d22 00010040
028: 262035e2 00010040 // 26203812 00010040 // 26203a42 00010040 //
2f3d8232 00010040
02c: 2f3d8462 00010040 // 2f3d8692 00010040 // 2f3d8f52 00010040 //
2f3d9182 00010040
030: 2f3d93b2 00010040 // 2f3d9c72 00010040 // 2f9d8002 00010040 //
2f9d8232 00010040
034: 2f9d8af2 00010040 // 2f9d8d22 00010040 // 2f9d8f52 00010040 //
2f9d9812 00010040
038: 2f9d9a42 00010040 // 2f9d9c72 00010040 // 260f6692 00010040 //
260f68c2 00010040
03c: 260f6af2 00010040 // 260f73b2 00010040 // 260f75e2 00010040 //
260f7812 00010040
040: 35b3e002 00010040 // 35b3e232 00010040 // 35b3e462 00010040 //
35b3ed22 00010040
044: 35b3ef52 00010040 // 35b3f182 00010040 // 35b3f812 00010040 //
35b3fa42 00010040
048: 35b3fc72 00010040 // 3591a692 00010040 // 3591a8c2 00010040 //
3591aaf2 00010040
04c: 3591b3b2 00010040 // 3591b5e2 00010040 // 3591b812 00010040 //
33624232 00010040
050: 33624462 00010040 // 33624692 00010040 // 33624d22 00010040 //
33624f52 00010040
054: 33625182 00010040 // 33625812 00010040 // 33625a42 00010040 //
33625c72 00010040
058: 26194692 00010040 // 261948c2 00010040 // 26194af2 00010040 //
26194f52 00010040
05c: 26195182 00010040 // 261953b2 00010040 // 26195c72 00010040 //
2fa58002 00010040
060: 2fa58232 00010040 // 2fa58af2 00010040 // 2fa58d22 00010040 //
2fa58f52 00010040
064: 2fa595e2 00010040 // 2fa59812 00010040 // 2fa59a42 00010040 //
30a58462 00010040
068: 30a58692 00010040 // 30a588c2 00010040 // 30a59182 00010040 //
30a593b2 00010040
06c: 30a595e2 00010040 // 3583a002 00010040 // 3583a232 00010040 //
3583a462 00010040
070: 3583af52 00010040 // 3583b3b2 00010040 // 3583b812 00010040 //
33754232 00010040
074: 33754462 00010040 // 33754692 00010040 // 33754f52 00010040 //
33755182 00010040
078: 337553b2 00010040 // 33755c72 00010040 // 2075c002 00010040 //
2075c232 00010040
07c: 2075caf2 00010040 // 2075cd22 00010040 // 2075cf52 00010040 //
2075d812 00010040
080: 2075da42 00010040 // 2075dc72 00010040 // 30980692 00010040 //
309808c2 00010040
084: 30980af2 00010040 // 309813b2 00010040 // 309815e2 00010040 //
30981812 00010040
088: 2f93a002 00010040 // 2f93a232 00010040 // 2f93a462 00010040 //
2f93ad22 00010040
08c: 2f93af52 00010040 // 2f93b182 00010040 // 2f93ba42 00010040 //
2f93bc72 00010040
090: 205c0002 00010040 // 205c08c2 00010040 // 205c0af2 00010040 //
205c0d22 00010040
094: 205c15e2 00010040 // 205c1812 00010040 // 205c1c72 00010040 //
337bc692 00010040
098: 337bc8c2 00010040 // 337bcaf2 00010040 // 337bcd22 00010040 //
337bd5e2 00010040
09c: 337bd812 00010040 // 204a4462 00010040 // 204a4692 00010040 //
204a48c2 00010040
0a0: 204a5182 00010040 // 204a55e2 00010040 // 204a5812 00010040 //
26104002 00010040
0a4: 26104232 00010040 // 26104692 00010040 // 26104f52 00010040 //
26105182 00010040
0a8: 261053b2 00010040 // 261055e2 00010040 // 3598e002 00010040 //
3598e232 00010040
0ac: 3598ed22 00010040 // 3598ef52 00010040 // 3598f182 00010040 //
3598fa42 00010040
0b0: 30ae0002 00010040 // 2f2818ee 000105e6 // 2066d23e 000105e6 //
30b5c8b6 0001004e
0b4: 30b5d116 0001004e // 2f821116 0001004e // 2f865546 0001004e //
2f35629e 000105e6
0b8: 2f823296 0001004e // 2f820ce6 0001004e // 2f821da6 0001004e //
3672b6c6 0001004e
0bc: 2f8c5546 0001004e // 2f8c7af6 0001004e // 2f8c6606 0001004e //
2f92f296 0001004e
0c0: 2f94e1d6 0001004e // 2f94d546 0001004e // 0bdbaa36 0001004e //
0bdb9976 0001004e
0c4: 0bdb88b6 0001004e // 2f8fb6c6 0001004e // 2f8f9976 0001004e //
36b176c6 0001004e
0c8: 366ea94e 000105e6 // 0bdbbaf6 0001004e // 2f864ce6 0001004e //
2f8208b6 0001004e
0cc: 2f822a36 0001004e // 2f8221d6 0001004e // 36069116 0001004e //
366ed276 0001004a
0d0: 366eca46 0001004a // 366ea156 0001004a // 366e9926 0001004a //
366ec216 0001004a
0d4: 366e90f6 0001004a // 366eeb06 0001004a // 366f2156 0001004a //
30b5c486 0001004e
0d8: 366f005e 000105e6 // 30b5f6c6 0001004e // 30b5ee66 0001004e //
30b5d546 0001004e
0dc: 30b5e1d6 0001004e // 2f8f8ce6 0001004e // 2f8fbaf6 0001004e //
30b5e606 0001004e
0e0: 30b5ea36 0001004e // 2f8fd976 0001004e // 2f820056 0001004e //
2f92ea36 0001004e
0e4: 3606b6c6 0001004e // 2f8676c6 0001004e // 2f867296 0001004e //
2f867af6 0001004e
0e8: 2f8fee6e 00010046 // 2f864486 0001004e // 2f865116 0001004e //
366f39ae 000105e6
0ec: 2f8fd116 0001004e // 2f8661d6 0001004e // 0bdb8486 0001004e //
0bdb9546 0001004e
0f0: 205a8486 0001004e // 205a88b6 0001004e // 3025d27e 000105e6 //
205a8ce6 0001004e
0f4: 2f9cb182 00010040 // 17d0527e 000105e6 // 2f9caf52 00010040 //
2f9ca692 00010040
0f8: 2f9caaf2 00010040 // 2f9ca232 00010040 // 2f9ca002 00010040 //
2f9cba42 00010040
0fc: 34a4d5e2 00010040 // 34a4cf52 00010040 // 34a4c8c2 00010040 //
34a4caf2 00010040
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 2.6.30-rc1: e1000: list_add corruption
[not found] ` <a4423d670904082048i26585b9boacf7ae449e004b8c-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2009-04-09 4:16 ` Alexander Beregalov
2009-04-09 4:20 ` David Miller
0 siblings, 1 reply; 9+ messages in thread
From: Alexander Beregalov @ 2009-04-09 4:16 UTC (permalink / raw)
To: Kernel Testers List, Netdev,
e1000-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f
2009/4/9 Alexander Beregalov <a.beregalov-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
> 2009/4/9 Alexander Beregalov <a.beregalov-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
>> WARNING: at lib/list_debug.c:30 __list_add+0x89/0x90()
>> Hardware name:
>> list_add corruption. prev->next should be next (c06ea834), but was
>> f70244c8. (prev=c06ea834).
>> Modules linked in: w83627hf hwmon_vid i2c_nforce2
>> Pid: 1436, comm: portageq Not tainted 2.6.30-rc1 #1
>> Call Trace:
>> [<c0129d73>] warn_slowpath+0x73/0xd0
>> [<c03c6008>] ? __kfree_skb+0x38/0x90
>> [<c03f9b06>] ? tcp_data_snd_check+0x26/0xe0
>> [<c03fd67f>] ? tcp_rcv_established+0x2bf/0x5e0
>> [<c040557a>] ? tcp_v4_rcv+0x47a/0x610
>> [<c014cebd>] ? print_lock_contention_bug+0x1d/0x110
>> [<c044a967>] ? _spin_unlock+0x27/0x50
>> [<c040564b>] ? tcp_v4_rcv+0x54b/0x610
>> [<c02d86f9>] __list_add+0x89/0x90
>> [<c03ccff9>] __napi_schedule+0x29/0x60
>> [<c036946d>] e1000_intr+0xbd/0x1a0
>> [<c015c5de>] handle_IRQ_event+0x3e/0x120
>> [<c015e190>] handle_fasteoi_irq+0x60/0xd0
>> [<c0104fd4>] handle_irq+0x34/0x60
>> [<c015f748>] ? rcu_irq_enter+0x8/0x40
>> [<c0104b29>] do_IRQ+0x39/0xa0
>> [<c03c592c>] ? skb_release_head_state+0x2c/0x60
>> [<c01034ee>] common_interrupt+0x2e/0x34
>> [<c02d8601>] ? list_del+0x21/0x90
>> [<c014e54b>] ? trace_hardirqs_on+0xb/0x10
>> [<c03cd4da>] __napi_complete+0x1a/0x30
>> [<c0381971>] nv_napi_poll+0xd1/0x5c0
>> [<c014e54b>] ? trace_hardirqs_on+0xb/0x10
>> [<c03cd5f6>] net_rx_action+0x106/0x1b0
>> [<c012e8df>] __do_softirq+0x6f/0x100
>> [<c044a967>] ? _spin_unlock+0x27/0x50
>> [<c015e1b8>] ? handle_fasteoi_irq+0x88/0xd0
>> [<c012e9cd>] do_softirq+0x5d/0x70
>> [<c012ebad>] irq_exit+0x7d/0xa0
>> [<c0104b32>] do_IRQ+0x42/0xa0
>> [<c012e9b7>] ? do_softirq+0x47/0x70
>> [<c01034ee>] common_interrupt+0x2e/0x34
>>
> Addition:
>
> WARNING: at net/sched/sch_generic.c:226 dev_watchdog+0x243/0x250()
> Hardware name:
> NETDEV WATCHDOG: eth1 (forcedeth): transmit timed out
> Modules linked in: w83627hf hwmon_vid i2c_nforce2
> Pid: 0, comm: swapper Tainted: G W 2.6.30-rc1 #1
> Call Trace:
> [<c0129d73>] warn_slowpath+0x73/0xd0
> [<c014cebd>] ? print_lock_contention_bug+0x1d/0x110
> [<c014cebd>] ? print_lock_contention_bug+0x1d/0x110
> [<c014cebd>] ? print_lock_contention_bug+0x1d/0x110
> [<c014cebd>] ? print_lock_contention_bug+0x1d/0x110
> [<c01272f5>] ? enqueue_task_fair+0x1a5/0x330
> [<c014cebd>] ? print_lock_contention_bug+0x1d/0x110
> [<c014cebd>] ? print_lock_contention_bug+0x1d/0x110
> [<c03df1ad>] ? dev_watchdog+0x5d/0x250
> [<c02c699d>] ? strlcpy+0x1d/0x80
> [<c03cd1c0>] ? netdev_drivername+0x40/0x50
> [<c03df393>] dev_watchdog+0x243/0x250
> [<c0132ee0>] ? run_timer_softirq+0xc0/0x230
> [<c0132f86>] run_timer_softirq+0x166/0x230
> [<c0132ee0>] ? run_timer_softirq+0xc0/0x230
> [<c03df150>] ? dev_watchdog+0x0/0x250
> [<c012e8df>] __do_softirq+0x6f/0x100
> [<c0149537>] ? tick_program_event+0x17/0x20
> [<c012e9cd>] do_softirq+0x5d/0x70
> [<c012ebad>] irq_exit+0x7d/0xa0
> [<c0114a0c>] smp_apic_timer_interrupt+0x4c/0x80
> [<c02c97a4>] ? trace_hardirqs_off_thunk+0xc/0x18
> [<c0103557>] apic_timer_interrupt+0x2f/0x34
> [<c0101e73>] ? cpu_idle+0x43/0x80
> [<c0109a6f>] ? default_idle+0x4f/0x60
> [<c0101e79>] cpu_idle+0x49/0x80
> [<c043dee6>] rest_init+0x66/0x70
> [<c06fc9dd>] start_kernel+0x2a4/0x2e4
> [<c06fc4fc>] ? unknown_bootoption+0x0/0x1ec
> [<c06fc2f1>] i386_start_kernel+0x41/0x49
> ---[ end trace 6efabe98c03fc44a ]---
> eth1: Got tx_timeout. irq: 00000032
> eth1: Ring at 36b62000
> eth1: Dumping tx registers
> 0: 00000032 00000000 00000003 c1f4ffff 00000032 00000000 00000003 c1f4ffff
> 20: 00000032 00000000 00000003 beb2ffff 00000032 00000000 00000003 beb2ffff
> 40: 00000032 00000000 00000003 bb72ffff 00000032 00000000 00000003 bb72ffff
> 60: 00000032 00000000 00000003 b830ffff 00000032 00000000 00000003 b831ffff
> 80: 003b0f3c 00000001 00000000 007f0020 0000061c 00000001 00000000 00007f8f
> a0: 0015050f 00000016 f58d5000 00001611 005e0001 00000100 ffffffff 0000ffff
> c0: 00000001 00000001 00000001 00000001 00000001 00000001 00000001 00000001
> e0: 00000001 00000001 00000001 00000001 00000001 00000001 00000001 00000001
> 100: 36b63000 36b62000 01ff00ff 00008000 00010064 00000000 00000063 36b63588
> 120: 36b62340 2f2818ee 000105e6 366d98b0 8000061c 36b63590 36b62348 00200010
> 140: 00300010 00000008 00000000 00000000 00000000 00000000 00000000 00000000
> 160: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> 180: 00000016 00000008 01947869 00008103 00000025 000041e1 0194786d 0000c1e3
> 1a0: 0000001e 00000008 0194786d 00008103 00000025 000041e1 0194786d 0000c1e3
> 1c0: 0000001e 00000008 0194786d 00008103 00000025 000041e1 0194786d 0000c1e3
> 1e0: 0000001e 00000008 0194786d 00008103 00000025 000041e1 0194786d 0000c1e3
> 200: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> 220: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> 240: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> 260: 00000000 00000000 fe020001 00000100 00000000 00000000 fe020001 00000100
> eth1: Dumping tx ring
> 000: 34a4c002 00010040 // 3609b3b2 00010040 // 3609ba42 00010040 //
> 3609bc72 00010040
> 004: 2f1f4002 00010040 // 2f1f4232 00010040 // 2f1f4462 00010040 //
> 2f1f48c2 00010040
> 008: 2f1f4af2 00010040 // 2f1f4d22 00010040 // 2f1f55e2 00010040 //
> 2f1f5812 00010040
> 00c: 2f1f5a42 00010040 // 205ac232 00010040 // 205ac462 00010040 //
> 205ac692 00010040
> 010: 205acaf2 00010040 // 205acd22 00010040 // 205acf52 00010040 //
> 205ad5e2 00010040
> 014: 205ad812 00010040 // 205ada42 00010040 // 260d0232 00010040 //
> 260d0462 00010040
> 018: 260d0692 00010040 // 260d0d22 00010040 // 260d0f52 00010040 //
> 260d1182 00010040
> 01c: 260d1a42 00010040 // 260d1c72 00010040 // 33794002 00010040 //
> 33794692 00010040
> 020: 337948c2 00010040 // 33794af2 00010040 // 337953b2 00010040 //
> 337955e2 00010040
> 024: 33795812 00010040 // 26202462 00010040 // 262028c2 00010040 //
> 26202d22 00010040
> 028: 262035e2 00010040 // 26203812 00010040 // 26203a42 00010040 //
> 2f3d8232 00010040
> 02c: 2f3d8462 00010040 // 2f3d8692 00010040 // 2f3d8f52 00010040 //
> 2f3d9182 00010040
> 030: 2f3d93b2 00010040 // 2f3d9c72 00010040 // 2f9d8002 00010040 //
> 2f9d8232 00010040
> 034: 2f9d8af2 00010040 // 2f9d8d22 00010040 // 2f9d8f52 00010040 //
> 2f9d9812 00010040
> 038: 2f9d9a42 00010040 // 2f9d9c72 00010040 // 260f6692 00010040 //
> 260f68c2 00010040
> 03c: 260f6af2 00010040 // 260f73b2 00010040 // 260f75e2 00010040 //
> 260f7812 00010040
> 040: 35b3e002 00010040 // 35b3e232 00010040 // 35b3e462 00010040 //
> 35b3ed22 00010040
> 044: 35b3ef52 00010040 // 35b3f182 00010040 // 35b3f812 00010040 //
> 35b3fa42 00010040
> 048: 35b3fc72 00010040 // 3591a692 00010040 // 3591a8c2 00010040 //
> 3591aaf2 00010040
> 04c: 3591b3b2 00010040 // 3591b5e2 00010040 // 3591b812 00010040 //
> 33624232 00010040
> 050: 33624462 00010040 // 33624692 00010040 // 33624d22 00010040 //
> 33624f52 00010040
> 054: 33625182 00010040 // 33625812 00010040 // 33625a42 00010040 //
> 33625c72 00010040
> 058: 26194692 00010040 // 261948c2 00010040 // 26194af2 00010040 //
> 26194f52 00010040
> 05c: 26195182 00010040 // 261953b2 00010040 // 26195c72 00010040 //
> 2fa58002 00010040
> 060: 2fa58232 00010040 // 2fa58af2 00010040 // 2fa58d22 00010040 //
> 2fa58f52 00010040
> 064: 2fa595e2 00010040 // 2fa59812 00010040 // 2fa59a42 00010040 //
> 30a58462 00010040
> 068: 30a58692 00010040 // 30a588c2 00010040 // 30a59182 00010040 //
> 30a593b2 00010040
> 06c: 30a595e2 00010040 // 3583a002 00010040 // 3583a232 00010040 //
> 3583a462 00010040
> 070: 3583af52 00010040 // 3583b3b2 00010040 // 3583b812 00010040 //
> 33754232 00010040
> 074: 33754462 00010040 // 33754692 00010040 // 33754f52 00010040 //
> 33755182 00010040
> 078: 337553b2 00010040 // 33755c72 00010040 // 2075c002 00010040 //
> 2075c232 00010040
> 07c: 2075caf2 00010040 // 2075cd22 00010040 // 2075cf52 00010040 //
> 2075d812 00010040
> 080: 2075da42 00010040 // 2075dc72 00010040 // 30980692 00010040 //
> 309808c2 00010040
> 084: 30980af2 00010040 // 309813b2 00010040 // 309815e2 00010040 //
> 30981812 00010040
> 088: 2f93a002 00010040 // 2f93a232 00010040 // 2f93a462 00010040 //
> 2f93ad22 00010040
> 08c: 2f93af52 00010040 // 2f93b182 00010040 // 2f93ba42 00010040 //
> 2f93bc72 00010040
> 090: 205c0002 00010040 // 205c08c2 00010040 // 205c0af2 00010040 //
> 205c0d22 00010040
> 094: 205c15e2 00010040 // 205c1812 00010040 // 205c1c72 00010040 //
> 337bc692 00010040
> 098: 337bc8c2 00010040 // 337bcaf2 00010040 // 337bcd22 00010040 //
> 337bd5e2 00010040
> 09c: 337bd812 00010040 // 204a4462 00010040 // 204a4692 00010040 //
> 204a48c2 00010040
> 0a0: 204a5182 00010040 // 204a55e2 00010040 // 204a5812 00010040 //
> 26104002 00010040
> 0a4: 26104232 00010040 // 26104692 00010040 // 26104f52 00010040 //
> 26105182 00010040
> 0a8: 261053b2 00010040 // 261055e2 00010040 // 3598e002 00010040 //
> 3598e232 00010040
> 0ac: 3598ed22 00010040 // 3598ef52 00010040 // 3598f182 00010040 //
> 3598fa42 00010040
> 0b0: 30ae0002 00010040 // 2f2818ee 000105e6 // 2066d23e 000105e6 //
> 30b5c8b6 0001004e
> 0b4: 30b5d116 0001004e // 2f821116 0001004e // 2f865546 0001004e //
> 2f35629e 000105e6
> 0b8: 2f823296 0001004e // 2f820ce6 0001004e // 2f821da6 0001004e //
> 3672b6c6 0001004e
> 0bc: 2f8c5546 0001004e // 2f8c7af6 0001004e // 2f8c6606 0001004e //
> 2f92f296 0001004e
> 0c0: 2f94e1d6 0001004e // 2f94d546 0001004e // 0bdbaa36 0001004e //
> 0bdb9976 0001004e
> 0c4: 0bdb88b6 0001004e // 2f8fb6c6 0001004e // 2f8f9976 0001004e //
> 36b176c6 0001004e
> 0c8: 366ea94e 000105e6 // 0bdbbaf6 0001004e // 2f864ce6 0001004e //
> 2f8208b6 0001004e
> 0cc: 2f822a36 0001004e // 2f8221d6 0001004e // 36069116 0001004e //
> 366ed276 0001004a
> 0d0: 366eca46 0001004a // 366ea156 0001004a // 366e9926 0001004a //
> 366ec216 0001004a
> 0d4: 366e90f6 0001004a // 366eeb06 0001004a // 366f2156 0001004a //
> 30b5c486 0001004e
> 0d8: 366f005e 000105e6 // 30b5f6c6 0001004e // 30b5ee66 0001004e //
> 30b5d546 0001004e
> 0dc: 30b5e1d6 0001004e // 2f8f8ce6 0001004e // 2f8fbaf6 0001004e //
> 30b5e606 0001004e
> 0e0: 30b5ea36 0001004e // 2f8fd976 0001004e // 2f820056 0001004e //
> 2f92ea36 0001004e
> 0e4: 3606b6c6 0001004e // 2f8676c6 0001004e // 2f867296 0001004e //
> 2f867af6 0001004e
> 0e8: 2f8fee6e 00010046 // 2f864486 0001004e // 2f865116 0001004e //
> 366f39ae 000105e6
> 0ec: 2f8fd116 0001004e // 2f8661d6 0001004e // 0bdb8486 0001004e //
> 0bdb9546 0001004e
> 0f0: 205a8486 0001004e // 205a88b6 0001004e // 3025d27e 000105e6 //
> 205a8ce6 0001004e
> 0f4: 2f9cb182 00010040 // 17d0527e 000105e6 // 2f9caf52 00010040 //
> 2f9ca692 00010040
> 0f8: 2f9caaf2 00010040 // 2f9ca232 00010040 // 2f9ca002 00010040 //
> 2f9cba42 00010040
> 0fc: 34a4d5e2 00010040 // 34a4cf52 00010040 // 34a4c8c2 00010040 //
> 34a4caf2 00010040
>
The host has two interfaces
eth0 - e1000, eth1 - forcedeth.
Both are not working after that messages.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 2.6.30-rc1: e1000: list_add corruption
2009-04-09 4:16 ` Alexander Beregalov
@ 2009-04-09 4:20 ` David Miller
[not found] ` <20090408.212040.184687004.davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>
0 siblings, 1 reply; 9+ messages in thread
From: David Miller @ 2009-04-09 4:20 UTC (permalink / raw)
To: a.beregalov; +Cc: kernel-testers, netdev, e1000-devel
From: Alexander Beregalov <a.beregalov@gmail.com>
Date: Thu, 9 Apr 2009 08:16:00 +0400
> The host has two interfaces
> eth0 - e1000, eth1 - forcedeth.
> Both are not working after that messages.
It looks like the NAPI pending list is being modified in an illegal
state by forcedeth, and mid-stream to a list_del() an e1000 interrupt
comes in and this tries to schedule the e1000 NAPI context and we get
list corruption.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 2.6.30-rc1: e1000: list_add corruption
[not found] ` <20090408.212040.184687004.davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>
@ 2009-04-09 7:12 ` Jarek Poplawski
[not found] ` <20090409071242.GA5515-8HppEYmqbBCE+EvaaNYduQ@public.gmane.org>
0 siblings, 1 reply; 9+ messages in thread
From: Jarek Poplawski @ 2009-04-09 7:12 UTC (permalink / raw)
To: David Miller
Cc: a.beregalov-Re5JQEeQqe8AvxtiuMwx3w,
kernel-testers-u79uwXL29TY76Z2rM5mHXA,
netdev-u79uwXL29TY76Z2rM5mHXA,
e1000-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f
On 09-04-2009 06:20, David Miller wrote:
> From: Alexander Beregalov <a.beregalov-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
> Date: Thu, 9 Apr 2009 08:16:00 +0400
>
>> The host has two interfaces
>> eth0 - e1000, eth1 - forcedeth.
>> Both are not working after that messages.
>
> It looks like the NAPI pending list is being modified in an illegal
> state by forcedeth, and mid-stream to a list_del() an e1000 interrupt
> comes in and this tries to schedule the e1000 NAPI context and we get
> list corruption.
Good point! Feel free to send a patch. ;-)
Jarek P.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 2.6.30-rc1: e1000: list_add corruption
[not found] ` <20090409071242.GA5515-8HppEYmqbBCE+EvaaNYduQ@public.gmane.org>
@ 2009-04-09 7:51 ` David Miller
2009-04-09 8:01 ` Jarek Poplawski
0 siblings, 1 reply; 9+ messages in thread
From: David Miller @ 2009-04-09 7:51 UTC (permalink / raw)
To: jarkao2-Re5JQEeQqe8AvxtiuMwx3w
Cc: a.beregalov-Re5JQEeQqe8AvxtiuMwx3w,
kernel-testers-u79uwXL29TY76Z2rM5mHXA,
netdev-u79uwXL29TY76Z2rM5mHXA,
e1000-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f
From: Jarek Poplawski <jarkao2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Date: Thu, 9 Apr 2009 07:12:42 +0000
> On 09-04-2009 06:20, David Miller wrote:
>> From: Alexander Beregalov <a.beregalov-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
>> Date: Thu, 9 Apr 2009 08:16:00 +0400
>>
>>> The host has two interfaces
>>> eth0 - e1000, eth1 - forcedeth.
>>> Both are not working after that messages.
>>
>> It looks like the NAPI pending list is being modified in an illegal
>> state by forcedeth, and mid-stream to a list_del() an e1000 interrupt
>> comes in and this tries to schedule the e1000 NAPI context and we get
>> list corruption.
>
> Good point! Feel free to send a patch. ;-)
A real shame, since I sent that analysis in hoping someone else
would do the rest of the digging and write the fix. :-/
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 2.6.30-rc1: e1000: list_add corruption
2009-04-09 7:51 ` David Miller
@ 2009-04-09 8:01 ` Jarek Poplawski
2009-04-09 8:04 ` David Miller
0 siblings, 1 reply; 9+ messages in thread
From: Jarek Poplawski @ 2009-04-09 8:01 UTC (permalink / raw)
To: David Miller; +Cc: a.beregalov, kernel-testers, netdev, e1000-devel
On Thu, Apr 09, 2009 at 12:51:06AM -0700, David Miller wrote:
> From: Jarek Poplawski <jarkao2@gmail.com>
> Date: Thu, 9 Apr 2009 07:12:42 +0000
>
> > On 09-04-2009 06:20, David Miller wrote:
> >> From: Alexander Beregalov <a.beregalov@gmail.com>
> >> Date: Thu, 9 Apr 2009 08:16:00 +0400
> >>
> >>> The host has two interfaces
> >>> eth0 - e1000, eth1 - forcedeth.
> >>> Both are not working after that messages.
> >>
> >> It looks like the NAPI pending list is being modified in an illegal
> >> state by forcedeth, and mid-stream to a list_del() an e1000 interrupt
> >> comes in and this tries to schedule the e1000 NAPI context and we get
> >> list corruption.
> >
> > Good point! Feel free to send a patch. ;-)
>
> A real shame, since I sent that analysis in hoping someone else
> would do the rest of the digging and write the fix. :-/
Hmm... What rest? Alexander?
Jarek P.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 2.6.30-rc1: e1000: list_add corruption
2009-04-09 8:01 ` Jarek Poplawski
@ 2009-04-09 8:04 ` David Miller
0 siblings, 0 replies; 9+ messages in thread
From: David Miller @ 2009-04-09 8:04 UTC (permalink / raw)
To: jarkao2; +Cc: a.beregalov, kernel-testers, netdev, e1000-devel
From: Jarek Poplawski <jarkao2@gmail.com>
Date: Thu, 9 Apr 2009 08:01:59 +0000
> On Thu, Apr 09, 2009 at 12:51:06AM -0700, David Miller wrote:
>> From: Jarek Poplawski <jarkao2@gmail.com>
>> Date: Thu, 9 Apr 2009 07:12:42 +0000
>>
>> > On 09-04-2009 06:20, David Miller wrote:
>> >> From: Alexander Beregalov <a.beregalov@gmail.com>
>> >> Date: Thu, 9 Apr 2009 08:16:00 +0400
>> >>
>> >>> The host has two interfaces
>> >>> eth0 - e1000, eth1 - forcedeth.
>> >>> Both are not working after that messages.
>> >>
>> >> It looks like the NAPI pending list is being modified in an illegal
>> >> state by forcedeth, and mid-stream to a list_del() an e1000 interrupt
>> >> comes in and this tries to schedule the e1000 NAPI context and we get
>> >> list corruption.
>> >
>> > Good point! Feel free to send a patch. ;-)
>>
>> A real shame, since I sent that analysis in hoping someone else
>> would do the rest of the digging and write the fix. :-/
>
> Hmm... What rest? Alexander?
I mean the part where you actually look at the code and figure out
where interrupts are enabled during a list_del() where they shouldn't
be.
The initial trace is pretty detailed and has all the information you
need...
Nevermind if I have to sit here and explain all of this I might
as well fucking fix this myself :-/
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 2.6.30-rc1: e1000: list_add corruption
2009-04-09 3:45 2.6.30-rc1: e1000: list_add corruption Alexander Beregalov
[not found] ` <a4423d670904082045q41274747ha08d28a0645ae378-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2009-04-09 8:08 ` David Miller
1 sibling, 0 replies; 9+ messages in thread
From: David Miller @ 2009-04-09 8:08 UTC (permalink / raw)
To: a.beregalov; +Cc: e1000-devel, netdev, kernel-testers
From: Alexander Beregalov <a.beregalov@gmail.com>
Date: Thu, 9 Apr 2009 07:45:39 +0400
> WARNING: at lib/list_debug.c:30 __list_add+0x89/0x90()
> Hardware name:
> list_add corruption. prev->next should be next (c06ea834), but was
> f70244c8. (prev=c06ea834).
> Modules linked in: w83627hf hwmon_vid i2c_nforce2
> Pid: 1436, comm: portageq Not tainted 2.6.30-rc1 #1
> Call Trace:
> [<c0129d73>] warn_slowpath+0x73/0xd0
I'll push the following fix for this:
forcedeth: Use napi_complete() not __napi_complete().
It's not enough that forcedeth's interrupts are disabled,
local cpu interrupts have to unconditionally be off
when we remove the device from the poll list.
Based upon a crash report from
Alexander Beregalov <a.beregalov@gmail.com>:
WARNING: at lib/list_debug.c:30 __list_add+0x89/0x90()
Hardware name:
list_add corruption. prev->next should be next (c06ea834), but was
f70244c8. (prev=c06ea834).
Modules linked in: w83627hf hwmon_vid i2c_nforce2
Pid: 1436, comm: portageq Not tainted 2.6.30-rc1 #1
Call Trace:
[<c0129d73>] warn_slowpath+0x73/0xd0
[<c03c6008>] ? __kfree_skb+0x38/0x90
[<c03f9b06>] ? tcp_data_snd_check+0x26/0xe0
[<c03fd67f>] ? tcp_rcv_established+0x2bf/0x5e0
[<c040557a>] ? tcp_v4_rcv+0x47a/0x610
[<c014cebd>] ? print_lock_contention_bug+0x1d/0x110
[<c044a967>] ? _spin_unlock+0x27/0x50
[<c040564b>] ? tcp_v4_rcv+0x54b/0x610
[<c02d86f9>] __list_add+0x89/0x90
[<c03ccff9>] __napi_schedule+0x29/0x60
[<c036946d>] e1000_intr+0xbd/0x1a0
[<c015c5de>] handle_IRQ_event+0x3e/0x120
[<c015e190>] handle_fasteoi_irq+0x60/0xd0
[<c0104fd4>] handle_irq+0x34/0x60
[<c015f748>] ? rcu_irq_enter+0x8/0x40
[<c0104b29>] do_IRQ+0x39/0xa0
[<c03c592c>] ? skb_release_head_state+0x2c/0x60
[<c01034ee>] common_interrupt+0x2e/0x34
[<c02d8601>] ? list_del+0x21/0x90
[<c014e54b>] ? trace_hardirqs_on+0xb/0x10
[<c03cd4da>] __napi_complete+0x1a/0x30
[<c0381971>] nv_napi_poll+0xd1/0x5c0
[<c014e54b>] ? trace_hardirqs_on+0xb/0x10
[<c03cd5f6>] net_rx_action+0x106/0x1b0
[<c012e8df>] __do_softirq+0x6f/0x100
[<c044a967>] ? _spin_unlock+0x27/0x50
[<c015e1b8>] ? handle_fasteoi_irq+0x88/0xd0
[<c012e9cd>] do_softirq+0x5d/0x70
[<c012ebad>] irq_exit+0x7d/0xa0
[<c0104b32>] do_IRQ+0x42/0xa0
[<c012e9b7>] ? do_softirq+0x47/0x70
[<c01034ee>] common_interrupt+0x2e/0x34
Signed-off-by: David S. Miller <davem@davemloft.net>
diff --git a/drivers/net/forcedeth.c b/drivers/net/forcedeth.c
index d374650..11d5db1 100644
--- a/drivers/net/forcedeth.c
+++ b/drivers/net/forcedeth.c
@@ -3745,14 +3745,14 @@ static int nv_napi_poll(struct napi_struct *napi, int budget)
mod_timer(&np->nic_poll, jiffies + POLL_WAIT);
}
spin_unlock_irqrestore(&np->lock, flags);
- __napi_complete(napi);
+ napi_complete(napi);
return rx_work;
}
if (rx_work < budget) {
/* re-enable interrupts
(msix not enabled in napi) */
- __napi_complete(napi);
+ napi_complete(napi);
writel(np->irqmask, base + NvRegIrqMask);
}
------------------------------------------------------------------------------
This SF.net email is sponsored by:
High Quality Requirements in a Collaborative Environment.
Download a free trial of Rational Requirements Composer Now!
http://p.sf.net/sfu/www-ibm-com
^ permalink raw reply related [flat|nested] 9+ messages in thread
end of thread, other threads:[~2009-04-09 8:08 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2009-04-09 3:45 2.6.30-rc1: e1000: list_add corruption Alexander Beregalov
[not found] ` <a4423d670904082045q41274747ha08d28a0645ae378-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2009-04-09 3:48 ` Alexander Beregalov
[not found] ` <a4423d670904082048i26585b9boacf7ae449e004b8c-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2009-04-09 4:16 ` Alexander Beregalov
2009-04-09 4:20 ` David Miller
[not found] ` <20090408.212040.184687004.davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>
2009-04-09 7:12 ` Jarek Poplawski
[not found] ` <20090409071242.GA5515-8HppEYmqbBCE+EvaaNYduQ@public.gmane.org>
2009-04-09 7:51 ` David Miller
2009-04-09 8:01 ` Jarek Poplawski
2009-04-09 8:04 ` David Miller
2009-04-09 8:08 ` David Miller
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).