* 2.6.30-rc1: e1000: list_add corruption
@ 2009-04-09 3:45 Alexander Beregalov
[not found] ` <a4423d670904082045q41274747ha08d28a0645ae378-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2009-04-09 8:08 ` David Miller
0 siblings, 2 replies; 9+ messages in thread
From: Alexander Beregalov @ 2009-04-09 3:45 UTC (permalink / raw)
To: Kernel Testers List, Netdev,
e1000-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f
WARNING: at lib/list_debug.c:30 __list_add+0x89/0x90()
Hardware name:
list_add corruption. prev->next should be next (c06ea834), but was
f70244c8. (prev=c06ea834).
Modules linked in: w83627hf hwmon_vid i2c_nforce2
Pid: 1436, comm: portageq Not tainted 2.6.30-rc1 #1
Call Trace:
[<c0129d73>] warn_slowpath+0x73/0xd0
[<c03c6008>] ? __kfree_skb+0x38/0x90
[<c03f9b06>] ? tcp_data_snd_check+0x26/0xe0
[<c03fd67f>] ? tcp_rcv_established+0x2bf/0x5e0
[<c040557a>] ? tcp_v4_rcv+0x47a/0x610
[<c014cebd>] ? print_lock_contention_bug+0x1d/0x110
[<c044a967>] ? _spin_unlock+0x27/0x50
[<c040564b>] ? tcp_v4_rcv+0x54b/0x610
[<c02d86f9>] __list_add+0x89/0x90
[<c03ccff9>] __napi_schedule+0x29/0x60
[<c036946d>] e1000_intr+0xbd/0x1a0
[<c015c5de>] handle_IRQ_event+0x3e/0x120
[<c015e190>] handle_fasteoi_irq+0x60/0xd0
[<c0104fd4>] handle_irq+0x34/0x60
[<c015f748>] ? rcu_irq_enter+0x8/0x40
[<c0104b29>] do_IRQ+0x39/0xa0
[<c03c592c>] ? skb_release_head_state+0x2c/0x60
[<c01034ee>] common_interrupt+0x2e/0x34
[<c02d8601>] ? list_del+0x21/0x90
[<c014e54b>] ? trace_hardirqs_on+0xb/0x10
[<c03cd4da>] __napi_complete+0x1a/0x30
[<c0381971>] nv_napi_poll+0xd1/0x5c0
[<c014e54b>] ? trace_hardirqs_on+0xb/0x10
[<c03cd5f6>] net_rx_action+0x106/0x1b0
[<c012e8df>] __do_softirq+0x6f/0x100
[<c044a967>] ? _spin_unlock+0x27/0x50
[<c015e1b8>] ? handle_fasteoi_irq+0x88/0xd0
[<c012e9cd>] do_softirq+0x5d/0x70
[<c012ebad>] irq_exit+0x7d/0xa0
[<c0104b32>] do_IRQ+0x42/0xa0
[<c012e9b7>] ? do_softirq+0x47/0x70
[<c01034ee>] common_interrupt+0x2e/0x34
^ permalink raw reply [flat|nested] 9+ messages in thread[parent not found: <a4423d670904082045q41274747ha08d28a0645ae378-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: 2.6.30-rc1: e1000: list_add corruption [not found] ` <a4423d670904082045q41274747ha08d28a0645ae378-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2009-04-09 3:48 ` Alexander Beregalov [not found] ` <a4423d670904082048i26585b9boacf7ae449e004b8c-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 9+ messages in thread From: Alexander Beregalov @ 2009-04-09 3:48 UTC (permalink / raw) To: Kernel Testers List, Netdev, e1000-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f 2009/4/9 Alexander Beregalov <a.beregalov-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>: > WARNING: at lib/list_debug.c:30 __list_add+0x89/0x90() > Hardware name: > list_add corruption. prev->next should be next (c06ea834), but was > f70244c8. (prev=c06ea834). > Modules linked in: w83627hf hwmon_vid i2c_nforce2 > Pid: 1436, comm: portageq Not tainted 2.6.30-rc1 #1 > Call Trace: > [<c0129d73>] warn_slowpath+0x73/0xd0 > [<c03c6008>] ? __kfree_skb+0x38/0x90 > [<c03f9b06>] ? tcp_data_snd_check+0x26/0xe0 > [<c03fd67f>] ? tcp_rcv_established+0x2bf/0x5e0 > [<c040557a>] ? tcp_v4_rcv+0x47a/0x610 > [<c014cebd>] ? print_lock_contention_bug+0x1d/0x110 > [<c044a967>] ? _spin_unlock+0x27/0x50 > [<c040564b>] ? tcp_v4_rcv+0x54b/0x610 > [<c02d86f9>] __list_add+0x89/0x90 > [<c03ccff9>] __napi_schedule+0x29/0x60 > [<c036946d>] e1000_intr+0xbd/0x1a0 > [<c015c5de>] handle_IRQ_event+0x3e/0x120 > [<c015e190>] handle_fasteoi_irq+0x60/0xd0 > [<c0104fd4>] handle_irq+0x34/0x60 > [<c015f748>] ? rcu_irq_enter+0x8/0x40 > [<c0104b29>] do_IRQ+0x39/0xa0 > [<c03c592c>] ? skb_release_head_state+0x2c/0x60 > [<c01034ee>] common_interrupt+0x2e/0x34 > [<c02d8601>] ? list_del+0x21/0x90 > [<c014e54b>] ? trace_hardirqs_on+0xb/0x10 > [<c03cd4da>] __napi_complete+0x1a/0x30 > [<c0381971>] nv_napi_poll+0xd1/0x5c0 > [<c014e54b>] ? trace_hardirqs_on+0xb/0x10 > [<c03cd5f6>] net_rx_action+0x106/0x1b0 > [<c012e8df>] __do_softirq+0x6f/0x100 > [<c044a967>] ? _spin_unlock+0x27/0x50 > [<c015e1b8>] ? handle_fasteoi_irq+0x88/0xd0 > [<c012e9cd>] do_softirq+0x5d/0x70 > [<c012ebad>] irq_exit+0x7d/0xa0 > [<c0104b32>] do_IRQ+0x42/0xa0 > [<c012e9b7>] ? do_softirq+0x47/0x70 > [<c01034ee>] common_interrupt+0x2e/0x34 > Addition: WARNING: at net/sched/sch_generic.c:226 dev_watchdog+0x243/0x250() Hardware name: NETDEV WATCHDOG: eth1 (forcedeth): transmit timed out Modules linked in: w83627hf hwmon_vid i2c_nforce2 Pid: 0, comm: swapper Tainted: G W 2.6.30-rc1 #1 Call Trace: [<c0129d73>] warn_slowpath+0x73/0xd0 [<c014cebd>] ? print_lock_contention_bug+0x1d/0x110 [<c014cebd>] ? print_lock_contention_bug+0x1d/0x110 [<c014cebd>] ? print_lock_contention_bug+0x1d/0x110 [<c014cebd>] ? print_lock_contention_bug+0x1d/0x110 [<c01272f5>] ? enqueue_task_fair+0x1a5/0x330 [<c014cebd>] ? print_lock_contention_bug+0x1d/0x110 [<c014cebd>] ? print_lock_contention_bug+0x1d/0x110 [<c03df1ad>] ? dev_watchdog+0x5d/0x250 [<c02c699d>] ? strlcpy+0x1d/0x80 [<c03cd1c0>] ? netdev_drivername+0x40/0x50 [<c03df393>] dev_watchdog+0x243/0x250 [<c0132ee0>] ? run_timer_softirq+0xc0/0x230 [<c0132f86>] run_timer_softirq+0x166/0x230 [<c0132ee0>] ? run_timer_softirq+0xc0/0x230 [<c03df150>] ? dev_watchdog+0x0/0x250 [<c012e8df>] __do_softirq+0x6f/0x100 [<c0149537>] ? tick_program_event+0x17/0x20 [<c012e9cd>] do_softirq+0x5d/0x70 [<c012ebad>] irq_exit+0x7d/0xa0 [<c0114a0c>] smp_apic_timer_interrupt+0x4c/0x80 [<c02c97a4>] ? trace_hardirqs_off_thunk+0xc/0x18 [<c0103557>] apic_timer_interrupt+0x2f/0x34 [<c0101e73>] ? cpu_idle+0x43/0x80 [<c0109a6f>] ? default_idle+0x4f/0x60 [<c0101e79>] cpu_idle+0x49/0x80 [<c043dee6>] rest_init+0x66/0x70 [<c06fc9dd>] start_kernel+0x2a4/0x2e4 [<c06fc4fc>] ? unknown_bootoption+0x0/0x1ec [<c06fc2f1>] i386_start_kernel+0x41/0x49 ---[ end trace 6efabe98c03fc44a ]--- eth1: Got tx_timeout. irq: 00000032 eth1: Ring at 36b62000 eth1: Dumping tx registers 0: 00000032 00000000 00000003 c1f4ffff 00000032 00000000 00000003 c1f4ffff 20: 00000032 00000000 00000003 beb2ffff 00000032 00000000 00000003 beb2ffff 40: 00000032 00000000 00000003 bb72ffff 00000032 00000000 00000003 bb72ffff 60: 00000032 00000000 00000003 b830ffff 00000032 00000000 00000003 b831ffff 80: 003b0f3c 00000001 00000000 007f0020 0000061c 00000001 00000000 00007f8f a0: 0015050f 00000016 f58d5000 00001611 005e0001 00000100 ffffffff 0000ffff c0: 00000001 00000001 00000001 00000001 00000001 00000001 00000001 00000001 e0: 00000001 00000001 00000001 00000001 00000001 00000001 00000001 00000001 100: 36b63000 36b62000 01ff00ff 00008000 00010064 00000000 00000063 36b63588 120: 36b62340 2f2818ee 000105e6 366d98b0 8000061c 36b63590 36b62348 00200010 140: 00300010 00000008 00000000 00000000 00000000 00000000 00000000 00000000 160: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 180: 00000016 00000008 01947869 00008103 00000025 000041e1 0194786d 0000c1e3 1a0: 0000001e 00000008 0194786d 00008103 00000025 000041e1 0194786d 0000c1e3 1c0: 0000001e 00000008 0194786d 00008103 00000025 000041e1 0194786d 0000c1e3 1e0: 0000001e 00000008 0194786d 00008103 00000025 000041e1 0194786d 0000c1e3 200: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 220: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 240: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 260: 00000000 00000000 fe020001 00000100 00000000 00000000 fe020001 00000100 eth1: Dumping tx ring 000: 34a4c002 00010040 // 3609b3b2 00010040 // 3609ba42 00010040 // 3609bc72 00010040 004: 2f1f4002 00010040 // 2f1f4232 00010040 // 2f1f4462 00010040 // 2f1f48c2 00010040 008: 2f1f4af2 00010040 // 2f1f4d22 00010040 // 2f1f55e2 00010040 // 2f1f5812 00010040 00c: 2f1f5a42 00010040 // 205ac232 00010040 // 205ac462 00010040 // 205ac692 00010040 010: 205acaf2 00010040 // 205acd22 00010040 // 205acf52 00010040 // 205ad5e2 00010040 014: 205ad812 00010040 // 205ada42 00010040 // 260d0232 00010040 // 260d0462 00010040 018: 260d0692 00010040 // 260d0d22 00010040 // 260d0f52 00010040 // 260d1182 00010040 01c: 260d1a42 00010040 // 260d1c72 00010040 // 33794002 00010040 // 33794692 00010040 020: 337948c2 00010040 // 33794af2 00010040 // 337953b2 00010040 // 337955e2 00010040 024: 33795812 00010040 // 26202462 00010040 // 262028c2 00010040 // 26202d22 00010040 028: 262035e2 00010040 // 26203812 00010040 // 26203a42 00010040 // 2f3d8232 00010040 02c: 2f3d8462 00010040 // 2f3d8692 00010040 // 2f3d8f52 00010040 // 2f3d9182 00010040 030: 2f3d93b2 00010040 // 2f3d9c72 00010040 // 2f9d8002 00010040 // 2f9d8232 00010040 034: 2f9d8af2 00010040 // 2f9d8d22 00010040 // 2f9d8f52 00010040 // 2f9d9812 00010040 038: 2f9d9a42 00010040 // 2f9d9c72 00010040 // 260f6692 00010040 // 260f68c2 00010040 03c: 260f6af2 00010040 // 260f73b2 00010040 // 260f75e2 00010040 // 260f7812 00010040 040: 35b3e002 00010040 // 35b3e232 00010040 // 35b3e462 00010040 // 35b3ed22 00010040 044: 35b3ef52 00010040 // 35b3f182 00010040 // 35b3f812 00010040 // 35b3fa42 00010040 048: 35b3fc72 00010040 // 3591a692 00010040 // 3591a8c2 00010040 // 3591aaf2 00010040 04c: 3591b3b2 00010040 // 3591b5e2 00010040 // 3591b812 00010040 // 33624232 00010040 050: 33624462 00010040 // 33624692 00010040 // 33624d22 00010040 // 33624f52 00010040 054: 33625182 00010040 // 33625812 00010040 // 33625a42 00010040 // 33625c72 00010040 058: 26194692 00010040 // 261948c2 00010040 // 26194af2 00010040 // 26194f52 00010040 05c: 26195182 00010040 // 261953b2 00010040 // 26195c72 00010040 // 2fa58002 00010040 060: 2fa58232 00010040 // 2fa58af2 00010040 // 2fa58d22 00010040 // 2fa58f52 00010040 064: 2fa595e2 00010040 // 2fa59812 00010040 // 2fa59a42 00010040 // 30a58462 00010040 068: 30a58692 00010040 // 30a588c2 00010040 // 30a59182 00010040 // 30a593b2 00010040 06c: 30a595e2 00010040 // 3583a002 00010040 // 3583a232 00010040 // 3583a462 00010040 070: 3583af52 00010040 // 3583b3b2 00010040 // 3583b812 00010040 // 33754232 00010040 074: 33754462 00010040 // 33754692 00010040 // 33754f52 00010040 // 33755182 00010040 078: 337553b2 00010040 // 33755c72 00010040 // 2075c002 00010040 // 2075c232 00010040 07c: 2075caf2 00010040 // 2075cd22 00010040 // 2075cf52 00010040 // 2075d812 00010040 080: 2075da42 00010040 // 2075dc72 00010040 // 30980692 00010040 // 309808c2 00010040 084: 30980af2 00010040 // 309813b2 00010040 // 309815e2 00010040 // 30981812 00010040 088: 2f93a002 00010040 // 2f93a232 00010040 // 2f93a462 00010040 // 2f93ad22 00010040 08c: 2f93af52 00010040 // 2f93b182 00010040 // 2f93ba42 00010040 // 2f93bc72 00010040 090: 205c0002 00010040 // 205c08c2 00010040 // 205c0af2 00010040 // 205c0d22 00010040 094: 205c15e2 00010040 // 205c1812 00010040 // 205c1c72 00010040 // 337bc692 00010040 098: 337bc8c2 00010040 // 337bcaf2 00010040 // 337bcd22 00010040 // 337bd5e2 00010040 09c: 337bd812 00010040 // 204a4462 00010040 // 204a4692 00010040 // 204a48c2 00010040 0a0: 204a5182 00010040 // 204a55e2 00010040 // 204a5812 00010040 // 26104002 00010040 0a4: 26104232 00010040 // 26104692 00010040 // 26104f52 00010040 // 26105182 00010040 0a8: 261053b2 00010040 // 261055e2 00010040 // 3598e002 00010040 // 3598e232 00010040 0ac: 3598ed22 00010040 // 3598ef52 00010040 // 3598f182 00010040 // 3598fa42 00010040 0b0: 30ae0002 00010040 // 2f2818ee 000105e6 // 2066d23e 000105e6 // 30b5c8b6 0001004e 0b4: 30b5d116 0001004e // 2f821116 0001004e // 2f865546 0001004e // 2f35629e 000105e6 0b8: 2f823296 0001004e // 2f820ce6 0001004e // 2f821da6 0001004e // 3672b6c6 0001004e 0bc: 2f8c5546 0001004e // 2f8c7af6 0001004e // 2f8c6606 0001004e // 2f92f296 0001004e 0c0: 2f94e1d6 0001004e // 2f94d546 0001004e // 0bdbaa36 0001004e // 0bdb9976 0001004e 0c4: 0bdb88b6 0001004e // 2f8fb6c6 0001004e // 2f8f9976 0001004e // 36b176c6 0001004e 0c8: 366ea94e 000105e6 // 0bdbbaf6 0001004e // 2f864ce6 0001004e // 2f8208b6 0001004e 0cc: 2f822a36 0001004e // 2f8221d6 0001004e // 36069116 0001004e // 366ed276 0001004a 0d0: 366eca46 0001004a // 366ea156 0001004a // 366e9926 0001004a // 366ec216 0001004a 0d4: 366e90f6 0001004a // 366eeb06 0001004a // 366f2156 0001004a // 30b5c486 0001004e 0d8: 366f005e 000105e6 // 30b5f6c6 0001004e // 30b5ee66 0001004e // 30b5d546 0001004e 0dc: 30b5e1d6 0001004e // 2f8f8ce6 0001004e // 2f8fbaf6 0001004e // 30b5e606 0001004e 0e0: 30b5ea36 0001004e // 2f8fd976 0001004e // 2f820056 0001004e // 2f92ea36 0001004e 0e4: 3606b6c6 0001004e // 2f8676c6 0001004e // 2f867296 0001004e // 2f867af6 0001004e 0e8: 2f8fee6e 00010046 // 2f864486 0001004e // 2f865116 0001004e // 366f39ae 000105e6 0ec: 2f8fd116 0001004e // 2f8661d6 0001004e // 0bdb8486 0001004e // 0bdb9546 0001004e 0f0: 205a8486 0001004e // 205a88b6 0001004e // 3025d27e 000105e6 // 205a8ce6 0001004e 0f4: 2f9cb182 00010040 // 17d0527e 000105e6 // 2f9caf52 00010040 // 2f9ca692 00010040 0f8: 2f9caaf2 00010040 // 2f9ca232 00010040 // 2f9ca002 00010040 // 2f9cba42 00010040 0fc: 34a4d5e2 00010040 // 34a4cf52 00010040 // 34a4c8c2 00010040 // 34a4caf2 00010040 ^ permalink raw reply [flat|nested] 9+ messages in thread
[parent not found: <a4423d670904082048i26585b9boacf7ae449e004b8c-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: 2.6.30-rc1: e1000: list_add corruption [not found] ` <a4423d670904082048i26585b9boacf7ae449e004b8c-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2009-04-09 4:16 ` Alexander Beregalov 2009-04-09 4:20 ` David Miller 0 siblings, 1 reply; 9+ messages in thread From: Alexander Beregalov @ 2009-04-09 4:16 UTC (permalink / raw) To: Kernel Testers List, Netdev, e1000-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f 2009/4/9 Alexander Beregalov <a.beregalov-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>: > 2009/4/9 Alexander Beregalov <a.beregalov-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>: >> WARNING: at lib/list_debug.c:30 __list_add+0x89/0x90() >> Hardware name: >> list_add corruption. prev->next should be next (c06ea834), but was >> f70244c8. (prev=c06ea834). >> Modules linked in: w83627hf hwmon_vid i2c_nforce2 >> Pid: 1436, comm: portageq Not tainted 2.6.30-rc1 #1 >> Call Trace: >> [<c0129d73>] warn_slowpath+0x73/0xd0 >> [<c03c6008>] ? __kfree_skb+0x38/0x90 >> [<c03f9b06>] ? tcp_data_snd_check+0x26/0xe0 >> [<c03fd67f>] ? tcp_rcv_established+0x2bf/0x5e0 >> [<c040557a>] ? tcp_v4_rcv+0x47a/0x610 >> [<c014cebd>] ? print_lock_contention_bug+0x1d/0x110 >> [<c044a967>] ? _spin_unlock+0x27/0x50 >> [<c040564b>] ? tcp_v4_rcv+0x54b/0x610 >> [<c02d86f9>] __list_add+0x89/0x90 >> [<c03ccff9>] __napi_schedule+0x29/0x60 >> [<c036946d>] e1000_intr+0xbd/0x1a0 >> [<c015c5de>] handle_IRQ_event+0x3e/0x120 >> [<c015e190>] handle_fasteoi_irq+0x60/0xd0 >> [<c0104fd4>] handle_irq+0x34/0x60 >> [<c015f748>] ? rcu_irq_enter+0x8/0x40 >> [<c0104b29>] do_IRQ+0x39/0xa0 >> [<c03c592c>] ? skb_release_head_state+0x2c/0x60 >> [<c01034ee>] common_interrupt+0x2e/0x34 >> [<c02d8601>] ? list_del+0x21/0x90 >> [<c014e54b>] ? trace_hardirqs_on+0xb/0x10 >> [<c03cd4da>] __napi_complete+0x1a/0x30 >> [<c0381971>] nv_napi_poll+0xd1/0x5c0 >> [<c014e54b>] ? trace_hardirqs_on+0xb/0x10 >> [<c03cd5f6>] net_rx_action+0x106/0x1b0 >> [<c012e8df>] __do_softirq+0x6f/0x100 >> [<c044a967>] ? _spin_unlock+0x27/0x50 >> [<c015e1b8>] ? handle_fasteoi_irq+0x88/0xd0 >> [<c012e9cd>] do_softirq+0x5d/0x70 >> [<c012ebad>] irq_exit+0x7d/0xa0 >> [<c0104b32>] do_IRQ+0x42/0xa0 >> [<c012e9b7>] ? do_softirq+0x47/0x70 >> [<c01034ee>] common_interrupt+0x2e/0x34 >> > Addition: > > WARNING: at net/sched/sch_generic.c:226 dev_watchdog+0x243/0x250() > Hardware name: > NETDEV WATCHDOG: eth1 (forcedeth): transmit timed out > Modules linked in: w83627hf hwmon_vid i2c_nforce2 > Pid: 0, comm: swapper Tainted: G W 2.6.30-rc1 #1 > Call Trace: > [<c0129d73>] warn_slowpath+0x73/0xd0 > [<c014cebd>] ? print_lock_contention_bug+0x1d/0x110 > [<c014cebd>] ? print_lock_contention_bug+0x1d/0x110 > [<c014cebd>] ? print_lock_contention_bug+0x1d/0x110 > [<c014cebd>] ? print_lock_contention_bug+0x1d/0x110 > [<c01272f5>] ? enqueue_task_fair+0x1a5/0x330 > [<c014cebd>] ? print_lock_contention_bug+0x1d/0x110 > [<c014cebd>] ? print_lock_contention_bug+0x1d/0x110 > [<c03df1ad>] ? dev_watchdog+0x5d/0x250 > [<c02c699d>] ? strlcpy+0x1d/0x80 > [<c03cd1c0>] ? netdev_drivername+0x40/0x50 > [<c03df393>] dev_watchdog+0x243/0x250 > [<c0132ee0>] ? run_timer_softirq+0xc0/0x230 > [<c0132f86>] run_timer_softirq+0x166/0x230 > [<c0132ee0>] ? run_timer_softirq+0xc0/0x230 > [<c03df150>] ? dev_watchdog+0x0/0x250 > [<c012e8df>] __do_softirq+0x6f/0x100 > [<c0149537>] ? tick_program_event+0x17/0x20 > [<c012e9cd>] do_softirq+0x5d/0x70 > [<c012ebad>] irq_exit+0x7d/0xa0 > [<c0114a0c>] smp_apic_timer_interrupt+0x4c/0x80 > [<c02c97a4>] ? trace_hardirqs_off_thunk+0xc/0x18 > [<c0103557>] apic_timer_interrupt+0x2f/0x34 > [<c0101e73>] ? cpu_idle+0x43/0x80 > [<c0109a6f>] ? default_idle+0x4f/0x60 > [<c0101e79>] cpu_idle+0x49/0x80 > [<c043dee6>] rest_init+0x66/0x70 > [<c06fc9dd>] start_kernel+0x2a4/0x2e4 > [<c06fc4fc>] ? unknown_bootoption+0x0/0x1ec > [<c06fc2f1>] i386_start_kernel+0x41/0x49 > ---[ end trace 6efabe98c03fc44a ]--- > eth1: Got tx_timeout. irq: 00000032 > eth1: Ring at 36b62000 > eth1: Dumping tx registers > 0: 00000032 00000000 00000003 c1f4ffff 00000032 00000000 00000003 c1f4ffff > 20: 00000032 00000000 00000003 beb2ffff 00000032 00000000 00000003 beb2ffff > 40: 00000032 00000000 00000003 bb72ffff 00000032 00000000 00000003 bb72ffff > 60: 00000032 00000000 00000003 b830ffff 00000032 00000000 00000003 b831ffff > 80: 003b0f3c 00000001 00000000 007f0020 0000061c 00000001 00000000 00007f8f > a0: 0015050f 00000016 f58d5000 00001611 005e0001 00000100 ffffffff 0000ffff > c0: 00000001 00000001 00000001 00000001 00000001 00000001 00000001 00000001 > e0: 00000001 00000001 00000001 00000001 00000001 00000001 00000001 00000001 > 100: 36b63000 36b62000 01ff00ff 00008000 00010064 00000000 00000063 36b63588 > 120: 36b62340 2f2818ee 000105e6 366d98b0 8000061c 36b63590 36b62348 00200010 > 140: 00300010 00000008 00000000 00000000 00000000 00000000 00000000 00000000 > 160: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 > 180: 00000016 00000008 01947869 00008103 00000025 000041e1 0194786d 0000c1e3 > 1a0: 0000001e 00000008 0194786d 00008103 00000025 000041e1 0194786d 0000c1e3 > 1c0: 0000001e 00000008 0194786d 00008103 00000025 000041e1 0194786d 0000c1e3 > 1e0: 0000001e 00000008 0194786d 00008103 00000025 000041e1 0194786d 0000c1e3 > 200: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 > 220: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 > 240: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 > 260: 00000000 00000000 fe020001 00000100 00000000 00000000 fe020001 00000100 > eth1: Dumping tx ring > 000: 34a4c002 00010040 // 3609b3b2 00010040 // 3609ba42 00010040 // > 3609bc72 00010040 > 004: 2f1f4002 00010040 // 2f1f4232 00010040 // 2f1f4462 00010040 // > 2f1f48c2 00010040 > 008: 2f1f4af2 00010040 // 2f1f4d22 00010040 // 2f1f55e2 00010040 // > 2f1f5812 00010040 > 00c: 2f1f5a42 00010040 // 205ac232 00010040 // 205ac462 00010040 // > 205ac692 00010040 > 010: 205acaf2 00010040 // 205acd22 00010040 // 205acf52 00010040 // > 205ad5e2 00010040 > 014: 205ad812 00010040 // 205ada42 00010040 // 260d0232 00010040 // > 260d0462 00010040 > 018: 260d0692 00010040 // 260d0d22 00010040 // 260d0f52 00010040 // > 260d1182 00010040 > 01c: 260d1a42 00010040 // 260d1c72 00010040 // 33794002 00010040 // > 33794692 00010040 > 020: 337948c2 00010040 // 33794af2 00010040 // 337953b2 00010040 // > 337955e2 00010040 > 024: 33795812 00010040 // 26202462 00010040 // 262028c2 00010040 // > 26202d22 00010040 > 028: 262035e2 00010040 // 26203812 00010040 // 26203a42 00010040 // > 2f3d8232 00010040 > 02c: 2f3d8462 00010040 // 2f3d8692 00010040 // 2f3d8f52 00010040 // > 2f3d9182 00010040 > 030: 2f3d93b2 00010040 // 2f3d9c72 00010040 // 2f9d8002 00010040 // > 2f9d8232 00010040 > 034: 2f9d8af2 00010040 // 2f9d8d22 00010040 // 2f9d8f52 00010040 // > 2f9d9812 00010040 > 038: 2f9d9a42 00010040 // 2f9d9c72 00010040 // 260f6692 00010040 // > 260f68c2 00010040 > 03c: 260f6af2 00010040 // 260f73b2 00010040 // 260f75e2 00010040 // > 260f7812 00010040 > 040: 35b3e002 00010040 // 35b3e232 00010040 // 35b3e462 00010040 // > 35b3ed22 00010040 > 044: 35b3ef52 00010040 // 35b3f182 00010040 // 35b3f812 00010040 // > 35b3fa42 00010040 > 048: 35b3fc72 00010040 // 3591a692 00010040 // 3591a8c2 00010040 // > 3591aaf2 00010040 > 04c: 3591b3b2 00010040 // 3591b5e2 00010040 // 3591b812 00010040 // > 33624232 00010040 > 050: 33624462 00010040 // 33624692 00010040 // 33624d22 00010040 // > 33624f52 00010040 > 054: 33625182 00010040 // 33625812 00010040 // 33625a42 00010040 // > 33625c72 00010040 > 058: 26194692 00010040 // 261948c2 00010040 // 26194af2 00010040 // > 26194f52 00010040 > 05c: 26195182 00010040 // 261953b2 00010040 // 26195c72 00010040 // > 2fa58002 00010040 > 060: 2fa58232 00010040 // 2fa58af2 00010040 // 2fa58d22 00010040 // > 2fa58f52 00010040 > 064: 2fa595e2 00010040 // 2fa59812 00010040 // 2fa59a42 00010040 // > 30a58462 00010040 > 068: 30a58692 00010040 // 30a588c2 00010040 // 30a59182 00010040 // > 30a593b2 00010040 > 06c: 30a595e2 00010040 // 3583a002 00010040 // 3583a232 00010040 // > 3583a462 00010040 > 070: 3583af52 00010040 // 3583b3b2 00010040 // 3583b812 00010040 // > 33754232 00010040 > 074: 33754462 00010040 // 33754692 00010040 // 33754f52 00010040 // > 33755182 00010040 > 078: 337553b2 00010040 // 33755c72 00010040 // 2075c002 00010040 // > 2075c232 00010040 > 07c: 2075caf2 00010040 // 2075cd22 00010040 // 2075cf52 00010040 // > 2075d812 00010040 > 080: 2075da42 00010040 // 2075dc72 00010040 // 30980692 00010040 // > 309808c2 00010040 > 084: 30980af2 00010040 // 309813b2 00010040 // 309815e2 00010040 // > 30981812 00010040 > 088: 2f93a002 00010040 // 2f93a232 00010040 // 2f93a462 00010040 // > 2f93ad22 00010040 > 08c: 2f93af52 00010040 // 2f93b182 00010040 // 2f93ba42 00010040 // > 2f93bc72 00010040 > 090: 205c0002 00010040 // 205c08c2 00010040 // 205c0af2 00010040 // > 205c0d22 00010040 > 094: 205c15e2 00010040 // 205c1812 00010040 // 205c1c72 00010040 // > 337bc692 00010040 > 098: 337bc8c2 00010040 // 337bcaf2 00010040 // 337bcd22 00010040 // > 337bd5e2 00010040 > 09c: 337bd812 00010040 // 204a4462 00010040 // 204a4692 00010040 // > 204a48c2 00010040 > 0a0: 204a5182 00010040 // 204a55e2 00010040 // 204a5812 00010040 // > 26104002 00010040 > 0a4: 26104232 00010040 // 26104692 00010040 // 26104f52 00010040 // > 26105182 00010040 > 0a8: 261053b2 00010040 // 261055e2 00010040 // 3598e002 00010040 // > 3598e232 00010040 > 0ac: 3598ed22 00010040 // 3598ef52 00010040 // 3598f182 00010040 // > 3598fa42 00010040 > 0b0: 30ae0002 00010040 // 2f2818ee 000105e6 // 2066d23e 000105e6 // > 30b5c8b6 0001004e > 0b4: 30b5d116 0001004e // 2f821116 0001004e // 2f865546 0001004e // > 2f35629e 000105e6 > 0b8: 2f823296 0001004e // 2f820ce6 0001004e // 2f821da6 0001004e // > 3672b6c6 0001004e > 0bc: 2f8c5546 0001004e // 2f8c7af6 0001004e // 2f8c6606 0001004e // > 2f92f296 0001004e > 0c0: 2f94e1d6 0001004e // 2f94d546 0001004e // 0bdbaa36 0001004e // > 0bdb9976 0001004e > 0c4: 0bdb88b6 0001004e // 2f8fb6c6 0001004e // 2f8f9976 0001004e // > 36b176c6 0001004e > 0c8: 366ea94e 000105e6 // 0bdbbaf6 0001004e // 2f864ce6 0001004e // > 2f8208b6 0001004e > 0cc: 2f822a36 0001004e // 2f8221d6 0001004e // 36069116 0001004e // > 366ed276 0001004a > 0d0: 366eca46 0001004a // 366ea156 0001004a // 366e9926 0001004a // > 366ec216 0001004a > 0d4: 366e90f6 0001004a // 366eeb06 0001004a // 366f2156 0001004a // > 30b5c486 0001004e > 0d8: 366f005e 000105e6 // 30b5f6c6 0001004e // 30b5ee66 0001004e // > 30b5d546 0001004e > 0dc: 30b5e1d6 0001004e // 2f8f8ce6 0001004e // 2f8fbaf6 0001004e // > 30b5e606 0001004e > 0e0: 30b5ea36 0001004e // 2f8fd976 0001004e // 2f820056 0001004e // > 2f92ea36 0001004e > 0e4: 3606b6c6 0001004e // 2f8676c6 0001004e // 2f867296 0001004e // > 2f867af6 0001004e > 0e8: 2f8fee6e 00010046 // 2f864486 0001004e // 2f865116 0001004e // > 366f39ae 000105e6 > 0ec: 2f8fd116 0001004e // 2f8661d6 0001004e // 0bdb8486 0001004e // > 0bdb9546 0001004e > 0f0: 205a8486 0001004e // 205a88b6 0001004e // 3025d27e 000105e6 // > 205a8ce6 0001004e > 0f4: 2f9cb182 00010040 // 17d0527e 000105e6 // 2f9caf52 00010040 // > 2f9ca692 00010040 > 0f8: 2f9caaf2 00010040 // 2f9ca232 00010040 // 2f9ca002 00010040 // > 2f9cba42 00010040 > 0fc: 34a4d5e2 00010040 // 34a4cf52 00010040 // 34a4c8c2 00010040 // > 34a4caf2 00010040 > The host has two interfaces eth0 - e1000, eth1 - forcedeth. Both are not working after that messages. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 2.6.30-rc1: e1000: list_add corruption 2009-04-09 4:16 ` Alexander Beregalov @ 2009-04-09 4:20 ` David Miller [not found] ` <20090408.212040.184687004.davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org> 0 siblings, 1 reply; 9+ messages in thread From: David Miller @ 2009-04-09 4:20 UTC (permalink / raw) To: a.beregalov; +Cc: kernel-testers, netdev, e1000-devel From: Alexander Beregalov <a.beregalov@gmail.com> Date: Thu, 9 Apr 2009 08:16:00 +0400 > The host has two interfaces > eth0 - e1000, eth1 - forcedeth. > Both are not working after that messages. It looks like the NAPI pending list is being modified in an illegal state by forcedeth, and mid-stream to a list_del() an e1000 interrupt comes in and this tries to schedule the e1000 NAPI context and we get list corruption. ^ permalink raw reply [flat|nested] 9+ messages in thread
[parent not found: <20090408.212040.184687004.davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>]
* Re: 2.6.30-rc1: e1000: list_add corruption [not found] ` <20090408.212040.184687004.davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org> @ 2009-04-09 7:12 ` Jarek Poplawski [not found] ` <20090409071242.GA5515-8HppEYmqbBCE+EvaaNYduQ@public.gmane.org> 0 siblings, 1 reply; 9+ messages in thread From: Jarek Poplawski @ 2009-04-09 7:12 UTC (permalink / raw) To: David Miller Cc: a.beregalov-Re5JQEeQqe8AvxtiuMwx3w, kernel-testers-u79uwXL29TY76Z2rM5mHXA, netdev-u79uwXL29TY76Z2rM5mHXA, e1000-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f On 09-04-2009 06:20, David Miller wrote: > From: Alexander Beregalov <a.beregalov-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> > Date: Thu, 9 Apr 2009 08:16:00 +0400 > >> The host has two interfaces >> eth0 - e1000, eth1 - forcedeth. >> Both are not working after that messages. > > It looks like the NAPI pending list is being modified in an illegal > state by forcedeth, and mid-stream to a list_del() an e1000 interrupt > comes in and this tries to schedule the e1000 NAPI context and we get > list corruption. Good point! Feel free to send a patch. ;-) Jarek P. ^ permalink raw reply [flat|nested] 9+ messages in thread
[parent not found: <20090409071242.GA5515-8HppEYmqbBCE+EvaaNYduQ@public.gmane.org>]
* Re: 2.6.30-rc1: e1000: list_add corruption [not found] ` <20090409071242.GA5515-8HppEYmqbBCE+EvaaNYduQ@public.gmane.org> @ 2009-04-09 7:51 ` David Miller 2009-04-09 8:01 ` Jarek Poplawski 0 siblings, 1 reply; 9+ messages in thread From: David Miller @ 2009-04-09 7:51 UTC (permalink / raw) To: jarkao2-Re5JQEeQqe8AvxtiuMwx3w Cc: a.beregalov-Re5JQEeQqe8AvxtiuMwx3w, kernel-testers-u79uwXL29TY76Z2rM5mHXA, netdev-u79uwXL29TY76Z2rM5mHXA, e1000-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f From: Jarek Poplawski <jarkao2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> Date: Thu, 9 Apr 2009 07:12:42 +0000 > On 09-04-2009 06:20, David Miller wrote: >> From: Alexander Beregalov <a.beregalov-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> >> Date: Thu, 9 Apr 2009 08:16:00 +0400 >> >>> The host has two interfaces >>> eth0 - e1000, eth1 - forcedeth. >>> Both are not working after that messages. >> >> It looks like the NAPI pending list is being modified in an illegal >> state by forcedeth, and mid-stream to a list_del() an e1000 interrupt >> comes in and this tries to schedule the e1000 NAPI context and we get >> list corruption. > > Good point! Feel free to send a patch. ;-) A real shame, since I sent that analysis in hoping someone else would do the rest of the digging and write the fix. :-/ ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 2.6.30-rc1: e1000: list_add corruption 2009-04-09 7:51 ` David Miller @ 2009-04-09 8:01 ` Jarek Poplawski 2009-04-09 8:04 ` David Miller 0 siblings, 1 reply; 9+ messages in thread From: Jarek Poplawski @ 2009-04-09 8:01 UTC (permalink / raw) To: David Miller; +Cc: a.beregalov, kernel-testers, netdev, e1000-devel On Thu, Apr 09, 2009 at 12:51:06AM -0700, David Miller wrote: > From: Jarek Poplawski <jarkao2@gmail.com> > Date: Thu, 9 Apr 2009 07:12:42 +0000 > > > On 09-04-2009 06:20, David Miller wrote: > >> From: Alexander Beregalov <a.beregalov@gmail.com> > >> Date: Thu, 9 Apr 2009 08:16:00 +0400 > >> > >>> The host has two interfaces > >>> eth0 - e1000, eth1 - forcedeth. > >>> Both are not working after that messages. > >> > >> It looks like the NAPI pending list is being modified in an illegal > >> state by forcedeth, and mid-stream to a list_del() an e1000 interrupt > >> comes in and this tries to schedule the e1000 NAPI context and we get > >> list corruption. > > > > Good point! Feel free to send a patch. ;-) > > A real shame, since I sent that analysis in hoping someone else > would do the rest of the digging and write the fix. :-/ Hmm... What rest? Alexander? Jarek P. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 2.6.30-rc1: e1000: list_add corruption 2009-04-09 8:01 ` Jarek Poplawski @ 2009-04-09 8:04 ` David Miller 0 siblings, 0 replies; 9+ messages in thread From: David Miller @ 2009-04-09 8:04 UTC (permalink / raw) To: jarkao2; +Cc: a.beregalov, kernel-testers, netdev, e1000-devel From: Jarek Poplawski <jarkao2@gmail.com> Date: Thu, 9 Apr 2009 08:01:59 +0000 > On Thu, Apr 09, 2009 at 12:51:06AM -0700, David Miller wrote: >> From: Jarek Poplawski <jarkao2@gmail.com> >> Date: Thu, 9 Apr 2009 07:12:42 +0000 >> >> > On 09-04-2009 06:20, David Miller wrote: >> >> From: Alexander Beregalov <a.beregalov@gmail.com> >> >> Date: Thu, 9 Apr 2009 08:16:00 +0400 >> >> >> >>> The host has two interfaces >> >>> eth0 - e1000, eth1 - forcedeth. >> >>> Both are not working after that messages. >> >> >> >> It looks like the NAPI pending list is being modified in an illegal >> >> state by forcedeth, and mid-stream to a list_del() an e1000 interrupt >> >> comes in and this tries to schedule the e1000 NAPI context and we get >> >> list corruption. >> > >> > Good point! Feel free to send a patch. ;-) >> >> A real shame, since I sent that analysis in hoping someone else >> would do the rest of the digging and write the fix. :-/ > > Hmm... What rest? Alexander? I mean the part where you actually look at the code and figure out where interrupts are enabled during a list_del() where they shouldn't be. The initial trace is pretty detailed and has all the information you need... Nevermind if I have to sit here and explain all of this I might as well fucking fix this myself :-/ ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 2.6.30-rc1: e1000: list_add corruption 2009-04-09 3:45 2.6.30-rc1: e1000: list_add corruption Alexander Beregalov [not found] ` <a4423d670904082045q41274747ha08d28a0645ae378-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2009-04-09 8:08 ` David Miller 1 sibling, 0 replies; 9+ messages in thread From: David Miller @ 2009-04-09 8:08 UTC (permalink / raw) To: a.beregalov; +Cc: e1000-devel, netdev, kernel-testers From: Alexander Beregalov <a.beregalov@gmail.com> Date: Thu, 9 Apr 2009 07:45:39 +0400 > WARNING: at lib/list_debug.c:30 __list_add+0x89/0x90() > Hardware name: > list_add corruption. prev->next should be next (c06ea834), but was > f70244c8. (prev=c06ea834). > Modules linked in: w83627hf hwmon_vid i2c_nforce2 > Pid: 1436, comm: portageq Not tainted 2.6.30-rc1 #1 > Call Trace: > [<c0129d73>] warn_slowpath+0x73/0xd0 I'll push the following fix for this: forcedeth: Use napi_complete() not __napi_complete(). It's not enough that forcedeth's interrupts are disabled, local cpu interrupts have to unconditionally be off when we remove the device from the poll list. Based upon a crash report from Alexander Beregalov <a.beregalov@gmail.com>: WARNING: at lib/list_debug.c:30 __list_add+0x89/0x90() Hardware name: list_add corruption. prev->next should be next (c06ea834), but was f70244c8. (prev=c06ea834). Modules linked in: w83627hf hwmon_vid i2c_nforce2 Pid: 1436, comm: portageq Not tainted 2.6.30-rc1 #1 Call Trace: [<c0129d73>] warn_slowpath+0x73/0xd0 [<c03c6008>] ? __kfree_skb+0x38/0x90 [<c03f9b06>] ? tcp_data_snd_check+0x26/0xe0 [<c03fd67f>] ? tcp_rcv_established+0x2bf/0x5e0 [<c040557a>] ? tcp_v4_rcv+0x47a/0x610 [<c014cebd>] ? print_lock_contention_bug+0x1d/0x110 [<c044a967>] ? _spin_unlock+0x27/0x50 [<c040564b>] ? tcp_v4_rcv+0x54b/0x610 [<c02d86f9>] __list_add+0x89/0x90 [<c03ccff9>] __napi_schedule+0x29/0x60 [<c036946d>] e1000_intr+0xbd/0x1a0 [<c015c5de>] handle_IRQ_event+0x3e/0x120 [<c015e190>] handle_fasteoi_irq+0x60/0xd0 [<c0104fd4>] handle_irq+0x34/0x60 [<c015f748>] ? rcu_irq_enter+0x8/0x40 [<c0104b29>] do_IRQ+0x39/0xa0 [<c03c592c>] ? skb_release_head_state+0x2c/0x60 [<c01034ee>] common_interrupt+0x2e/0x34 [<c02d8601>] ? list_del+0x21/0x90 [<c014e54b>] ? trace_hardirqs_on+0xb/0x10 [<c03cd4da>] __napi_complete+0x1a/0x30 [<c0381971>] nv_napi_poll+0xd1/0x5c0 [<c014e54b>] ? trace_hardirqs_on+0xb/0x10 [<c03cd5f6>] net_rx_action+0x106/0x1b0 [<c012e8df>] __do_softirq+0x6f/0x100 [<c044a967>] ? _spin_unlock+0x27/0x50 [<c015e1b8>] ? handle_fasteoi_irq+0x88/0xd0 [<c012e9cd>] do_softirq+0x5d/0x70 [<c012ebad>] irq_exit+0x7d/0xa0 [<c0104b32>] do_IRQ+0x42/0xa0 [<c012e9b7>] ? do_softirq+0x47/0x70 [<c01034ee>] common_interrupt+0x2e/0x34 Signed-off-by: David S. Miller <davem@davemloft.net> diff --git a/drivers/net/forcedeth.c b/drivers/net/forcedeth.c index d374650..11d5db1 100644 --- a/drivers/net/forcedeth.c +++ b/drivers/net/forcedeth.c @@ -3745,14 +3745,14 @@ static int nv_napi_poll(struct napi_struct *napi, int budget) mod_timer(&np->nic_poll, jiffies + POLL_WAIT); } spin_unlock_irqrestore(&np->lock, flags); - __napi_complete(napi); + napi_complete(napi); return rx_work; } if (rx_work < budget) { /* re-enable interrupts (msix not enabled in napi) */ - __napi_complete(napi); + napi_complete(napi); writel(np->irqmask, base + NvRegIrqMask); } ------------------------------------------------------------------------------ This SF.net email is sponsored by: High Quality Requirements in a Collaborative Environment. Download a free trial of Rational Requirements Composer Now! http://p.sf.net/sfu/www-ibm-com ^ permalink raw reply related [flat|nested] 9+ messages in thread
end of thread, other threads:[~2009-04-09 8:08 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2009-04-09 3:45 2.6.30-rc1: e1000: list_add corruption Alexander Beregalov
[not found] ` <a4423d670904082045q41274747ha08d28a0645ae378-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2009-04-09 3:48 ` Alexander Beregalov
[not found] ` <a4423d670904082048i26585b9boacf7ae449e004b8c-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2009-04-09 4:16 ` Alexander Beregalov
2009-04-09 4:20 ` David Miller
[not found] ` <20090408.212040.184687004.davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>
2009-04-09 7:12 ` Jarek Poplawski
[not found] ` <20090409071242.GA5515-8HppEYmqbBCE+EvaaNYduQ@public.gmane.org>
2009-04-09 7:51 ` David Miller
2009-04-09 8:01 ` Jarek Poplawski
2009-04-09 8:04 ` David Miller
2009-04-09 8:08 ` David Miller
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).