From: Simon Kirby <sim@hostway.ca>
To: Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
netdev@vger.kernel.org
Subject: Re: Linux 3.1-rc9
Date: Mon, 17 Oct 2011 22:40:51 -0700 [thread overview]
Message-ID: <20111018054051.GA19085@hostway.ca> (raw)
In-Reply-To: <20111012213555.GC24461@hostway.ca>
On Wed, Oct 12, 2011 at 02:35:55PM -0700, Simon Kirby wrote:
> > > patching file kernel/posix-cpu-timers.c
> > > patching file kernel/sched_stats.h
> >
> > yes that would be fine.
>
> This patch (s/raw_//) has been stable on 5 boxes for a day. I'll push to
> another 15 shortly and confirm tomorrow. Meanwhile, we had another ~4
> boxes lock up on 3.1-rc9 _with_ d670ec13 reverted (all CPUs spinning),
> but there weren't enough serial cables to log all of them and we haven't
> been lucky enough to capture anything other than what fits on 80x25.
> I'm hoping it's just the same bug you've already fixed.
Looks to be a different bug. It just happened on a box with serial
console logging, on the same build I was testing the above patch on --
Linus master circa Oct 7th. This seems to be specific to TCP. I'm not
sure what is with all of the doubled backtraces. I've only seen this on
a couple of different boxes so far.
Full log at http://0x.ca/sim/ref/3.1-rc9/3.1-rc9-tcp-lockup.log
First 100 lines:
[516112.140013] BUG: soft lockup - CPU#0 stuck for 22s! [swapper:0]
[516112.144001] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2
[516112.144001] CPU 0
[516112.144001] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2
[516112.144001]
[516112.144001] Pid: 0, comm: swapper Not tainted 3.1.0-rc9-hw+ #48 Dell Inc. PowerEdge 1950/0UR033
[516112.144001] RIP: 0010:[<ffffffff816b6694>] [<ffffffff816b6694>] _raw_spin_lock+0x14/0x20
[516112.144001] RSP: 0018:ffff88022fc03e10 EFLAGS: 00000297
[516112.144001] RAX: 0000000000000100 RBX: ffffffff81022674 RCX: ffffffff81b4df20
[516112.144001] RDX: ffff8801002aebe0 RSI: dead000000200200 RDI: ffff8801002ad188
[516112.144001] RBP: ffff88022fc03e10 R08: 00000000000000f7 R09: 0000000000000000
[516112.144001] R10: 0000000000000000 R11: 0000000000000010 R12: ffff88022fc03d88
[516112.144001] R13: ffffffff816bed1e R14: ffff88022fc03e10 R15: ffffffff81b4df00
[516112.144001] FS: 0000000000000000(0000) GS:ffff88022fc00000(0000) knlGS:0000000000000000
[516112.244020] BUG: soft lockup - CPU#1 stuck for 22s! [kworker/0:0:0]
[516112.244024] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2
[516112.244033] CPU 1
[516112.244035] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2
[516112.244041]
[516112.244044] Pid: 0, comm: kworker/0:0 Not tainted 3.1.0-rc9-hw+ #48 Dell Inc. PowerEdge 1950/0UR033
[516112.244048] RIP: 0010:[<ffffffff816b6694>] [<ffffffff816b6694>] _raw_spin_lock+0x14/0x20
[516112.244057] RSP: 0018:ffff88022fc43e10 EFLAGS: 00000297
[516112.244059] RAX: 0000000000000100 RBX: ffffffff81022674 RCX: ffff880226888020
[516112.244062] RDX: ffff88001ece1aa0 RSI: dead000000200200 RDI: ffff88001ece1f88
[516112.244064] RBP: ffff88022fc43e10 R08: 00000000000000df R09: 0000000000000000
[516112.244066] R10: 0000000000000000 R11: 0000000000000010 R12: ffff88022fc43d88
[516112.244068] R13: ffffffff816bed1e R14: ffff88022fc43e10 R15: ffff880226888000
[516112.244071] FS: 0000000000000000(0000) GS:ffff88022fc40000(0000) knlGS:0000000000000000
[516112.244074] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[516112.244076] CR2: ffffffffff600400 CR3: 0000000126d93000 CR4: 00000000000006e0
[516112.244078] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[516112.244081] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[516112.244083] Process kworker/0:0 (pid: 0, threadinfo ffff880226918000, task ffff880226911640)
[516112.244085] Stack:
[516112.244086] ffff88022fc43e40 ffffffff8162a613 0000000000000000 0000000000000000
[516112.244090] ffff880226888000 ffff88001ece20e0 ffff88022fc43ee0 ffffffff810692dc
[516112.244094] 0000000000000000 ffff880226919fd8 ffff880226919fd8 ffff880226919fd8
[516112.244098] Call Trace:
[516112.244099] <IRQ>
[516112.244105] [<ffffffff8162a613>] tcp_keepalive_timer+0x23/0x260
[516112.244110] [<ffffffff810692dc>] run_timer_softirq+0x1ac/0x310
[516112.244113] [<ffffffff8162a5f0>] ? tcp_init_xmit_timers+0x20/0x20
[516112.244118] [<ffffffff8102e838>] ? lapic_next_event+0x18/0x20
[516112.244121] [<ffffffff81060bf0>] __do_softirq+0xe0/0x1d0
[516112.244125] [<ffffffff816c04ac>] call_softirq+0x1c/0x30
[516112.244129] [<ffffffff81014255>] do_softirq+0x65/0xa0
[516112.244132] [<ffffffff810608fd>] irq_exit+0xad/0xe0
[516112.244135] [<ffffffff8102f569>] smp_apic_timer_interrupt+0x69/0xa0
[516112.244139] [<ffffffff816bed1e>] apic_timer_interrupt+0x6e/0x80
[516112.244140] <EOI>
[516112.244144] [<ffffffff8101a337>] ? mwait_idle+0x117/0x120
[516112.244147] [<ffffffff810120c6>] cpu_idle+0x86/0xe0
[516112.244151] [<ffffffff816ae77c>] start_secondary+0x1a3/0x1e7
[516112.244153] Code: 0f b6 c2 85 c0 c9 0f 95 c0 0f b6 c0 c3 66 2e 0f 1f 84 00 00 00 00 00 55 b8 00 01 00 00 48 89 e5 f0 66 0f c1 07 38 e0 74 06 f3 90 <8a> 07 eb f6 c9 c3 66 0f 1f 44 00 00 55 48 89 e5 9c 58 66 66 90
[516112.244173] Call Trace:
[516112.244174] <IRQ> [<ffffffff8162a613>] tcp_keepalive_timer+0x23/0x260
[516112.244179] [<ffffffff810692dc>] run_timer_softirq+0x1ac/0x310
[516112.244182] [<ffffffff8162a5f0>] ? tcp_init_xmit_timers+0x20/0x20
[516112.244185] [<ffffffff8102e838>] ? lapic_next_event+0x18/0x20
[516112.244188] [<ffffffff81060bf0>] __do_softirq+0xe0/0x1d0
[516112.244191] [<ffffffff816c04ac>] call_softirq+0x1c/0x30
[516112.244194] [<ffffffff81014255>] do_softirq+0x65/0xa0
[516112.244197] [<ffffffff810608fd>] irq_exit+0xad/0xe0
[516112.244199] [<ffffffff8102f569>] smp_apic_timer_interrupt+0x69/0xa0
[516112.244202] [<ffffffff816bed1e>] apic_timer_interrupt+0x6e/0x80
[516112.244204] <EOI> [<ffffffff8101a337>] ? mwait_idle+0x117/0x120
[516112.244209] [<ffffffff810120c6>] cpu_idle+0x86/0xe0
[516112.244212] [<ffffffff816ae77c>] start_secondary+0x1a3/0x1e7
[516112.344023] BUG: soft lockup - CPU#2 stuck for 23s! [php:1486]
[516112.344025] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2
[516112.344033] CPU 2
[516112.344034] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2
[516112.344040]
[516112.344042] Pid: 1486, comm: php Not tainted 3.1.0-rc9-hw+ #48 Dell Inc. PowerEdge 1950/0UR033
[516112.344046] RIP: 0010:[<ffffffff816b6694>] [<ffffffff816b6694>] _raw_spin_lock+0x14/0x20
[516112.344051] RSP: 0000:ffff88022fc83e10 EFLAGS: 00000297
[516112.344053] RAX: 0000000000000100 RBX: ffffffff81022674 RCX: ffff880226920020
[516112.344056] RDX: ffff88022198c660 RSI: dead000000200200 RDI: ffff8800ac758cc8
[516112.344058] RBP: ffff88022fc83e10 R08: 00000000000000ef R09: 0000000000000000
[516112.344060] R10: 000000000000018b R11: 0000000000000010 R12: ffff88022fc83d88
[516112.344062] R13: ffffffff816bed1e R14: ffff88022fc83e10 R15: ffff880226920000
[516112.344065] FS: 00007faafda03720(0000) GS:ffff88022fc80000(0000) knlGS:0000000000000000
[516112.344068] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[516112.344070] CR2: ffffffffff600400 CR3: 00000002223de000 CR4: 00000000000006e0
[516112.344072] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[516112.344075] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[516112.344077] Process php (pid: 1486, threadinfo ffff880039262000, task ffff88003e675900)
[516112.344079] Stack:
[516112.344081] ffff88022fc83e40 ffffffff8162a613 0000000000000000 0000000000000000
[516112.344084] ffff880226920000 ffff8800ac758e20 ffff88022fc83ee0 ffffffff810692dc
[516112.344088] 0000000000000001 ffff880039263fd8 ffff880039263fd8 ffff880039263fd8
[516112.344091] Call Trace:
[516112.344093] <IRQ>
[516112.344099] [<ffffffff8162a613>] tcp_keepalive_timer+0x23/0x260
[516112.344104] [<ffffffff810692dc>] run_timer_softirq+0x1ac/0x310
[516112.344107] [<ffffffff8162a5f0>] ? tcp_init_xmit_timers+0x20/0x20
[516112.344111] [<ffffffff8102e838>] ? lapic_next_event+0x18/0x20
[516112.344115] [<ffffffff81060bf0>] __do_softirq+0xe0/0x1d0
[516112.344119] [<ffffffff816c04ac>] call_softirq+0x1c/0x30
[516112.344123] [<ffffffff81014255>] do_softirq+0x65/0xa0
Simon-
next parent reply other threads:[~2011-10-18 5:40 UTC|newest]
Thread overview: 31+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <CA+55aFxPNszU5UHFrDDYnshLEMupaviFwhgEsgmPkqpmuWNZ8A@mail.gmail.com>
[not found] ` <20111007070842.GA27555@hostway.ca>
[not found] ` <20111007174848.GA11011@hostway.ca>
[not found] ` <1318010515.398.8.camel@twins>
[not found] ` <20111008005035.GC22843@hostway.ca>
[not found] ` <1318060551.8395.0.camel@twins>
[not found] ` <20111012213555.GC24461@hostway.ca>
2011-10-18 5:40 ` Simon Kirby [this message]
[not found] <1318847658.6594.40.camel@twins>
[not found] ` <CA+55aFxaGKEyhXdHXNxvPrPQ-SGSpbXdfoeXrxfjPx3VXsgvtg@mail.gmail.com>
[not found] ` <1318874090.4172.84.camel@twins>
[not found] ` <CA+55aFwCBy=4YK6amE=H-BYu9-boj4Po2Zkgf4V261mCx0DC4A@mail.gmail.com>
[not found] ` <1318879396.4172.92.camel@twins>
[not found] ` <alpine.LFD.2.02.1110172237030.3240@ionos>
[not found] ` <alpine.LFD.2.02.1110181037120.3240@ionos>
[not found] ` <1318928713.21167.4.camel@twins>
[not found] ` <20111018182046.GF1309@hostway.ca>
[not found] ` <alpine.LFD.2.02.1110182146440.3240@ionos>
[not found] ` <20111024190203.GA24410@hostway.ca>
2011-10-25 7:13 ` Linux 3.1-rc9 Linus Torvalds
2011-10-25 9:01 ` David Miller
2011-10-25 12:30 ` Thomas Gleixner
2011-10-25 23:18 ` David Miller
2011-10-25 20:20 ` Simon Kirby
2011-10-31 17:32 ` Simon Kirby
2011-11-02 16:40 ` Thomas Gleixner
2011-11-02 17:27 ` Eric Dumazet
2011-11-02 17:46 ` Linus Torvalds
2011-11-02 17:53 ` Eric Dumazet
2011-11-02 18:00 ` Linus Torvalds
2011-11-02 18:05 ` Eric Dumazet
2011-11-02 18:10 ` Linus Torvalds
2011-11-02 17:49 ` Eric Dumazet
2011-11-02 17:58 ` Eric Dumazet
2011-11-02 19:16 ` Simon Kirby
2011-11-02 22:42 ` Eric Dumazet
2011-11-03 0:24 ` Thomas Gleixner
2011-11-03 0:52 ` Simon Kirby
2011-11-03 22:07 ` David Miller
2011-11-03 6:06 ` Jörg-Volker Peetz
2011-11-02 17:54 ` Thomas Gleixner
2011-11-02 18:04 ` Eric Dumazet
2011-11-02 18:28 ` Simon Kirby
2011-11-02 18:30 ` Thomas Gleixner
2011-11-02 22:10 ` Steven Rostedt
2011-11-02 23:00 ` Steven Rostedt
2011-11-03 0:09 ` Simon Kirby
2011-11-03 0:15 ` Steven Rostedt
2011-11-03 0:17 ` Simon Kirby
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20111018054051.GA19085@hostway.ca \
--to=sim@hostway.ca \
--cc=linux-kernel@vger.kernel.org \
--cc=netdev@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).