From: Andres Freund <andres@anarazel.de>
To: Jarek Poplawski <jarkao2@gmail.com>
Cc: LKML <linux-kernel@vger.kernel.org>,
netdev@vger.kernel.org, Stephen Hemminger <shemminger@vyatta.com>,
Patrick McHardy <kaber@trash.net>
Subject: Re: Soft-Lockup/Race in networking in 2.6.31-rc1+195 (possibly caused by netem)
Date: Thu, 02 Jul 2009 13:11:56 +0200 [thread overview]
Message-ID: <4A4C95FC.7040305@anarazel.de> (raw)
In-Reply-To: <20090702101207.GA7056@ff.dom.local>
[-- Attachment #1: Type: text/plain, Size: 932 bytes --]
On 07/02/2009 12:12 PM, Jarek Poplawski wrote:
> On Thu, Jul 02, 2009 at 09:30:31AM +0000, Jarek Poplawski wrote:
>> On Thu, Jul 02, 2009 at 02:37:24AM +0200, Andres Freund wrote:
>> ...
>>> So I tried - and I did not catch any lockdep output before the crash.
>>> Unfortunately I do not have another machine on the same local network to
>>> catch any messages after the crash... So I could be missing some warning
>>> (I did synchronous logging though).
>>> Will check with netconsole tomorrow.
>>
>> Could you try if this patch changes anything?
>
> ...and maybe CONFIG_PACKET_MMAP turned off.
Ok. Removed the skb_orphan and turned of CONFIG_PACKET_MMAP. Seemingly
the same game.
I now had another computer to catch the netconsole output. Still no
lockdep warnings.
Unfortunately the other computer was a windows machine with its strange
terminal, so long lines are wrapped at 80cols, but that shouldn't be too
bad.
Andres
[-- Attachment #2: dump_no_orphan_no_mmap.log --]
[-- Type: text/plain, Size: 7665 bytes --]
[ 215.208044] netem: version 1.2
[ 350.040136] BUG: soft lockup - CPU#1 stuck for 61s! [openvpn:4248]
[ 350.040136] Modules linked in: sch_netem sbs sbshc pcmcia snd_hda_codec_conex
ant yenta_socket rsrc_nonstatic snd_hda_intel snd_hda_codec thinkpad_acpi iwlagn
pcmcia_core btusb snd_hwdep ehci_hcd uhci_hcd
[ 350.040136] irq event stamp: 149925
[ 350.040136] hardirqs last enabled at (149924): [<ffffffff81036a10>] restore_
args+0x0/0x30
[ 350.040136] hardirqs last disabled at (149925): [<ffffffff81035d3a>] save_arg
s+0x6a/0x70
[ 350.040136] softirqs last enabled at (19946): [<ffffffff815528ad>] lock_sock
_nested+0x8d/0x130
[ 350.040136] softirqs last disabled at (19952): [<ffffffff815627a8>] dev_queue
_xmit+0x58/0x4b0
[ 350.040136] CPU 1:
[ 350.040136] Modules linked in: sch_netem sbs sbshc pcmcia snd_hda_codec_conex
ant yenta_socket rsrc_nonstatic snd_hda_intel snd_hda_codec thinkpad_acpi iwlagn
pcmcia_core btusb snd_hwdep ehci_hcd uhci_hcd
[ 350.040136] Pid: 4248, comm: openvpn Not tainted 2.6.31-rc1-andres-00437-gde7
327a-dirty #61 208252G
[ 350.040136] RIP: 0010:[<ffffffff8103e276>] [<ffffffff8103e276>] native_read_
tsc+0x6/0x20
[ 350.040136] RSP: 0018:ffff8801271c78b8 EFLAGS: 00000206
[ 350.040136] RAX: 000000000ae20bd8 RBX: ffff8801271c78b8 RCX: 000000000ae20b00
[ 350.040136] RDX: 00000000000000b0 RSI: 0000000000006040 RDI: 0000000000000001
[ 350.040136] RBP: ffffffff81036b6e R08: ffffffff82175180 R09: 0000000000000000
[ 350.040136] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8801271c6000
[ 350.040136] R13: 0000000000000000 R14: ffff88002efec400 R15: 0000000000000000
[ 350.040136] FS: 00007f98ddc646f0(0000) GS:ffff88002efde000(0000) knlGS:00000
00000000000
[ 350.040136] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 350.040136] CR2: 0000000005593008 CR3: 000000012742f000 CR4: 00000000000026e0
[ 350.040136] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 350.040136] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 350.040136] Call Trace:
[ 350.040136] [<ffffffff812a96c2>] ? delay_tsc+0x22/0x80
[ 350.040136] [<ffffffff812a95da>] ? __delay+0xa/0x10
[ 350.040136] [<ffffffff812addbd>] ? _raw_spin_lock+0xfd/0x170
[ 350.040136] [<ffffffff816e92f1>] ? _spin_lock+0x51/0x70
[ 350.040136] [<ffffffff81562836>] ? dev_queue_xmit+0xe6/0x4b0
[ 350.040136] [<ffffffff81562836>] ? dev_queue_xmit+0xe6/0x4b0
[ 350.040136] [<ffffffff815627a3>] ? dev_queue_xmit+0x53/0x4b0
[ 350.040136] [<ffffffff81594bac>] ? ip_finish_output+0x13c/0x320
[ 350.040136] [<ffffffff81594e0b>] ? ip_output+0x7b/0xd0
[ 350.040136] [<ffffffff81593bf0>] ? ip_local_out+0x20/0x30
[ 350.040136] [<ffffffff815943c5>] ? ip_queue_xmit+0x165/0x3b0
[ 350.040136] [<ffffffff815a8d49>] ? tcp_transmit_skb+0x3e9/0x780
[ 350.040136] [<ffffffff815ab3b7>] ? tcp_write_xmit+0x1e7/0x9d0
[ 350.040136] [<ffffffff815abc0b>] ? __tcp_push_pending_frames+0x2b/0xd0
[ 350.040136] [<ffffffff8159e327>] ? tcp_sendmsg+0x887/0xb90
[ 350.040136] [<ffffffff8154fa86>] ? sock_sendmsg+0x126/0x140
[ 350.040136] [<ffffffff81097b60>] ? autoremove_wake_function+0x0/0x40
[ 350.040136] [<ffffffff81097b60>] ? autoremove_wake_function+0x0/0x40
[ 350.040136] [<ffffffff810ab0e7>] ? mark_held_locks+0x67/0x90
[ 350.040136] [<ffffffff816e90bb>] ? _spin_unlock_irqrestore+0x3b/0x70
[ 350.040136] [<ffffffff815509c0>] ? sys_sendto+0xf0/0x130
[ 350.040136] [<ffffffff810ab3fd>] ? trace_hardirqs_on_caller+0x14d/0x190
[ 350.040136] [<ffffffff810ab44d>] ? trace_hardirqs_on+0xd/0x10
[ 350.040136] [<ffffffff810a1c27>] ? getnstimeofday+0x57/0xe0
[ 350.040136] [<ffffffff8109bbf1>] ? ktime_get_ts+0x51/0x70
[ 350.040136] [<ffffffff81035ec2>] ? system_call_fastpath+0x16/0x1b
[ 415.538136] BUG: soft lockup - CPU#1 stuck for 61s! [openvpn:4248]
[ 415.538136] Modules linked in: sch_netem sbs sbshc pcmcia snd_hda_codec_conex
ant yenta_socket rsrc_nonstatic snd_hda_intel snd_hda_codec thinkpad_acpi iwlagn
pcmcia_core btusb snd_hwdep ehci_hcd uhci_hcd
[ 415.538136] irq event stamp: 281051
[ 415.538136] hardirqs last enabled at (281050): [<ffffffff81036a10>] restore_
args+0x0/0x30
[ 415.538136] hardirqs last disabled at (281051): [<ffffffff81035d3a>] save_arg
s+0x6a/0x70
[ 415.538136] softirqs last enabled at (19946): [<ffffffff815528ad>] lock_sock
_nested+0x8d/0x130
[ 415.538136] softirqs last disabled at (19952): [<ffffffff815627a8>] dev_queue
_xmit+0x58/0x4b0
[ 415.538136] CPU 1:
[ 415.538136] Modules linked in: sch_netem sbs sbshc pcmcia snd_hda_codec_conex
ant yenta_socket rsrc_nonstatic snd_hda_intel snd_hda_codec thinkpad_acpi iwlagn
pcmcia_core btusb snd_hwdep ehci_hcd uhci_hcd
[ 415.538136] Pid: 4248, comm: openvpn Not tainted 2.6.31-rc1-andres-00437-gde7
327a-dirty #61 208252G
[ 415.538136] RIP: 0010:[<ffffffff8103e276>] [<ffffffff8103e276>] native_read_
tsc+0x6/0x20
[ 415.538136] RSP: 0018:ffff8801271c78b8 EFLAGS: 00000206
[ 415.538136] RAX: 000000008cf9059c RBX: ffff8801271c78b8 RCX: 000000008cf9050c
[ 415.538136] RDX: 00000000000000d4 RSI: 0000000000006040 RDI: 0000000000000001
[ 415.538136] RBP: ffffffff81036b6e R08: ffffffff82175180 R09: 0000000000000000
[ 415.538136] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000003fed
[ 415.538136] R13: ffff88002efde000 R14: ffff8801271c6000 R15: 0000000000000000
[ 415.538136] FS: 00007f98ddc646f0(0000) GS:ffff88002efde000(0000) knlGS:00000
00000000000
[ 415.538136] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 415.538136] CR2: 0000000005593008 CR3: 000000012742f000 CR4: 00000000000026e0
[ 415.538136] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 415.538136] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 415.538136] Call Trace:
[ 415.538136] [<ffffffff812a96ea>] ? delay_tsc+0x4a/0x80
[ 415.538136] [<ffffffff812a95da>] ? __delay+0xa/0x10
[ 415.538136] [<ffffffff812addbd>] ? _raw_spin_lock+0xfd/0x170
[ 415.538136] [<ffffffff816e92f1>] ? _spin_lock+0x51/0x70
[ 415.538136] [<ffffffff81562836>] ? dev_queue_xmit+0xe6/0x4b0
[ 415.538136] [<ffffffff81562836>] ? dev_queue_xmit+0xe6/0x4b0
[ 415.538136] [<ffffffff815627a3>] ? dev_queue_xmit+0x53/0x4b0
[ 415.538136] [<ffffffff81594bac>] ? ip_finish_output+0x13c/0x320
[ 415.538136] [<ffffffff81594e0b>] ? ip_output+0x7b/0xd0
[ 415.538136] [<ffffffff81593bf0>] ? ip_local_out+0x20/0x30
[ 415.538136] [<ffffffff815943c5>] ? ip_queue_xmit+0x165/0x3b0
[ 415.538136] [<ffffffff815a8d49>] ? tcp_transmit_skb+0x3e9/0x780
[ 415.538136] [<ffffffff815ab3b7>] ? tcp_write_xmit+0x1e7/0x9d0
[ 415.538136] [<ffffffff815abc0b>] ? __tcp_push_pending_frames+0x2b/0xd0
[ 415.538136] [<ffffffff8159e327>] ? tcp_sendmsg+0x887/0xb90
[ 415.538136] [<ffffffff8154fa86>] ? sock_sendmsg+0x126/0x140
[ 415.538136] [<ffffffff81097b60>] ? autoremove_wake_function+0x0/0x40
[ 415.538136] [<ffffffff81097b60>] ? autoremove_wake_function+0x0/0x40
[ 415.538136] [<ffffffff810ab0e7>] ? mark_held_locks+0x67/0x90
[ 415.538136] [<ffffffff816e90bb>] ? _spin_unlock_irqrestore+0x3b/0x70
[ 415.538136] [<ffffffff815509c0>] ? sys_sendto+0xf0/0x130
[ 415.538136] [<ffffffff810ab3fd>] ? trace_hardirqs_on_caller+0x14d/0x190
[ 415.538136] [<ffffffff810ab44d>] ? trace_hardirqs_on+0xd/0x10
[ 415.538136] [<ffffffff810a1c27>] ? getnstimeofday+0x57/0xe0
[ 415.538136] [<ffffffff8109bbf1>] ? ktime_get_ts+0x51/0x70
[ 415.538136] [<ffffffff81035ec2>] ? system_call_fastpath+0x16/0x1b
next prev parent reply other threads:[~2009-07-02 11:11 UTC|newest]
Thread overview: 65+ messages / expand[flat|nested] mbox.gz Atom feed top
2009-06-30 23:20 Soft-Lockup/Race in networking in 2.6.31-rc1+195 (possibly caused by netem) Andres Freund
2009-07-01 18:39 ` Jarek Poplawski
2009-07-01 21:22 ` Andres Freund
2009-07-02 0:37 ` Andres Freund
2009-07-02 9:30 ` Jarek Poplawski
2009-07-02 10:12 ` Jarek Poplawski
2009-07-02 10:51 ` Joao Correia
2009-07-02 11:09 ` Jarek Poplawski
2009-07-02 11:11 ` Andres Freund [this message]
2009-07-02 11:43 ` Jarek Poplawski
2009-07-02 11:43 ` Andres Freund
2009-07-02 11:54 ` Jarek Poplawski
2009-07-02 11:59 ` Andres Freund
-- strict thread matches above, loose matches on Subject: below --
2009-07-03 1:31 Soft-Lockup/Race in networking in 2.6.31-rc1+195 ( possibly " Andres Freund
2009-07-03 6:12 ` Soft-Lockup/Race in networking in 2.6.31-rc1+195 ( possibly?caused " Jarek Poplawski
2009-07-03 11:26 ` Andres Freund
2009-07-03 12:03 ` Jarek Poplawski
2009-07-03 12:30 ` Andres Freund
2009-07-03 20:22 ` David Miller
2009-07-03 22:56 ` Jarek Poplawski
2009-07-04 1:55 ` David Miller
2009-07-04 6:36 ` Jarek Poplawski
2009-07-04 15:18 ` Jarek Poplawski
2009-07-06 4:53 ` Joao Correia
2009-07-06 8:14 ` Jarek Poplawski
2009-07-06 11:28 ` Joao Correia
2009-07-06 14:19 ` Jarek Poplawski
2009-07-06 16:13 ` Andres Freund
2009-07-06 16:31 ` Jarek Poplawski
2009-07-06 17:23 ` Joao Correia
2009-07-06 17:26 ` Andres Freund
2009-07-07 6:50 ` Jarek Poplawski
2009-07-07 10:40 ` Joao Correia
2009-07-07 10:47 ` Andres Freund
2009-07-07 13:18 ` Jarek Poplawski
2009-07-07 13:22 ` Andres Freund
2009-07-07 13:29 ` Jarek Poplawski
2009-07-07 13:34 ` Andres Freund
2009-07-07 13:57 ` Jarek Poplawski
2009-07-07 16:11 ` Andres Freund
2009-07-08 8:08 ` Jarek Poplawski
2009-07-08 8:29 ` Andres Freund
2009-07-08 9:13 ` Jarek Poplawski
2009-07-08 21:44 ` Joao Correia
2009-07-08 22:07 ` Jarek Poplawski
2009-07-08 22:27 ` Joao Correia
2009-07-08 22:42 ` Jarek Poplawski
2009-07-08 22:48 ` Joao Correia
2009-07-08 22:23 ` Andres Freund
2009-07-08 22:48 ` Jarek Poplawski
2009-07-09 10:31 ` Thomas Gleixner
2009-07-09 10:44 ` Jarek Poplawski
2009-07-09 12:03 ` Thomas Gleixner
2009-07-09 13:22 ` Jarek Poplawski
2009-07-09 14:15 ` Thomas Gleixner
2009-07-09 14:24 ` Jarek Poplawski
2009-07-09 14:25 ` Joao Correia
2009-07-09 14:28 ` Thomas Gleixner
2009-07-09 15:28 ` Andres Freund
2009-07-09 16:01 ` Thomas Gleixner
2009-07-09 16:46 ` Andres Freund
2009-07-09 17:44 ` Thomas Gleixner
2009-07-09 21:19 ` Joao Correia
2009-07-07 13:20 ` Jarek Poplawski
2009-07-06 17:24 ` Andres Freund
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4A4C95FC.7040305@anarazel.de \
--to=andres@anarazel.de \
--cc=jarkao2@gmail.com \
--cc=kaber@trash.net \
--cc=linux-kernel@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=shemminger@vyatta.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).