From: Eric Dumazet <dada1@cosmosbay.com>
To: David Miller <davem@davemloft.net>
Cc: denys@visp.net.lb, netdev@vger.kernel.org, kaber@trash.net,
netfilter-devel@vger.kernel.org
Subject: Re: kernel 2.6.25-rc7 highly unstable on high load
Date: Thu, 27 Mar 2008 08:03:25 +0100 [thread overview]
Message-ID: <47EB46BD.2080001@cosmosbay.com> (raw)
In-Reply-To: <20080326.234049.256533912.davem@davemloft.net>
David Miller a écrit :
> From: "Denys Fedoryshchenko" <denys@visp.net.lb>
> Date: Thu, 27 Mar 2008 08:35:06 +0200
>
>> It seems i am having very bad luck with 2.6.27. As Linus told, it have to be
>> released soon, but it is crashing like hell on high network load.
>
> That's amazing, you've taken a trip into the future and are running
> 2.6.27 already, please let me borrow your time machine :-)
>
> More seriously, there is obviously something very unique to your
> setup or else everyone would be reporting this crash, and we have
> to find out what that might be.
>
> There seems to be bunch of netfilter stuff in your traces, but
> the top of the trace is somewhere totally unrelated. This is
> a common reoccurance in your crash traces, making them less
> useful than they could be.
>
> I know you asked before what can be done to improve the traces,
> but I'm not an x86 expert so I have no idea how to help you
> in that area.
>
> Patrick, could you see if you can make any sense of his log?
> I see conttrack a lot in the backtraces.
I can see rt_garbage_collect() involved here. This one might explain very long
delays in softirq processing, and eventually crashes...
Denys, could you post :
grep . /proc/sys/net/ipv4/route/*
rtstat -c1 -i10
So that we can check if you should first change route cache tunables :)
>
> Thanks.
>
>> Here is a message i got over syslog on last crash (it was 2.6.25-rc6-git6),
>> available also at http://www.nuclearcat.com/files/crash_2.6.25.txt
>>
>> Mar 26 02:27:14 ROUTER [ 4698.694693] BUG: NMI Watchdog detected LOCKUP
>> Mar 26 02:27:14 ROUTER on CPU1, ip c02ad109, registers:
>> Mar 26 02:27:14 ROUTER [ 4698.694693] Process snmpd (pid: 2327, ti=c092e000
>> task=f7459080 task.ti=f70b7000)
>> Mar 26 02:27:14 ROUTER
>> Mar 26 02:27:14 ROUTER [ 4698.694693] Stack:
>> Mar 26 02:27:14 ROUTER c092eb14
>> Mar 26 02:27:14 ROUTER c011991e
>> Mar 26 02:27:14 ROUTER f750d600
>> Mar 26 02:27:14 ROUTER f750d600
>> Mar 26 02:27:14 ROUTER c0378058
>> Mar 26 02:27:14 ROUTER 00000001
>> Mar 26 02:27:14 ROUTER c092eb34
>> Mar 26 02:27:14 ROUTER c0119b3b
>> Mar 26 02:27:14 ROUTER
>> Mar 26 02:27:14 ROUTER [ 4698.694693]
>> Mar 26 02:27:14 ROUTER 00000000
>> Mar 26 02:27:14 ROUTER 00000001
>> Mar 26 02:27:14 ROUTER 00000082
>> Mar 26 02:27:14 ROUTER f708af88
>> Mar 26 02:27:14 ROUTER c0378058
>> Mar 26 02:27:14 ROUTER 00000001
>> Mar 26 02:27:14 ROUTER c092eb3c
>> Mar 26 02:27:14 ROUTER c0119bfe
>> Mar 26 02:27:14 ROUTER
>> Mar 26 02:27:14 ROUTER [ 4698.694693]
>> Mar 26 02:27:14 ROUTER c092eb50
>> Mar 26 02:27:14 ROUTER c012f19c
>> Mar 26 02:27:14 ROUTER 00000000
>> Mar 26 02:27:14 ROUTER f708af88
>> Mar 26 02:27:14 ROUTER c0378058
>> Mar 26 02:27:14 ROUTER c092eb74
>> Mar 26 02:27:14 ROUTER c011652a
>> Mar 26 02:27:14 ROUTER 00000000
>> Mar 26 02:27:14 ROUTER
>> Mar 26 02:27:14 ROUTER [ 4698.694693] Call Trace:
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<c011991e>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER task_rq_lock+0x31/0x58
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<c0119b3b>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER try_to_wake_up+0x19/0xd1
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<c0119bfe>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER default_wake_function+0xb/0xd
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<c012f19c>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER autoremove_wake_function+0xf/0x33
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<c011652a>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER __wake_up_common+0x2f/0x5a
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<c01189b8>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER __wake_up+0x28/0x3b
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<c01201a3>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER wake_up_klogd+0x2e/0x31
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<c012033d>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER release_console_sem+0x197/0x19f
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<c0120747>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER vprintk+0x295/0x2e5
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<f899634c>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER death_by_timeout+0x8b/0xa3 [nf_conntrack]
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<f8999d08>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER tcp_packet+0x931/0x9e5 [nf_conntrack]
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<c01207ac>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER printk+0x15/0x17
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<c011fb65>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER warn_on_slowpath+0x2a/0x51
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<c011764a>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER __update_rq_clock+0x1c/0x126
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<c0116ab3>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER update_curr+0x48/0x64
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<f89961ed>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER nf_ct_invert_tuple+0x63/0x6f [nf_conntrack]
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<f8996cca>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER nf_conntrack_tuple_taken+0xf8/0x100 [nf_conntrack]
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<f899850c>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER __nf_ct_helper_find+0x2c/0x90 [nf_conntrack]
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<f8996b95>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER nf_conntrack_alter_reply+0x4a/0x87 [nf_conntrack]
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<f8974976>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER nf_nat_setup_info+0x3cc/0x55a [nf_nat]
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<c011701c>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER dequeue_rt_entity+0x88/0x171
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<c0117127>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER dequeue_rt_stack+0x22/0x27
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<c0117425>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER enqueue_task_rt+0x19/0x2c
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<c011617f>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER enqueue_task+0xd/0x18
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<c01161c0>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER activate_task+0x1e/0x2b
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<c0119bb1>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER try_to_wake_up+0x8f/0xd1
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<c0119c1b>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER wake_up_process+0xf/0x11
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<c013dfa1>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER softlockup_tick+0x9d/0x10b
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<c0126f5c>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER run_local_timers+0x17/0x19
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<c01272fa>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER update_process_times+0x24/0x49
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<c0135f4c>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER tick_periodic+0x62/0x6e
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<c0135f71>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER tick_handle_periodic+0x19/0x68
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<c010e87b>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER smp_apic_timer_interrupt+0x6c/0x81
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<c0104344>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER apic_timer_interrupt+0x28/0x30
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<c02ad202>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER _spin_lock_bh+0x20/0x22
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<c02751fa>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER rt_garbage_collect+0x132/0x27a
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<c0262d95>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER dst_alloc+0x19/0x63
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<c0276eb1>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER ip_route_input+0x6b9/0xbd9
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<c0278898>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER ip_rcv_finish+0x2c/0x29a
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<c0278ef8>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER ip_rcv+0x202/0x22c
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<c025ee4e>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER netif_receive_skb+0x33e/0x3a9
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<c02612c2>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER process_backlog+0x62/0xb5
>> Mar 26 02:27:14 ROUTER [ 4698.694693] [<c0260d27>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER net_rx_action+0x8f/0x191
>> Mar 26 02:27:14 ROUTER [ 4698.694694] [<c01240a7>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER __do_softirq+0x64/0xcd
>> Mar 26 02:27:14 ROUTER [ 4698.694694] [<c0105f0a>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER do_softirq+0x55/0x89
>> Mar 26 02:27:14 ROUTER [ 4698.694694] [<c0123f88>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER local_bh_enable+0x61/0x6d
>> Mar 26 02:27:14 ROUTER [ 4698.694694] [<c0257689>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER lock_sock_nested+0x83/0x8b
>> Mar 26 02:27:14 ROUTER [ 4698.694694] [<c0292e58>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER udp_destroy_sock+0xd/0x20
>> Mar 26 02:27:14 ROUTER [ 4698.694694] [<c0257b9e>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER sk_common_release+0x15/0x60
>> Mar 26 02:27:14 ROUTER [ 4698.694694] [<c02924a4>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER udp_lib_close+0x8/0xa
>> Mar 26 02:27:14 ROUTER [ 4698.694694] [<c0299006>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER inet_release+0x42/0x48
>> Mar 26 02:27:14 ROUTER [ 4698.694694] [<c025625b>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER sock_release+0x14/0x60
>> Mar 26 02:27:14 ROUTER [ 4698.694694] [<c02565d9>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER sock_close+0x29/0x30
>> Mar 26 02:27:14 ROUTER [ 4698.694694] [<c015a6a2>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER __fput+0x93/0x135
>> Mar 26 02:27:14 ROUTER [ 4698.694694] [<c015a8e2>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER fput+0x17/0x19
>> Mar 26 02:27:14 ROUTER [ 4698.694694] [<c01583dc>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER filp_close+0x47/0x51
>> Mar 26 02:27:14 ROUTER [ 4698.694694] [<c0159414>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER sys_close+0x68/0x9d
>> Mar 26 02:27:14 ROUTER [ 4698.694694] [<c0103876>]
>> Mar 26 02:27:14 ROUTER ?
>> Mar 26 02:27:14 ROUTER sysenter_past_esp+0x5f/0x85
>> Mar 26 02:27:14 ROUTER [ 4698.694694] =======================
>> Mar 26 02:27:14 ROUTER [ 4698.694694] Code:
>> Mar 26 02:27:14 ROUTER 94
>> Mar 26 02:27:14 ROUTER c0
>> Mar 26 02:27:14 ROUTER 84
>> Mar 26 02:27:14 ROUTER c0
>> Mar 26 02:27:14 ROUTER b9
>> Mar 26 02:27:14 ROUTER 01
>> Mar 26 02:27:14 ROUTER 00
>> Mar 26 02:27:14 ROUTER 00
>> Mar 26 02:27:14 ROUTER 00
>> Mar 26 02:27:14 ROUTER 75
>> Mar 26 02:27:14 ROUTER 09
>> Mar 26 02:27:14 ROUTER f0
>> Mar 26 02:27:14 ROUTER 81
>> Mar 26 02:27:14 ROUTER 02
>> Mar 26 02:27:14 ROUTER 00
>> Mar 26 02:27:14 ROUTER 00
>> Mar 26 02:27:14 ROUTER 00
>> Mar 26 02:27:14 ROUTER 01
>> Mar 26 02:27:14 ROUTER 30
>> Mar 26 02:27:14 ROUTER c9
>> Mar 26 02:27:14 ROUTER 5d
>> Mar 26 02:27:14 ROUTER 89
>> Mar 26 02:27:14 ROUTER c8
>> Mar 26 02:27:14 ROUTER c3
>> Mar 26 02:27:14 ROUTER 55
>> Mar 26 02:27:14 ROUTER ba
>> Mar 26 02:27:14 ROUTER 00
>> Mar 26 02:27:14 ROUTER 01
>> Mar 26 02:27:14 ROUTER 00
>> Mar 26 02:27:14 ROUTER 00
>> Mar 26 02:27:14 ROUTER 89
>> Mar 26 02:27:14 ROUTER e5
>> Mar 26 02:27:14 ROUTER f0
>> Mar 26 02:27:14 ROUTER 66
>> Mar 26 02:27:14 ROUTER 0f
>> Mar 26 02:27:14 ROUTER c1
>> Mar 26 02:27:14 ROUTER 10
>> Mar 26 02:27:14 ROUTER 38
>> Mar 26 02:27:14 ROUTER f2
>> Mar 26 02:27:14 ROUTER 74
>> Mar 26 02:27:14 ROUTER 06
>> Mar 26 02:27:14 ROUTER f3
>> Mar 26 02:27:14 ROUTER 90
>> Mar 26 02:27:14 ROUTER unparseable log message: "<8a> "
>> Mar 26 02:27:14 ROUTER 10
>> Mar 26 02:27:14 ROUTER eb
>> Mar 26 02:27:14 ROUTER f6
>> Mar 26 02:27:14 ROUTER 5d
>> Mar 26 02:27:14 ROUTER c3
>> Mar 26 02:27:14 ROUTER 55
>> Mar 26 02:27:14 ROUTER 89
>> Mar 26 02:27:14 ROUTER e5
>> Mar 26 02:27:14 ROUTER f0
>> Mar 26 02:27:14 ROUTER 81
>> Mar 26 02:27:14 ROUTER 28
>> Mar 26 02:27:14 ROUTER 00
>> Mar 26 02:27:14 ROUTER 00
>> Mar 26 02:27:14 ROUTER 00
>> Mar 26 02:27:14 ROUTER 01
>> Mar 26 02:27:14 ROUTER 74
>> Mar 26 02:27:14 ROUTER 05
>> Mar 26 02:27:14 ROUTER e8
>> Mar 26 02:27:14 ROUTER 64
>> Mar 26 02:27:14 ROUTER fd
>> Mar 26 02:27:14 ROUTER
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
next prev parent reply other threads:[~2008-03-27 7:03 UTC|newest]
Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top
2008-03-27 6:35 kernel 2.6.25-rc7 highly unstable on high load Denys Fedoryshchenko
2008-03-27 6:40 ` David Miller
2008-03-27 7:03 ` Eric Dumazet [this message]
2008-03-27 8:36 ` Denys Fedoryshchenko
2008-03-27 8:48 ` Denys Fedoryshchenko
2008-03-27 14:41 ` Denys Fedoryshchenko
2008-03-27 14:52 ` Eric Dumazet
[not found] ` <20080327151520.M89250@visp.net.lb>
2008-03-27 16:07 ` Eric Dumazet
2008-03-27 16:29 ` Eric Dumazet
2008-03-27 18:37 ` Denys Fedoryshchenko
2008-03-27 18:56 ` Eric Dumazet
2008-03-27 19:05 ` Denis V. Lunev
2008-03-27 22:03 ` David Miller
2008-03-28 0:47 ` Denys Fedoryshchenko
2008-03-28 4:50 ` Denys Fedoryshchenko
2008-03-28 5:25 ` Denys Fedoryshchenko
2008-03-28 5:49 ` Eric Dumazet
2008-03-28 7:38 ` Denys Fedoryshchenko
2008-03-28 7:46 ` Eric Dumazet
2008-03-28 9:50 ` Denys Fedoryshchenko
2008-03-28 15:57 ` Eric Dumazet
2008-03-28 20:23 ` Stephen Hemminger
2008-03-28 20:45 ` Denys Fedoryshchenko
2008-03-28 21:17 ` Eric Dumazet
2008-03-28 23:21 ` David Miller
2008-03-29 9:35 ` Denys Fedoryshchenko
[not found] ` <20080328220902.M5377@visp.net.lb>
2008-03-29 8:31 ` Eric Dumazet
2008-03-27 7:07 ` Denys Fedoryshchenko
2008-03-27 15:01 ` Patrick McHardy
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=47EB46BD.2080001@cosmosbay.com \
--to=dada1@cosmosbay.com \
--cc=davem@davemloft.net \
--cc=denys@visp.net.lb \
--cc=kaber@trash.net \
--cc=netdev@vger.kernel.org \
--cc=netfilter-devel@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).