From: Patrick McHardy <kaber@trash.net>
To: Pekka Enberg <penberg@cs.helsinki.fi>
Cc: Ingo Molnar <mingo@elte.hu>, David Miller <davem@davemloft.net>,
herbert@gondor.apana.org.au, w@1wt.eu, davidn@davidnewall.com,
torvalds@linux-foundation.org, akpm@linux-foundation.org,
netdev@vger.kernel.org, linux-kernel@vger.kernel.org,
stefanr@s5r6.in-berlin.de, rjw@sisk.pl,
ilpo.jarvinen@helsinki.fi, Dave Jones <davej@redhat.com>
Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference
Date: Thu, 24 Jul 2008 14:49:53 +0200 [thread overview]
Message-ID: <48887A71.5010209@trash.net> (raw)
In-Reply-To: <84144f020807240544q507e1b7cv220d1afbae0ee0f0@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1335 bytes --]
Pekka Enberg wrote:
> On Thu, Jul 24, 2008 at 3:03 PM, Patrick McHardy <kaber@trash.net> wrote:
>> Ingo Molnar wrote:
>>> * Ingo Molnar <mingo@elte.hu> wrote:
>>>
>>>> here's a new type of crash a -tip testbox triggered today:
>>>>
>>>> BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
>>>> IP: [<0000000000000000>] 0x0
>>>> ...
>>>> Call Trace:
>>>> [<ffffffff80777f25>] ? ipv4_confirm+0x8d/0x122
>>>> [<ffffffff80733107>] nf_iterate+0x43/0xa5
>> Does reverting 31d8519c fix this?
>
> Hmm, why do you think it's related? It looks like elem->hook() is a
> NULL pointer but my patch shouldn't make any difference here...
No, its already in ipv4_confirm, so its most likely helper->help()
thats NULL, which is contained in an extension.
The reason why I think its this patch is (quoting from an old
email that I never got a response to):
---
Your patch introduced a use-after-free and double-free.
krealloc() frees the old pointer, but it is still used
for the ->move operations, then freed again.
To fix this I think we need a __krealloc() that doesn't
free the old memory, especially since it must not be
freed immediately because it may still be used in a RCU
read side (see the last part in the patch attached to
this mail (based on a kernel without your patch)).
---
I've attached that patch again.
[-- Attachment #2: 01.diff --]
[-- Type: text/x-diff, Size: 3306 bytes --]
netfilter: nf_nat: fix RCU races
Fix three ct_extend/NAT extension related races:
- When cleaning up the extension area and removing it from the bysource hash,
the nat->ct pointer must not be set to NULL since it may still be used in
a RCU read side
- When replacing a NAT extension area in the bysource hash, the nat->ct
pointer must be assigned before performing the replacement
- When reallocating extension storage in ct_extend, the old memory must
not be freed immediately since it may still be used by a RCU read side
Possibly fixes https://bugzilla.redhat.com/show_bug.cgi?id=449315
and/or http://bugzilla.kernel.org/show_bug.cgi?id=10875
Signed-off-by: Patrick McHardy <kaber@trash.net>
---
commit 8d4c178a5e17c19cf7a781b0e5e416c4e22b1ff2
tree 2c4651788906d120cb7636006e2178dbd7a283c4
parent ec0a196626bd12e0ba108d7daa6d95a4fb25c2c5
author Patrick McHardy <kaber@trash.net> Sat, 14 Jun 2008 12:42:45 +0200
committer Patrick McHardy <kaber@trash.net> Sat, 14 Jun 2008 12:42:45 +0200
include/net/netfilter/nf_conntrack_extend.h | 1 +
net/ipv4/netfilter/nf_nat_core.c | 3 +--
net/netfilter/nf_conntrack_extend.c | 9 ++++++++-
3 files changed, 10 insertions(+), 3 deletions(-)
diff --git a/include/net/netfilter/nf_conntrack_extend.h b/include/net/netfilter/nf_conntrack_extend.h
index f736e84..f80c0ed 100644
--- a/include/net/netfilter/nf_conntrack_extend.h
+++ b/include/net/netfilter/nf_conntrack_extend.h
@@ -15,6 +15,7 @@ enum nf_ct_ext_id
/* Extensions: optional stuff which isn't permanently in struct. */
struct nf_ct_ext {
+ struct rcu_head rcu;
u8 offset[NF_CT_EXT_NUM];
u8 len;
char data[0];
diff --git a/net/ipv4/netfilter/nf_nat_core.c b/net/ipv4/netfilter/nf_nat_core.c
index 0457859..d2a887f 100644
--- a/net/ipv4/netfilter/nf_nat_core.c
+++ b/net/ipv4/netfilter/nf_nat_core.c
@@ -556,7 +556,6 @@ static void nf_nat_cleanup_conntrack(struct nf_conn *ct)
spin_lock_bh(&nf_nat_lock);
hlist_del_rcu(&nat->bysource);
- nat->ct = NULL;
spin_unlock_bh(&nf_nat_lock);
}
@@ -570,8 +569,8 @@ static void nf_nat_move_storage(void *new, void *old)
return;
spin_lock_bh(&nf_nat_lock);
- hlist_replace_rcu(&old_nat->bysource, &new_nat->bysource);
new_nat->ct = ct;
+ hlist_replace_rcu(&old_nat->bysource, &new_nat->bysource);
spin_unlock_bh(&nf_nat_lock);
}
diff --git a/net/netfilter/nf_conntrack_extend.c b/net/netfilter/nf_conntrack_extend.c
index bcc19fa..8a3f8b3 100644
--- a/net/netfilter/nf_conntrack_extend.c
+++ b/net/netfilter/nf_conntrack_extend.c
@@ -59,12 +59,19 @@ nf_ct_ext_create(struct nf_ct_ext **ext, enum nf_ct_ext_id id, gfp_t gfp)
if (!*ext)
return NULL;
+ INIT_RCU_HEAD(&(*ext)->rcu);
(*ext)->offset[id] = off;
(*ext)->len = len;
return (void *)(*ext) + off;
}
+static void __nf_ct_ext_free_rcu(struct rcu_head *head)
+{
+ struct nf_ct_ext *ext = container_of(head, struct nf_ct_ext, rcu);
+ kfree(ext);
+}
+
void *__nf_ct_ext_add(struct nf_conn *ct, enum nf_ct_ext_id id, gfp_t gfp)
{
struct nf_ct_ext *new;
@@ -106,7 +113,7 @@ void *__nf_ct_ext_add(struct nf_conn *ct, enum nf_ct_ext_id id, gfp_t gfp)
(void *)ct->ext + ct->ext->offset[i]);
rcu_read_unlock();
}
- kfree(ct->ext);
+ call_rcu(&ct->ext->rcu, __nf_ct_ext_free_rcu);
ct->ext = new;
}
next prev parent reply other threads:[~2008-07-24 12:49 UTC|newest]
Thread overview: 204+ messages / expand[flat|nested] mbox.gz Atom feed top
2008-07-20 17:44 [GIT]: Networking David Miller
2008-07-20 17:59 ` Arjan van de Ven
2008-07-20 23:52 ` David Miller
2008-07-21 20:32 ` David Miller
2008-07-21 0:54 ` Linus Torvalds
2008-07-21 1:03 ` David Miller
2008-07-21 1:09 ` Alexey Dobriyan
2008-07-21 1:14 ` David Miller
2008-07-21 1:22 ` Alexey Dobriyan
2008-07-21 2:40 ` Alexey Dobriyan
2008-07-21 2:48 ` David Miller
2008-07-21 5:11 ` David Miller
2008-07-21 9:48 ` Alexander Beregalov
2008-07-21 10:16 ` Ben Hutchings
2008-07-21 15:35 ` David Miller
2008-07-21 16:04 ` Alexander Beregalov
2008-07-21 11:57 ` Alexey Dobriyan
2008-07-21 15:27 ` David Miller
2008-07-21 16:49 ` Linus Torvalds
2008-07-21 16:53 ` David Miller
2008-07-21 1:20 ` Patrick McHardy
2008-07-21 11:28 ` Stefan Richter
2008-07-21 11:45 ` James Morris
2008-07-21 12:05 ` Patrick McHardy
2008-07-21 17:28 ` David Miller
2008-07-21 17:40 ` Linus Torvalds
2008-07-21 20:33 ` Patrick McHardy
2008-07-23 23:42 ` David Miller
2008-07-21 1:07 ` Linus Torvalds
2008-07-21 1:17 ` David Miller
2008-07-21 8:36 ` iwlwifi: fix build bug in "iwlwifi: fix LED stall" Ingo Molnar
2008-07-21 10:02 ` Winkler, Tomas
2008-07-21 10:53 ` Ingo Molnar
2008-07-21 12:12 ` [PATCH] iwlwifi: RS small compile warnings without CONFIG_IWLWIFI_DEBUG Tomas Winkler
2008-07-21 12:12 ` [PATCH] iwlwifi: " Tomas Winkler
2008-07-21 12:12 ` [PATCH] iwlwifi: compilation error when CONFIG_IWLWIFI_DEBUG is not set Tomas Winkler
2008-07-21 13:30 ` [crash, bisected] Kernel BUG at ffffffff8079afb1 (__netif_schedule()) Ingo Molnar
2008-07-21 13:45 ` [crash] BUG: unable to handle kernel NULL pointer dereference at 0000000000000370 Ingo Molnar
2008-07-21 14:30 ` Ingo Molnar
2008-07-21 15:04 ` Ingo Molnar
2008-07-21 15:24 ` David Miller
2008-07-21 18:18 ` Ian Schram
2008-07-21 19:06 ` Ingo Molnar
[not found] ` <20080721190646.GA19044-X9Un+BFzKDI@public.gmane.org>
2008-07-21 19:13 ` Larry Finger
[not found] ` <4884DFEE.4060003-tQ5ms3gMjBLk1uMJSBkQmQ@public.gmane.org>
2008-07-21 19:34 ` Ingo Molnar
[not found] ` <20080721193425.GA29287-X9Un+BFzKDI@public.gmane.org>
2008-07-21 19:43 ` Larry Finger
[not found] ` <4884E6E6.7010800-tQ5ms3gMjBLk1uMJSBkQmQ@public.gmane.org>
2008-07-21 19:47 ` Linus Torvalds
2008-07-21 20:15 ` David Miller
[not found] ` <alpine.LFD.1.10.0807211246280.31863-5CScLwifNT1QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org>
2008-07-21 20:28 ` Larry Finger
2008-07-21 20:21 ` David Miller
[not found] ` <20080721.132100.128525904.davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>
2008-07-21 20:38 ` Larry Finger
2008-07-21 20:46 ` David Miller
2008-07-21 20:51 ` Patrick McHardy
2008-07-21 21:01 ` David Miller
2008-07-21 21:06 ` Patrick McHardy
[not found] ` <4884FA3F.70109-dcUjhNyLwpNeoWH0uzbU5w@public.gmane.org>
2008-07-21 21:35 ` Patrick McHardy
[not found] ` <48850121.9090106-dcUjhNyLwpNeoWH0uzbU5w@public.gmane.org>
2008-07-21 21:42 ` Patrick McHardy
2008-07-21 21:51 ` Larry Finger
[not found] ` <488504D9.5090100-tQ5ms3gMjBLk1uMJSBkQmQ@public.gmane.org>
2008-07-21 22:04 ` Patrick McHardy
[not found] ` <488507FB.9050101-dcUjhNyLwpNeoWH0uzbU5w@public.gmane.org>
2008-07-21 22:40 ` Larry Finger
[not found] ` <4885104A.2070201-tQ5ms3gMjBLk1uMJSBkQmQ@public.gmane.org>
2008-07-21 23:15 ` David Miller
2008-07-22 6:34 ` Larry Finger
[not found] ` <48857F74.2040406-tQ5ms3gMjBLk1uMJSBkQmQ@public.gmane.org>
2008-07-22 10:51 ` Jarek Poplawski
2008-07-22 11:32 ` David Miller
2008-07-22 12:52 ` Larry Finger
2008-07-22 20:43 ` David Miller
2008-07-22 13:02 ` Larry Finger
[not found] ` <4885DA49.50703-tQ5ms3gMjBLk1uMJSBkQmQ@public.gmane.org>
2008-07-22 14:53 ` Patrick McHardy
[not found] ` <4885F46A.30309-dcUjhNyLwpNeoWH0uzbU5w@public.gmane.org>
2008-07-22 21:17 ` David Miller
2008-07-22 16:39 ` Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98() Larry Finger
[not found] ` <48860D4B.8070003-tQ5ms3gMjBLk1uMJSBkQmQ@public.gmane.org>
2008-07-22 17:20 ` Patrick McHardy
2008-07-22 18:39 ` Larry Finger
2008-07-22 18:44 ` Patrick McHardy
[not found] ` <48862A76.8030307-dcUjhNyLwpNeoWH0uzbU5w@public.gmane.org>
2008-07-22 19:30 ` Larry Finger
2008-07-22 23:04 ` David Miller
2008-07-23 6:20 ` Jarek Poplawski
2008-07-23 7:59 ` David Miller
[not found] ` <20080723.005921.113868915.davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>
2008-07-23 8:54 ` Jarek Poplawski
[not found] ` <20080723085452.GB4561-8HppEYmqbBCE+EvaaNYduQ@public.gmane.org>
2008-07-23 9:03 ` Peter Zijlstra
2008-07-23 9:35 ` Jarek Poplawski
[not found] ` <20080723093459.GC4561-8HppEYmqbBCE+EvaaNYduQ@public.gmane.org>
2008-07-23 9:50 ` Peter Zijlstra
2008-07-23 10:13 ` Jarek Poplawski
2008-07-23 10:58 ` Peter Zijlstra
2008-07-23 11:35 ` Jarek Poplawski
[not found] ` <20080723113519.GE4561-8HppEYmqbBCE+EvaaNYduQ@public.gmane.org>
2008-07-23 11:49 ` Jarek Poplawski
2008-07-23 20:16 ` David Miller
2008-07-23 20:43 ` Jarek Poplawski
[not found] ` <20080723204335.GA14788-dUp/P3zyUuaNj9Bq2fkWzw@public.gmane.org>
2008-07-23 20:55 ` David Miller
[not found] ` <20080723.131607.79681752.davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>
2008-07-24 9:10 ` Peter Zijlstra
2008-07-24 9:20 ` David Miller
[not found] ` <20080724.022040.23129457.davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>
2008-07-24 9:27 ` Peter Zijlstra
2008-07-24 9:32 ` David Miller
[not found] ` <20080724.023210.229338550.davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>
2008-07-24 10:08 ` Peter Zijlstra
2008-07-24 10:38 ` Nick Piggin
[not found] ` <200807242038.36693.nickpiggin-/E1597aS9LT0CCvOHzKKcA@public.gmane.org>
2008-07-24 10:55 ` Miklos Szeredi
[not found] ` <E1KLyUI-0006ZS-Cg-8f8m9JG5TPIdUIPVzhDTVZP2KDSNp7ea@public.gmane.org>
2008-07-24 11:06 ` Nick Piggin
[not found] ` <200807242106.52672.nickpiggin-/E1597aS9LT0CCvOHzKKcA@public.gmane.org>
2008-08-01 21:10 ` Paul E. McKenney
2008-07-24 10:59 ` Peter Zijlstra
2008-08-01 21:10 ` Paul E. McKenney
2008-07-23 20:14 ` David Miller
[not found] ` <20080723.131441.200166513.davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>
2008-07-24 7:00 ` Peter Zijlstra
2008-07-25 17:04 ` Ingo Oeser
[not found] ` <200807251904.37302.netdev-BkyiQQGWkgE@public.gmane.org>
2008-07-25 18:36 ` Jarek Poplawski
2008-07-25 19:16 ` Johannes Berg
[not found] ` <1217013384.4758.5.camel-YfaajirXv214zXjbi5bjpg@public.gmane.org>
2008-07-25 19:34 ` Jarek Poplawski
2008-07-25 19:36 ` Johannes Berg
2008-07-25 20:01 ` Jarek Poplawski
[not found] ` <20080725200137.GC3107-dUp/P3zyUuaNj9Bq2fkWzw@public.gmane.org>
2008-07-26 9:18 ` David Miller
2008-07-26 10:53 ` Jarek Poplawski
[not found] ` <20080726.021846.236624483.davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>
2008-07-26 13:18 ` Jarek Poplawski
[not found] ` <20080726131838.GA2873-dUp/P3zyUuaNj9Bq2fkWzw@public.gmane.org>
2008-07-27 0:34 ` David Miller
[not found] ` <20080726.173434.48036095.davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>
2008-07-27 20:37 ` Jarek Poplawski
[not found] ` <20080727203757.GA2527-dUp/P3zyUuaNj9Bq2fkWzw@public.gmane.org>
2008-07-31 12:29 ` David Miller
[not found] ` <20080731.052932.110299354.davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>
2008-07-31 12:38 ` Nick Piggin
2008-07-31 12:44 ` David Miller
2008-08-01 4:27 ` David Miller
[not found] ` <20080731.212729.138736823.davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>
2008-08-01 7:09 ` Peter Zijlstra
2008-08-01 6:48 ` Jarek Poplawski
[not found] ` <20080801064810.GA4435-8HppEYmqbBCE+EvaaNYduQ@public.gmane.org>
2008-08-01 7:00 ` David Miller
2008-08-01 7:01 ` Jarek Poplawski
[not found] ` <20080801070150.GB4435-8HppEYmqbBCE+EvaaNYduQ@public.gmane.org>
2008-08-01 7:01 ` David Miller
2008-08-01 7:41 ` Jarek Poplawski
2008-07-25 6:20 ` [lockdep warning] AOE / networking: aoenet_xmit: noop_qdisc.q.lock, INFO: inconsistent lock state at 0000000000000370 Ingo Molnar
2008-07-25 6:25 ` David Miller
2008-07-25 7:26 ` Ingo Molnar
2008-07-25 8:23 ` David Miller
[not found] ` <20080721143023.GA32451-X9Un+BFzKDI@public.gmane.org>
2008-07-21 15:10 ` [crash] BUG: unable to handle kernel NULL pointer dereference " David Miller
2008-07-21 18:23 ` [crash] kernel BUG at net/core/dev.c:1328! Ingo Molnar
2008-07-21 18:35 ` Linus Torvalds
2008-07-21 18:46 ` Ingo Molnar
2008-07-21 19:30 ` Ingo Molnar
2008-07-22 11:21 ` [TCP bug] stuck distcc connections in latest -git Ingo Molnar
2008-07-22 13:45 ` David Newall
2008-07-22 13:57 ` Ingo Molnar
2008-07-22 14:54 ` David Newall
2008-07-22 15:34 ` Ingo Molnar
2008-07-22 21:12 ` Willy Tarreau
2008-07-23 8:26 ` Ingo Molnar
2008-07-24 6:04 ` [TCP bug, regression] " Ingo Molnar
2008-07-24 6:32 ` Ingo Molnar
2008-07-24 7:33 ` Willy Tarreau
2008-07-24 8:35 ` Ingo Molnar
2008-07-24 7:53 ` Herbert Xu
2008-07-24 8:24 ` Willy Tarreau
2008-07-24 8:27 ` Ingo Molnar
2008-07-24 8:36 ` David Miller
2008-07-24 9:05 ` Herbert Xu
2008-07-24 9:22 ` David Miller
2008-07-24 9:34 ` Ingo Molnar
2008-07-24 11:56 ` [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference Ingo Molnar
2008-07-24 11:59 ` Ingo Molnar
2008-07-24 12:03 ` Patrick McHardy
2008-07-24 12:22 ` Herbert Xu
2008-07-24 12:40 ` Pekka Enberg
2008-07-24 12:50 ` Herbert Xu
2008-07-24 12:56 ` Nick Piggin
2008-07-24 13:04 ` Herbert Xu
2008-07-24 13:13 ` Nick Piggin
2008-07-24 13:32 ` Pekka Enberg
2008-07-24 19:21 ` Matt Mackall
2008-07-25 9:09 ` Nick Piggin
2008-07-24 13:11 ` Matt Mackall
2008-07-24 14:37 ` Herbert Xu
2008-07-24 17:47 ` Matt Mackall
2008-07-25 1:39 ` Herbert Xu
2008-07-25 2:59 ` Matt Mackall
2008-07-24 12:44 ` Pekka Enberg
2008-07-24 12:49 ` Patrick McHardy [this message]
2008-07-24 13:23 ` Pekka Enberg
2008-07-24 13:31 ` Patrick McHardy
2008-07-24 13:34 ` Pekka Enberg
2008-07-24 18:51 ` Andrew Morton
2008-07-24 18:55 ` Pekka Enberg
2008-07-24 20:58 ` David Miller
2008-07-25 8:02 ` Dieter Ries
2008-07-25 10:41 ` Pekka Enberg
2008-07-24 19:35 ` Ingo Molnar
2008-07-26 16:09 ` Patrick McHardy
2008-07-26 17:34 ` Ingo Molnar
2008-07-26 13:43 ` Patrick McHardy
2008-07-24 21:13 ` Linus Torvalds
2008-07-24 22:09 ` David Miller
2008-07-26 13:47 ` Patrick McHardy
2008-08-01 21:10 ` Paul E. McKenney
2008-07-24 14:23 ` Ingo Molnar
2008-07-24 15:23 ` Patrick McHardy
2008-07-24 15:32 ` Ingo Molnar
2008-07-24 15:34 ` Patrick McHardy
2008-07-24 18:00 ` Krzysztof Oledzki
2008-07-24 13:01 ` [TCP bug, regression] stuck distcc connections in latest -git Willy Tarreau
2008-07-24 9:25 ` Ingo Molnar
2008-07-24 9:29 ` David Miller
2008-07-24 11:12 ` Herbert Xu
2008-07-24 9:36 ` Ilpo Järvinen
2008-07-24 10:03 ` Ilpo Järvinen
2008-07-21 19:00 ` [crash] kernel BUG at net/core/dev.c:1328! David Miller
2008-07-21 19:20 ` Stefan Richter
2008-07-21 20:11 ` David Miller
2008-07-21 21:26 ` Stefan Richter
2008-07-21 19:44 ` Ingo Molnar
2008-07-21 20:20 ` David Miller
2008-07-21 15:07 ` [crash, bisected] Kernel BUG at ffffffff8079afb1 (__netif_schedule()) David Miller
2008-07-21 13:50 ` [GIT]: Networking Ingo Molnar
2008-07-21 14:15 ` Stefan Richter
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=48887A71.5010209@trash.net \
--to=kaber@trash.net \
--cc=akpm@linux-foundation.org \
--cc=davej@redhat.com \
--cc=davem@davemloft.net \
--cc=davidn@davidnewall.com \
--cc=herbert@gondor.apana.org.au \
--cc=ilpo.jarvinen@helsinki.fi \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@elte.hu \
--cc=netdev@vger.kernel.org \
--cc=penberg@cs.helsinki.fi \
--cc=rjw@sisk.pl \
--cc=stefanr@s5r6.in-berlin.de \
--cc=torvalds@linux-foundation.org \
--cc=w@1wt.eu \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).