From: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
To: linux-kernel@vger.kernel.org
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
stable@vger.kernel.org, Filipe Manana <fdmanana@suse.com>
Subject: [PATCH 4.3 48/71] Btrfs: fix truncation of compressed and inlined extents
Date: Sat, 12 Dec 2015 12:06:12 -0800 [thread overview]
Message-ID: <20151212200539.129607533@linuxfoundation.org> (raw)
In-Reply-To: <20151212200536.761001328@linuxfoundation.org>
4.3-stable review patch. If anyone has any objections, please let me know.
------------------
From: Filipe Manana <fdmanana@suse.com>
commit 0305cd5f7fca85dae392b9ba85b116896eb7c1c7 upstream.
When truncating a file to a smaller size which consists of an inline
extent that is compressed, we did not discard (or made unusable) the
data between the new file size and the old file size, wasting metadata
space and allowing for the truncated data to be leaked and the data
corruption/loss mentioned below.
We were also not correctly decrementing the number of bytes used by the
inode, we were setting it to zero, giving a wrong report for callers of
the stat(2) syscall. The fsck tool also reported an error about a mismatch
between the nbytes of the file versus the real space used by the file.
Now because we weren't discarding the truncated region of the file, it
was possible for a caller of the clone ioctl to actually read the data
that was truncated, allowing for a security breach without requiring root
access to the system, using only standard filesystem operations. The
scenario is the following:
1) User A creates a file which consists of an inline and compressed
extent with a size of 2000 bytes - the file is not accessible to
any other users (no read, write or execution permission for anyone
else);
2) The user truncates the file to a size of 1000 bytes;
3) User A makes the file world readable;
4) User B creates a file consisting of an inline extent of 2000 bytes;
5) User B issues a clone operation from user A's file into its own
file (using a length argument of 0, clone the whole range);
6) User B now gets to see the 1000 bytes that user A truncated from
its file before it made its file world readbale. User B also lost
the bytes in the range [1000, 2000[ bytes from its own file, but
that might be ok if his/her intention was reading stale data from
user A that was never supposed to be public.
Note that this contrasts with the case where we truncate a file from 2000
bytes to 1000 bytes and then truncate it back from 1000 to 2000 bytes. In
this case reading any byte from the range [1000, 2000[ will return a value
of 0x00, instead of the original data.
This problem exists since the clone ioctl was added and happens both with
and without my recent data loss and file corruption fixes for the clone
ioctl (patch "Btrfs: fix file corruption and data loss after cloning
inline extents").
So fix this by truncating the compressed inline extents as we do for the
non-compressed case, which involves decompressing, if the data isn't already
in the page cache, compressing the truncated version of the extent, writing
the compressed content into the inline extent and then truncate it.
The following test case for fstests reproduces the problem. In order for
the test to pass both this fix and my previous fix for the clone ioctl
that forbids cloning a smaller inline extent into a larger one,
which is titled "Btrfs: fix file corruption and data loss after cloning
inline extents", are needed. Without that other fix the test fails in a
different way that does not leak the truncated data, instead part of
destination file gets replaced with zeroes (because the destination file
has a larger inline extent than the source).
seq=`basename $0`
seqres=$RESULT_DIR/$seq
echo "QA output created by $seq"
tmp=/tmp/$$
status=1 # failure is the default!
trap "_cleanup; exit \$status" 0 1 2 3 15
_cleanup()
{
rm -f $tmp.*
}
# get standard environment, filters and checks
. ./common/rc
. ./common/filter
# real QA test starts here
_need_to_be_root
_supported_fs btrfs
_supported_os Linux
_require_scratch
_require_cloner
rm -f $seqres.full
_scratch_mkfs >>$seqres.full 2>&1
_scratch_mount "-o compress"
# Create our test files. File foo is going to be the source of a clone operation
# and consists of a single inline extent with an uncompressed size of 512 bytes,
# while file bar consists of a single inline extent with an uncompressed size of
# 256 bytes. For our test's purpose, it's important that file bar has an inline
# extent with a size smaller than foo's inline extent.
$XFS_IO_PROG -f -c "pwrite -S 0xa1 0 128" \
-c "pwrite -S 0x2a 128 384" \
$SCRATCH_MNT/foo | _filter_xfs_io
$XFS_IO_PROG -f -c "pwrite -S 0xbb 0 256" $SCRATCH_MNT/bar | _filter_xfs_io
# Now durably persist all metadata and data. We do this to make sure that we get
# on disk an inline extent with a size of 512 bytes for file foo.
sync
# Now truncate our file foo to a smaller size. Because it consists of a
# compressed and inline extent, btrfs did not shrink the inline extent to the
# new size (if the extent was not compressed, btrfs would shrink it to 128
# bytes), it only updates the inode's i_size to 128 bytes.
$XFS_IO_PROG -c "truncate 128" $SCRATCH_MNT/foo
# Now clone foo's inline extent into bar.
# This clone operation should fail with errno EOPNOTSUPP because the source
# file consists only of an inline extent and the file's size is smaller than
# the inline extent of the destination (128 bytes < 256 bytes). However the
# clone ioctl was not prepared to deal with a file that has a size smaller
# than the size of its inline extent (something that happens only for compressed
# inline extents), resulting in copying the full inline extent from the source
# file into the destination file.
#
# Note that btrfs' clone operation for inline extents consists of removing the
# inline extent from the destination inode and copy the inline extent from the
# source inode into the destination inode, meaning that if the destination
# inode's inline extent is larger (N bytes) than the source inode's inline
# extent (M bytes), some bytes (N - M bytes) will be lost from the destination
# file. Btrfs could copy the source inline extent's data into the destination's
# inline extent so that we would not lose any data, but that's currently not
# done due to the complexity that would be needed to deal with such cases
# (specially when one or both extents are compressed), returning EOPNOTSUPP, as
# it's normally not a very common case to clone very small files (only case
# where we get inline extents) and copying inline extents does not save any
# space (unlike for normal, non-inlined extents).
$CLONER_PROG -s 0 -d 0 -l 0 $SCRATCH_MNT/foo $SCRATCH_MNT/bar
# Now because the above clone operation used to succeed, and due to foo's inline
# extent not being shinked by the truncate operation, our file bar got the whole
# inline extent copied from foo, making us lose the last 128 bytes from bar
# which got replaced by the bytes in range [128, 256[ from foo before foo was
# truncated - in other words, data loss from bar and being able to read old and
# stale data from foo that should not be possible to read anymore through normal
# filesystem operations. Contrast with the case where we truncate a file from a
# size N to a smaller size M, truncate it back to size N and then read the range
# [M, N[, we should always get the value 0x00 for all the bytes in that range.
# We expected the clone operation to fail with errno EOPNOTSUPP and therefore
# not modify our file's bar data/metadata. So its content should be 256 bytes
# long with all bytes having the value 0xbb.
#
# Without the btrfs bug fix, the clone operation succeeded and resulted in
# leaking truncated data from foo, the bytes that belonged to its range
# [128, 256[, and losing data from bar in that same range. So reading the
# file gave us the following content:
#
# 0000000 a1 a1 a1 a1 a1 a1 a1 a1 a1 a1 a1 a1 a1 a1 a1 a1
# *
# 0000200 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a
# *
# 0000400
echo "File bar's content after the clone operation:"
od -t x1 $SCRATCH_MNT/bar
# Also because the foo's inline extent was not shrunk by the truncate
# operation, btrfs' fsck, which is run by the fstests framework everytime a
# test completes, failed reporting the following error:
#
# root 5 inode 257 errors 400, nbytes wrong
status=0
exit
Signed-off-by: Filipe Manana <fdmanana@suse.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
---
fs/btrfs/inode.c | 82 +++++++++++++++++++++++++++++++++++++++++++++----------
1 file changed, 68 insertions(+), 14 deletions(-)
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -4217,6 +4217,47 @@ static int truncate_space_check(struct b
}
+static int truncate_inline_extent(struct inode *inode,
+ struct btrfs_path *path,
+ struct btrfs_key *found_key,
+ const u64 item_end,
+ const u64 new_size)
+{
+ struct extent_buffer *leaf = path->nodes[0];
+ int slot = path->slots[0];
+ struct btrfs_file_extent_item *fi;
+ u32 size = (u32)(new_size - found_key->offset);
+ struct btrfs_root *root = BTRFS_I(inode)->root;
+
+ fi = btrfs_item_ptr(leaf, slot, struct btrfs_file_extent_item);
+
+ if (btrfs_file_extent_compression(leaf, fi) != BTRFS_COMPRESS_NONE) {
+ loff_t offset = new_size;
+ loff_t page_end = ALIGN(offset, PAGE_CACHE_SIZE);
+
+ /*
+ * Zero out the remaining of the last page of our inline extent,
+ * instead of directly truncating our inline extent here - that
+ * would be much more complex (decompressing all the data, then
+ * compressing the truncated data, which might be bigger than
+ * the size of the inline extent, resize the extent, etc).
+ * We release the path because to get the page we might need to
+ * read the extent item from disk (data not in the page cache).
+ */
+ btrfs_release_path(path);
+ return btrfs_truncate_page(inode, offset, page_end - offset, 0);
+ }
+
+ btrfs_set_file_extent_ram_bytes(leaf, fi, size);
+ size = btrfs_file_extent_calc_inline_size(size);
+ btrfs_truncate_item(root, path, size, 1);
+
+ if (test_bit(BTRFS_ROOT_REF_COWS, &root->state))
+ inode_sub_bytes(inode, item_end + 1 - new_size);
+
+ return 0;
+}
+
/*
* this can truncate away extent items, csum items and directory items.
* It starts at a high offset and removes keys until it can't find
@@ -4411,27 +4452,40 @@ search_again:
* special encodings
*/
if (!del_item &&
- btrfs_file_extent_compression(leaf, fi) == 0 &&
btrfs_file_extent_encryption(leaf, fi) == 0 &&
btrfs_file_extent_other_encoding(leaf, fi) == 0) {
- u32 size = new_size - found_key.offset;
-
- if (test_bit(BTRFS_ROOT_REF_COWS, &root->state))
- inode_sub_bytes(inode, item_end + 1 -
- new_size);
/*
- * update the ram bytes to properly reflect
- * the new size of our item
+ * Need to release path in order to truncate a
+ * compressed extent. So delete any accumulated
+ * extent items so far.
*/
- btrfs_set_file_extent_ram_bytes(leaf, fi, size);
- size =
- btrfs_file_extent_calc_inline_size(size);
- btrfs_truncate_item(root, path, size, 1);
+ if (btrfs_file_extent_compression(leaf, fi) !=
+ BTRFS_COMPRESS_NONE && pending_del_nr) {
+ err = btrfs_del_items(trans, root, path,
+ pending_del_slot,
+ pending_del_nr);
+ if (err) {
+ btrfs_abort_transaction(trans,
+ root,
+ err);
+ goto error;
+ }
+ pending_del_nr = 0;
+ }
+
+ err = truncate_inline_extent(inode, path,
+ &found_key,
+ item_end,
+ new_size);
+ if (err) {
+ btrfs_abort_transaction(trans,
+ root, err);
+ goto error;
+ }
} else if (test_bit(BTRFS_ROOT_REF_COWS,
&root->state)) {
- inode_sub_bytes(inode, item_end + 1 -
- found_key.offset);
+ inode_sub_bytes(inode, item_end + 1 - new_size);
}
}
delete:
next prev parent reply other threads:[~2015-12-12 20:26 UTC|newest]
Thread overview: 93+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-12-12 20:05 [PATCH 4.3 00/71] 4.3.3-stable review Greg Kroah-Hartman
2015-12-12 20:05 ` [PATCH 4.3 01/71] certs: add .gitignore to stop git nagging about x509_certificate_list Greg Kroah-Hartman
2015-12-12 20:05 ` [PATCH 4.3 02/71] r8169: fix kasan reported skb use-after-free Greg Kroah-Hartman
2015-12-12 20:05 ` [PATCH 4.3 03/71] af-unix: fix use-after-free with concurrent readers while splicing Greg Kroah-Hartman
2015-12-12 20:05 ` [PATCH 4.3 04/71] af_unix: dont append consumed skbs to sk_receive_queue Greg Kroah-Hartman
2015-12-12 20:05 ` [PATCH 4.3 05/71] af_unix: take receive queue lock while appending new skb Greg Kroah-Hartman
2015-12-12 20:05 ` [PATCH 4.3 06/71] unix: avoid use-after-free in ep_remove_wait_queue Greg Kroah-Hartman
2015-12-12 20:05 ` [PATCH 4.3 07/71] af-unix: passcred support for sendpage Greg Kroah-Hartman
2015-12-12 20:05 ` [PATCH 4.3 08/71] ipv6: Avoid creating RTF_CACHE from a rt that is not managed by fib6 tree Greg Kroah-Hartman
2015-12-12 20:05 ` [PATCH 4.3 09/71] ipv6: Check expire on DST_NOCACHE route Greg Kroah-Hartman
2015-12-12 20:05 ` [PATCH 4.3 10/71] ipv6: Check rt->dst.from for the " Greg Kroah-Hartman
2015-12-12 20:05 ` [PATCH 4.3 11/71] Revert "ipv6: ndisc: inherit metadata dst when creating ndisc requests" Greg Kroah-Hartman
2015-12-12 20:05 ` [PATCH 4.3 12/71] tools/net: Use include/uapi with __EXPORTED_HEADERS__ Greg Kroah-Hartman
2015-12-12 20:05 ` [PATCH 4.3 13/71] packet: do skb_probe_transport_header when we actually have data Greg Kroah-Hartman
2015-12-12 20:05 ` [PATCH 4.3 14/71] packet: always probe for transport header Greg Kroah-Hartman
2015-12-12 20:05 ` [PATCH 4.3 15/71] packet: only allow extra vlan len on ethernet devices Greg Kroah-Hartman
2015-12-12 20:05 ` [PATCH 4.3 16/71] packet: infer protocol from ethernet header if unset Greg Kroah-Hartman
2015-12-12 20:05 ` [PATCH 4.3 17/71] packet: fix tpacket_snd max frame len Greg Kroah-Hartman
2015-12-12 20:05 ` [PATCH 4.3 18/71] sctp: translate host order to network order when setting a hmacid Greg Kroah-Hartman
2015-12-12 20:05 ` [PATCH 4.3 19/71] net/mlx5e: Added self loopback prevention Greg Kroah-Hartman
2015-12-12 20:05 ` [PATCH 4.3 20/71] net/mlx4_core: Fix sleeping while holding spinlock at rem_slave_counters Greg Kroah-Hartman
2015-12-12 20:05 ` [PATCH 4.3 21/71] ip_tunnel: disable preemption when updating per-cpu tstats Greg Kroah-Hartman
2015-12-12 20:05 ` [PATCH 4.3 22/71] net: switchdev: fix return code of fdb_dump stub Greg Kroah-Hartman
2015-12-12 20:05 ` [PATCH 4.3 23/71] net: thunder: Check for driver data in nicvf_remove() Greg Kroah-Hartman
2015-12-14 7:17 ` Pavel Fedin
2015-12-14 14:16 ` 'Greg Kroah-Hartman'
2015-12-14 14:51 ` Pavel Fedin
2015-12-12 20:05 ` [PATCH 4.3 24/71] snmp: Remove duplicate OUTMCAST stat increment Greg Kroah-Hartman
2015-12-12 20:05 ` [PATCH 4.3 25/71] net/ip6_tunnel: fix dst leak Greg Kroah-Hartman
2015-12-12 20:05 ` [PATCH 4.3 27/71] tcp: md5: fix lockdep annotation Greg Kroah-Hartman
2015-12-12 20:05 ` [PATCH 4.3 28/71] tcp: disable Fast Open on timeouts after handshake Greg Kroah-Hartman
2015-12-12 20:05 ` [PATCH 4.3 29/71] tcp: fix potential huge kmalloc() calls in TCP_REPAIR Greg Kroah-Hartman
2015-12-12 20:05 ` [PATCH 4.3 30/71] tcp: initialize tp->copied_seq in case of cross SYN connection Greg Kroah-Hartman
2015-12-12 20:05 ` [PATCH 4.3 31/71] net, scm: fix PaX detected msg_controllen overflow in scm_detach_fds Greg Kroah-Hartman
2015-12-12 20:05 ` [PATCH 4.3 32/71] net: ipmr: fix static mfc/dev leaks on table destruction Greg Kroah-Hartman
2015-12-12 20:05 ` [PATCH 4.3 33/71] net: ip6mr: " Greg Kroah-Hartman
2015-12-12 20:05 ` [PATCH 4.3 34/71] vrf: fix double free and memory corruption on register_netdevice failure Greg Kroah-Hartman
2015-12-14 17:45 ` Ben Hutchings
2015-12-14 18:59 ` David Ahern
2015-12-15 5:40 ` Greg Kroah-Hartman
2015-12-15 15:12 ` [PATCH 4.3] vrf: Fix memory leak on registration failure in vrf_newlink() Ben Hutchings
2015-12-15 15:15 ` David Ahern
2015-12-15 15:26 ` Ben Hutchings
2015-12-15 15:31 ` [PATCH 4.3 1/2] Revert "vrf: fix double free and memory corruption on register_netdevice failure" Ben Hutchings
2015-12-15 15:49 ` David Ahern
2015-12-17 22:43 ` Patch "Revert "vrf: fix double free and memory corruption on register_netdevice failure"" has been added to the 4.3-stable tree gregkh
2015-12-15 15:32 ` [PATCH 4.3 2/2] vrf: fix double free and memory corruption on register_netdevice failure Nikolay Aleksandrov
2015-12-15 15:50 ` David Ahern
2015-12-15 17:02 ` Ben Hutchings
2015-12-17 22:43 ` Patch "vrf: fix double free and memory corruption on register_netdevice failure" has been added to the 4.3-stable tree gregkh
2015-12-15 17:48 ` [PATCH 4.3] vrf: Fix memory leak on registration failure in vrf_newlink() David Miller
2015-12-12 20:05 ` [PATCH 4.3 35/71] broadcom: fix PHY_ID_BCM5481 entry in the id table Greg Kroah-Hartman
2015-12-12 20:06 ` [PATCH 4.3 36/71] tipc: fix error handling of expanding buffer headroom Greg Kroah-Hartman
2015-12-14 17:46 ` Ben Hutchings
2015-12-14 23:52 ` Greg Kroah-Hartman
2015-12-12 20:06 ` [PATCH 4.3 37/71] ipv6: distinguish frag queues by device for multicast and link-local packets Greg Kroah-Hartman
2015-12-12 20:06 ` [PATCH 4.3 38/71] RDS: fix race condition when sending a message on unbound socket Greg Kroah-Hartman
2015-12-12 20:06 ` [PATCH 4.3 39/71] bpf, array: fix heap out-of-bounds access when updating elements Greg Kroah-Hartman
2015-12-12 20:06 ` [PATCH 4.3 40/71] ipv6: add complete rcu protection around np->opt Greg Kroah-Hartman
2015-12-12 20:06 ` [PATCH 4.3 41/71] net/neighbour: fix crash at dumping device-agnostic proxy entries Greg Kroah-Hartman
2015-12-12 20:06 ` [PATCH 4.3 42/71] ipv6: sctp: implement sctp_v6_destroy_sock() Greg Kroah-Hartman
2015-12-12 20:06 ` [PATCH 4.3 43/71] openvswitch: fix hangup on vxlan/gre/geneve device deletion Greg Kroah-Hartman
2015-12-12 20:06 ` [PATCH 4.3 44/71] net_sched: fix qdisc_tree_decrease_qlen() races Greg Kroah-Hartman
2015-12-12 20:06 ` [PATCH 4.3 45/71] btrfs: fix resending received snapshot with parent Greg Kroah-Hartman
2015-12-12 20:06 ` [PATCH 4.3 46/71] btrfs: check unsupported filters in balance arguments Greg Kroah-Hartman
2015-12-12 20:06 ` [PATCH 4.3 47/71] Btrfs: fix file corruption and data loss after cloning inline extents Greg Kroah-Hartman
2015-12-12 20:06 ` Greg Kroah-Hartman [this message]
2015-12-12 20:06 ` [PATCH 4.3 50/71] Btrfs: fix race leading to incorrect item deletion when dropping extents Greg Kroah-Hartman
2015-12-12 20:06 ` [PATCH 4.3 51/71] Btrfs: fix race leading to BUG_ON when running delalloc for nodatacow Greg Kroah-Hartman
2015-12-12 20:06 ` [PATCH 4.3 52/71] Btrfs: fix race when listing an inodes xattrs Greg Kroah-Hartman
2015-12-12 20:06 ` [PATCH 4.3 53/71] btrfs: fix signed overflows in btrfs_sync_file Greg Kroah-Hartman
2015-12-12 20:06 ` [PATCH 4.3 54/71] rbd: dont put snap_context twice in rbd_queue_workfn() Greg Kroah-Hartman
2015-12-12 20:06 ` [PATCH 4.3 55/71] ext4 crypto: fix memory leak in ext4_bio_write_page() Greg Kroah-Hartman
2015-12-12 20:06 ` [PATCH 4.3 56/71] ext4 crypto: fix bugs in ext4_encrypted_zeroout() Greg Kroah-Hartman
2015-12-12 20:06 ` [PATCH 4.3 57/71] ext4: fix potential use after free in __ext4_journal_stop Greg Kroah-Hartman
2015-12-12 20:06 ` [PATCH 4.3 58/71] ext4, jbd2: ensure entering into panic after recording an error in superblock Greg Kroah-Hartman
2015-12-12 20:06 ` [PATCH 4.3 59/71] firewire: ohci: fix JMicron JMB38x IT context discovery Greg Kroah-Hartman
2015-12-12 20:06 ` [PATCH 4.3 60/71] nfsd: serialize state seqid morphing operations Greg Kroah-Hartman
2015-12-12 20:06 ` [PATCH 4.3 61/71] nfsd: eliminate sending duplicate and repeated delegations Greg Kroah-Hartman
2015-12-12 20:06 ` [PATCH 4.3 62/71] debugfs: fix refcount imbalance in start_creating Greg Kroah-Hartman
2015-12-12 20:06 ` [PATCH 4.3 63/71] nfs4: start callback_ident at idr 1 Greg Kroah-Hartman
2015-12-12 20:06 ` [PATCH 4.3 64/71] nfs4: resend LAYOUTGET when there is a race that changes the seqid Greg Kroah-Hartman
2015-12-12 20:06 ` [PATCH 4.3 65/71] nfs: if we have no valid attrs, then dont declare the attribute cache valid Greg Kroah-Hartman
2015-12-12 20:06 ` [PATCH 4.3 66/71] ocfs2: fix umask ignored issue Greg Kroah-Hartman
2015-12-12 20:06 ` [PATCH 4.3 67/71] block: fix segment split Greg Kroah-Hartman
2015-12-12 20:06 ` [PATCH 4.3 68/71] ceph: fix message length computation Greg Kroah-Hartman
2015-12-12 20:06 ` [PATCH 4.3 69/71] ALSA: pci: depend on ZONE_DMA Greg Kroah-Hartman
2015-12-12 20:06 ` [PATCH 4.3 70/71] ALSA: hda/hdmi - apply Skylake fix-ups to Broxton display codec Greg Kroah-Hartman
2015-12-12 20:06 ` [PATCH 4.3 71/71] [media] cobalt: fix Kconfig dependency Greg Kroah-Hartman
2015-12-13 3:05 ` [PATCH 4.3 00/71] 4.3.3-stable review Shuah Khan
2015-12-13 3:46 ` Greg Kroah-Hartman
2015-12-13 16:01 ` Guenter Roeck
2015-12-14 3:28 ` Greg Kroah-Hartman
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20151212200539.129607533@linuxfoundation.org \
--to=gregkh@linuxfoundation.org \
--cc=fdmanana@suse.com \
--cc=linux-kernel@vger.kernel.org \
--cc=stable@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).