From: Kevin Wolf <kwolf@redhat.com>
To: qemu-block@nongnu.org
Cc: kwolf@redhat.com, qemu-devel@nongnu.org
Subject: [Qemu-devel] [PULL 49/76] docs: incremental backup documentation
Date: Tue, 28 Apr 2015 17:00:31 +0200 [thread overview]
Message-ID: <1430233258-31807-50-git-send-email-kwolf@redhat.com> (raw)
In-Reply-To: <1430233258-31807-1-git-send-email-kwolf@redhat.com>
From: John Snow <jsnow@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 1429314609-29776-2-git-send-email-jsnow@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
docs/bitmaps.md | 352 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 352 insertions(+)
create mode 100644 docs/bitmaps.md
diff --git a/docs/bitmaps.md b/docs/bitmaps.md
new file mode 100644
index 0000000..f066b48
--- /dev/null
+++ b/docs/bitmaps.md
@@ -0,0 +1,352 @@
+<!--
+Copyright 2015 John Snow <jsnow@redhat.com> and Red Hat, Inc.
+All rights reserved.
+
+This file is licensed via The FreeBSD Documentation License, the full text of
+which is included at the end of this document.
+-->
+
+# Dirty Bitmaps and Incremental Backup
+
+* Dirty Bitmaps are objects that track which data needs to be backed up for the
+ next incremental backup.
+
+* Dirty bitmaps can be created at any time and attached to any node
+ (not just complete drives.)
+
+## Dirty Bitmap Names
+
+* A dirty bitmap's name is unique to the node, but bitmaps attached to different
+ nodes can share the same name.
+
+## Bitmap Modes
+
+* A Bitmap can be "frozen," which means that it is currently in-use by a backup
+ operation and cannot be deleted, renamed, written to, reset,
+ etc.
+
+## Basic QMP Usage
+
+### Supported Commands ###
+
+* block-dirty-bitmap-add
+* block-dirty-bitmap-remove
+* block-dirty-bitmap-clear
+
+### Creation
+
+* To create a new bitmap, enabled, on the drive with id=drive0:
+
+```json
+{ "execute": "block-dirty-bitmap-add",
+ "arguments": {
+ "node": "drive0",
+ "name": "bitmap0"
+ }
+}
+```
+
+* This bitmap will have a default granularity that matches the cluster size of
+ its associated drive, if available, clamped to between [4KiB, 64KiB].
+ The current default for qcow2 is 64KiB.
+
+* To create a new bitmap that tracks changes in 32KiB segments:
+
+```json
+{ "execute": "block-dirty-bitmap-add",
+ "arguments": {
+ "node": "drive0",
+ "name": "bitmap0",
+ "granularity": 32768
+ }
+}
+```
+
+### Deletion
+
+* Bitmaps that are frozen cannot be deleted.
+
+* Deleting the bitmap does not impact any other bitmaps attached to the same
+ node, nor does it affect any backups already created from this node.
+
+* Because bitmaps are only unique to the node to which they are attached,
+ you must specify the node/drive name here, too.
+
+```json
+{ "execute": "block-dirty-bitmap-remove",
+ "arguments": {
+ "node": "drive0",
+ "name": "bitmap0"
+ }
+}
+```
+
+### Resetting
+
+* Resetting a bitmap will clear all information it holds.
+
+* An incremental backup created from an empty bitmap will copy no data,
+ as if nothing has changed.
+
+```json
+{ "execute": "block-dirty-bitmap-clear",
+ "arguments": {
+ "node": "drive0",
+ "name": "bitmap0"
+ }
+}
+```
+
+## Transactions (Not yet implemented)
+
+* Transactional commands are forthcoming in a future version,
+ and are not yet available for use. This section serves as
+ documentation of intent for their design and usage.
+
+### Justification
+
+Bitmaps can be safely modified when the VM is paused or halted by using
+the basic QMP commands. For instance, you might perform the following actions:
+
+1. Boot the VM in a paused state.
+2. Create a full drive backup of drive0.
+3. Create a new bitmap attached to drive0.
+4. Resume execution of the VM.
+5. Incremental backups are ready to be created.
+
+At this point, the bitmap and drive backup would be correctly in sync,
+and incremental backups made from this point forward would be correctly aligned
+to the full drive backup.
+
+This is not particularly useful if we decide we want to start incremental
+backups after the VM has been running for a while, for which we will need to
+perform actions such as the following:
+
+1. Boot the VM and begin execution.
+2. Using a single transaction, perform the following operations:
+ * Create bitmap0.
+ * Create a full drive backup of drive0.
+3. Incremental backups are now ready to be created.
+
+### Supported Bitmap Transactions
+
+* block-dirty-bitmap-add
+* block-dirty-bitmap-clear
+
+The usages are identical to their respective QMP commands, but see below
+for examples.
+
+### Example: New Incremental Backup
+
+As outlined in the justification, perhaps we want to create a new incremental
+backup chain attached to a drive.
+
+```json
+{ "execute": "transaction",
+ "arguments": {
+ "actions": [
+ {"type": "block-dirty-bitmap-add",
+ "data": {"node": "drive0", "name": "bitmap0"} },
+ {"type": "drive-backup",
+ "data": {"device": "drive0", "target": "/path/to/full_backup.img",
+ "sync": "full", "format": "qcow2"} }
+ ]
+ }
+}
+```
+
+### Example: New Incremental Backup Anchor Point
+
+Maybe we just want to create a new full backup with an existing bitmap and
+want to reset the bitmap to track the new chain.
+
+```json
+{ "execute": "transaction",
+ "arguments": {
+ "actions": [
+ {"type": "block-dirty-bitmap-clear",
+ "data": {"node": "drive0", "name": "bitmap0"} },
+ {"type": "drive-backup",
+ "data": {"device": "drive0", "target": "/path/to/new_full_backup.img",
+ "sync": "full", "format": "qcow2"} }
+ ]
+ }
+}
+```
+
+## Incremental Backups
+
+The star of the show.
+
+**Nota Bene!** Only incremental backups of entire drives are supported for now.
+So despite the fact that you can attach a bitmap to any arbitrary node, they are
+only currently useful when attached to the root node. This is because
+drive-backup only supports drives/devices instead of arbitrary nodes.
+
+### Example: First Incremental Backup
+
+1. Create a full backup and sync it to the dirty bitmap, as in the transactional
+examples above; or with the VM offline, manually create a full copy and then
+create a new bitmap before the VM begins execution.
+
+ * Let's assume the full backup is named 'full_backup.img'.
+ * Let's assume the bitmap you created is 'bitmap0' attached to 'drive0'.
+
+2. Create a destination image for the incremental backup that utilizes the
+full backup as a backing image.
+
+ * Let's assume it is named 'incremental.0.img'.
+
+ ```sh
+ # qemu-img create -f qcow2 incremental.0.img -b full_backup.img -F qcow2
+ ```
+
+3. Issue the incremental backup command:
+
+ ```json
+ { "execute": "drive-backup",
+ "arguments": {
+ "device": "drive0",
+ "bitmap": "bitmap0",
+ "target": "incremental.0.img",
+ "format": "qcow2",
+ "sync": "dirty-bitmap",
+ "mode": "existing"
+ }
+ }
+ ```
+
+### Example: Second Incremental Backup
+
+1. Create a new destination image for the incremental backup that points to the
+ previous one, e.g.: 'incremental.1.img'
+
+ ```sh
+ # qemu-img create -f qcow2 incremental.1.img -b incremental.0.img -F qcow2
+ ```
+
+2. Issue a new incremental backup command. The only difference here is that we
+ have changed the target image below.
+
+ ```json
+ { "execute": "drive-backup",
+ "arguments": {
+ "device": "drive0",
+ "bitmap": "bitmap0",
+ "target": "incremental.1.img",
+ "format": "qcow2",
+ "sync": "dirty-bitmap",
+ "mode": "existing"
+ }
+ }
+ ```
+
+## Errors
+
+* In the event of an error that occurs after a backup job is successfully
+ launched, either by a direct QMP command or a QMP transaction, the user
+ will receive a BLOCK_JOB_COMPLETE event with a failure message, accompanied
+ by a BLOCK_JOB_ERROR event.
+
+* In the case of an event being cancelled, the user will receive a
+ BLOCK_JOB_CANCELLED event instead of a pair of COMPLETE and ERROR events.
+
+* In either case, the incremental backup data contained within the bitmap is
+ safely rolled back, and the data within the bitmap is not lost. The image
+ file created for the failed attempt can be safely deleted.
+
+* Once the underlying problem is fixed (e.g. more storage space is freed up),
+ you can simply retry the incremental backup command with the same bitmap.
+
+### Example
+
+1. Create a target image:
+
+ ```sh
+ # qemu-img create -f qcow2 incremental.0.img -b full_backup.img -F qcow2
+ ```
+
+2. Attempt to create an incremental backup via QMP:
+
+ ```json
+ { "execute": "drive-backup",
+ "arguments": {
+ "device": "drive0",
+ "bitmap": "bitmap0",
+ "target": "incremental.0.img",
+ "format": "qcow2",
+ "sync": "dirty-bitmap",
+ "mode": "existing"
+ }
+ }
+ ```
+
+3. Receive an event notifying us of failure:
+
+ ```json
+ { "timestamp": { "seconds": 1424709442, "microseconds": 844524 },
+ "data": { "speed": 0, "offset": 0, "len": 67108864,
+ "error": "No space left on device",
+ "device": "drive1", "type": "backup" },
+ "event": "BLOCK_JOB_COMPLETED" }
+ ```
+
+4. Delete the failed incremental, and re-create the image.
+
+ ```sh
+ # rm incremental.0.img
+ # qemu-img create -f qcow2 incremental.0.img -b full_backup.img -F qcow2
+ ```
+
+5. Retry the command after fixing the underlying problem,
+ such as freeing up space on the backup volume:
+
+ ```json
+ { "execute": "drive-backup",
+ "arguments": {
+ "device": "drive0",
+ "bitmap": "bitmap0",
+ "target": "incremental.0.img",
+ "format": "qcow2",
+ "sync": "dirty-bitmap",
+ "mode": "existing"
+ }
+ }
+ ```
+
+6. Receive confirmation that the job completed successfully:
+
+ ```json
+ { "timestamp": { "seconds": 1424709668, "microseconds": 526525 },
+ "data": { "device": "drive1", "type": "backup",
+ "speed": 0, "len": 67108864, "offset": 67108864},
+ "event": "BLOCK_JOB_COMPLETED" }
+ ```
+
+<!--
+The FreeBSD Documentation License
+
+Redistribution and use in source (Markdown) and 'compiled' forms (SGML, HTML,
+PDF, PostScript, RTF and so forth) with or without modification, are permitted
+provided that the following conditions are met:
+
+Redistributions of source code (Markdown) must retain the above copyright
+notice, this list of conditions and the following disclaimer of this file
+unmodified.
+
+Redistributions in compiled form (transformed to other DTDs, converted to PDF,
+PostScript, RTF and other formats) must reproduce the above copyright notice,
+this list of conditions and the following disclaimer in the documentation and/or
+other materials provided with the distribution.
+
+THIS DOCUMENTATION IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+THIS DOCUMENTATION, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+-->
--
1.8.3.1
next prev parent reply other threads:[~2015-04-28 15:02 UTC|newest]
Thread overview: 80+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-04-28 14:59 [Qemu-devel] [PULL 00/76] Block patches Kevin Wolf
2015-04-28 14:59 ` [Qemu-devel] [PULL 01/76] savevm: create snapshot failed when id_str already exists Kevin Wolf
2015-04-28 14:59 ` [Qemu-devel] [PULL 02/76] MAINTAINERS: Add myself as the maintainer of the Quorum driver Kevin Wolf
2015-04-28 14:59 ` [Qemu-devel] [PULL 03/76] bt-sdp: fix broken uuids power-of-2 calculation Kevin Wolf
2015-04-28 14:59 ` [Qemu-devel] [PULL 04/76] hw/arm/nseries: convert ffs(3) to ctz32() Kevin Wolf
2015-04-28 14:59 ` [Qemu-devel] [PULL 05/76] uninorth: " Kevin Wolf
2015-04-28 14:59 ` [Qemu-devel] [PULL 06/76] Convert (ffs(val) - 1) to ctz32(val) Kevin Wolf
2015-04-28 14:59 ` [Qemu-devel] [PULL 07/76] Convert ffs() != 0 callers to ctz32() Kevin Wolf
2015-04-28 14:59 ` [Qemu-devel] [PULL 08/76] sd: convert sd_normal_command() ffs(3) call " Kevin Wolf
2015-04-28 14:59 ` [Qemu-devel] [PULL 09/76] omap_intc: convert ffs(3) to ctz32() in omap_inth_sir_update() Kevin Wolf
2015-04-28 14:59 ` [Qemu-devel] [PULL 10/76] os-win32: drop ffs(3) prototype Kevin Wolf
2015-04-28 14:59 ` [Qemu-devel] [PULL 11/76] checkpatch: complain about ffs(3) calls Kevin Wolf
2015-04-28 14:59 ` [Qemu-devel] [PULL 12/76] block: Switch to host monotonic clock for IO throttling Kevin Wolf
2015-04-28 14:59 ` [Qemu-devel] [PULL 13/76] aio-posix: move pollfds to thread-local storage Kevin Wolf
2015-04-28 14:59 ` [Qemu-devel] [PULL 14/76] AioContext: acquire/release AioContext during aio_poll Kevin Wolf
2015-04-28 14:59 ` [Qemu-devel] [PULL 15/76] iothread: release iothread around aio_poll Kevin Wolf
2015-04-28 14:59 ` [Qemu-devel] [PULL 16/76] block-backend: Expose bdrv_write_zeroes() Kevin Wolf
2015-04-28 14:59 ` [Qemu-devel] [PULL 17/76] qemu-img convert: Rewrite copying logic Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 18/76] qemu-iotests: Some qemu-img convert tests Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 19/76] blkdebug: Add bdrv_truncate() Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 20/76] vhdx: Fix zero-fill iov length Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 21/76] thread-pool: clean up thread_pool_completion_bh() Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 22/76] scripts: add 'qemu coroutine' command to qemu-gdb.py Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 23/76] block/null: Latency simulation by adding new option "latency-ns" Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 24/76] block/null: Support reopen Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 25/76] MAINTAINERS: Add Fam Zheng as Null block driver maintainer Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 26/76] blockjob: Allow nested pause Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 27/76] block: Pause block jobs in bdrv_drain_all Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 28/76] qemu-iotests: Test that "stop" doesn't drain block jobs Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 29/76] blockjob: Update function name in comments Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 30/76] block: avoid unnecessary bottom halves Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 31/76] virtio_blk: comment fix Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 32/76] m25p80: add missing blk_attach_dev_nofail Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 33/76] m25p80: fix s->blk usage before assignment Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 34/76] block: document block-stream in qmp-commands.hx Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 35/76] block: add bdrv_get_device_or_node_name() Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 36/76] block: use bdrv_get_device_or_node_name() in error messages Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 37/76] block: add 'node-name' field to BLOCK_IMAGE_CORRUPTED Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 38/76] Revert "hmp: fix crash in 'info block -n -v'" Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 39/76] qmp: fill in the image field in BlockDeviceInfo Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 40/76] block/iscsi: do not forget to logout from target Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 41/76] block/iscsi: change all iscsilun properties from uint8_t to bool Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 42/76] block/iscsi: rename iscsi_write_protected and let it return void Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 43/76] block/iscsi: store DPOFUA bit from the modesense command Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 44/76] block/iscsi: optimize WRITE10/16 if cache.writeback is not set Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 45/76] block/iscsi: increase retry count Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 46/76] block/iscsi: handle SCSI_STATUS_TASK_SET_FULL Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 47/76] block/iscsi: bump year in copyright notice Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 48/76] block/iscsi: use the allocationmap also if cache.direct=on Kevin Wolf
2015-04-28 15:00 ` Kevin Wolf [this message]
2015-04-28 15:00 ` [Qemu-devel] [PULL 50/76] qapi: Add optional field "name" to block dirty bitmap Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 51/76] qmp: Ensure consistent granularity type Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 52/76] qmp: Add block-dirty-bitmap-add and block-dirty-bitmap-remove Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 53/76] block: Introduce bdrv_dirty_bitmap_granularity() Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 54/76] hbitmap: cache array lengths Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 55/76] hbitmap: add hbitmap_merge Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 56/76] block: Add bitmap disabled status Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 57/76] block: Add bitmap successors Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 58/76] qmp: Add support of "dirty-bitmap" sync mode for drive-backup Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 59/76] qmp: add block-dirty-bitmap-clear Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 60/76] qmp: Add dirty bitmap status field in query-block Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 61/76] block: add BdrvDirtyBitmap documentation Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 62/76] block: Ensure consistent bitmap function prototypes Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 63/76] block: Resize bitmaps on bdrv_truncate Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 64/76] hbitmap: truncate tests Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 65/76] iotests: add invalid input incremental backup tests Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 66/76] iotests: add QMP event waiting queue Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 67/76] iotests: add simple incremental backup case Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 68/76] iotests: add incremental backup failure recovery test Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 69/76] iotests: add incremental backup granularity tests Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 70/76] block/mirror: Always call block_job_sleep_ns() Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 71/76] block/dmg: make it modular Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 72/76] vmdk: Widen before shifting 32 bit header field Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 73/76] block: replace bdrv_states iteration with bdrv_next() Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 74/76] block: add bdrv_set_dirty()/bdrv_reset_dirty() to block_int.h Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 75/76] block: extract bdrv_setup_io_funcs() Kevin Wolf
2015-04-28 15:00 ` [Qemu-devel] [PULL 76/76] block: move I/O request processing to block/io.c Kevin Wolf
2015-04-28 17:15 ` [Qemu-devel] [Qemu-block] " Eric Blake
2015-04-29 8:27 ` Kevin Wolf
2015-04-28 17:58 ` [Qemu-devel] [PULL 00/76] Block patches Peter Maydell
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1430233258-31807-50-git-send-email-kwolf@redhat.com \
--to=kwolf@redhat.com \
--cc=qemu-block@nongnu.org \
--cc=qemu-devel@nongnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).