From: Mike Snitzer <snitzer@redhat.com>
To: Sarthak Kukreti <sarthakkukreti@chromium.org>
Cc: Jens Axboe <axboe@kernel.dk>,
Gwendal Grignou <gwendal@google.com>,
Theodore Ts'o <tytso@mit.edu>,
"Michael S . Tsirkin" <mst@redhat.com>,
Jason Wang <jasowang@redhat.com>,
Bart Van Assche <bvanassche@google.com>,
Mike Snitzer <snitzer@kernel.org>,
linux-kernel@vger.kernel.org,
virtualization@lists.linux-foundation.org,
linux-block@vger.kernel.org, dm-devel@redhat.com,
Andreas Dilger <adilger.kernel@dilger.ca>,
Daniil Lunev <dlunev@google.com>,
Stefan Hajnoczi <stefanha@redhat.com>,
Paolo Bonzini <pbonzini@redhat.com>,
linux-ext4@vger.kernel.org, Evan Green <evgreen@google.com>,
Alasdair Kergon <agk@redhat.com>
Subject: Re: [dm-devel] [PATCH RFC 2/8] dm: Add support for block provisioning
Date: Fri, 23 Sep 2022 10:23:25 -0400 [thread overview]
Message-ID: <Yy3BXc9wf4PH6Rby@redhat.com> (raw)
In-Reply-To: <20220915164826.1396245-3-sarthakkukreti@google.com>
On Thu, Sep 15 2022 at 12:48P -0400,
Sarthak Kukreti <sarthakkukreti@chromium.org> wrote:
> From: Sarthak Kukreti <sarthakkukreti@chromium.org>
>
> Add support to dm devices for REQ_OP_PROVISION. The default mode
> is to pass through the request and dm-thin will utilize it to provision
> blocks.
>
> Signed-off-by: Sarthak Kukreti <sarthakkukreti@chromium.org>
> ---
> drivers/md/dm-crypt.c | 4 +-
> drivers/md/dm-linear.c | 1 +
> drivers/md/dm-table.c | 17 +++++++
> drivers/md/dm-thin.c | 86 +++++++++++++++++++++++++++++++++--
> drivers/md/dm.c | 4 ++
> include/linux/device-mapper.h | 6 +++
> 6 files changed, 113 insertions(+), 5 deletions(-)
>
> diff --git a/drivers/md/dm-crypt.c b/drivers/md/dm-crypt.c
> index 159c6806c19b..357f0899cfb6 100644
> --- a/drivers/md/dm-crypt.c
> +++ b/drivers/md/dm-crypt.c
> @@ -3081,6 +3081,8 @@ static int crypt_ctr_optional(struct dm_target *ti, unsigned int argc, char **ar
> if (ret)
> return ret;
>
> + ti->num_provision_bios = 1;
> +
> while (opt_params--) {
> opt_string = dm_shift_arg(&as);
> if (!opt_string) {
> @@ -3384,7 +3386,7 @@ static int crypt_map(struct dm_target *ti, struct bio *bio)
> * - for REQ_OP_DISCARD caller must use flush if IO ordering matters
> */
> if (unlikely(bio->bi_opf & REQ_PREFLUSH ||
> - bio_op(bio) == REQ_OP_DISCARD)) {
> + bio_op(bio) == REQ_OP_DISCARD || bio_op(bio) == REQ_OP_PROVISION)) {
> bio_set_dev(bio, cc->dev->bdev);
> if (bio_sectors(bio))
> bio->bi_iter.bi_sector = cc->start +
> diff --git a/drivers/md/dm-linear.c b/drivers/md/dm-linear.c
> index 3212ef6aa81b..1aa782149428 100644
> --- a/drivers/md/dm-linear.c
> +++ b/drivers/md/dm-linear.c
> @@ -61,6 +61,7 @@ static int linear_ctr(struct dm_target *ti, unsigned int argc, char **argv)
> ti->num_discard_bios = 1;
> ti->num_secure_erase_bios = 1;
> ti->num_write_zeroes_bios = 1;
> + ti->num_provision_bios = 1;
> ti->private = lc;
> return 0;
>
> diff --git a/drivers/md/dm-table.c b/drivers/md/dm-table.c
> index 332f96b58252..b7f9cb66b7ba 100644
> --- a/drivers/md/dm-table.c
> +++ b/drivers/md/dm-table.c
> @@ -1853,6 +1853,18 @@ static bool dm_table_supports_write_zeroes(struct dm_table *t)
> return true;
> }
>
> +static bool dm_table_supports_provision(struct dm_table *t)
> +{
> + for (unsigned int i = 0; i < t->num_targets; i++) {
> + struct dm_target *ti = dm_table_get_target(t, i);
> +
> + if (ti->num_provision_bios)
> + return true;
> + }
> +
> + return false;
> +}
> +
This needs to go a step further and verify a device in the stack
actually services REQ_OP_PROVISION.
Please see dm_table_supports_discards(): it iterates all devices in
the table and checks that support is advertised.
For discard, DM requires that _all_ devices in a table advertise
support (that is pretty strict and likely could be relaxed to _any_).
You'll need ti->provision_supported (like ->discards_supported) to
advertise actual support is provided by dm-thinp (even if underlying
devices don't support it).
And yeah, dm-thinp passdown support for REQ_OP_PROVISION can follow
later as needed (if there actual HW that would benefit from
REQ_OP_PROVISION).
Mike
--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel
WARNING: multiple messages have this Message-ID (diff)
From: Mike Snitzer <snitzer@redhat.com>
To: Sarthak Kukreti <sarthakkukreti@chromium.org>
Cc: dm-devel@redhat.com, linux-block@vger.kernel.org,
linux-ext4@vger.kernel.org, linux-kernel@vger.kernel.org,
virtualization@lists.linux-foundation.org,
Jens Axboe <axboe@kernel.dk>,
"Michael S . Tsirkin" <mst@redhat.com>,
Jason Wang <jasowang@redhat.com>,
Paolo Bonzini <pbonzini@redhat.com>,
Stefan Hajnoczi <stefanha@redhat.com>,
Alasdair Kergon <agk@redhat.com>,
Mike Snitzer <snitzer@kernel.org>, Theodore Ts'o <tytso@mit.edu>,
Andreas Dilger <adilger.kernel@dilger.ca>,
Bart Van Assche <bvanassche@google.com>,
Daniil Lunev <dlunev@google.com>, Evan Green <evgreen@google.com>,
Gwendal Grignou <gwendal@google.com>
Subject: Re: [PATCH RFC 2/8] dm: Add support for block provisioning
Date: Fri, 23 Sep 2022 10:23:25 -0400 [thread overview]
Message-ID: <Yy3BXc9wf4PH6Rby@redhat.com> (raw)
In-Reply-To: <20220915164826.1396245-3-sarthakkukreti@google.com>
On Thu, Sep 15 2022 at 12:48P -0400,
Sarthak Kukreti <sarthakkukreti@chromium.org> wrote:
> From: Sarthak Kukreti <sarthakkukreti@chromium.org>
>
> Add support to dm devices for REQ_OP_PROVISION. The default mode
> is to pass through the request and dm-thin will utilize it to provision
> blocks.
>
> Signed-off-by: Sarthak Kukreti <sarthakkukreti@chromium.org>
> ---
> drivers/md/dm-crypt.c | 4 +-
> drivers/md/dm-linear.c | 1 +
> drivers/md/dm-table.c | 17 +++++++
> drivers/md/dm-thin.c | 86 +++++++++++++++++++++++++++++++++--
> drivers/md/dm.c | 4 ++
> include/linux/device-mapper.h | 6 +++
> 6 files changed, 113 insertions(+), 5 deletions(-)
>
> diff --git a/drivers/md/dm-crypt.c b/drivers/md/dm-crypt.c
> index 159c6806c19b..357f0899cfb6 100644
> --- a/drivers/md/dm-crypt.c
> +++ b/drivers/md/dm-crypt.c
> @@ -3081,6 +3081,8 @@ static int crypt_ctr_optional(struct dm_target *ti, unsigned int argc, char **ar
> if (ret)
> return ret;
>
> + ti->num_provision_bios = 1;
> +
> while (opt_params--) {
> opt_string = dm_shift_arg(&as);
> if (!opt_string) {
> @@ -3384,7 +3386,7 @@ static int crypt_map(struct dm_target *ti, struct bio *bio)
> * - for REQ_OP_DISCARD caller must use flush if IO ordering matters
> */
> if (unlikely(bio->bi_opf & REQ_PREFLUSH ||
> - bio_op(bio) == REQ_OP_DISCARD)) {
> + bio_op(bio) == REQ_OP_DISCARD || bio_op(bio) == REQ_OP_PROVISION)) {
> bio_set_dev(bio, cc->dev->bdev);
> if (bio_sectors(bio))
> bio->bi_iter.bi_sector = cc->start +
> diff --git a/drivers/md/dm-linear.c b/drivers/md/dm-linear.c
> index 3212ef6aa81b..1aa782149428 100644
> --- a/drivers/md/dm-linear.c
> +++ b/drivers/md/dm-linear.c
> @@ -61,6 +61,7 @@ static int linear_ctr(struct dm_target *ti, unsigned int argc, char **argv)
> ti->num_discard_bios = 1;
> ti->num_secure_erase_bios = 1;
> ti->num_write_zeroes_bios = 1;
> + ti->num_provision_bios = 1;
> ti->private = lc;
> return 0;
>
> diff --git a/drivers/md/dm-table.c b/drivers/md/dm-table.c
> index 332f96b58252..b7f9cb66b7ba 100644
> --- a/drivers/md/dm-table.c
> +++ b/drivers/md/dm-table.c
> @@ -1853,6 +1853,18 @@ static bool dm_table_supports_write_zeroes(struct dm_table *t)
> return true;
> }
>
> +static bool dm_table_supports_provision(struct dm_table *t)
> +{
> + for (unsigned int i = 0; i < t->num_targets; i++) {
> + struct dm_target *ti = dm_table_get_target(t, i);
> +
> + if (ti->num_provision_bios)
> + return true;
> + }
> +
> + return false;
> +}
> +
This needs to go a step further and verify a device in the stack
actually services REQ_OP_PROVISION.
Please see dm_table_supports_discards(): it iterates all devices in
the table and checks that support is advertised.
For discard, DM requires that _all_ devices in a table advertise
support (that is pretty strict and likely could be relaxed to _any_).
You'll need ti->provision_supported (like ->discards_supported) to
advertise actual support is provided by dm-thinp (even if underlying
devices don't support it).
And yeah, dm-thinp passdown support for REQ_OP_PROVISION can follow
later as needed (if there actual HW that would benefit from
REQ_OP_PROVISION).
Mike
WARNING: multiple messages have this Message-ID (diff)
From: Mike Snitzer <snitzer@redhat.com>
To: Sarthak Kukreti <sarthakkukreti@chromium.org>
Cc: Jens Axboe <axboe@kernel.dk>,
Gwendal Grignou <gwendal@google.com>,
Theodore Ts'o <tytso@mit.edu>,
"Michael S . Tsirkin" <mst@redhat.com>,
Bart Van Assche <bvanassche@google.com>,
Mike Snitzer <snitzer@kernel.org>,
linux-kernel@vger.kernel.org,
virtualization@lists.linux-foundation.org,
linux-block@vger.kernel.org, dm-devel@redhat.com,
Andreas Dilger <adilger.kernel@dilger.ca>,
Daniil Lunev <dlunev@google.com>,
Stefan Hajnoczi <stefanha@redhat.com>,
Paolo Bonzini <pbonzini@redhat.com>,
linux-ext4@vger.kernel.org, Evan Green <evgreen@google.com>,
Alasdair Kergon <agk@redhat.com>
Subject: Re: [PATCH RFC 2/8] dm: Add support for block provisioning
Date: Fri, 23 Sep 2022 10:23:25 -0400 [thread overview]
Message-ID: <Yy3BXc9wf4PH6Rby@redhat.com> (raw)
In-Reply-To: <20220915164826.1396245-3-sarthakkukreti@google.com>
On Thu, Sep 15 2022 at 12:48P -0400,
Sarthak Kukreti <sarthakkukreti@chromium.org> wrote:
> From: Sarthak Kukreti <sarthakkukreti@chromium.org>
>
> Add support to dm devices for REQ_OP_PROVISION. The default mode
> is to pass through the request and dm-thin will utilize it to provision
> blocks.
>
> Signed-off-by: Sarthak Kukreti <sarthakkukreti@chromium.org>
> ---
> drivers/md/dm-crypt.c | 4 +-
> drivers/md/dm-linear.c | 1 +
> drivers/md/dm-table.c | 17 +++++++
> drivers/md/dm-thin.c | 86 +++++++++++++++++++++++++++++++++--
> drivers/md/dm.c | 4 ++
> include/linux/device-mapper.h | 6 +++
> 6 files changed, 113 insertions(+), 5 deletions(-)
>
> diff --git a/drivers/md/dm-crypt.c b/drivers/md/dm-crypt.c
> index 159c6806c19b..357f0899cfb6 100644
> --- a/drivers/md/dm-crypt.c
> +++ b/drivers/md/dm-crypt.c
> @@ -3081,6 +3081,8 @@ static int crypt_ctr_optional(struct dm_target *ti, unsigned int argc, char **ar
> if (ret)
> return ret;
>
> + ti->num_provision_bios = 1;
> +
> while (opt_params--) {
> opt_string = dm_shift_arg(&as);
> if (!opt_string) {
> @@ -3384,7 +3386,7 @@ static int crypt_map(struct dm_target *ti, struct bio *bio)
> * - for REQ_OP_DISCARD caller must use flush if IO ordering matters
> */
> if (unlikely(bio->bi_opf & REQ_PREFLUSH ||
> - bio_op(bio) == REQ_OP_DISCARD)) {
> + bio_op(bio) == REQ_OP_DISCARD || bio_op(bio) == REQ_OP_PROVISION)) {
> bio_set_dev(bio, cc->dev->bdev);
> if (bio_sectors(bio))
> bio->bi_iter.bi_sector = cc->start +
> diff --git a/drivers/md/dm-linear.c b/drivers/md/dm-linear.c
> index 3212ef6aa81b..1aa782149428 100644
> --- a/drivers/md/dm-linear.c
> +++ b/drivers/md/dm-linear.c
> @@ -61,6 +61,7 @@ static int linear_ctr(struct dm_target *ti, unsigned int argc, char **argv)
> ti->num_discard_bios = 1;
> ti->num_secure_erase_bios = 1;
> ti->num_write_zeroes_bios = 1;
> + ti->num_provision_bios = 1;
> ti->private = lc;
> return 0;
>
> diff --git a/drivers/md/dm-table.c b/drivers/md/dm-table.c
> index 332f96b58252..b7f9cb66b7ba 100644
> --- a/drivers/md/dm-table.c
> +++ b/drivers/md/dm-table.c
> @@ -1853,6 +1853,18 @@ static bool dm_table_supports_write_zeroes(struct dm_table *t)
> return true;
> }
>
> +static bool dm_table_supports_provision(struct dm_table *t)
> +{
> + for (unsigned int i = 0; i < t->num_targets; i++) {
> + struct dm_target *ti = dm_table_get_target(t, i);
> +
> + if (ti->num_provision_bios)
> + return true;
> + }
> +
> + return false;
> +}
> +
This needs to go a step further and verify a device in the stack
actually services REQ_OP_PROVISION.
Please see dm_table_supports_discards(): it iterates all devices in
the table and checks that support is advertised.
For discard, DM requires that _all_ devices in a table advertise
support (that is pretty strict and likely could be relaxed to _any_).
You'll need ti->provision_supported (like ->discards_supported) to
advertise actual support is provided by dm-thinp (even if underlying
devices don't support it).
And yeah, dm-thinp passdown support for REQ_OP_PROVISION can follow
later as needed (if there actual HW that would benefit from
REQ_OP_PROVISION).
Mike
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
next prev parent reply other threads:[~2022-09-23 14:23 UTC|newest]
Thread overview: 102+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-09-15 16:48 [dm-devel] [PATCH RFC 0/8] Introduce provisioning primitives for thinly provisioned storage Sarthak Kukreti
2022-09-15 16:48 ` Sarthak Kukreti
2022-09-15 16:48 ` [dm-devel] [PATCH RFC 1/8] block: Introduce provisioning primitives Sarthak Kukreti
2022-09-15 16:48 ` Sarthak Kukreti
2022-09-23 15:15 ` [dm-devel] " Mike Snitzer
2022-09-23 15:15 ` Mike Snitzer
2022-09-23 15:15 ` Mike Snitzer
2022-12-29 8:17 ` [dm-devel] " Sarthak Kukreti
2022-12-29 8:17 ` Sarthak Kukreti
2022-09-15 16:48 ` [dm-devel] [PATCH RFC 2/8] dm: Add support for block provisioning Sarthak Kukreti
2022-09-15 16:48 ` Sarthak Kukreti
2022-09-23 14:23 ` Mike Snitzer [this message]
2022-09-23 14:23 ` Mike Snitzer
2022-09-23 14:23 ` Mike Snitzer
2022-12-29 8:22 ` [dm-devel] " Sarthak Kukreti
2022-12-29 8:22 ` Sarthak Kukreti
2022-09-15 16:48 ` [dm-devel] [PATCH RFC 3/8] virtio_blk: Add support for provision requests Sarthak Kukreti
2022-09-15 16:48 ` Sarthak Kukreti
2022-09-16 5:48 ` [dm-devel] " Stefan Hajnoczi
2022-09-16 5:48 ` Stefan Hajnoczi
2022-09-16 5:48 ` Stefan Hajnoczi
2022-09-20 2:33 ` [dm-devel] " Sarthak Kukreti
2022-09-20 2:33 ` Sarthak Kukreti
2022-09-27 21:37 ` [dm-devel] " Michael S. Tsirkin
2022-09-27 21:37 ` Michael S. Tsirkin
2022-09-27 21:37 ` Michael S. Tsirkin
2022-09-15 16:48 ` [dm-devel] [PATCH RFC 4/8] fs: Introduce FALLOC_FL_PROVISION Sarthak Kukreti
2022-09-15 16:48 ` Sarthak Kukreti
2022-09-16 11:56 ` [dm-devel] " Brian Foster
2022-09-16 11:56 ` Brian Foster
2022-09-16 11:56 ` Brian Foster
2022-09-16 21:02 ` [dm-devel] " Sarthak Kukreti
2022-09-16 21:02 ` Sarthak Kukreti
2022-09-21 15:39 ` [dm-devel] " Brian Foster
2022-09-21 15:39 ` Brian Foster
2022-09-21 15:39 ` Brian Foster
2022-09-22 8:04 ` [dm-devel] " Sarthak Kukreti
2022-09-22 8:04 ` Sarthak Kukreti
2022-09-22 18:29 ` [dm-devel] " Brian Foster
2022-09-22 18:29 ` Brian Foster
2022-09-22 18:29 ` Brian Foster
2022-12-29 8:13 ` [dm-devel] " Sarthak Kukreti
2022-12-29 8:13 ` Sarthak Kukreti
2022-09-20 7:49 ` [dm-devel] " Christoph Hellwig
2022-09-20 7:49 ` Christoph Hellwig
2022-09-20 7:49 ` Christoph Hellwig
2022-09-21 5:54 ` [dm-devel] " Sarthak Kukreti
2022-09-21 5:54 ` Sarthak Kukreti
2022-09-21 15:21 ` [dm-devel] " Mike Snitzer
2022-09-21 15:21 ` Mike Snitzer
2022-09-21 15:21 ` Mike Snitzer
2022-09-22 8:08 ` [dm-devel] " Sarthak Kukreti
2022-09-22 8:08 ` Sarthak Kukreti
2022-09-23 8:45 ` [dm-devel] " Christoph Hellwig
2022-09-23 8:45 ` Christoph Hellwig
2022-09-23 8:45 ` Christoph Hellwig
2022-12-29 8:14 ` [dm-devel] " Sarthak Kukreti
2022-12-29 8:14 ` Sarthak Kukreti
2022-09-15 16:48 ` [dm-devel] [PATCH RFC 5/8] loop: Add support for provision requests Sarthak Kukreti
2022-09-15 16:48 ` Sarthak Kukreti
2022-09-15 16:48 ` [dm-devel] [PATCH RFC 6/8] ext4: Add support for FALLOC_FL_PROVISION Sarthak Kukreti
2022-09-15 16:48 ` Sarthak Kukreti
2022-09-15 16:48 ` [dm-devel] [PATCH RFC 7/8] ext4: Add mount option for provisioning blocks during allocations Sarthak Kukreti
2022-09-15 16:48 ` Sarthak Kukreti
2022-09-15 16:48 ` [dm-devel] [PATCH RFC 8/8] ext4: Add a per-file provision override xattr Sarthak Kukreti
2022-09-15 16:48 ` Sarthak Kukreti
2022-09-16 6:09 ` [dm-devel] [PATCH RFC 0/8] Introduce provisioning primitives for thinly provisioned storage Stefan Hajnoczi
2022-09-16 6:09 ` Stefan Hajnoczi
2022-09-16 6:09 ` Stefan Hajnoczi
2022-09-16 18:48 ` [dm-devel] " Sarthak Kukreti
2022-09-16 18:48 ` Sarthak Kukreti
2022-09-16 20:01 ` [dm-devel] " Bart Van Assche
2022-09-16 20:01 ` Bart Van Assche
2022-09-16 20:01 ` Bart Van Assche
2022-09-16 21:59 ` [dm-devel] " Sarthak Kukreti
2022-09-16 21:59 ` Sarthak Kukreti
2022-09-20 7:46 ` [dm-devel] " Christoph Hellwig
2022-09-20 7:46 ` Christoph Hellwig
2022-09-20 7:46 ` Christoph Hellwig
2022-09-20 10:17 ` [dm-devel] " Daniil Lunev
2022-09-20 11:30 ` Christoph Hellwig
2022-09-20 11:30 ` Christoph Hellwig
2022-09-20 11:30 ` Christoph Hellwig
2022-09-20 21:48 ` [dm-devel] " Daniil Lunev
2022-09-21 15:08 ` Mike Snitzer
2022-09-21 15:08 ` Mike Snitzer
2022-09-21 15:08 ` Mike Snitzer
2022-09-23 8:51 ` [dm-devel] " Christoph Hellwig
2022-09-23 8:51 ` Christoph Hellwig
2022-09-23 8:51 ` Christoph Hellwig
2022-09-23 14:08 ` [dm-devel] " Mike Snitzer
2022-09-23 14:08 ` Mike Snitzer
2022-09-23 14:08 ` Mike Snitzer
2022-12-29 8:17 ` [dm-devel] " Sarthak Kukreti
2022-12-29 8:17 ` Sarthak Kukreti
2022-09-17 3:03 ` [dm-devel] " Darrick J. Wong
2022-09-17 3:03 ` Darrick J. Wong
2022-09-17 19:46 ` Sarthak Kukreti
2022-09-17 19:46 ` Sarthak Kukreti
2022-09-19 16:36 ` Stefan Hajnoczi
2022-09-19 16:36 ` Stefan Hajnoczi
2022-09-19 16:36 ` Stefan Hajnoczi
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=Yy3BXc9wf4PH6Rby@redhat.com \
--to=snitzer@redhat.com \
--cc=adilger.kernel@dilger.ca \
--cc=agk@redhat.com \
--cc=axboe@kernel.dk \
--cc=bvanassche@google.com \
--cc=dlunev@google.com \
--cc=dm-devel@redhat.com \
--cc=evgreen@google.com \
--cc=gwendal@google.com \
--cc=jasowang@redhat.com \
--cc=linux-block@vger.kernel.org \
--cc=linux-ext4@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mst@redhat.com \
--cc=pbonzini@redhat.com \
--cc=sarthakkukreti@chromium.org \
--cc=snitzer@kernel.org \
--cc=stefanha@redhat.com \
--cc=tytso@mit.edu \
--cc=virtualization@lists.linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.