From: Laurence Oberman <loberman@redhat.com>
To: Bart Van Assche <bart.vanassche@sandisk.com>
Cc: Mike Snitzer <snitzer@redhat.com>,
dm-devel@redhat.com, linux-scsi@vger.kernel.org
Subject: Re: [dm-devel] dm-mq and end_clone_request()
Date: Wed, 3 Aug 2016 14:03:05 -0400 (EDT) [thread overview]
Message-ID: <1165785378.8091030.1470247385715.JavaMail.zimbra@redhat.com> (raw)
In-Reply-To: <4b015567-0fdb-dbf5-96e2-627e76b48a03@sandisk.com>
----- Original Message -----
> From: "Bart Van Assche" <bart.vanassche@sandisk.com>
> To: "Laurence Oberman" <loberman@redhat.com>, "Mike Snitzer" <snitzer@redhat.com>
> Cc: dm-devel@redhat.com, linux-scsi@vger.kernel.org
> Sent: Wednesday, August 3, 2016 12:06:17 PM
> Subject: Re: [dm-devel] dm-mq and end_clone_request()
>
> On 08/02/2016 06:33 PM, Laurence Oberman wrote:
> > #!/bin/bash
> > for i in /sys/class/srp_remote_ports/*
> > do
> > echo "Deleting host $i, it will re-connect via srp_daemon"
> > echo 1 > $i/delete
> > sleep 10
> > done
>
> Hello Laurence,
>
> Sorry but the above looks wrong to me. There should be a second loop
> around this loop and the sleep statement should be moved from the inner
> loop to the outer loop. The above code logs out one (initiator, target)
> port pair at a time instead of logging out all paths at once.
>
> Bart.
>
Hi Bart
Latest tests are still good on our side.
I am now taking both paths out at the same time but still we seem stable here.
First test removed sleep and we still had a delay, second test add a background so they ran as close as possible to the same time.
Both tests passed.
I will email messages log just to you.
With no sleep we still have a gap when we delete paths of 9s and we are good.
Aug 3 13:41:21 jumpclient multipathd: 360001ff0b035d000000000008d700001: remaining active paths: 1
Aug 3 13:41:22 jumpclient multipathd: 360001ff0b035d000000000028d720003: remaining active paths: 1
Aug 3 13:41:22 jumpclient multipathd: 360001ff0b035d000000000048d740005: remaining active paths: 1
Aug 3 13:41:22 jumpclient multipathd: 360001ff0b035d000000000068d760007: remaining active paths: 1
Aug 3 13:41:23 jumpclient multipathd: 360001ff0b035d0000000000b8d7b000c: remaining active paths: 1
Aug 3 13:41:23 jumpclient multipathd: 360001ff0b035d0000000000d8d7d000e: remaining active paths: 1
Aug 3 13:41:23 jumpclient multipathd: 360001ff0b035d000000000118d810012: remaining active paths: 1
Aug 3 13:41:24 jumpclient multipathd: 360001ff0b035d000000000138d830014: remaining active paths: 1
Aug 3 13:41:24 jumpclient multipathd: 360001ff0b035d000000000158d850016: remaining active paths: 1
Aug 3 13:41:25 jumpclient multipathd: 360001ff0b035d000000000178d870018: remaining active paths: 1
Aug 3 13:41:25 jumpclient multipathd: 360001ff0b035d000000000198d89001a: remaining active paths: 1
Aug 3 13:41:25 jumpclient multipathd: 360001ff0b035d0000000001a8d8a001b: remaining active paths: 1
Aug 3 13:41:25 jumpclient multipathd: 360001ff0b035d0000000001c8d8c001d: remaining active paths: 1
Aug 3 13:41:26 jumpclient multipathd: 360001ff0b035d0000000001e8d8e001f: remaining active paths: 1
Aug 3 13:41:26 jumpclient multipathd: 360001ff0b035d0000000001f8d8f0020: remaining active paths: 1
Aug 3 13:41:26 jumpclient multipathd: 360001ff0b035d000000000208d900021: remaining active paths: 1
Aug 3 13:41:26 jumpclient multipathd: 360001ff0b035d000000000228d920023: remaining active paths: 1
Aug 3 13:41:28 jumpclient multipathd: 360001ff0b035d000000000248d940025: remaining active paths: 1
Aug 3 13:41:29 jumpclient multipathd: 360001ff0b035d000000000268d960027: remaining active paths: 1
Aug 3 13:41:29 jumpclient multipathd: 360001ff0b035d000000000278d970028: remaining active paths: 1
Aug 3 13:41:30 jumpclient multipathd: 360001ff0b035d000000000288d980029: remaining active paths: 1
Aug 3 13:41:35 jumpclient multipathd: 360001ff0b035d000000000008d700001: remaining active paths: 0
Aug 3 13:41:36 jumpclient multipathd: 360001ff0b035d000000000028d720003: remaining active paths: 0
Aug 3 13:41:37 jumpclient multipathd: 360001ff0b035d000000000048d740005: remaining active paths: 0
Aug 3 13:41:37 jumpclient multipathd: 360001ff0b035d000000000068d760007: remaining active paths: 0
Aug 3 13:41:38 jumpclient multipathd: 360001ff0b035d0000000000b8d7b000c: remaining active paths: 0
Aug 3 13:41:38 jumpclient multipathd: 360001ff0b035d0000000000d8d7d000e: remaining active paths: 0
Aug 3 13:41:38 jumpclient multipathd: 360001ff0b035d000000000108d800011: remaining active paths: 0
Aug 3 13:41:38 jumpclient multipathd: 360001ff0b035d000000000118d810012: remaining active paths: 0
Aug 3 13:41:38 jumpclient multipathd: 360001ff0b035d000000000138d830014: remaining active paths: 0
Aug 3 13:41:39 jumpclient multipathd: 360001ff0b035d000000000158d850016: remaining active paths: 0
Aug 3 13:41:39 jumpclient multipathd: 360001ff0b035d000000000178d870018: remaining active paths: 0
Aug 3 13:41:39 jumpclient multipathd: 360001ff0b035d000000000198d89001a: remaining active paths: 0
Aug 3 13:41:39 jumpclient multipathd: 360001ff0b035d0000000001a8d8a001b: remaining active paths: 0
Aug 3 13:41:39 jumpclient multipathd: 360001ff0b035d0000000001c8d8c001d: remaining active paths: 0
Aug 3 13:41:39 jumpclient multipathd: 360001ff0b035d0000000001e8d8e001f: remaining active paths: 0
Aug 3 13:41:41 jumpclient multipathd: 360001ff0b035d0000000001f8d8f0020: remaining active paths: 0
Aug 3 13:41:41 jumpclient multipathd: 360001ff0b035d000000000208d900021: remaining active paths: 0
Aug 3 13:41:43 jumpclient multipathd: 360001ff0b035d000000000248d940025: remaining active paths: 0
Aug 3 13:41:43 jumpclient multipathd: 360001ff0b035d000000000268d960027: remaining active paths: 0
Aug 3 13:41:44 jumpclient multipathd: 360001ff0b035d000000000288d980029: remaining active paths: 0
Aug 3 13:42:44 jumpclient multipathd: 360001ff0b035d000000000138d830014: remaining active paths: 2
Aug 3 13:42:44 jumpclient multipathd: 360001ff0b035d000000000158d850016: remaining active paths: 2
These are the only errors and they are expected.
Aug 3 13:41:22 jumpclient kernel: blk_update_request: I/O error, dev sdd, sector 31141264880
Aug 3 13:41:22 jumpclient kernel: blk_update_request: I/O error, dev sdd, sector 79928
Aug 3 13:41:22 jumpclient kernel: blk_update_request: I/O error, dev sdd, sector 65264
Aug 3 13:41:22 jumpclient kernel: blk_update_request: I/O error, dev sdd, sector 55232
Aug 3 13:41:22 jumpclient kernel: blk_update_request: I/O error, dev sdd, sector 14152
Aug 3 13:41:22 jumpclient kernel: blk_update_request: I/O error, dev sdd, sector 168880
Aug 3 13:41:22 jumpclient kernel: blk_update_request: I/O error, dev sdd, sector 269392
Aug 3 13:41:22 jumpclient kernel: blk_update_request: I/O error, dev sdd, sector 309200
Aug 3 13:41:22 jumpclient kernel: blk_update_request: I/O error, dev sdd, sector 87520
Aug 3 13:41:22 jumpclient kernel: blk_update_request: I/O error, dev sdd, sector 7744
Aug 3 13:41:28 jumpclient kernel: blk_update_request: I/O error, dev sdca, sector 119984
Aug 3 13:41:29 jumpclient kernel: blk_update_request: I/O error, dev sdcd, sector 31139908984
Aug 3 13:41:29 jumpclient kernel: blk_update_request: I/O error, dev sdcd, sector 131136
Aug 3 13:41:29 jumpclient kernel: blk_update_request: I/O error, dev sdcd, sector 97536
Aug 3 13:41:29 jumpclient kernel: blk_update_request: I/O error, dev sdcd, sector 123264
Aug 3 13:41:29 jumpclient kernel: blk_update_request: I/O error, dev sdcd, sector 110336
Aug 3 13:41:29 jumpclient kernel: blk_update_request: I/O error, dev sdcd, sector 158136
Aug 3 13:41:29 jumpclient kernel: blk_update_request: I/O error, dev sdcd, sector 156136
Aug 3 13:41:29 jumpclient kernel: blk_update_request: I/O error, dev sdcd, sector 173072
Aug 3 13:41:29 jumpclient kernel: blk_update_request: I/O error, dev sdcd, sector 6984
Aug 3 13:41:35 jumpclient kernel: blk_update_request: I/O error, dev sdc, sector 130224
Aug 3 13:41:35 jumpclient kernel: blk_update_request: I/O error, dev sdc, sector 225816
Aug 3 13:41:36 jumpclient kernel: blk_update_request: I/O error, dev sdc, sector 248120
Aug 3 13:41:36 jumpclient kernel: blk_update_request: I/O error, dev sdc, sector 242528
Aug 3 13:41:36 jumpclient kernel: blk_update_request: I/O error, dev sdk, sector 251248
Aug 3 13:41:36 jumpclient kernel: blk_update_request: I/O error, dev sdk, sector 242032
Aug 3 13:41:36 jumpclient kernel: blk_update_request: I/O error, dev sdk, sector 203736
Aug 3 13:41:36 jumpclient kernel: blk_update_request: I/O error, dev sdk, sector 31141107808
Aug 3 13:41:36 jumpclient kernel: blk_update_request: I/O error, dev sdk, sector 233336
Aug 3 13:41:36 jumpclient kernel: blk_update_request: I/O error, dev sdk, sector 187944
Aug 3 13:41:41 jumpclient kernel: blk_update_request: I/O error, dev sdbl, sector 85800
Aug 3 13:41:41 jumpclient kernel: blk_update_request: I/O error, dev sdbl, sector 74120
Aug 3 13:41:41 jumpclient kernel: blk_update_request: I/O error, dev sdbl, sector 78216
Aug 3 13:41:41 jumpclient kernel: blk_update_request: I/O error, dev sdbl, sector 79976
Aug 3 13:41:41 jumpclient kernel: blk_update_request: I/O error, dev sdbl, sector 79552
Aug 3 13:41:41 jumpclient kernel: blk_update_request: I/O error, dev sdbl, sector 87888
Aug 3 13:41:43 jumpclient kernel: blk_update_request: I/O error, dev sdbt, sector 274368
Aug 3 13:41:43 jumpclient kernel: blk_update_request: I/O error, dev sdbt, sector 31139814080
Aug 3 13:41:43 jumpclient kernel: blk_update_request: I/O error, dev sdbx, sector 6776
Aug 3 13:41:43 jumpclient kernel: blk_update_request: I/O error, dev sdbx, sector 302152
Changing script to add & we take both away at the same time but still we seem to survive here.
This is my configuration:
360001ff0b035d000000000078d770008 dm-9 DDN ,SFA14K
size=29T features='3 queue_if_no_path queue_mode mq' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=90 status=active
| `- 1:0:0:7 sday 67:32 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
`- 2:0:0:7 sdj 8:144 active ready running
device {
vendor "DDN"
product "SFA14K"
path_grouping_policy group_by_prio
prio alua
path_selector "round-robin 0"
path_checker tur
failback 2
rr_weight uniform
no_path_retry 12
dev_loss_tmo 10
fast_io_fail_tmo 5
features "1 queue_if_no_path"
}
With &
#!/bin/bash
for i in /sys/class/srp_remote_ports/*
do
echo "Deleting host $i, it will re-connect via srp_daemon"
echo 1 > $i/delete &
#sleep 3
done
Here we lose both paths at the same time.
[root@jumpclient bart_tests]# grep remaining messages
Aug 3 13:49:38 jumpclient multipathd: 360001ff0b035d000000000008d700001: remaining active paths: 0
Aug 3 13:49:38 jumpclient multipathd: 360001ff0b035d000000000028d720003: remaining active paths: 0
Aug 3 13:49:38 jumpclient multipathd: 360001ff0b035d000000000048d740005: remaining active paths: 0
Aug 3 13:49:41 jumpclient multipathd: 360001ff0b035d000000000068d760007: remaining active paths: 0
Aug 3 13:49:42 jumpclient multipathd: 360001ff0b035d0000000000d8d7d000e: remaining active paths: 0
Aug 3 13:49:45 jumpclient multipathd: 360001ff0b035d000000000118d810012: remaining active paths: 0
Aug 3 13:49:45 jumpclient multipathd: 360001ff0b035d000000000108d800011: remaining active paths: 0
Aug 3 13:49:47 jumpclient multipathd: 360001ff0b035d000000000158d850016: remaining active paths: 0
Aug 3 13:49:48 jumpclient multipathd: 360001ff0b035d000000000178d870018: remaining active paths: 0
Aug 3 13:49:48 jumpclient multipathd: 360001ff0b035d000000000198d89001a: remaining active paths: 0
Aug 3 13:49:48 jumpclient multipathd: 360001ff0b035d0000000001a8d8a001b: remaining active paths: 0
Aug 3 13:49:55 jumpclient multipathd: 360001ff0b035d0000000001e8d8e001f: remaining active paths: 0
Aug 3 13:49:55 jumpclient multipathd: 360001ff0b035d0000000001f8d8f0020: remaining active paths: 0
Aug 3 13:49:58 jumpclient multipathd: 360001ff0b035d000000000248d940025: remaining active paths: 0
Aug 3 13:49:59 jumpclient multipathd: 360001ff0b035d000000000268d960027: remaining active paths: 0
Aug 3 13:50:00 jumpclient multipathd: 360001ff0b035d000000000288d980029: remaining active paths: 0
Aug 3 13:51:17 jumpclient multipathd: 360001ff0b035d000000000038d730004: remaining active paths: 2
Aug 3 13:51:17 jumpclient multipathd: 360001ff0b035d000000000028d720003: remaining active paths: 2
Aug 3 13:51:19 jumpclient multipathd: 360001ff0b035d000000000078d770008: remaining active paths: 2
Aug 3 13:51:20 jumpclient multipathd: 360001ff0b035d0000000000a8d7a000b: remaining active paths: 2
Aug 3 13:51:23 jumpclient multipathd: 360001ff0b035d0000000000d8d7d000e: remaining active paths: 2
Aug 3 13:51:24 jumpclient multipathd: 360001ff0b035d000000000108d800011: remaining active paths: 2
Aug 3 13:51:25 jumpclient multipathd: 360001ff0b035d0000000000f8d7f0010: remaining active paths: 2
Aug 3 13:51:26 jumpclient multipathd: 360001ff0b035d000000000128d820013: remaining active paths: 2
Aug 3 13:51:29 jumpclient multipathd: 360001ff0b035d0000000001c8d8c001d: remaining active paths: 2
Aug 3 13:51:33 jumpclient multipathd: 360001ff0b035d000000000228d920023: remaining active paths: 2
Aug 3 13:51:34 jumpclient multipathd: 360001ff0b035d000000000238d930024: remaining active paths: 2
We still survive.
[root@jumpclient bart_tests]# grep -i error messages
Aug 3 13:49:38 jumpclient kernel: blk_update_request: I/O error, dev sdc, sector 98288
Aug 3 13:49:38 jumpclient kernel: blk_update_request: I/O error, dev sdc, sector 98320
Aug 3 13:49:38 jumpclient kernel: blk_update_request: I/O error, dev sdc, sector 46976
Aug 3 13:49:38 jumpclient kernel: blk_update_request: I/O error, dev sde, sector 216720
Aug 3 13:49:38 jumpclient kernel: blk_update_request: I/O error, dev sdg, sector 130672
Aug 3 13:49:41 jumpclient kernel: blk_update_request: I/O error, dev sdi, sector 56984
Aug 3 13:49:41 jumpclient kernel: blk_update_request: I/O error, dev sdi, sector 56120
Aug 3 13:49:41 jumpclient kernel: blk_update_request: I/O error, dev sdi, sector 62112
Aug 3 13:49:42 jumpclient kernel: blk_update_request: I/O error, dev sdp, sector 156944
Aug 3 13:49:42 jumpclient kernel: blk_update_request: I/O error, dev sdp, sector 31140975496
Aug 3 13:49:44 jumpclient kernel: blk_update_request: I/O error, dev sdt, sector 207392
Aug 3 13:49:44 jumpclient kernel: blk_update_request: I/O error, dev sdt, sector 200568
Aug 3 13:49:44 jumpclient kernel: blk_update_request: I/O error, dev sdt, sector 251048
Aug 3 13:49:44 jumpclient kernel: blk_update_request: I/O error, dev sdt, sector 247616
Aug 3 13:49:44 jumpclient kernel: blk_update_request: I/O error, dev sdt, sector 210592
Aug 3 13:49:44 jumpclient kernel: blk_update_request: I/O error, dev sdt, sector 200120
Aug 3 13:49:44 jumpclient kernel: blk_update_request: I/O error, dev sdt, sector 203000
Aug 3 13:49:44 jumpclient kernel: blk_update_request: I/O error, dev sdt, sector 248640
Aug 3 13:49:47 jumpclient kernel: blk_update_request: I/O error, dev sdx, sector 48232
Aug 3 13:49:48 jumpclient kernel: blk_update_request: I/O error, dev sdz, sector 9984
Aug 3 13:49:55 jumpclient kernel: blk_update_request: I/O error, dev sdag, sector 130512
Aug 3 13:49:58 jumpclient kernel: blk_update_request: I/O error, dev sdai, sector 39040
Aug 3 13:49:58 jumpclient kernel: blk_update_request: I/O error, dev sdam, sector 31140570528
Aug 3 13:49:59 jumpclient kernel: blk_update_request: I/O error, dev sdao, sector 204552
Aug 3 13:50:00 jumpclient kernel: blk_update_request: I/O error, dev sdaq, sector 31142052904
next prev parent reply other threads:[~2016-08-03 18:03 UTC|newest]
Thread overview: 74+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-07-19 22:57 dm-mq and end_clone_request() Bart Van Assche
2016-07-20 14:08 ` Mike Snitzer
2016-07-20 14:27 ` Mike Snitzer
2016-07-20 17:37 ` Bart Van Assche
2016-07-20 18:33 ` Mike Snitzer
2016-07-21 20:58 ` [dm-devel] " Bart Van Assche
2016-07-25 17:53 ` Mike Snitzer
2016-07-25 21:23 ` Mike Snitzer
2016-07-25 22:00 ` Bart Van Assche
2016-07-26 1:16 ` Mike Snitzer
2016-07-26 22:51 ` Bart Van Assche
2016-07-27 14:08 ` Mike Snitzer
2016-07-27 15:52 ` [dm-devel] " Benjamin Marzinski
2016-07-27 19:06 ` Bart Van Assche
2016-07-27 19:54 ` Mike Snitzer
2016-07-27 20:09 ` Mike Snitzer
2016-07-27 23:05 ` Bart Van Assche
2016-07-28 13:33 ` Mike Snitzer
2016-07-28 15:23 ` Bart Van Assche
2016-07-28 15:40 ` Mike Snitzer
2016-07-29 6:28 ` [dm-devel] " Hannes Reinecke
2016-07-26 6:02 ` Hannes Reinecke
2016-07-26 16:06 ` Mike Snitzer
[not found] ` <317679447.7168375.1469729769593.JavaMail.zimbra@redhat.com>
[not found] ` <6880321d-e14f-169b-d100-6e460dd9bd09@sandisk.com>
[not found] ` <1110327939.7305916.1469819453678.JavaMail.zimbra@redhat.com>
[not found] ` <a5c1a149-b1a2-b5a4-2207-bdaf32db3cbd@sandisk.com>
[not found] ` <757522831.7667712.1470059860543.JavaMail.zimbra@redhat.com>
[not found] ` <536022978.7668211.1470060125271.JavaMail.zimbra@redhat.com>
[not found] ` <931235537.7668834.1470060339483.JavaMail.zimbra@redhat.com>
[not found] ` <1264951811.7684268.1470065187014.JavaMail.zimbra@redhat.com>
[not found] ` <17da3ab0-233a-2cec-f921-bfd42c953ccc@sandisk.com>
2016-08-01 17:59 ` Mike Snitzer
2016-08-01 18:55 ` Bart Van Assche
2016-08-01 19:15 ` Mike Snitzer
2016-08-01 20:46 ` Mike Snitzer
2016-08-01 22:41 ` Bart Van Assche
2016-08-01 22:41 ` Bart Van Assche
2016-08-02 17:45 ` Mike Snitzer
2016-08-03 0:19 ` Bart Van Assche
2016-08-03 0:40 ` Mike Snitzer
2016-08-03 1:33 ` Laurence Oberman
2016-08-03 2:10 ` Mike Snitzer
2016-08-03 2:18 ` Laurence Oberman
2016-08-03 2:55 ` Laurence Oberman
2016-08-03 15:10 ` Laurence Oberman
2016-08-03 16:06 ` Bart Van Assche
2016-08-03 17:25 ` Laurence Oberman
2016-08-03 18:03 ` Laurence Oberman [this message]
2016-08-03 16:55 ` Bart Van Assche
2016-08-04 9:53 ` Hannes Reinecke
2016-08-04 10:09 ` Hannes Reinecke
2016-08-04 15:10 ` Mike Snitzer
2016-08-04 16:10 ` Mike Snitzer
2016-08-04 17:42 ` Bart Van Assche
2016-08-04 23:58 ` Mike Snitzer
2016-08-05 1:07 ` Laurence Oberman
2016-08-05 11:43 ` Laurence Oberman
2016-08-05 15:39 ` Laurence Oberman
2016-08-05 15:43 ` Bart Van Assche
2016-08-05 18:42 ` [dm-devel] " Bart Van Assche
2016-08-06 14:47 ` Laurence Oberman
2016-08-07 22:31 ` [dm-devel] " Bart Van Assche
2016-08-08 12:45 ` Laurence Oberman
2016-08-08 13:44 ` Johannes Thumshirn
2016-08-08 13:44 ` Johannes Thumshirn
2016-08-08 14:32 ` Laurence Oberman
2016-08-08 14:54 ` Bart Van Assche
2016-08-08 15:11 ` Bart Van Assche
2016-08-08 15:26 ` Laurence Oberman
2016-08-08 15:28 ` Bart Van Assche
2016-08-08 22:39 ` Bart Van Assche
2016-08-08 22:52 ` Laurence Oberman
2016-08-09 0:09 ` Laurence Oberman
2016-08-09 15:51 ` Bart Van Assche
2016-08-09 17:12 ` [dm-devel] " Laurence Oberman
2016-08-09 17:16 ` Bart Van Assche
2016-08-09 17:21 ` Laurence Oberman
2016-08-10 21:38 ` Laurence Oberman
2016-08-11 16:51 ` Laurence Oberman
2016-08-05 18:40 ` Bart Van Assche
2016-07-21 20:32 ` Mike Snitzer
2016-07-21 20:40 ` [dm-devel] " Bart Van Assche
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1165785378.8091030.1470247385715.JavaMail.zimbra@redhat.com \
--to=loberman@redhat.com \
--cc=bart.vanassche@sandisk.com \
--cc=dm-devel@redhat.com \
--cc=linux-scsi@vger.kernel.org \
--cc=snitzer@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.