rq-based mpath failed path on SCSI target failure (malformed discard)

rq-based mpath failed path on SCSI target failure (malformed discard)

[Mike Snitzer]

Hi,

I have been implementing discard support for the various DM targets.  I
have a LUN that requires a 4K alignment for all discard requests.  I
forced the issue (by misconfiguring the alignment such that discards
would be misaligned).  Multipath failed the path once the SCSI layer got
the following error.

My question: shouldn't mpath _not_ fail the path for "Illegal
Request".. "Parameter value invalid"?  The same request would fail on
any path given that the failure is on the target side (rejecting the
malformed discard request).  Following setup is ext4 FS ontop of linear
DM mapping (dm-2) ontop of request-based mpath mapping (dm-1):

EXT4-fs (dm-2): mounted filesystem with ordered data mode
sd 3:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 3:0:0:0: [sda] Sense Key : Illegal Request [current] 
Info fld=0x0
sd 3:0:0:0: [sda] Add. Sense: Parameter value invalid
sd 3:0:0:0: [sda] CDB: Write same(16): 93 08 00 00 00 00 00 00 45 82 00 00 03 00 00 00
end_request: I/O error, dev sda, sector 17794
device-mapper: multipath: Failing path 8:0.
end_request: I/O error, dev dm-1, sector 17794
end_request: I/O error, dev dm-1, sector 17794
end_request: I/O error, dev dm-1, sector 0
end_request: I/O error, dev dm-1, sector 386
quiet_error: 1228 callbacks suppressed
Buffer I/O error on device dm-2, logical block 1
lost page write due to I/O error on dm-2
Buffer I/O error on device dm-2, logical block 2
lost page write due to I/O error on dm-2
end_request: I/O error, dev dm-1, sector 904
Buffer I/O error on device dm-2, logical block 260
lost page write due to I/O error on dm-2
end_request: I/O error, dev dm-1, sector 926
Buffer I/O error on device dm-2, logical block 271
lost page write due to I/O error on dm-2
end_request: I/O error, dev dm-1, sector 934
Buffer I/O error on device dm-2, logical block 275
lost page write due to I/O error on dm-2
end_request: I/O error, dev dm-1, sector 966
Buffer I/O error on device dm-2, logical block 291
lost page write due to I/O error on dm-2
Buffer I/O error on device dm-2, logical block 292
lost page write due to I/O error on dm-2

Here is the relevant scsi trace:

        kdmflush-3658  [000] 952692.258948: scsi_dispatch_cmd_start: host_no=3 channel=0 id=0 lun=0 data_sgl=1 prot_sgl=0 cmnd=(WRITE_SAME_16 lba=17794 txlen=768 protect=0 unmap=1 raw=93 08 00 00 00 00 00 00 45 82 00 00 03 00 00 00)
          <idle>-0     [000] 952692.259650: scsi_dispatch_cmd_done: host_no=3 channel=0 id=0 lun=0 data_sgl=1 prot_sgl=0 cmnd=(WRITE_SAME_16 lba=17794 txlen=768 protect=0 unmap=1 raw=93 08 00 00 00 00 00 00 45 82 00 00 03 00 00 00) result=(driver=DRIVER_OK host=DID_OK message=COMMAND_COMPLETE status=SAM_STAT_CHECK_CONDITION)
      multipathd-3661  [000] 952692.261484: scsi_dispatch_cmd_start: host_no=3 channel=0 id=0 lun=0 data_sgl=1 prot_sgl=0 cmnd=(READ_10 lba=0 txlen=8 protect=0 raw=28 00 00 00 00 00 00 00 08 00)

Re: rq-based mpath failed path on SCSI target failure (malformed discard)

[Mike Snitzer]

On Mon, Jul 26 2010 at  9:30am -0400,
Mike Snitzer <snitzer@redhat.com> wrote:

> Hi,
> 
> I have been implementing discard support for the various DM targets.  I
> have a LUN that requires a 4K alignment for all discard requests.  I
> forced the issue (by misconfiguring the alignment such that discards
> would be misaligned).  Multipath failed the path once the SCSI layer got
> the following error.
> 
> My question: shouldn't mpath _not_ fail the path for "Illegal
> Request".. "Parameter value invalid"?  The same request would fail on
> any path given that the failure is on the target side (rejecting the
> malformed discard request).  Following setup is ext4 FS ontop of linear
> DM mapping (dm-2) ontop of request-based mpath mapping (dm-1):
> 
> EXT4-fs (dm-2): mounted filesystem with ordered data mode
> sd 3:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> sd 3:0:0:0: [sda] Sense Key : Illegal Request [current] 
> Info fld=0x0
> sd 3:0:0:0: [sda] Add. Sense: Parameter value invalid
> sd 3:0:0:0: [sda] CDB: Write same(16): 93 08 00 00 00 00 00 00 45 82 00 00 03 00 00 00
> end_request: I/O error, dev sda, sector 17794
> device-mapper: multipath: Failing path 8:0.

The following patch sorts this out in multipath; special-cases discard
requests so that they don't enduce path failures:

diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c
index ec2b43e..125c80f 100644
--- a/drivers/md/dm-mpath.c
+++ b/drivers/md/dm-mpath.c
@@ -1272,6 +1272,9 @@ static int do_end_io(struct multipath *m, struct request *clone,
 	if (error == -EOPNOTSUPP)
 		return error;
 
+	if (clone->cmd_flags & REQ_DISCARD)
+		return error;   /* pass all discard request failures up */
+
 	if (mpio->pgpath)
 		fail_path(mpio->pgpath);
 

Re: rq-based mpath failed path on SCSI target failure (malformed discard)

[Mike Christie]

On 07/26/2010 10:00 AM, Mike Snitzer wrote:
>
> diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c
> index ec2b43e..125c80f 100644
> --- a/drivers/md/dm-mpath.c
> +++ b/drivers/md/dm-mpath.c
> @@ -1272,6 +1272,9 @@ static int do_end_io(struct multipath *m, struct request *clone,
>   	if (error == -EOPNOTSUPP)
>   		return error;
>
> +	if (clone->cmd_flags&  REQ_DISCARD)
> +		return error;   /* pass all discard request failures up */
> +


If a discard where failed due to a transport problem then you are going 
to fail the IO instead of just retrying on another path.

You need something like Hannes's work to pass up the sense, or something 
like generic block error codes like I started to work on, so you know 
why the IO failed.

Re: rq-based mpath failed path on SCSI target failure (malformed discard)

[Mike Snitzer]

On Mon, Jul 26 2010 at 12:23pm -0400,
Mike Christie <michaelc@cs.wisc.edu> wrote:

> On 07/26/2010 10:00 AM, Mike Snitzer wrote:
> >
> >diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c
> >index ec2b43e..125c80f 100644
> >--- a/drivers/md/dm-mpath.c
> >+++ b/drivers/md/dm-mpath.c
> >@@ -1272,6 +1272,9 @@ static int do_end_io(struct multipath *m, struct request *clone,
> >  	if (error == -EOPNOTSUPP)
> >  		return error;
> >
> >+	if (clone->cmd_flags&  REQ_DISCARD)
> >+		return error;   /* pass all discard request failures up */
> >+
> 
> 
> If a discard where failed due to a transport problem then you are
> going to fail the IO instead of just retrying on another path.
> 
> You need something like Hannes's work to pass up the sense, or
> something like generic block error codes like I started to work on,
> so you know why the IO failed.

Thanks for reminding me about previous work done in the area of error
propagation.

Yes, we certainly need a more intelligent check but since neither of the
solutions you referenced are available yet we're better off failing the
discard in the near-term.  Clearly not ideal but failing a path when an
optional discard IO failure occurs isn't going to fly either.

I'll slap a FIXME on this interim "fix".

In parallel I'll see if I can track down Hannes' and/or your work,
rebase it as needed, etc.  If you have updated patches please let me
know.

Thanks,
Mike