[BUG] dm-mpath and scsi persistent reservation

[BUG] dm-mpath and scsi persistent reservation

[Christophe Varoqui]

Hi,

the dm-mpath multipathing driver enqueues write ios returned by
the scsi layer with a "reservation conflict" error (on assymetric
storage controllers, like Clariion, where queue_if_no_path is enabled).

This is wrong, and a potent data-corrupter : wio sent to a reserved
scsi device should never be retried on this same device.

Can someone advise on a viable solution for this problem ? I
understand the Device Mapper being a block remapper, it is not expected
to receive scsi errors directly ... but maybe some kind of translation
might be acceptable. Or is there hope in the request-based remapper ?

Jens, James, Alasdair, ... as maintainers of the involved subsystems,
would you care to give some advise on the issue.

Regards,
cvaroqui

Re: [BUG] dm-mpath and scsi persistent reservation

[Chandra Seetharaman]

Christophe,

The SCSI Hardware handler is created for devices with special needs. It
is available in 2.6.27. One can be written for any device to translate
the scsi sense code to a different error code that the dm understands.
Is this something that coule help this situation ?

Have a look at drivers/scsi/device_handler directory.

Hope this helps,

chandra
On Tue, 2008-10-21 at 23:19 +0200, Christophe Varoqui wrote:
> Hi,
> 
> the dm-mpath multipathing driver enqueues write ios returned by
> the scsi layer with a "reservation conflict" error (on assymetric
> storage controllers, like Clariion, where queue_if_no_path is enabled).
> 
> This is wrong, and a potent data-corrupter : wio sent to a reserved
> scsi device should never be retried on this same device.
> 
> Can someone advise on a viable solution for this problem ? I
> understand the Device Mapper being a block remapper, it is not expected
> to receive scsi errors directly ... but maybe some kind of translation
> might be acceptable. Or is there hope in the request-based remapper ?
> 
> Jens, James, Alasdair, ... as maintainers of the involved subsystems,
> would you care to give some advise on the issue.
> 
> Regards,
> cvaroqui
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG] dm-mpath and scsi persistent reservation

[Christophe Varoqui]

It seems to me the device handler infrastructure proposes to translate
scsi error codes from requests generated by the device handler itself. I
don't know how we can detect a reservation conflict from a device
handler without submitting a dangerous write io.

I don't see how we could use a device handler to translate an scsi error
code from a write io submitted to the multipath device map. Do you ?

Regards,
cvaroqui
> 
> The SCSI Hardware handler is created for devices with special needs.
> It is available in 2.6.27. One can be written for any device to
> translate the scsi sense code to a different error code that the dm
> understands. Is this something that coule help this situation ?
> 
> Have a look at drivers/scsi/device_handler directory.
> 
> Hope this helps,
> 
> chandra
> On Tue, 2008-10-21 at 23:19 +0200, Christophe Varoqui wrote:
> > Hi,
> > 
> > the dm-mpath multipathing driver enqueues write ios returned by
> > the scsi layer with a "reservation conflict" error (on assymetric
> > storage controllers, like Clariion, where queue_if_no_path is
> > enabled).
> > 
> > This is wrong, and a potent data-corrupter : wio sent to a reserved
> > scsi device should never be retried on this same device.
> > 
> > Can someone advise on a viable solution for this problem ? I
> > understand the Device Mapper being a block remapper, it is not
> > expected to receive scsi errors directly ... but maybe some kind of
> > translation might be acceptable. Or is there hope in the
> > request-based remapper ?
> > 
> > Jens, James, Alasdair, ... as maintainers of the involved
> > subsystems, would you care to give some advise on the issue.
> > 
> > 

Re: [dm-devel] Re: [BUG] dm-mpath and scsi persistent reservation

[James Bottomley]

On Wed, 2008-10-22 at 21:54 +0200, Christophe Varoqui wrote:
> It seems to me the device handler infrastructure proposes to translate
> scsi error codes from requests generated by the device handler itself. I
> don't know how we can detect a reservation conflict from a device
> handler without submitting a dangerous write io.

For SCSI-2 reservations, Test Unit Ready will do this for you.

For SCSI-3, you're right, it's more complex.  You actually have to use
the PR IN commands to read the reservations if you don't want to test
what they'll actually do with an I/O.

> I don't see how we could use a device handler to translate an scsi error
> code from a write io submitted to the multipath device map. Do you ?

Well, there is a problem.  Reservation Conflict should be treated as a
device error and passed straight up ... it shouldn't really have any
effect on dm mp because a path switch is unlikely to fix any issues.  So
dm mp shouldn't be intercepting this type of error at all.

James

Re: [BUG] dm-mpath and scsi persistent reservation

[Chandra Seetharaman]

On Wed, 2008-10-22 at 21:54 +0200, Christophe Varoqui wrote:
> It seems to me the device handler infrastructure proposes to translate
> scsi error codes from requests generated by the device handler itself. I

No, the handler doesn't generate the requests.

Device handler's check_sense() function is called from the
scsi_error.c:scsi_check_sense() function whenever a sense code is
returned from the device (on normal I/Os).

When in the sense function, the device handler can do any action (like
closing the other path(s)) and returning an appropriate error code (such
that dm doesn't retry the I/O on other paths).

> don't know how we can detect a reservation conflict from a device
> handler without submitting a dangerous write io.
> 
> I don't see how we could use a device handler to translate an scsi error
> code from a write io submitted to the multipath device map. Do you ?

That is what it actually does, it intercepts the sense code returned
from the device on normal I/Os.

> 
> Regards,
> cvaroqui
> > 
> > The SCSI Hardware handler is created for devices with special needs.
> > It is available in 2.6.27. One can be written for any device to
> > translate the scsi sense code to a different error code that the dm
> > understands. Is this something that coule help this situation ?
> > 
> > Have a look at drivers/scsi/device_handler directory.
> > 
> > Hope this helps,
> > 
> > chandra
> > On Tue, 2008-10-21 at 23:19 +0200, Christophe Varoqui wrote:
> > > Hi,
> > > 
> > > the dm-mpath multipathing driver enqueues write ios returned by
> > > the scsi layer with a "reservation conflict" error (on assymetric
> > > storage controllers, like Clariion, where queue_if_no_path is
> > > enabled).
> > > 
> > > This is wrong, and a potent data-corrupter : wio sent to a reserved
> > > scsi device should never be retried on this same device.
> > > 
> > > Can someone advise on a viable solution for this problem ? I
> > > understand the Device Mapper being a block remapper, it is not
> > > expected to receive scsi errors directly ... but maybe some kind of
> > > translation might be acceptable. Or is there hope in the
> > > request-based remapper ?
> > > 
> > > Jens, James, Alasdair, ... as maintainers of the involved
> > > subsystems, would you care to give some advise on the issue.
> > > 
> > > 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [dm-devel] Re: [BUG] dm-mpath and scsi persistent reservation

[Mike Christie]

James Bottomley wrote:
>> I don't see how we could use a device handler to translate an scsi error
>> code from a write io submitted to the multipath device map. Do you ?
> 
> Well, there is a problem.  Reservation Conflict should be treated as a
> device error and passed straight up ... it shouldn't really have any
> effect on dm mp because a path switch is unlikely to fix any issues.  So
> dm mp shouldn't be intercepting this type of error at all.
> 

I think what Christophe was asking for is something like this:

[RFC PATCH 1/4] convert block layer drivers to blkerr
http://marc.info/?l=linux-scsi&m=112487427230642&w=2
[RFC PATCH 2/4] convert dm to blkerr error values
http://marc.info/?l=linux-scsi&m=112487427306501&w=2
[RFC PATCH 3/4] convert dm-multipath to blkerr error
http://marc.info/?l=linux-scsi&m=112487431524436&w=2
[RFC PATCH 4/4] convert scsi to blkerr error values
http://marc.info/?l=linux-scsi&m=112487431524350&w=2

something that allows lower layers to give the upper layers some extra 
info. In the patches the scsi layer would return a fatal device error, 
and device mapper multipath would see that and just fail the IO instead 
of retrying on a new path.

I do not like my implementation in those patches, but I did not have 
time in the past to rework them. I can now though if you guys have any 
comments. I am really struggling on the definition of the block layer 
errors codes and what info they should convey. For example I was not 
sure if they should they give a hint about if the error is fatal or 
retryable like in the patches above or should they describe what happened?

Re: [BUG] dm-mpath and scsi persistent reservation

[Christophe Varoqui]

Le Wed, 22 Oct 2008 14:21:57 -0700,
Chandra Seetharaman <sekharan@us.ibm.com> a écrit :

> 
> On Wed, 2008-10-22 at 21:54 +0200, Christophe Varoqui wrote:
> > It seems to me the device handler infrastructure proposes to
> > translate scsi error codes from requests generated by the device
> > handler itself. I
> 
> No, the handler doesn't generate the requests.
> 
> Device handler's check_sense() function is called from the
> scsi_error.c:scsi_check_sense() function whenever a sense code is
> returned from the device (on normal I/Os).
> 
> When in the sense function, the device handler can do any action (like
> closing the other path(s)) and returning an appropriate error code
> (such that dm doesn't retry the I/O on other paths).
> 
Thanks for the clarification. I shouldn't have stopped after
reading the hp_sw handler, which does not implement a .check_sense

So now, device handlers indeed look promising. I still see difficulties
though :
1/ reservation conflict is a scmd status, and as such short-cuts the
scsi_check_sense function altogether. Sould we hook device handlers to
status parser too ?
2/ should we create an trivial device handler and load multi-handlers
multipath maps or should we make this device handler implicitly always
active ?

Regards,
cvaroqui
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG] dm-mpath and scsi persistent reservation

[Christophe Varoqui]

> On Wed, 2008-10-22 at 21:54 +0200, Christophe Varoqui wrote:
> > It seems to me the device handler infrastructure proposes to
> > translate scsi error codes from requests generated by the device
> > handler itself. I don't know how we can detect a reservation
> > conflict from a device handler without submitting a dangerous write
> > io.
> 
> For SCSI-2 reservations, Test Unit Ready will do this for you.
> 
> For SCSI-3, you're right, it's more complex.  You actually have to use
> the PR IN commands to read the reservations if you don't want to test
> what they'll actually do with an I/O.
> 
The PR-IN "READ FULL STATUS" looked promising indeed, bu does not work
on any storage hardware I tried. Always ends up there (in
sg_persist.c) :

  293         else if (SG_LIB_CAT_ILLEGAL_REQ == res)
  294             fprintf(stderr, "PR in: bad field in cdb including "
  295                     "unsupported service action\n");

Other PR-INs would list the registration keys and reservation,
but how would we know from the kernel or from userspace multipath which
keys are associated with host's I_T nexus ? The key would likely have
been registered by a userspace clusterware for fencing purpose.

PR-OUT "REGISTER" with params rk=0 sark=0 may trigger a significative
difference when send over a registered I_T or not (based on SPC-3 -
Table 33 — Register behaviors for a REGISTER service action).

Did you have something different in mind ?

> > I don't see how we could use a device handler to translate an scsi
> > error code from a write io submitted to the multipath device map.
> > Do you ?
> 
> Well, there is a problem.  Reservation Conflict should be treated as a
> device error and passed straight up ... it shouldn't really have any
> effect on dm mp because a path switch is unlikely to fix any issues.
> So dm mp shouldn't be intercepting this type of error at all.
> 
Well, a path switch might be a valid behaviour considering persistent
reservations are per I_T nexus ... the io may succeed if submited from
another intiator for example. But anyway, if all paths failed the io
should not be queued.

Regards,
cvaroqui
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html