Clarification for Persistent Reservation + All Registrants

Clarification for Persistent Reservation + All Registrants

[Nicholas A. Bellinger]

Hello T10/SCSI folks,

A question came up on the LIO target-devel mailing list recently wrt the
proper handling of Persistent Reservations + All Registrants logic in a
couple of different scenarios.

The crux of the confusion stems from how registrations are treated
before an explicit PR-OUT RESERVE service action with All Registrants
type occurs, and after that explicit PR-OUT RESERVE of All Registrants
type has been released.

My original interpretation was that an explicit PR-OUT RESERVE with All
Registrants type had to first occur before other non reservation holding
registrations (existing or not) would gain the implicit reservation.

This was primarily because the reservation type was not known until the
explicit PR-OUT RESERVE occurred, and AFAICT at the time there where no
host implementations that actually specified the reservation type at
registration time.

So Is it correct to assume that one explicit PR-OUT RESERVE with All
Registrants type needs to occur before other non reservation holding
registrations gain an implicit reservation..?

How about when the explicit reservation is removed..?  Do the implicit
reservations need to remain in effect for the existing registrations,
even though the explicit reservation of All Registrants type no longer
exists..?

The second point that Ilias raised is what should be the correct
R_HOLDER bit status for PRIN READ_FULL_STATUS descriptors when All
Registrants type is specified.  My original interpretation was that
R_HOLDER was to signal the explicit reservation holder obtained with
PR-OUT RESERVE, and not for the implicit reservations from the other
registrations.  Is this the correct approach..?

Also, does anyone have any experience with a host side implementation of
All Registrants to use as a reference for what's already out in the
wild..?

Thank you,

--nab

FYI, the pointer to Ilias's original question and my response with
additional background is available here:

http://comments.gmane.org/gmane.linux.scsi.target.devel/7632

Re: Clarification for Persistent Reservation + All Registrants

[James Bottomley]

On Fri, 2014-12-12 at 14:59 -0800, Nicholas A. Bellinger wrote:
> Hello T10/SCSI folks,
> 
> A question came up on the LIO target-devel mailing list recently wrt the
> proper handling of Persistent Reservations + All Registrants logic in a
> couple of different scenarios.

I can tell you how it was originally designed over a decade ago to
support clustering products (being in the business then of negotiating
with the manufacturers).

> The crux of the confusion stems from how registrations are treated
> before an explicit PR-OUT RESERVE service action with All Registrants
> type occurs, and after that explicit PR-OUT RESERVE of All Registrants
> type has been released.
> 
> My original interpretation was that an explicit PR-OUT RESERVE with All
> Registrants type had to first occur before other non reservation holding
> registrations (existing or not) would gain the implicit reservation.

You seem to be asking in the presence of registrations would an
unregistered initiator receive a reservation conflict before the PR_OUT
RESERVE is issued?  The answer, categorially is no because there's no
reservation until a registered initiator establishes one.

> This was primarily because the reservation type was not known until the
> explicit PR-OUT RESERVE occurred, and AFAICT at the time there where no
> host implementations that actually specified the reservation type at
> registration time.
> 
> So Is it correct to assume that one explicit PR-OUT RESERVE with All
> Registrants type needs to occur before other non reservation holding
> registrations gain an implicit reservation..?

I don't understand what you mean implicit reservation.  Once an all
registrants reservation is established, *all* registered initiators
become the reservation holder ... they're all equal.  This is mandated
in SPC-3 5.6.9

> How about when the explicit reservation is removed..?  Do the implicit
> reservations need to remain in effect for the existing registrations,
> even though the explicit reservation of All Registrants type no longer
> exists..?

Again, I don't understand the distinction between implicit and explicit.
For an all registrants reservation, all registered initiators are the
reservation holder so any one of them may release the reservation.  Once
the reservation is released, there's no reservation at all.

> The second point that Ilias raised is what should be the correct
> R_HOLDER bit status for PRIN READ_FULL_STATUS descriptors when All
> Registrants type is specified.  My original interpretation was that
> R_HOLDER was to signal the explicit reservation holder obtained with
> PR-OUT RESERVE, and not for the implicit reservations from the other
> registrations.  Is this the correct approach..?

R_HOLDER is set for any initiator that would be defined to be the
persistent reservation holder in section 5.6.9

> Also, does anyone have any experience with a host side implementation of
> All Registrants to use as a reference for what's already out in the
> wild..?

I know what was negotiated to support clustering (which is a pretty tiny
subset).  However, I think your point of confusion is trying to
distinguish between what you call implicit and explicit reservations ...
there's no such thing.  There's only a reservation holder which may be
many initiators, So it's binary: a given initiator either is or is not
the reservation holder.  Section 5.6 needs to be read in that context.

James

Re: Clarification for Persistent Reservation + All Registrants

[Nicholas A. Bellinger]

On Sat, 2014-12-13 at 00:52 +0100, James Bottomley wrote:
> On Fri, 2014-12-12 at 14:59 -0800, Nicholas A. Bellinger wrote:
> > Hello T10/SCSI folks,
> > 
> > A question came up on the LIO target-devel mailing list recently wrt the
> > proper handling of Persistent Reservations + All Registrants logic in a
> > couple of different scenarios.
> 
> I can tell you how it was originally designed over a decade ago to
> support clustering products (being in the business then of negotiating
> with the manufacturers).
> 
> > The crux of the confusion stems from how registrations are treated
> > before an explicit PR-OUT RESERVE service action with All Registrants
> > type occurs, and after that explicit PR-OUT RESERVE of All Registrants
> > type has been released.
> > 
> > My original interpretation was that an explicit PR-OUT RESERVE with All
> > Registrants type had to first occur before other non reservation holding
> > registrations (existing or not) would gain the implicit reservation.
> 
> You seem to be asking in the presence of registrations would an
> unregistered initiator receive a reservation conflict before the PR_OUT
> RESERVE is issued?  The answer, categorially is no because there's no
> reservation until a registered initiator establishes one.
> 

Not exactly.  The question is if a RESERVE with type=All Registrants
still needs to happen in order for the other registrations to have their
'implicit' reservation activated.

As mentioned below, the reasoning was because I'd only ever seen RESERVE
populate the reservation TYPE field (eg: not REGISTER), which would
imply that a 'explicit' RESERVE needs to happen from one initiator in
order for the target to know if All Registrants logic needs to kick in
for other existing + future registrations.

> > This was primarily because the reservation type was not known until the
> > explicit PR-OUT RESERVE occurred, and AFAICT at the time there where no
> > host implementations that actually specified the reservation type at
> > registration time.
> > 
> > So Is it correct to assume that one explicit PR-OUT RESERVE with All
> > Registrants type needs to occur before other non reservation holding
> > registrations gain an implicit reservation..?
> 
> I don't understand what you mean implicit reservation.  Once an all
> registrants reservation is established, *all* registered initiators
> become the reservation holder ... they're all equal.  This is mandated
> in SPC-3 5.6.9

Implicit reservation meaning a registration for an I_T nexus to a device
with a different I_T nexus holding an All Registrants reservation.

Eg, an I_T Nexus issued a RESERVE with type=All Registrants, and now the
other I_T nexus with an active registration on the same device have an
implicit reservation.

> 
> > How about when the explicit reservation is removed..?  Do the implicit
> > reservations need to remain in effect for the existing registrations,
> > even though the explicit reservation of All Registrants type no longer
> > exists..?
> 
> Again, I don't understand the distinction between implicit and explicit.
> For an all registrants reservation, all registered initiators are the
> reservation holder so any one of them may release the reservation.  Once
> the reservation is released, there's no reservation at all.

So some initiator still needs to perform the actual RESERVE, right..?

That is, a number of registrations don't become 'implicit' reservation
holders until one initiator performs an 'explicit' RESERVE with type=All
Registrants.

> 
> > The second point that Ilias raised is what should be the correct
> > R_HOLDER bit status for PRIN READ_FULL_STATUS descriptors when All
> > Registrants type is specified.  My original interpretation was that
> > R_HOLDER was to signal the explicit reservation holder obtained with
> > PR-OUT RESERVE, and not for the implicit reservations from the other
> > registrations.  Is this the correct approach..?
> 
> R_HOLDER is set for any initiator that would be defined to be the
> persistent reservation holder in section 5.6.9
> 

Well, section 5.6.9 only mentions the use of RESERVE, so that would seem
to indicate the original interpretation + current code for
READ_FULL_STATUS is correct.

> > Also, does anyone have any experience with a host side implementation of
> > All Registrants to use as a reference for what's already out in the
> > wild..?
> 
> I know what was negotiated to support clustering (which is a pretty tiny
> subset).  However, I think your point of confusion is trying to
> distinguish between what you call implicit and explicit reservations ...
> there's no such thing.  There's only a reservation holder which may be
> many initiators, So it's binary: a given initiator either is or is not
> the reservation holder.  Section 5.6 needs to be read in that context.
> 

The confusion is around when registrations who don't issue an explicit
RESERVE become implicit reservation holders.  Is it when one initiator
does an explicit RESERVE + type=All Registrants, or is it when REGISTER
+ type=All Registrants occurs..?

--nab

Re: Clarification for Persistent Reservation + All Registrants

[James Bottomley]

On Fri, 2014-12-12 at 17:53 -0800, Nicholas A. Bellinger wrote:
> On Sat, 2014-12-13 at 00:52 +0100, James Bottomley wrote:
> > On Fri, 2014-12-12 at 14:59 -0800, Nicholas A. Bellinger wrote:
> > > Hello T10/SCSI folks,
> > > 
> > > A question came up on the LIO target-devel mailing list recently wrt the
> > > proper handling of Persistent Reservations + All Registrants logic in a
> > > couple of different scenarios.
> > 
> > I can tell you how it was originally designed over a decade ago to
> > support clustering products (being in the business then of negotiating
> > with the manufacturers).
> > 
> > > The crux of the confusion stems from how registrations are treated
> > > before an explicit PR-OUT RESERVE service action with All Registrants
> > > type occurs, and after that explicit PR-OUT RESERVE of All Registrants
> > > type has been released.
> > > 
> > > My original interpretation was that an explicit PR-OUT RESERVE with All
> > > Registrants type had to first occur before other non reservation holding
> > > registrations (existing or not) would gain the implicit reservation.
> > 
> > You seem to be asking in the presence of registrations would an
> > unregistered initiator receive a reservation conflict before the PR_OUT
> > RESERVE is issued?  The answer, categorially is no because there's no
> > reservation until a registered initiator establishes one.
> > 
> 
> Not exactly.  The question is if a RESERVE with type=All Registrants
> still needs to happen in order for the other registrations to have their
> 'implicit' reservation activated.
> 
> As mentioned below, the reasoning was because I'd only ever seen RESERVE
> populate the reservation TYPE field (eg: not REGISTER), which would
> imply that a 'explicit' RESERVE needs to happen from one initiator in
> order for the target to know if All Registrants logic needs to kick in
> for other existing + future registrations.

I'm not sure I can be more clear: You can't have a reservation without
and initiator issuing a reserve command.

The TYPE parameter is ignored for REGISTER (specified in table 116).

> > > This was primarily because the reservation type was not known until the
> > > explicit PR-OUT RESERVE occurred, and AFAICT at the time there where no
> > > host implementations that actually specified the reservation type at
> > > registration time.
> > > 
> > > So Is it correct to assume that one explicit PR-OUT RESERVE with All
> > > Registrants type needs to occur before other non reservation holding
> > > registrations gain an implicit reservation..?
> > 
> > I don't understand what you mean implicit reservation.  Once an all
> > registrants reservation is established, *all* registered initiators
> > become the reservation holder ... they're all equal.  This is mandated
> > in SPC-3 5.6.9
> 
> Implicit reservation meaning a registration for an I_T nexus to a device
> with a different I_T nexus holding an All Registrants reservation.
> 
> Eg, an I_T Nexus issued a RESERVE with type=All Registrants, and now the
> other I_T nexus with an active registration on the same device have an
> implicit reservation.

There's no such distinction in the standards; all registered initators
become the reservation holder once an all registrants reservation is
issued. 

> > > How about when the explicit reservation is removed..?  Do the implicit
> > > reservations need to remain in effect for the existing registrations,
> > > even though the explicit reservation of All Registrants type no longer
> > > exists..?
> > 
> > Again, I don't understand the distinction between implicit and explicit.
> > For an all registrants reservation, all registered initiators are the
> > reservation holder so any one of them may release the reservation.  Once
> > the reservation is released, there's no reservation at all.
> 
> So some initiator still needs to perform the actual RESERVE, right..?

Yes, there's no reservation until a reserve is issued.

> That is, a number of registrations don't become 'implicit' reservation
> holders until one initiator performs an 'explicit' RESERVE with type=All
> Registrants.

Once one initiator issues an all registrants reservation, all registants
become the reservation holder.

> > > The second point that Ilias raised is what should be the correct
> > > R_HOLDER bit status for PRIN READ_FULL_STATUS descriptors when All
> > > Registrants type is specified.  My original interpretation was that
> > > R_HOLDER was to signal the explicit reservation holder obtained with
> > > PR-OUT RESERVE, and not for the implicit reservations from the other
> > > registrations.  Is this the correct approach..?
> > 
> > R_HOLDER is set for any initiator that would be defined to be the
> > persistent reservation holder in section 5.6.9
> > 
> 
> Well, section 5.6.9 only mentions the use of RESERVE, so that would seem
> to indicate the original interpretation + current code for
> READ_FULL_STATUS is correct.

It should be set for every registered initiator in the all registrants
case (because they're all the reservation holder).

> > > Also, does anyone have any experience with a host side implementation of
> > > All Registrants to use as a reference for what's already out in the
> > > wild..?
> > 
> > I know what was negotiated to support clustering (which is a pretty tiny
> > subset).  However, I think your point of confusion is trying to
> > distinguish between what you call implicit and explicit reservations ...
> > there's no such thing.  There's only a reservation holder which may be
> > many initiators, So it's binary: a given initiator either is or is not
> > the reservation holder.  Section 5.6 needs to be read in that context.
> > 
> 
> The confusion is around when registrations who don't issue an explicit
> RESERVE become implicit reservation holders.  Is it when one initiator
> does an explicit RESERVE + type=All Registrants, or is it when REGISTER
> + type=All Registrants occurs..?

There is no reservation until a reserve command is issued.  Once an all
registrants reservation is issued, by any registered initator, all
registered initators then become the reservation holder.

James

Re: Clarification for Persistent Reservation + All Registrants

[Nicholas A. Bellinger]

On Sat, 2014-12-13 at 08:33 +0100, James Bottomley wrote:
> On Fri, 2014-12-12 at 17:53 -0800, Nicholas A. Bellinger wrote:
> > On Sat, 2014-12-13 at 00:52 +0100, James Bottomley wrote:
> > > On Fri, 2014-12-12 at 14:59 -0800, Nicholas A. Bellinger wrote:
> > > > Hello T10/SCSI folks,
> > > > 
> > > > A question came up on the LIO target-devel mailing list recently wrt the
> > > > proper handling of Persistent Reservations + All Registrants logic in a
> > > > couple of different scenarios.

<SNIP>

> > > > How about when the explicit reservation is removed..?  Do the implicit
> > > > reservations need to remain in effect for the existing registrations,
> > > > even though the explicit reservation of All Registrants type no longer
> > > > exists..?
> > > 
> > > Again, I don't understand the distinction between implicit and explicit.
> > > For an all registrants reservation, all registered initiators are the
> > > reservation holder so any one of them may release the reservation.  Once
> > > the reservation is released, there's no reservation at all.
> > 
> > So some initiator still needs to perform the actual RESERVE, right..?
> 
> Yes, there's no reservation until a reserve is issued.
> 
> > That is, a number of registrations don't become 'implicit' reservation
> > holders until one initiator performs an 'explicit' RESERVE with type=All
> > Registrants.
> 
> Once one initiator issues an all registrants reservation, all registants
> become the reservation holder.
> 
> > > > The second point that Ilias raised is what should be the correct
> > > > R_HOLDER bit status for PRIN READ_FULL_STATUS descriptors when All
> > > > Registrants type is specified.  My original interpretation was that
> > > > R_HOLDER was to signal the explicit reservation holder obtained with
> > > > PR-OUT RESERVE, and not for the implicit reservations from the other
> > > > registrations.  Is this the correct approach..?
> > > 
> > > R_HOLDER is set for any initiator that would be defined to be the
> > > persistent reservation holder in section 5.6.9
> > > 
> > 
> > Well, section 5.6.9 only mentions the use of RESERVE, so that would seem
> > to indicate the original interpretation + current code for
> > READ_FULL_STATUS is correct.
> 
> It should be set for every registered initiator in the all registrants
> case (because they're all the reservation holder).
> 

Thanks for the clarification on this.

> > > > Also, does anyone have any experience with a host side implementation of
> > > > All Registrants to use as a reference for what's already out in the
> > > > wild..?
> > > 
> > > I know what was negotiated to support clustering (which is a pretty tiny
> > > subset).  However, I think your point of confusion is trying to
> > > distinguish between what you call implicit and explicit reservations ...
> > > there's no such thing.  There's only a reservation holder which may be
> > > many initiators, So it's binary: a given initiator either is or is not
> > > the reservation holder.  Section 5.6 needs to be read in that context.
> > > 
> > 
> > The confusion is around when registrations who don't issue an explicit
> > RESERVE become implicit reservation holders.  Is it when one initiator
> > does an explicit RESERVE + type=All Registrants, or is it when REGISTER
> > + type=All Registrants occurs..?
> 
> There is no reservation until a reserve command is issued.  Once an all
> registrants reservation is issued, by any registered initator, all
> registered initators then become the reservation holder.
> 

So based on the above, the issue raised by Ilias where once the original
All Registrants reservation is released, the other registrations lose
their 'implicit' reservation until someone does another RESERVE +
type=All Registrants is a bug.

Working on a proper bugfix for this now.

Thanks James!

--nab

Re: Clarification for Persistent Reservation + All Registrants

[Lee Duncan]

Sorry to come to this late, but I have one addition ...

On 12/13/2014 10:27 PM, Nicholas A. Bellinger wrote:
> On Sat, 2014-12-13 at 08:33 +0100, James Bottomley wrote:
>> On Fri, 2014-12-12 at 17:53 -0800, Nicholas A. Bellinger wrote:
>>> On Sat, 2014-12-13 at 00:52 +0100, James Bottomley wrote:
>>>> On Fri, 2014-12-12 at 14:59 -0800, Nicholas A. Bellinger wrote:
> lots of SNIP
>>
>> There is no reservation until a reserve command is issued.  Once an all
>> registrants reservation is issued, by any registered initator, all
>> registered initators then become the reservation holder.
>>
> 
> So based on the above, the issue raised by Ilias where once the original
> All Registrants reservation is released, the other registrations lose
> their 'implicit' reservation until someone does another RESERVE +
> type=All Registrants is a bug.

U belive your target Nicholas is working correctly. See that other
thread for more details.

> 
> Working on a proper bugfix for this now.
> 
> Thanks James!
> 
> --nab
> 


-- 
Lee Duncan