SRP automatic target discovery

SRP automatic target discovery

[Craig Prescott]

Hi;

I'm having a problem with SRP automatic target discovery.  I am using 
srp_daemon with OFED 1.5.2 on the initiator.  My one target machine with 
several LUNs also uses OFED 1.5.2, and I am using scst 2.0.0.1 and srpt 
2.0.0 on top of that.

On the initiator, I have SRP_LOAD=yes and SRP_DAEMON_ENABLE=yes.  This 
seems to work fine, and after rebooting, I can see the SRP targets on 
the initiator, and srp_daemon running in the background:

[root@hpcoss2 ~]# ps auxww | grep srp
root      6635  0.0  0.0  63840  1176 ?        S    12:04   0:00 
/bin/bash /usr/sbin/srp_daemon.sh
root      6638  0.0  0.0  63836  1164 ?        S    12:04   0:00 
/bin/bash /usr/sbin/run_srp_daemon -e -c -n -i mlx4_1 -p 1 -R 60 -V
root      6647  0.0  0.0  31012   996 ?        SLl  12:04   0:00 
srp_daemon -e -c -n -i mlx4_1 -p 1 -R 60 -V

But if I add an additional target on the same target machine with, say:

[root@hpcoss1 ~]# echo "add 7:0:1:0 3" 
 >/proc/scsi_tgt/groups/Default/devices

the srp_daemon on the initiator never sees the new storage and 
restarting srp_daemon doesn't seem to help.

If I remove the SRP scsi devices, unload the ib_srp module, and reload 
it, I can see the new LUN (and I don't need to restart srp_daemon, 
either).  But I don't think that is how it is supposed to work.

Am I doing something wrong?

Thanks,
Craig Prescott
UF HPC Center



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: SRP automatic target discovery

[David Dillow]

On Wed, 2011-02-02 at 15:27 -0500, Craig Prescott wrote:
> But if I add an additional target on the same target machine with, say:
> 
> [root@hpcoss1 ~]# echo "add 7:0:1:0 3" 
>  >/proc/scsi_tgt/groups/Default/devices
> 
> the srp_daemon on the initiator never sees the new storage and 
> restarting srp_daemon doesn't seem to help.
> 
> If I remove the SRP scsi devices, unload the ib_srp module, and reload 
> it, I can see the new LUN (and I don't need to restart srp_daemon, 
> either).  But I don't think that is how it is supposed to work.

Sounds about right for the existing code. There's no notification to the
initiator that you added a new LUN on an existing connection, so the LUN
will not show up on the client. There is a mechanism in the SCSI
standard for such things, but the Linux SCSI mid-layer does not
currently implement it.

Restarting the daemon without removing the ib_srp module doesn't help
you, because when you restart the daemon, it notices you already have a
connection to that particular SRP target. Unloading/reloading the ib_srp
module tears down the existing connections, so srp_daemon will cause a
reconnection when it notices.

If you want to avoid having to unload/reload right now, you can manually
tell the client's SCSI layer to scan that LUN to get it to show up.

-- 
Dave Dillow
National Center for Computational Science
Oak Ridge National Laboratory
(865) 241-6602 office

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: SRP automatic target discovery

[Vladislav Bolkhovitin]

David Dillow, on 02/02/2011 07:44 PM wrote:
> On Wed, 2011-02-02 at 15:27 -0500, Craig Prescott wrote:
>> But if I add an additional target on the same target machine with, say:
>>
>> [root@hpcoss1 ~]# echo "add 7:0:1:0 3" 
>>  >/proc/scsi_tgt/groups/Default/devices
>>
>> the srp_daemon on the initiator never sees the new storage and 
>> restarting srp_daemon doesn't seem to help.
>>
>> If I remove the SRP scsi devices, unload the ib_srp module, and reload 
>> it, I can see the new LUN (and I don't need to restart srp_daemon, 
>> either).  But I don't think that is how it is supposed to work.
> 
> Sounds about right for the existing code. There's no notification to the
> initiator that you added a new LUN on an existing connection, so the LUN
> will not show up on the client. There is a mechanism in the SCSI
> standard for such things, but the Linux SCSI mid-layer does not
> currently implement it.

I'm sorry, but I need to correct you. There is notification to the
initiator that you added a new LUN on an existing connection. It is
REPORT LUNS DATA HAVE CHANGED Unit Attention. SCST sends it. But it
isn't processed anyhow on the initiator.

Vlad
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: SRP automatic target discovery

[Bart Van Assche]

On Wed, Feb 2, 2011 at 9:27 PM, Craig Prescott <prescott-rKMKKiNZRMeVc3sceRu5cw@public.gmane.org> wrote:
>
> I'm having a problem with SRP automatic target discovery.  I am using srp_daemon with OFED 1.5.2 on the initiator.  My one target machine with several LUNs also uses OFED 1.5.2, and I am using scst 2.0.0.1 and srpt 2.0.0 on top of that.
>
> On the initiator, I have SRP_LOAD=yes and SRP_DAEMON_ENABLE=yes.  This seems to work fine, and after rebooting, I can see the SRP targets on the initiator, and srp_daemon running in the background:
>
> [root@hpcoss2 ~]# ps auxww | grep srp
> root      6635  0.0  0.0  63840  1176 ?        S    12:04   0:00 /bin/bash /usr/sbin/srp_daemon.sh
> root      6638  0.0  0.0  63836  1164 ?        S    12:04   0:00 /bin/bash /usr/sbin/run_srp_daemon -e -c -n -i mlx4_1 -p 1 -R 60 -V
> root      6647  0.0  0.0  31012   996 ?        SLl  12:04   0:00 srp_daemon -e -c -n -i mlx4_1 -p 1 -R 60 -V
>
> But if I add an additional target on the same target machine with, say:
>
> [root@hpcoss1 ~]# echo "add 7:0:1:0 3" >/proc/scsi_tgt/groups/Default/devices
>
> the srp_daemon on the initiator never sees the new storage and restarting srp_daemon doesn't seem to help.
>
> If I remove the SRP scsi devices, unload the ib_srp module, and reload it, I can see the new LUN (and I don't need to restart srp_daemon, either).  But I don't think that is how it is supposed to work.

Hello Craig,

In addition to what Dave and Vlad already wrote:
* In the SCSI Architectural Model (SAM) it is specified that a logical
unit shall generate a unit attention condition whenever e.g. the
INQUIRY data has changed. The mechanism for reporting unit attention
conditions to an initiator is transport specific.
* As Vlad explained, the SCST core supports this mechanism. The
ib_srpt driver however ignores unit attention conditions and does not
propagate these to the SRP initiator.
* The mechanism defined in the SRP draft for reporting unit attention
conditions by a target to an initiator is the SRP_AER_REQ information
unit.
* Neither the Linux SRP initiator nor the VMware ESX initiator nor the
Windows SRP initiator currently support the SRP_AER_REQ information
unit. Even worse, some versions of these initiators cause a kernel
crash upon reception of a SRP_AER_REQ information unit.
* Support in the Linux SCSI layer (at the initiator side) for unit
attention conditions is very limited. Currently only media change
events are supported (SDEV_EVT_MEDIA_CHANGE).
* As Dave already wrote, you can tell the initiator SCSI layer to
rescan LUNs manually. This script can be convenient to rescan LUNs:
http://www.garloff.de/kurt/linux/rescan-scsi-bus.sh-1.48.
* The goal of srp_daemon is to discover targets. It is out of scope
for srp_daemon to detect LUN changes of a target.

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html