[PATCH 0/2] scsi_transport_fc: LUN masking

[PATCH 0/2] scsi_transport_fc: LUN masking

[Hannes Reinecke]

Hi all,

having been subjected to the pain of trying to bootstrap a really
large machine with systemd I decided to implement LUN masking in
scsi_transport_fc.
The principle is simple: disallow the automated LUN scanning when
discovering a rport, and create udev rules which selectively
enable individual LUNs by echoing the relevant values in the 'scan'
attribute of the SCSI host.
With that I'm able to boot an arbitrary large machine without
running into any udev or systemd imposed timeout.
To _disable_ LUN masking and restoring the original behaviour
I've noticed that the 'scan' sysfs attribute is actually synchronous,
ie the calling process will be blocked until the entire LUN scan
is completed.
So I've added another module parameter 'async_user_scan' to
move the scanning onto the existing scan workqueue, and unblock
the calling process.

As usual, comments and reviews are welcome.

Hannes Reinecke (2):
  scsi_transport_fc: implement 'disable_target_scan' module parameter
  scsi_transport_fc: Implement 'async_user_scan' module parameter

 drivers/scsi/scsi_transport_fc.c | 47 +++++++++++++++++++++++++++++++++++++---
 1 file changed, 44 insertions(+), 3 deletions(-)

-- 
1.8.5.6

[PATCH 1/2] scsi_transport_fc: implement 'disable_target_scan' module parameter

[Hannes Reinecke]

On larger installations it makes sense to disable the target scan
per default on boot, and scan the required LUNs directly via udev
rules.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/scsi/scsi_transport_fc.c | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git a/drivers/scsi/scsi_transport_fc.c b/drivers/scsi/scsi_transport_fc.c
index 454cc28..72954a8 100644
--- a/drivers/scsi/scsi_transport_fc.c
+++ b/drivers/scsi/scsi_transport_fc.c
@@ -71,6 +71,18 @@ MODULE_PARM_DESC(dev_loss_tmo,
 		 " fast_io_fail_tmo is not set.");
 
 /*
+ * disable_target_scan: Disable target scan per default
+ *   useful on larger installations where only a small
+ *   number of LUNs are required for booting.
+ */
+static bool fc_disable_target_scan;
+
+module_param_named(disable_target_scan, fc_disable_target_scan,
+		   bool, S_IRUGO|S_IWUSR);
+MODULE_PARM_DESC(disable_target_scan,
+		 "Disable target scan on remote ports (default=0)");
+
+/*
  * Redefine so that we can have same named attributes in the
  * sdev/starget/host objects.
  */
@@ -3282,10 +3294,14 @@ fc_scsi_scan_rport(struct work_struct *work)
 	struct Scsi_Host *shost = rport_to_shost(rport);
 	struct fc_internal *i = to_fc_internal(shost->transportt);
 	unsigned long flags;
+	bool disable_target_scan;
+
+	disable_target_scan = fc_disable_target_scan ?
+		fc_disable_target_scan : i->f->disable_target_scan;
 
 	if ((rport->port_state == FC_PORTSTATE_ONLINE) &&
 	    (rport->roles & FC_PORT_ROLE_FCP_TARGET) &&
-	    !(i->f->disable_target_scan)) {
+	    !disable_target_scan) {
 		scsi_scan_target(&rport->dev, rport->channel,
 			rport->scsi_target_id, SCAN_WILD_CARD, 1);
 	}
-- 
1.8.5.6

RE: [PATCH 1/2] scsi_transport_fc: implement 'disable_target_scan' module parameter

[Seymour, Shane M]

Reviewed-by: Shane Seymour <shane.seymour@hpe.com>

[PATCH 2/2] scsi_transport_fc: Implement 'async_user_scan' module parameter

[Hannes Reinecke]

When invoking a scan via the sysfs 'scan' attribute the process
will be blocked until the scan is completed, which can take a
very long time on large installations.
Enabling the 'async_user_scan' parameter moves the actual
LUN scanning to a workqueue, thereby unblocking the process.

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 drivers/scsi/scsi_transport_fc.c | 29 +++++++++++++++++++++++++++--
 1 file changed, 27 insertions(+), 2 deletions(-)

diff --git a/drivers/scsi/scsi_transport_fc.c b/drivers/scsi/scsi_transport_fc.c
index 72954a8..af1d35b 100644
--- a/drivers/scsi/scsi_transport_fc.c
+++ b/drivers/scsi/scsi_transport_fc.c
@@ -83,6 +83,21 @@ MODULE_PARM_DESC(disable_target_scan,
 		 "Disable target scan on remote ports (default=0)");
 
 /*
+ * async_user_scan: make 'scan' sysfs attribute asynchronous
+ *   on larger installations scanning can take a very long time
+ *   during which the process invoking the scan will be blocked
+ *   on writing to the 'scan' attribute. Enabling this attribute
+ *   will move scanning to a work queue, allowing the process
+ *   to return immediately.
+ */
+static bool fc_async_user_scan;
+
+module_param_named(async_user_scan, fc_async_user_scan,
+		   bool, S_IRUGO|S_IWUSR);
+MODULE_PARM_DESC(async_user_scan,
+		 "Allow for asynchronous user LUN scanning (default=0)");
+
+/*
  * Redefine so that we can have same named attributes in the
  * sdev/starget/host objects.
  */
@@ -2121,8 +2136,18 @@ fc_user_scan_tgt(struct Scsi_Host *shost, uint channel, uint id, uint lun)
 
 		if ((channel == rport->channel) &&
 		    (id == rport->scsi_target_id)) {
-			spin_unlock_irqrestore(shost->host_lock, flags);
-			scsi_scan_target(&rport->dev, channel, id, lun, 1);
+			if (lun == SCAN_WILD_CARD &&
+			    fc_async_user_scan) {
+				if (!(rport->flags & FC_RPORT_SCAN_PENDING)) {
+					rport->flags |= FC_RPORT_SCAN_PENDING;
+					scsi_queue_work(shost,
+							&rport->scan_work);
+				}
+				spin_unlock_irqrestore(shost->host_lock, flags);
+			} else {
+				spin_unlock_irqrestore(shost->host_lock, flags);
+				scsi_scan_target(&rport->dev, channel, id, lun, 1);
+			}
 			return;
 		}
 	}
-- 
1.8.5.6

Re: [PATCH 0/2] scsi_transport_fc: LUN masking

[Steffen Maier]

Hi Hannes,

On 11/11/2015 11:34 AM, Hannes Reinecke wrote:
> having been subjected to the pain of trying to bootstrap a really
> large machine with systemd I decided to implement LUN masking in

This is very interesting.

I also thought about initiator-based LUN masking a while ago.
While scsi_transport_fc would be sufficient to allow zfcp to get rid of 
its own implementation, I wonder why we wouldn't do it in the midlayer's 
scsi_scan.c. This would provide initiator-based LUN masking for any and 
all transports and LLDDs?

(I also thought about initiator-based zoning, and that would indeed be 
something for scsi_transport_fc because it is inherently transport 
specific.)

> scsi_transport_fc.
> The principle is simple: disallow the automated LUN scanning when
> discovering a rport, and create udev rules which selectively
> enable individual LUNs by echoing the relevant values in the 'scan'
> attribute of the SCSI host.

I imagined exactly the same existing user space interface.

Now we "only" need some user-friendly user space tooling, which provides 
transport specific user arguments automatically translated into the 
midlayer naming scheme, such as converting FC SAN addressing (HBA device 
bus-ID,WWPN,FCPLUN) into scsi_device names (Scsi_Host,0,scsi_id,scsi_lun),
and some persistency mechanism (such as managing udev rules).

> With that I'm able to boot an arbitrary large machine without
> running into any udev or systemd imposed timeout.
> To _disable_ LUN masking and restoring the original behaviour
> I've noticed that the 'scan' sysfs attribute is actually synchronous,
> ie the calling process will be blocked until the entire LUN scan
> is completed.
> So I've added another module parameter 'async_user_scan' to
> move the scanning onto the existing scan workqueue, and unblock
> the calling process.

Just curious: Which means would a user of async_user_scan have to sync 
against the completion?
Or is this only meant for user space--I currently cannot imagine--which 
does not care about the appearance of what was (newly) discovered?

>
> As usual, comments and reviews are welcome.
>
> Hannes Reinecke (2):
>    scsi_transport_fc: implement 'disable_target_scan' module parameter
>    scsi_transport_fc: Implement 'async_user_scan' module parameter
>
>   drivers/scsi/scsi_transport_fc.c | 47 +++++++++++++++++++++++++++++++++++++---
>   1 file changed, 44 insertions(+), 3 deletions(-)
>

-- 
Mit freundlichen Grüßen / Kind regards
Steffen Maier

Linux on z Systems Development

IBM Deutschland Research & Development GmbH
Vorsitzende des Aufsichtsrats: Martina Koederitz
Geschaeftsfuehrung: Dirk Wittkopp
Sitz der Gesellschaft: Boeblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294

--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/2] scsi_transport_fc: LUN masking

[Hannes Reinecke]

[-- Attachment #1: Type: text/plain, Size: 3669 bytes --]

On 11/12/2015 02:33 PM, Steffen Maier wrote:
> Hi Hannes,
> 
> On 11/11/2015 11:34 AM, Hannes Reinecke wrote:
>> having been subjected to the pain of trying to bootstrap a really
>> large machine with systemd I decided to implement LUN masking in
> 
> This is very interesting.
> 
> I also thought about initiator-based LUN masking a while ago.
> While scsi_transport_fc would be sufficient to allow zfcp to get rid
> of its own implementation, I wonder why we wouldn't do it in the
> midlayer's scsi_scan.c. This would provide initiator-based LUN
> masking for any and all transports and LLDDs?
> 
> (I also thought about initiator-based zoning, and that would indeed
> be something for scsi_transport_fc because it is inherently
> transport specific.)
> 
Well, it's not _that_ easy.

The one problem you're facing here is addressing; at one point
you'll have to identify which LUNs you'd like to unmask.
While it's true that the LUN itself is stable, the (logical) target
number from the SCSI-ID is not (disregarding SCSI parallel, of course).
The same goes for the 'bus' and 'host' number.

So how would you go about identifying them from the SCSI midlayer?

Hence my approach of moving the addressing into the transport layer,
where we _can_ identify the targets and hosts uniquely.

>> scsi_transport_fc.
>> The principle is simple: disallow the automated LUN scanning when
>> discovering a rport, and create udev rules which selectively
>> enable individual LUNs by echoing the relevant values in the 'scan'
>> attribute of the SCSI host.
> 
> I imagined exactly the same existing user space interface.
> 
> Now we "only" need some user-friendly user space tooling, which
> provides transport specific user arguments automatically translated
> into the midlayer naming scheme, such as converting FC SAN
> addressing (HBA device bus-ID,WWPN,FCPLUN) into scsi_device names
> (Scsi_Host,0,scsi_id,scsi_lun),
> and some persistency mechanism (such as managing udev rules).
> 
Oh, that's easy.
Attached is a simple dracut module which will enable LUN-masking
during booting.

>> With that I'm able to boot an arbitrary large machine without
>> running into any udev or systemd imposed timeout.
>> To _disable_ LUN masking and restoring the original behaviour
>> I've noticed that the 'scan' sysfs attribute is actually synchronous,
>> ie the calling process will be blocked until the entire LUN scan
>> is completed.
>> So I've added another module parameter 'async_user_scan' to
>> move the scanning onto the existing scan workqueue, and unblock
>> the calling process.
> 
> Just curious: Which means would a user of async_user_scan have to
> sync against the completion?
> Or is this only meant for user space--I currently cannot
> imagine--which does not care about the appearance of what was
> (newly) discovered?
> 
The problem I've been running into is that I have a machine where
LUN enumeration alone take 1800s (and no, that's not a typo).
So with the current implementation of 'scan' the calling process
would be stuck for that time, too.
If you were to try to rescan all LUNs during startup with say a
systemd service it would be killed halfway through as the
initialisation would take too long.
Plus any sanely defined service requiring any of those disks would
only startup once all disks are discovered, so there is no need to
wait for the discovery to finish.

Or that's the hope, at least :-)

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		               zSeries & Storage
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-95lunmask-Add-module-to-handle-LUN-masking.patch --]
[-- Type: text/x-patch; name="0001-95lunmask-Add-module-to-handle-LUN-masking.patch", Size: 4555 bytes --]

From 1784a28f26011ff0dd29b1c40494cadfa71df564 Mon Sep 17 00:00:00 2001
From: Hannes Reinecke <hare@suse.de>
Date: Wed, 11 Nov 2015 12:20:33 +0100
Subject: [PATCH] 95lunmask: Add module to handle LUN masking

Using the module option 'scsi_transport_fc.disable_target_scan'
this implements LUN masking by selectively enable only those
devices required for booting.

References: bsc#954600

Signed-off-by: Hannes Reinecke <hare@suse.de>
---
 modules.d/95lunmask/fc_transport_scan_lun.sh | 26 +++++++++++++
 modules.d/95lunmask/module-setup.sh          | 55 ++++++++++++++++++++++++++++
 modules.d/95lunmask/parse-lunmask.sh         | 28 ++++++++++++++
 3 files changed, 109 insertions(+)
 create mode 100755 modules.d/95lunmask/fc_transport_scan_lun.sh
 create mode 100755 modules.d/95lunmask/module-setup.sh
 create mode 100755 modules.d/95lunmask/parse-lunmask.sh

diff --git a/modules.d/95lunmask/fc_transport_scan_lun.sh b/modules.d/95lunmask/fc_transport_scan_lun.sh
new file mode 100755
index 0000000..d9f84a3
--- /dev/null
+++ b/modules.d/95lunmask/fc_transport_scan_lun.sh
@@ -0,0 +1,26 @@
+#!/bin/bash
+#
+# fc_transport_lun_scan
+#
+# Selectively enable individual LUNs behind an FC remote port
+#
+# ACTION=="add", SUBSYSTEM=="fc_transport", ATTR{port_name}=="wwpn", \
+#    PROGRAM="fc_transport_lun_scan lun"
+#
+
+[ -z $DEVPATH ] && exit 1
+
+if [ -n "$1" ] ; then
+    LUN=$1
+else
+    LUN=-
+fi
+ID=${DEVPATH##*/rport-}
+HOST=${ID%%:*}
+CHANNEL=${ID%%-*}
+CHANNEL=${CHANNEL#*:}
+if [ -f /sys$DEVPATH/scsi_target_id ] ; then
+    TARGET=$(cat /sys$DEVPATH/scsi_target_id)
+fi
+[ -z "$TARGET" ] && exit 1
+echo $CHANNEL $TARGET $LUN > /sys/class/scsi_host/host$HOST/scan
diff --git a/modules.d/95lunmask/module-setup.sh b/modules.d/95lunmask/module-setup.sh
new file mode 100755
index 0000000..d8d559b
--- /dev/null
+++ b/modules.d/95lunmask/module-setup.sh
@@ -0,0 +1,55 @@
+#!/bin/bash
+# -*- mode: shell-script; indent-tabs-mode: nil; sh-basic-offset: 4; -*-
+# ex: ts=8 sw=4 sts=4 et filetype=sh
+
+# called by dracut
+cmdline() {
+    get_lunmask() {
+        local _dev=$1
+        local _devpath=$(cd -P /sys/dev/block/$_dev ; echo $PWD)
+        local _sdev _lun _rport _classdev _wwpn
+
+        [ "${_devpath#*/sd}" == "$_devpath" ] && return 1
+        _sdev="${_devpath%%/block/*}"
+        _lun="${_sdev##*:}"
+        _rport="${_devpath##*/rport-}"
+        [ "$_rport" == "$_devpath" ] && return 1
+        _rport="${_rport%%/*}"
+        _classdev="/sys/class/fc_remote_ports/rport-${_rport}"
+        [ -d "$_classdev" ] || return 1
+        _wwpn=$(cat ${_classdev}/port_name)
+        echo "rd.lunmask=${_wwpn},${_lun}"
+        return 0
+    }
+    [[ $hostonly ]] || [[ $mount_needs ]] && {
+        for_each_host_dev_and_slaves_all get_lunmask
+    } | sort | uniq
+}
+
+# called by dracut
+check() {
+    [[ $hostonly ]] || [[ $mount_needs ]] && {
+        [ -f /sys/module/scsi_transport_fc/parameters/disable_target_scan ] \
+            && return 0
+        return 255
+    }
+    return 0
+}
+
+# called by dracut
+depends() {
+    return 0
+}
+
+# called by dracut
+install() {
+    inst_script "$moddir/fc_transport_scan_lun.sh" /usr/lib/udev/fc_transport_scan_lun.sh
+    inst_hook cmdline 30 "$moddir/parse-lunmask.sh"
+    if [[ $hostonly_cmdline == "yes" ]] ; then
+        local _lunmask
+
+        for _lunmask in $(cmdline) ; do
+            printf "%s\n" "$_lunmask" >> "${initdir}/etc/cmdline.d/95lunmask.conf"
+        done
+    fi
+}
diff --git a/modules.d/95lunmask/parse-lunmask.sh b/modules.d/95lunmask/parse-lunmask.sh
new file mode 100755
index 0000000..af990dd
--- /dev/null
+++ b/modules.d/95lunmask/parse-lunmask.sh
@@ -0,0 +1,28 @@
+#!/bin/sh
+# -*- mode: shell-script; indent-tabs-mode: nil; sh-basic-offset: 4; -*-
+# ex: ts=8 sw=4 sts=4 et filetype=sh
+
+create_udev_rule() {
+    local wwpn=$1
+    local lun=$2
+    local _rule=/etc/udev/rules.d/51-fc-lunmask-${wwpn}.rules
+    local _cu_type _dev_type
+
+    [ -e ${_rule} ] && return 0
+
+    if [ ! -f "$_rule" ] ; then
+        cat > $_rule <<EOF
+ACTION=="add", SUBSYSTEM=="fc_remote_ports", ATTR{port_name}=="$wwpn", PROGRAM="fc_transport_scan_lun.sh $lun"
+EOF
+    fi
+}
+
+for zfcp_arg in $(getargs rd.lunmask); do
+    (
+        local OLDIFS="$IFS"
+        local IFS=","
+        set $zfcp_arg
+        IFS="$OLDIFS"
+        create_udev_rule $1 $2
+    )
+done
-- 
1.8.4.5

Re: [PATCH 0/2] scsi_transport_fc: LUN masking

[Hannes Reinecke]

On 11/11/2015 11:34 AM, Hannes Reinecke wrote:
> Hi all,
> 
> having been subjected to the pain of trying to bootstrap a really
> large machine with systemd I decided to implement LUN masking in
> scsi_transport_fc.
> The principle is simple: disallow the automated LUN scanning when
> discovering a rport, and create udev rules which selectively
> enable individual LUNs by echoing the relevant values in the 'scan'
> attribute of the SCSI host.
> With that I'm able to boot an arbitrary large machine without
> running into any udev or systemd imposed timeout.
> To _disable_ LUN masking and restoring the original behaviour
> I've noticed that the 'scan' sysfs attribute is actually synchronous,
> ie the calling process will be blocked until the entire LUN scan
> is completed.
> So I've added another module parameter 'async_user_scan' to
> move the scanning onto the existing scan workqueue, and unblock
> the calling process.
> 
> As usual, comments and reviews are welcome.
> 
> Hannes Reinecke (2):
>   scsi_transport_fc: implement 'disable_target_scan' module parameter
>   scsi_transport_fc: Implement 'async_user_scan' module parameter
> 
>  drivers/scsi/scsi_transport_fc.c | 47 +++++++++++++++++++++++++++++++++++++---
>  1 file changed, 44 insertions(+), 3 deletions(-)
> 
Hmm. this seemed to have fallen through

-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/2] scsi_transport_fc: LUN masking

[Hannes Reinecke]

On 01/29/2016 03:57 PM, Hannes Reinecke wrote:
> On 11/11/2015 11:34 AM, Hannes Reinecke wrote:
>> Hi all,
>>
>> having been subjected to the pain of trying to bootstrap a really
>> large machine with systemd I decided to implement LUN masking in
>> scsi_transport_fc.
>> The principle is simple: disallow the automated LUN scanning when
>> discovering a rport, and create udev rules which selectively
>> enable individual LUNs by echoing the relevant values in the 'scan'
>> attribute of the SCSI host.
>> With that I'm able to boot an arbitrary large machine without
>> running into any udev or systemd imposed timeout.
>> To _disable_ LUN masking and restoring the original behaviour
>> I've noticed that the 'scan' sysfs attribute is actually synchronous,
>> ie the calling process will be blocked until the entire LUN scan
>> is completed.
>> So I've added another module parameter 'async_user_scan' to
>> move the scanning onto the existing scan workqueue, and unblock
>> the calling process.
>>
>> As usual, comments and reviews are welcome.
>>
>> Hannes Reinecke (2):
>>   scsi_transport_fc: implement 'disable_target_scan' module parameter
>>   scsi_transport_fc: Implement 'async_user_scan' module parameter
>>
>>  drivers/scsi/scsi_transport_fc.c | 47 +++++++++++++++++++++++++++++++++++++---
>>  1 file changed, 44 insertions(+), 3 deletions(-)
>>
> Hmm. this seemed to have fallen through
> 
Bah. Hit send too early.

Any chance of a review for this one?
Shall I resend the patches?

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/2] scsi_transport_fc: LUN masking

[Martin K. Petersen]

>>>>> "Hannes" == Hannes Reinecke <hare@suse.de> writes:

Hannes> Hmm. this seemed to have fallen through

-ENOREVIEWS

So, yes. Please resend.

-- 
Martin K. Petersen	Oracle Linux Engineering