Upcall prioritizer for multipath

Upcall prioritizer for multipath

[Oliver Mangold]

[-- Attachment #1: Type: text/plain, Size: 989 bytes --]

I'm working with JBODs and would like to access the disks inside through 
dm-multipath. For performance reasons I would like to use to prioritizer 
to pin individual disks to different paths. For testing purposes this 
can be done with the weighted_path prioritizer, but for production use 
this is unsatisfactory, as neither the device name nor the SCSI address 
are persistent over reboots. Thus I wrote a new, simple prioritizer to 
call an external program to determine what the prio should be. I would 
like to ask if there is interest to include this (or something similar) 
upstream (see attached patch).

Remark: After I did this, I've seen the wwn regex mode in the upcoming 
0.6 version, which would also provide a solution to the problem I 
started out with. But using this would still means one had to recreate 
multipath.conf each time one replaces a failed disk, so I still believe 
having an external upcall makes sense in some situations.

Any thoughts?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-libmultipath-adding-new-prioritizer-upcall-to-call-e.patch --]
[-- Type: text/x-patch; name="0001-libmultipath-adding-new-prioritizer-upcall-to-call-e.patch", Size: 2431 bytes --]

From 124b54531dd1859755175ca5e9e4882d15cdd87d Mon Sep 17 00:00:00 2001
From: Oliver Mangold <oliver.mangold@emea.nec.com>
Date: Fri, 22 Apr 2016 12:53:55 +0200
Subject: [PATCH] libmultipath: adding new prioritizer 'upcall' to call
 external program

---
 libmultipath/prioritizers/Makefile |  1 +
 libmultipath/prioritizers/upcall.c | 25 +++++++++++++++++++++++++
 multipath/multipath.conf.5         |  6 ++++++
 3 files changed, 32 insertions(+)
 create mode 100644 libmultipath/prioritizers/upcall.c

diff --git a/libmultipath/prioritizers/Makefile b/libmultipath/prioritizers/Makefile
index 6cfac88..1b32d00 100644
--- a/libmultipath/prioritizers/Makefile
+++ b/libmultipath/prioritizers/Makefile
@@ -15,6 +15,7 @@ LIBS = \
 	libpriodatacore.so \
 	libpriohds.so \
 	libprioweightedpath.so \
+	libprioupcall.so \
 	libprioiet.so
 
 CFLAGS += -I..
diff --git a/libmultipath/prioritizers/upcall.c b/libmultipath/prioritizers/upcall.c
new file mode 100644
index 0000000..175b8fe
--- /dev/null
+++ b/libmultipath/prioritizers/upcall.c
@@ -0,0 +1,25 @@
+#define _GNU_SOURCE
+#include <stdio.h>
+#include <stdlib.h>
+#include <time.h>
+
+#include <prio.h>
+#include <structs.h>
+
+int getprio (struct path * pp, char * args)
+{
+    char* cmd;
+    int ret = asprintf(&cmd, "%s %s", args, pp->dev);
+    if (ret == -1)
+      return -1;
+    FILE* pipe = popen( cmd, "r" );
+    free(cmd);
+    int prio;
+    int count = fscanf(pipe, "%i", &prio);
+    ret = pclose(pipe);
+    if (ret != 0)
+      return -1;
+    if (count != 1)
+        return -1;
+    return prio;
+}
diff --git a/multipath/multipath.conf.5 b/multipath/multipath.conf.5
index 0d4df0f..d8d6ad6 100644
--- a/multipath/multipath.conf.5
+++ b/multipath/multipath.conf.5
@@ -217,6 +217,9 @@ Generate a random priority between 1 and 10.
 Generate the path priority based on the regular expression and the 
 priority provided as argument. requires prio_args keyword.
 .TP
+.B upcall
+Call an external program to calculate the priority.
+.TP
 Default value is \fBnone\fR.
 .RE
 .TP
@@ -243,6 +246,9 @@ If
 .I exclusive_pref_bit
 is set, paths with the TPGS pref bit set will always be in their own path
 group.
+.TP
+.B upcall
+Cmdline to execute to determine device priority. The name of the component device of interest is added as the last argument.
 .RE
 .TP
 .B features
-- 
2.8.0


[-- Attachment #3: Type: text/plain, Size: 0 bytes --]

Upcall prioritizer for multipath

[Oliver Mangold]

[-- Attachment #1: Type: text/plain, Size: 989 bytes --]

I'm working with JBODs and would like to access the disks inside through 
dm-multipath. For performance reasons I would like to use to prioritizer 
to pin individual disks to different paths. For testing purposes this 
can be done with the weighted_path prioritizer, but for production use 
this is unsatisfactory, as neither the device name nor the SCSI address 
are persistent over reboots. Thus I wrote a new, simple prioritizer to 
call an external program to determine what the prio should be. I would 
like to ask if there is interest to include this (or something similar) 
upstream (see attached patch).

Remark: After I did this, I've seen the wwn regex mode in the upcoming 
0.6 version, which would also provide a solution to the problem I 
started out with. But using this would still means one had to recreate 
multipath.conf each time one replaces a failed disk, so I still believe 
having an external upcall makes sense in some situations.

Any thoughts?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-libmultipath-adding-new-prioritizer-upcall-to-call-e.patch --]
[-- Type: text/x-patch; name="0001-libmultipath-adding-new-prioritizer-upcall-to-call-e.patch", Size: 2433 bytes --]

From 124b54531dd1859755175ca5e9e4882d15cdd87d Mon Sep 17 00:00:00 2001
From: Oliver Mangold <oliver.mangold@emea.nec.com>
Date: Fri, 22 Apr 2016 12:53:55 +0200
Subject: [PATCH] libmultipath: adding new prioritizer 'upcall' to call
 external program

---
 libmultipath/prioritizers/Makefile |  1 +
 libmultipath/prioritizers/upcall.c | 25 +++++++++++++++++++++++++
 multipath/multipath.conf.5         |  6 ++++++
 3 files changed, 32 insertions(+)
 create mode 100644 libmultipath/prioritizers/upcall.c

diff --git a/libmultipath/prioritizers/Makefile b/libmultipath/prioritizers/Makefile
index 6cfac88..1b32d00 100644
--- a/libmultipath/prioritizers/Makefile
+++ b/libmultipath/prioritizers/Makefile
@@ -15,6 +15,7 @@ LIBS = \
 	libpriodatacore.so \
 	libpriohds.so \
 	libprioweightedpath.so \
+	libprioupcall.so \
 	libprioiet.so
 
 CFLAGS += -I..
diff --git a/libmultipath/prioritizers/upcall.c b/libmultipath/prioritizers/upcall.c
new file mode 100644
index 0000000..175b8fe
--- /dev/null
+++ b/libmultipath/prioritizers/upcall.c
@@ -0,0 +1,25 @@
+#define _GNU_SOURCE
+#include <stdio.h>
+#include <stdlib.h>
+#include <time.h>
+
+#include <prio.h>
+#include <structs.h>
+
+int getprio (struct path * pp, char * args)
+{
+    char* cmd;
+    int ret = asprintf(&cmd, "%s %s", args, pp->dev);
+    if (ret == -1)
+      return -1;
+    FILE* pipe = popen( cmd, "r" );
+    free(cmd);
+    int prio;
+    int count = fscanf(pipe, "%i", &prio);
+    ret = pclose(pipe);
+    if (ret != 0)
+      return -1;
+    if (count != 1)
+        return -1;
+    return prio;
+}
diff --git a/multipath/multipath.conf.5 b/multipath/multipath.conf.5
index 0d4df0f..d8d6ad6 100644
--- a/multipath/multipath.conf.5
+++ b/multipath/multipath.conf.5
@@ -217,6 +217,9 @@ Generate a random priority between 1 and 10.
 Generate the path priority based on the regular expression and the 
 priority provided as argument. requires prio_args keyword.
 .TP
+.B upcall
+Call an external program to calculate the priority.
+.TP
 Default value is \fBnone\fR.
 .RE
 .TP
@@ -243,6 +246,9 @@ If
 .I exclusive_pref_bit
 is set, paths with the TPGS pref bit set will always be in their own path
 group.
+.TP
+.B upcall
+Cmdline to execute to determine device priority. The name of the component device of interest is added as the last argument.
 .RE
 .TP
 .B features
-- 
2.8.0



[-- Attachment #3: Type: text/plain, Size: 0 bytes --]

Re: Upcall prioritizer for multipath

[Hannes Reinecke]

On 04/22/2016 02:48 PM, Oliver Mangold wrote:
> I'm working with JBODs and would like to access the disks inside through 
> dm-multipath. For performance reasons I would like to use to prioritizer 
> to pin individual disks to different paths. For testing purposes this 
> can be done with the weighted_path prioritizer, but for production use 
> this is unsatisfactory, as neither the device name nor the SCSI address 
> are persistent over reboots. Thus I wrote a new, simple prioritizer to 
> call an external program to determine what the prio should be. I would 
> like to ask if there is interest to include this (or something similar) 
> upstream (see attached patch).
> 
> Remark: After I did this, I've seen the wwn regex mode in the upcoming 
> 0.6 version, which would also provide a solution to the problem I 
> started out with. But using this would still means one had to recreate 
> multipath.conf each time one replaces a failed disk, so I still believe 
> having an external upcall makes sense in some situations.
> 
> Any thoughts?
> 
Ah, the good old days.
You do remember (of course, as you as a diligent developer would
have had a look through the archives, right?) that we original did
precisely that, namely using 'prio' as an external callout.

The reason why we moved away from this is a deadlock:
When all paths are down (ie the _one_ situation where you absolutely
_need_ multipath to work) you cannot reinstate any paths, as for
reinstating you would need to call the prio program.
Which happens to be an external program, which would need to be
loaded from disk.
Which is not available.
Say goodbye to your machine :-(

So we've moved to using a shared library mechanism, which can be
mlock()ed at boot to avoid these situations.

Hence I would really _not_ use this approach.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)