OpenSM plugin patch request

OpenSM plugin patch request

[Karl Schulz]

[-- Attachment #1: Type: text/plain, Size: 503 bytes --]

Hello,

Would it be possible to consider exposing light-sweep events to the current OpenSM plugin interface?  Something along the lines of the attached patch is what we are using locally against opensm-3.3.9.

The motivation is to support a plugin module we have written to expose topology information to a query service which can be used by MPI stacks (in user-space) for topology-aware optimization and for topology-aware scheduling via batch systems. 

Thanks for the consideration,

Karl


[-- Attachment #2: opensm.patch --]
[-- Type: application/octet-stream, Size: 1316 bytes --]

--- opensm/osm_state_mgr.c	2011-08-08 18:36:38.000000000 -0500
+++ opensm/osm_state_mgr.c	2011-10-12 15:55:39.000000000 -0500
@@ -620,6 +620,11 @@
 	}
 
 	OSM_LOG_MSG_BOX(sm->p_log, OSM_LOG_VERBOSE, "INITIATING LIGHT SWEEP");
+
+	osm_opensm_report_event(sm->p_subn->p_osm,
+				OSM_EVENT_ID_LIGHT_SWEEP_START, NULL);
+
+
 	CL_PLOCK_ACQUIRE(sm->p_lock);
 	cl_qmap_apply_func(p_sw_tbl, state_mgr_get_sw_info, sm);
 	CL_PLOCK_RELEASE(sm->p_lock);
@@ -663,6 +668,9 @@
 
 	CL_PLOCK_RELEASE(sm->p_lock);
 
+	osm_opensm_report_event(sm->p_subn->p_osm,
+				OSM_EVENT_ID_LIGHT_SWEEP_DONE, NULL);
+
 _exit:
 	OSM_LOG_EXIT(sm->p_log);
 	return status;
--- include/opensm/osm_event_plugin.h	2011-08-08 18:40:19.000000000 -0500
+++ include/opensm/osm_event_plugin.h	2011-10-12 16:02:31.000000000 -0500
@@ -66,6 +66,7 @@
 /** =========================================================================
  * Event types
  */
+
 typedef enum {
 	OSM_EVENT_ID_PORT_ERRORS = 0,
 	OSM_EVENT_ID_PORT_DATA_COUNTERS,
@@ -77,7 +78,9 @@
 	OSM_EVENT_ID_UCAST_ROUTING_DONE,
 	OSM_EVENT_ID_STATE_CHANGE,
 	OSM_EVENT_ID_SA_DB_DUMPED,
-	OSM_EVENT_ID_MAX
+	OSM_EVENT_ID_MAX,
+	OSM_EVENT_ID_LIGHT_SWEEP_START,	
+	OSM_EVENT_ID_LIGHT_SWEEP_DONE,	
 } osm_epi_event_id_t;
 
 typedef struct osm_epi_port_id {

Re: OpenSM plugin patch request

[Ira Weiny]

I think this is a reasonable idea.

Comment on the patch.

-	OSM_EVENT_ID_MAX
+	OSM_EVENT_ID_MAX,
+	OSM_EVENT_ID_LIGHT_SWEEP_START,	
+	OSM_EVENT_ID_LIGHT_SWEEP_DONE,	

Traditionally the "MAX" value in the enum is kept at the end.

Ira


On Wed, 12 Oct 2011 14:31:07 -0700
Karl Schulz <karl-Tmo6s3XX3Xr5xfTjtVVUew@public.gmane.org> wrote:

> Hello,
> 
> Would it be possible to consider exposing light-sweep events to the current OpenSM plugin interface?  Something along the lines of the attached patch is what we are using locally against opensm-3.3.9.
> 
> The motivation is to support a plugin module we have written to expose topology information to a query service which can be used by MPI stacks (in user-space) for topology-aware optimization and for topology-aware scheduling via batch systems. 
> 
> Thanks for the consideration,
> 
> Karl
> 


-- 
Ira Weiny
Member of Technical Staff
Lawrence Livermore National Lab
925-423-8008
weiny2-i2BcT+NCU+M@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OpenSM plugin patch request

[Karl Schulz]

Thanks Ira,

My apologies for not following the conventional standard.  Should I generate a new patch?

-k

On Oct 12, 2011, at 5:13 PM, Ira Weiny wrote:

> I think this is a reasonable idea.
> 
> Comment on the patch.
> 
> -	OSM_EVENT_ID_MAX
> +	OSM_EVENT_ID_MAX,
> +	OSM_EVENT_ID_LIGHT_SWEEP_START,	
> +	OSM_EVENT_ID_LIGHT_SWEEP_DONE,	
> 
> Traditionally the "MAX" value in the enum is kept at the end.
> 
> Ira
> 
> 
> On Wed, 12 Oct 2011 14:31:07 -0700
> Karl Schulz <karl-Tmo6s3XX3Xr5xfTjtVVUew@public.gmane.org> wrote:
> 
>> Hello,
>> 
>> Would it be possible to consider exposing light-sweep events to the current OpenSM plugin interface?  Something along the lines of the attached patch is what we are using locally against opensm-3.3.9.
>> 
>> The motivation is to support a plugin module we have written to expose topology information to a query service which can be used by MPI stacks (in user-space) for topology-aware optimization and for topology-aware scheduling via batch systems. 
>> 
>> Thanks for the consideration,
>> 
>> Karl
>> 
> 
> 
> -- 
> Ira Weiny
> Member of Technical Staff
> Lawrence Livermore National Lab
> 925-423-8008
> weiny2-i2BcT+NCU+M@public.gmane.org

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OpenSM plugin patch request

[Yevgeny Kliteynik]

Hi Karl,


The idea in that if the received event is >= OSM_EVENT_ID_MAX,

then the plug-in knows that something went wrong (wrong version,

something changed in API, etc).


Also, you might want to know about more stuff if you're pulling
topology/routing info from SM, such as when SM became stand-by
and when it became master, and when re-route was done.

Please see this patch for reference:


http://www.mail-archive.com/linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org/msg02206.html


It wasn't applied as is back then, but you might want it now.


-- YK



On 13-Oct-11 4:50 AM, Karl Schulz wrote:

> Thanks Ira,
>
> My apologies for not following the conventional standard.  Should I generate a new patch?
>
> -k
>
> On Oct 12, 2011, at 5:13 PM, Ira Weiny wrote:
>
>> I think this is a reasonable idea.
>>
>> Comment on the patch.
>>
>> -	OSM_EVENT_ID_MAX
>> +	OSM_EVENT_ID_MAX,
>> +	OSM_EVENT_ID_LIGHT_SWEEP_START,	
>> +	OSM_EVENT_ID_LIGHT_SWEEP_DONE,	
>>
>> Traditionally the "MAX" value in the enum is kept at the end.
>>
>> Ira
>>
>>
>> On Wed, 12 Oct 2011 14:31:07 -0700
>> Karl Schulz<karl-Tmo6s3XX3Xr5xfTjtVVUew@public.gmane.org>  wrote:
>>
>>> Hello,
>>>
>>> Would it be possible to consider exposing light-sweep events to the current OpenSM plugin interface?  Something along the lines of the attached patch is what we are using locally against opensm-3.3.9.
>>>
>>> The motivation is to support a plugin module we have written to expose topology information to a query service which can be used by MPI stacks (in user-space) for topology-aware optimization and for topology-aware scheduling via batch systems.
>>>
>>> Thanks for the consideration,
>>>
>>> Karl
>>>
>>
>> -- 
>> Ira Weiny
>> Member of Technical Staff
>> Lawrence Livermore National Lab
>> 925-423-8008
>> weiny2-i2BcT+NCU+M@public.gmane.org
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OpenSM plugin patch request

[Karl Schulz]

[-- Attachment #1: Type: text/plain, Size: 2743 bytes --]

Thanks for explaining the enum convention - quite logical indeed.  Also, thanks for some of the off-list discussion where we tried to see if any of the current event callbacks could be used to achieve the desired support.  As it stands, it still sounds like including light sweep events is the simplest solution and I'm attaching an updated patch for consideration which honors the MAX convention.

Thanks,

Karl


On Oct 13, 2011, at 7:50 AM, Yevgeny Kliteynik wrote:

> Hi Karl,
> 
> 
> The idea in that if the received event is >= OSM_EVENT_ID_MAX,
> 
> then the plug-in knows that something went wrong (wrong version,
> 
> something changed in API, etc).
> 
> 
> Also, you might want to know about more stuff if you're pulling
> topology/routing info from SM, such as when SM became stand-by
> and when it became master, and when re-route was done.
> 
> Please see this patch for reference:
> 
> 
> http://www.mail-archive.com/linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org/msg02206.html
> 
> 
> It wasn't applied as is back then, but you might want it now.
> 
> 
> -- YK
> 
> 
> 
> On 13-Oct-11 4:50 AM, Karl Schulz wrote:
> 
>> Thanks Ira,
>> 
>> My apologies for not following the conventional standard.  Should I generate a new patch?
>> 
>> -k
>> 
>> On Oct 12, 2011, at 5:13 PM, Ira Weiny wrote:
>> 
>>> I think this is a reasonable idea.
>>> 
>>> Comment on the patch.
>>> 
>>> -   OSM_EVENT_ID_MAX
>>> +   OSM_EVENT_ID_MAX,
>>> +   OSM_EVENT_ID_LIGHT_SWEEP_START, 
>>> +   OSM_EVENT_ID_LIGHT_SWEEP_DONE,  
>>> 
>>> Traditionally the "MAX" value in the enum is kept at the end.
>>> 
>>> Ira
>>> 
>>> 
>>> On Wed, 12 Oct 2011 14:31:07 -0700
>>> Karl Schulz<karl-Tmo6s3XX3Xr5xfTjtVVUew@public.gmane.org>  wrote:
>>> 
>>>> Hello,
>>>> 
>>>> Would it be possible to consider exposing light-sweep events to the current OpenSM plugin interface?  Something along the lines of the attached patch is what we are using locally against opensm-3.3.9.
>>>> 
>>>> The motivation is to support a plugin module we have written to expose topology information to a query service which can be used by MPI stacks (in user-space) for topology-aware optimization and for topology-aware scheduling via batch systems.
>>>> 
>>>> Thanks for the consideration,
>>>> 
>>>> Karl
>>>> 
>>> 
>>> -- 
>>> Ira Weiny
>>> Member of Technical Staff
>>> Lawrence Livermore National Lab
>>> 925-423-8008
>>> weiny2-i2BcT+NCU+M@public.gmane.org
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> 


[-- Attachment #2: opensm.patch --]
[-- Type: application/octet-stream, Size: 1316 bytes --]

--- opensm/osm_state_mgr.c	2011-08-08 18:36:38.000000000 -0500
+++ opensm/osm_state_mgr.c	2011-10-12 16:18:43.000000000 -0500
@@ -620,6 +620,11 @@
 	}
 
 	OSM_LOG_MSG_BOX(sm->p_log, OSM_LOG_VERBOSE, "INITIATING LIGHT SWEEP");
+
+	osm_opensm_report_event(sm->p_subn->p_osm,
+				OSM_EVENT_ID_LIGHT_SWEEP_START, NULL);
+
+
 	CL_PLOCK_ACQUIRE(sm->p_lock);
 	cl_qmap_apply_func(p_sw_tbl, state_mgr_get_sw_info, sm);
 	CL_PLOCK_RELEASE(sm->p_lock);
@@ -663,6 +668,9 @@
 
 	CL_PLOCK_RELEASE(sm->p_lock);
 
+	osm_opensm_report_event(sm->p_subn->p_osm,
+				OSM_EVENT_ID_LIGHT_SWEEP_DONE, NULL);
+
 _exit:
 	OSM_LOG_EXIT(sm->p_log);
 	return status;
--- include/opensm/osm_event_plugin.h	2011-08-08 18:40:19.000000000 -0500
+++ include/opensm/osm_event_plugin.h	2011-10-17 16:10:43.000000000 -0500
@@ -66,6 +66,7 @@
 /** =========================================================================
  * Event types
  */
+
 typedef enum {
 	OSM_EVENT_ID_PORT_ERRORS = 0,
 	OSM_EVENT_ID_PORT_DATA_COUNTERS,
@@ -77,7 +78,9 @@
 	OSM_EVENT_ID_UCAST_ROUTING_DONE,
 	OSM_EVENT_ID_STATE_CHANGE,
 	OSM_EVENT_ID_SA_DB_DUMPED,
-	OSM_EVENT_ID_MAX
+	OSM_EVENT_ID_LIGHT_SWEEP_START,	
+	OSM_EVENT_ID_LIGHT_SWEEP_DONE,	
+	OSM_EVENT_ID_MAX,
 } osm_epi_event_id_t;
 
 typedef struct osm_epi_port_id {

Re: OpenSM plugin patch request

[Hal Rosenstock]

Hi Karl,

On 10/17/2011 5:43 PM, Karl Schulz wrote:
> Thanks for explaining the enum convention - quite logical indeed.  Also, thanks for some 
> of the off-list discussion where we tried to see if any of the current event callbacks 
> could be used to achieve the desired support.  As it stands, it still sounds like including 
> light sweep events is the simplest solution and I'm attaching an updated patch for consideration which honors the MAX convention.

A couple of minor comments:

1. Why not move the OSM_EVENT_ID_LIGHT_SWEEP_DONE event to where "LIGHT
SWEEP COMPLETE" is indicated in osm_state_mgr.c ? That's actually more
accurate as all the transactions have completed.

2. The new events should be added to osmeventplugin/src/osmeventplugin.c
example.

Thanks.

-- Hal

> Thanks,
> 
> Karl
> 
> 
> On Oct 13, 2011, at 7:50 AM, Yevgeny Kliteynik wrote:
> 
>> Hi Karl,
>>
>>
>> The idea in that if the received event is >= OSM_EVENT_ID_MAX,
>>
>> then the plug-in knows that something went wrong (wrong version,
>>
>> something changed in API, etc).
>>
>>
>> Also, you might want to know about more stuff if you're pulling
>> topology/routing info from SM, such as when SM became stand-by
>> and when it became master, and when re-route was done.
>>
>> Please see this patch for reference:
>>
>>
>> http://www.mail-archive.com/linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org/msg02206.html
>>
>>
>> It wasn't applied as is back then, but you might want it now.
>>
>>
>> -- YK
>>
>>
>>
>> On 13-Oct-11 4:50 AM, Karl Schulz wrote:
>>
>>> Thanks Ira,
>>>
>>> My apologies for not following the conventional standard.  Should I generate a new patch?
>>>
>>> -k
>>>
>>> On Oct 12, 2011, at 5:13 PM, Ira Weiny wrote:
>>>
>>>> I think this is a reasonable idea.
>>>>
>>>> Comment on the patch.
>>>>
>>>> -   OSM_EVENT_ID_MAX
>>>> +   OSM_EVENT_ID_MAX,
>>>> +   OSM_EVENT_ID_LIGHT_SWEEP_START, 
>>>> +   OSM_EVENT_ID_LIGHT_SWEEP_DONE,  
>>>>
>>>> Traditionally the "MAX" value in the enum is kept at the end.
>>>>
>>>> Ira
>>>>
>>>>
>>>> On Wed, 12 Oct 2011 14:31:07 -0700
>>>> Karl Schulz<karl-Tmo6s3XX3Xr5xfTjtVVUew@public.gmane.org>  wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> Would it be possible to consider exposing light-sweep events to the current OpenSM plugin interface?  Something along the lines of the attached patch is what we are using locally against opensm-3.3.9.
>>>>>
>>>>> The motivation is to support a plugin module we have written to expose topology information to a query service which can be used by MPI stacks (in user-space) for topology-aware optimization and for topology-aware scheduling via batch systems.
>>>>>
>>>>> Thanks for the consideration,
>>>>>
>>>>> Karl
>>>>>
>>>>
>>>> -- 
>>>> Ira Weiny
>>>> Member of Technical Staff
>>>> Lawrence Livermore National Lab
>>>> 925-423-8008
>>>> weiny2-i2BcT+NCU+M@public.gmane.org
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OpenSM plugin patch request

[Karl Schulz]

[-- Attachment #1: Type: text/plain, Size: 1218 bytes --]

On Oct 17, 2011, at 5:28 PM, Hal Rosenstock wrote:

> 
> A couple of minor comments:
> 
> 1. Why not move the OSM_EVENT_ID_LIGHT_SWEEP_DONE event to where "LIGHT
> SWEEP COMPLETE" is indicated in osm_state_mgr.c ? That's actually more
> accurate as all the transactions have completed.
> 
> 2. The new events should be added to osmeventplugin/src/osmeventplugin.c
> example.
> 
> Thanks.
> 
> -- Hal
> 

Thanks for the comments.  The attached osm_light_sweep.patch addresses the points above by moving the OSM_EVENT_ID_LIGHT_SWEEP_DONE event to be with it's corresponding LIGHT SWEEP COMPLETE log message.  Also, the light sweep START/DONE events are included in the example event plugin. 

While adding to the example event plugin, I realized that the autoconf setup in 3.3.9 does not appear to be propagating all user settings down into the default plugin.  Consequently, I was not able to build the plugin when OFED headers were not in the default location.   In case it's useful (and hasn't already been addressed), the second attached patch adds  $(OSMV_INCLUDES) to the automake directives for the plugin which was sufficient in my case to resolve necessary headers.

Thanks,

Karl




[-- Attachment #2: osm_light_sweep.patch --]
[-- Type: application/octet-stream, Size: 1738 bytes --]

--- include/opensm/osm_event_plugin.h	2011-08-08 18:40:19.000000000 -0500
+++ include/opensm/osm_event_plugin.h	2011-10-24 08:13:05.000000000 -0500
@@ -77,7 +77,9 @@
 	OSM_EVENT_ID_UCAST_ROUTING_DONE,
 	OSM_EVENT_ID_STATE_CHANGE,
 	OSM_EVENT_ID_SA_DB_DUMPED,
-	OSM_EVENT_ID_MAX
+	OSM_EVENT_ID_LIGHT_SWEEP_START,
+	OSM_EVENT_ID_LIGHT_SWEEP_DONE,
+	OSM_EVENT_ID_MAX,
 } osm_epi_event_id_t;

 typedef struct osm_epi_port_id {
--- opensm/osm_state_mgr.c.orig	2011-08-08 18:36:38.000000000 -0500
+++ opensm/osm_state_mgr.c	2011-10-24 08:57:48.000000000 -0500
@@ -620,6 +620,9 @@
 	}

 	OSM_LOG_MSG_BOX(sm->p_log, OSM_LOG_VERBOSE, "INITIATING LIGHT SWEEP");
+	osm_opensm_report_event(sm->p_subn->p_osm,
+				OSM_EVENT_ID_LIGHT_SWEEP_START, NULL);
+
 	CL_PLOCK_ACQUIRE(sm->p_lock);
 	cl_qmap_apply_func(p_sw_tbl, state_mgr_get_sw_info, sm);
 	CL_PLOCK_RELEASE(sm->p_lock);
@@ -1113,6 +1116,8 @@
 					OSM_EVENT_ID_SA_DB_DUMPED, NULL);
 			OSM_LOG_MSG_BOX(sm->p_log, OSM_LOG_VERBOSE,
 					"LIGHT SWEEP COMPLETE");
+			osm_opensm_report_event(sm->p_subn->p_osm,
+						OSM_EVENT_ID_LIGHT_SWEEP_DONE, NULL);
 			return;
 		}
 	}
--- osmeventplugin/src/osmeventplugin.c	2011-10-24 08:05:04.000000000 -0500
+++ osmeventplugin/src/osmeventplugin.c	2011-10-24 08:05:55.000000000 -0500
@@ -176,6 +176,12 @@
 	case OSM_EVENT_ID_SUBNET_UP:
 		fprintf(log->log_file, "Subnet up reported\n");
 		break;
+	case OSM_EVENT_ID_LIGHT_SWEEP_START:
+		fprintf(log->log_file, "Light sweep started\n");
+		break;
+	case OSM_EVENT_ID_LIGHT_SWEEP_DONE:
+		fprintf(log->log_file, "Light sweep completed\n");
+		break;
 	case OSM_EVENT_ID_HEAVY_SWEEP_START:
 		fprintf(log->log_file, "Heavy sweep started\n");
 		break;

[-- Attachment #3: build.patch --]
[-- Type: application/octet-stream, Size: 324 bytes --]

--- osmeventplugin/Makefile.am	2011-10-24 09:21:25.000000000 -0500
+++ osmeventplugin/Makefile.am	2011-10-24 09:18:00.000000000 -0500
@@ -1,6 +1,6 @@
 
 INCLUDES = -I$(srcdir)/../include \
-	   -I$(includedir)/infiniband
+	   -I$(includedir)/infiniband $(OSMV_INCLUDES)
 
 lib_LTLIBRARIES = libosmeventplugin.la