[PATCH 0/3] staging: lustre: lnet: bug fixs for 4.7-rc2

[PATCH 0/3] staging: lustre: lnet: bug fixs for 4.7-rc2

[James Simmons]

Here are the latest fixes for LNet. One fix covers a bug
for the infiniband driver that happens when the o2iblnd
intereface is pinged before it is finishing setting up.
The next patch set from Bruno adds kmem_caches for the
LNet layer.

Bruno Faccini (2):
  staging: lustre: lnet: Allocate MEs and small MDs in own kmem_caches
  staging: lustre: lnet: optimize memory foot print for lnet_libmd

Doug Oucharek (1):
  staging: lustre: lnet: Don't access NULL NI on failure path

 .../staging/lustre/include/linux/lnet/lib-lnet.h   |   36 ++++++++++++++--
 .../staging/lustre/include/linux/lnet/lib-types.h  |    2 +-
 .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c |    7 ++-
 drivers/staging/lustre/lnet/lnet/api-ni.c          |   45 ++++++++++++++++++++
 4 files changed, 82 insertions(+), 8 deletions(-)

[PATCH 1/3] staging: lustre: lnet: Don't access NULL NI on failure path

[James Simmons]

From: Doug Oucharek <doug.s.oucharek@intel.com>

In kiblnd_passive_connect(), if we are failing the connection
attempt because we cannot find a valid NI (we have a NULL NI),
we were coring after the "goto fail" because the failure
path was assuming non-NULL NI.

This patch ensures we don't dereference a NULL NI on that
failure path.

Signed-off-by: Doug Oucharek <doug.s.oucharek@intel.com>
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-8022
Reviewed-on: http://review.whamcloud.com/19614
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Matt Ezell <ezellma@ornl.gov>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
---
 .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c |    7 ++++---
 1 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
index 0f7e3a1..dbc26f1 100644
--- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
+++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
@@ -2529,12 +2529,13 @@ kiblnd_passive_connect(struct rdma_cm_id *cmid, void *priv, int priv_nob)
 	return 0;
 
  failed:
-	if (ni)
+	if (ni) {
 		lnet_ni_decref(ni);
+		rej.ibr_cp.ibcp_queue_depth = kiblnd_msg_queue_size(version, ni);
+		rej.ibr_cp.ibcp_max_frags = kiblnd_rdma_frags(version, ni);
+	}
 
 	rej.ibr_version             = version;
-	rej.ibr_cp.ibcp_queue_depth = kiblnd_msg_queue_size(version, ni);
-	rej.ibr_cp.ibcp_max_frags = kiblnd_rdma_frags(version, ni);
 	kiblnd_reject(cmid, &rej);
 
 	return -ECONNREFUSED;
-- 
1.7.1

[PATCH 2/3] staging: lustre: lnet: Allocate MEs and small MDs in own kmem_caches

[James Simmons]

From: Bruno Faccini <bruno.faccini@intel.com>

As part of LU-3848 and LU-4330, it has been discovered that LNET
MEs and small MDs (<=128 Bytes) are allocated in <size-128> kmem_cache
and thus can suffer quite frequent corruptions, from other modules or
Kernel parts, that occur there. To avoid this, MEs and small-MDs
specific kmem_cache have been created.

Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-4430
Reviewed-on: http://review.whamcloud.com/18586
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Doug Oucharek <doug.s.oucharek@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Signed-off-by: James Simmons <jsimmons@infradead.org>
---
 .../staging/lustre/include/linux/lnet/lib-lnet.h   |   36 ++++++++++++++--
 drivers/staging/lustre/lnet/lnet/api-ni.c          |   45 ++++++++++++++++++++
 2 files changed, 77 insertions(+), 4 deletions(-)

diff --git a/drivers/staging/lustre/include/linux/lnet/lib-lnet.h b/drivers/staging/lustre/include/linux/lnet/lib-lnet.h
index 513a822..51ad729 100644
--- a/drivers/staging/lustre/include/linux/lnet/lib-lnet.h
+++ b/drivers/staging/lustre/include/linux/lnet/lib-lnet.h
@@ -178,6 +178,11 @@ lnet_net_lock_current(void)
 
 #define MAX_PORTALS		64
 
+#define LNET_SMALL_MD_SIZE	offsetof(lnet_libmd_t, md_iov.iov[1])
+extern struct kmem_cache *lnet_mes_cachep;	/* MEs kmem_cache */
+extern struct kmem_cache *lnet_small_mds_cachep;/* <= LNET_SMALL_MD_SIZE bytes
+						 * MDs kmem_cache
+						 */
 static inline lnet_eq_t *
 lnet_eq_alloc(void)
 {
@@ -208,7 +213,19 @@ lnet_md_alloc(lnet_md_t *umd)
 		size = offsetof(lnet_libmd_t, md_iov.iov[niov]);
 	}
 
-	LIBCFS_ALLOC(md, size);
+	if (size <= LNET_SMALL_MD_SIZE) {
+		md = kmem_cache_alloc(lnet_small_mds_cachep,
+				      GFP_NOFS | __GFP_ZERO);
+		if (md) {
+			CDEBUG(D_MALLOC, "slab-alloced 'md' of size %u at %p.\n",
+			       size, md);
+		} else {
+			CDEBUG(D_MALLOC, "failed to allocate 'md' of size %u\n",
+			       size);
+		}
+	} else {
+		LIBCFS_ALLOC(md, size);
+	}
 
 	if (md) {
 		/* Set here in case of early free */
@@ -230,7 +247,12 @@ lnet_md_free(lnet_libmd_t *md)
 	else
 		size = offsetof(lnet_libmd_t, md_iov.iov[md->md_niov]);
 
-	LIBCFS_FREE(md, size);
+	if (size <= LNET_SMALL_MD_SIZE) {
+		CDEBUG(D_MALLOC, "slab-freed 'md' at %p.\n", md);
+		kmem_cache_free(lnet_small_mds_cachep, md);
+	} else {
+		LIBCFS_FREE(md, size);
+	}
 }
 
 static inline lnet_me_t *
@@ -238,14 +260,20 @@ lnet_me_alloc(void)
 {
 	lnet_me_t *me;
 
-	LIBCFS_ALLOC(me, sizeof(*me));
+	me = kmem_cache_alloc(lnet_mes_cachep, GFP_NOFS | __GFP_ZERO);
+	if (me)
+		CDEBUG(D_MALLOC, "slab-alloced 'me' at %p.\n", me);
+	else
+		CDEBUG(D_MALLOC, "failed to allocate 'me'\n");
+
 	return me;
 }
 
 static inline void
 lnet_me_free(lnet_me_t *me)
 {
-	LIBCFS_FREE(me, sizeof(*me));
+	CDEBUG(D_MALLOC, "slab-freed 'me' at %p.\n", me);
+	kmem_cache_free(lnet_mes_cachep, me);
 }
 
 static inline lnet_msg_t *
diff --git a/drivers/staging/lustre/lnet/lnet/api-ni.c b/drivers/staging/lustre/lnet/lnet/api-ni.c
index fe0dbe7..9db0ff1 100644
--- a/drivers/staging/lustre/lnet/lnet/api-ni.c
+++ b/drivers/staging/lustre/lnet/lnet/api-ni.c
@@ -103,6 +103,46 @@ lnet_init_locks(void)
 	mutex_init(&the_lnet.ln_api_mutex);
 }
 
+struct kmem_cache *lnet_mes_cachep;		/* MEs kmem_cache */
+struct kmem_cache *lnet_small_mds_cachep;	/* <= LNET_SMALL_MD_SIZE bytes
+						 *  MDs kmem_cache
+						 */
+static int
+lnet_descriptor_setup(void)
+{
+	/*
+	 * create specific kmem_cache for MEs and small MDs (i.e., originally
+	 * allocated in <size-xxx> kmem_cache).
+	 */
+	lnet_mes_cachep = kmem_cache_create("lnet_MEs", sizeof(lnet_me_t),
+					    0, 0, NULL);
+	if (!lnet_mes_cachep)
+		return -ENOMEM;
+
+	lnet_small_mds_cachep = kmem_cache_create("lnet_small_MDs",
+						  LNET_SMALL_MD_SIZE, 0, 0,
+						  NULL);
+	if (!lnet_small_mds_cachep)
+		return -ENOMEM;
+
+	return 0;
+}
+
+static void
+lnet_descriptor_cleanup(void)
+{
+
+	if (lnet_small_mds_cachep) {
+		kmem_cache_destroy(lnet_small_mds_cachep);
+		lnet_small_mds_cachep = NULL;
+	}
+
+	if (lnet_mes_cachep) {
+		kmem_cache_destroy(lnet_mes_cachep);
+		lnet_mes_cachep = NULL;
+	}
+}
+
 static int
 lnet_create_remote_nets_table(void)
 {
@@ -553,6 +593,10 @@ lnet_prepare(lnet_pid_t requested_pid)
 	INIT_LIST_HEAD(&the_lnet.ln_drop_rules);
 	INIT_LIST_HEAD(&the_lnet.ln_delay_rules);
 
+	rc = lnet_descriptor_setup();
+	if (rc)
+		goto failed;
+
 	rc = lnet_create_remote_nets_table();
 	if (rc)
 		goto failed;
@@ -652,6 +696,7 @@ lnet_unprepare(void)
 		the_lnet.ln_counters = NULL;
 	}
 	lnet_destroy_remote_nets_table();
+	lnet_descriptor_cleanup();
 
 	return 0;
 }
-- 
1.7.1

Re: [PATCH 2/3] staging: lustre: lnet: Allocate MEs and small MDs in own kmem_caches

[Greg Kroah-Hartman]

On Thu, Jun 09, 2016 at 06:45:46PM -0400, James Simmons wrote:
> From: Bruno Faccini <bruno.faccini@intel.com>
> 
> As part of LU-3848 and LU-4330, it has been discovered that LNET
> MEs and small MDs (<=128 Bytes) are allocated in <size-128> kmem_cache
> and thus can suffer quite frequent corruptions, from other modules or
> Kernel parts, that occur there. To avoid this, MEs and small-MDs
> specific kmem_cache have been created.

What?  Who corrupts them?  That shouldn't be possible, and on some
systems, even if you do ask for a separate slab, it will be merged
togther with others of the same size.  So this patch doesn't do all that
much.

I think you are having some other problem here, changing to a separate
memory cache shouldn't solve corruption issues.

sorry,

greg k-h

Re: [PATCH 2/3] staging: lustre: lnet: Allocate MEs and small MDs in own kmem_caches

[Faccini, Bruno]

Hello,
The intent of this patch is not to solve the corruptions for sure, but only to avoid the concerned MEs/small-MDs LNet structs to be quite frequently impacted due to their high allocation/free rate. This may also possibly help to save cycles due to high usage and contention when using a generic kmem_cache (when they stay separate from others, thanks for the precision!).
Bye,
Bruno.

> Le Jun 10, 2016 à 03:28, Greg Kroah-Hartman <gregkh@linuxfoundation.org> a écrit :
> 
> On Thu, Jun 09, 2016 at 06:45:46PM -0400, James Simmons wrote:
>> From: Bruno Faccini <bruno.faccini@intel.com>
>> 
>> As part of LU-3848 and LU-4330, it has been discovered that LNET
>> MEs and small MDs (<=128 Bytes) are allocated in <size-128> kmem_cache
>> and thus can suffer quite frequent corruptions, from other modules or
>> Kernel parts, that occur there. To avoid this, MEs and small-MDs
>> specific kmem_cache have been created.
> 
> What?  Who corrupts them?  That shouldn't be possible, and on some
> systems, even if you do ask for a separate slab, it will be merged
> togther with others of the same size.  So this patch doesn't do all that
> much.
> 
> I think you are having some other problem here, changing to a separate
> memory cache shouldn't solve corruption issues.
> 
> sorry,
> 
> greg k-h

---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris, 
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Re: [PATCH 2/3] staging: lustre: lnet: Allocate MEs and small MDs in own kmem_caches

[Greg Kroah-Hartman]

A: No.
Q: Should I include quotations after my reply?

http://daringfireball.net/2007/07/on_top

On Fri, Jun 10, 2016 at 03:25:28PM +0000, Faccini, Bruno wrote:
> Hello,
> The intent of this patch is not to solve the corruptions for sure, but
> only to avoid the concerned MEs/small-MDs LNet structs to be quite
> frequently impacted due to their high allocation/free rate.

But that's not what the patch description said :(

And again, putting them in a separate cache is not going to save much of
anything, given that your caches might have been merged together anyway.

> This may also possibly help to save cycles due to high usage and
> contention when using a generic kmem_cache (when they stay separate
> from others, thanks for the precision!).

Have you measured this?

This isn't applicable for 4.7-rc at this time, _unless_ it fixes a bug,
which is why I pushed back on this.  If you want your own cache for
these variables, fine, I don't care, but that makes it a 4.8-rc1 patch
instead.

hope that helps explain things better,

greg k-h

Re: [PATCH 2/3] staging: lustre: lnet: Allocate MEs and small MDs in own kmem_caches

[James Simmons]

> > This may also possibly help to save cycles due to high usage and
> > contention when using a generic kmem_cache (when they stay separate
> > from others, thanks for the precision!).
> 
> Have you measured this?
> 
> This isn't applicable for 4.7-rc at this time, _unless_ it fixes a bug,
> which is why I pushed back on this.  If you want your own cache for
> these variables, fine, I don't care, but that makes it a 4.8-rc1 patch
> instead.
> 
> hope that helps explain things better,

As a side question when is the window to push patches of this class?
Is it when 4.7-rc7 is merged to staging? 

Re: [PATCH 2/3] staging: lustre: lnet: Allocate MEs and small MDs in own kmem_caches

[Greg Kroah-Hartman]

On Wed, Jun 15, 2016 at 04:02:40AM +0100, James Simmons wrote:
> 
> > > This may also possibly help to save cycles due to high usage and
> > > contention when using a generic kmem_cache (when they stay separate
> > > from others, thanks for the precision!).
> > 
> > Have you measured this?
> > 
> > This isn't applicable for 4.7-rc at this time, _unless_ it fixes a bug,
> > which is why I pushed back on this.  If you want your own cache for
> > these variables, fine, I don't care, but that makes it a 4.8-rc1 patch
> > instead.
> > 
> > hope that helps explain things better,
> 
> As a side question when is the window to push patches of this class?
> Is it when 4.7-rc7 is merged to staging? 

You can send them to me anytime, I'll queue them up in my "-next" branch
to be merged in the next merge window.  Like I do for almost all lustre
patches that aren't bugfixes or regressions.

But I think you have a bigger problem here that you need to debug, using
a separate cache isn't going to solve that bug, only postpone you
finding it...

thanks,

greg k-h

[PATCH 3/3] staging: lustre: lnet: optimize memory foot print for lnet_libmd

[James Simmons]

From: Bruno Faccini <bruno.faccini@intel.com>

The lnet_libmd struct fields have been re-ordered to optimize its
memory foot-print.

Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-4430
Reviewed-on: http://review.whamcloud.com/18586
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Doug Oucharek <doug.s.oucharek@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Signed-off-by: James Simmons <jsimmons@infradead.org>
---
 .../staging/lustre/include/linux/lnet/lib-types.h  |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/drivers/staging/lustre/include/linux/lnet/lib-types.h b/drivers/staging/lustre/include/linux/lnet/lib-types.h
index 7967b01..79a4ecf 100644
--- a/drivers/staging/lustre/include/linux/lnet/lib-types.h
+++ b/drivers/staging/lustre/include/linux/lnet/lib-types.h
@@ -151,9 +151,9 @@ typedef struct lnet_libmd {
 	int			 md_refcount;
 	unsigned int		 md_options;
 	unsigned int		 md_flags;
+	unsigned int		 md_niov;	/* # frags at end of struct */
 	void			*md_user_ptr;
 	lnet_eq_t		*md_eq;
-	unsigned int		 md_niov;	/* # frags */
 	union {
 		struct kvec	iov[LNET_MAX_IOV];
 		lnet_kiov_t	kiov[LNET_MAX_IOV];
-- 
1.7.1

Re: [PATCH 3/3] staging: lustre: lnet: optimize memory foot print for lnet_libmd

[Greg Kroah-Hartman]

On Thu, Jun 09, 2016 at 06:45:47PM -0400, James Simmons wrote:
> From: Bruno Faccini <bruno.faccini@intel.com>
> 
> The lnet_libmd struct fields have been re-ordered to optimize its
> memory foot-print.

This isn't a regression, so isn't ok for 4.7-rc releases, sorry.

greg k-h