[PATCH 0/3] Fixes for NFS/RDMA device removal

[PATCH 0/3] Fixes for NFS/RDMA device removal

[cel]

From: Chuck Lever <chuck.lever@oracle.com>

Fix a handful of nits for the NFS/RDMA device removal code that
went into v6.11-rc.

Chuck Lever (3):
  rpcrdma: Device kref is over-incremented on error from xa_alloc
  rpcrdma: Use XA_FLAGS_ALLOC instead of XA_FLAGS_ALLOC1
  rpcrdma: Trace connection registration and unregistration

 include/trace/events/rpcrdma.h  | 36 +++++++++++++++++++++++++++++++++
 net/sunrpc/xprtrdma/ib_client.c |  6 ++++--
 2 files changed, 40 insertions(+), 2 deletions(-)

-- 
2.45.1

[PATCH 1/3] rpcrdma: Device kref is over-incremented on error from xa_alloc

[cel]

From: Chuck Lever <chuck.lever@oracle.com>

If the device's reference count is too high, the device completion
callback never fires.

Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
 net/sunrpc/xprtrdma/ib_client.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/net/sunrpc/xprtrdma/ib_client.c b/net/sunrpc/xprtrdma/ib_client.c
index a938c19c3490..4d1e9fa89573 100644
--- a/net/sunrpc/xprtrdma/ib_client.c
+++ b/net/sunrpc/xprtrdma/ib_client.c
@@ -62,9 +62,9 @@ int rpcrdma_rn_register(struct ib_device *device,
 	if (!rd || test_bit(RPCRDMA_RD_F_REMOVING, &rd->rd_flags))
 		return -ENETUNREACH;
 
-	kref_get(&rd->rd_kref);
 	if (xa_alloc(&rd->rd_xa, &rn->rn_index, rn, xa_limit_32b, GFP_KERNEL) < 0)
 		return -ENOMEM;
+	kref_get(&rd->rd_kref);
 	rn->rn_done = done;
 	return 0;
 }
-- 
2.45.1

[PATCH 2/3] rpcrdma: Use XA_FLAGS_ALLOC instead of XA_FLAGS_ALLOC1

[cel]

From: Chuck Lever <chuck.lever@oracle.com>

Nit: The built-in xa_limit_32b range starts at 0, but
XA_FLAGS_ALLOC1 configures the xarray's allocator to start at 1.
Adopt the more conventional XA_FLAGS_ALLOC because there's no
mechanical reason to skip 0.

Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
 net/sunrpc/xprtrdma/ib_client.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/net/sunrpc/xprtrdma/ib_client.c b/net/sunrpc/xprtrdma/ib_client.c
index 4d1e9fa89573..7913d7bad23d 100644
--- a/net/sunrpc/xprtrdma/ib_client.c
+++ b/net/sunrpc/xprtrdma/ib_client.c
@@ -111,7 +111,7 @@ static int rpcrdma_add_one(struct ib_device *device)
 		return -ENOMEM;
 
 	kref_init(&rd->rd_kref);
-	xa_init_flags(&rd->rd_xa, XA_FLAGS_ALLOC1);
+	xa_init_flags(&rd->rd_xa, XA_FLAGS_ALLOC);
 	rd->rd_device = device;
 	init_completion(&rd->rd_done);
 	ib_set_client_data(device, &rpcrdma_ib_client, rd);
-- 
2.45.1

[PATCH 3/3] rpcrdma: Trace connection registration and unregistration

[cel]

From: Chuck Lever <chuck.lever@oracle.com>

These new trace points record xarray indices and the time of
endpoint registration and unregistration, to co-ordinate with
device removal events.

Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
 include/trace/events/rpcrdma.h  | 36 +++++++++++++++++++++++++++++++++
 net/sunrpc/xprtrdma/ib_client.c |  2 ++
 2 files changed, 38 insertions(+)

diff --git a/include/trace/events/rpcrdma.h b/include/trace/events/rpcrdma.h
index 9141398591e0..e6a72646c507 100644
--- a/include/trace/events/rpcrdma.h
+++ b/include/trace/events/rpcrdma.h
@@ -2300,6 +2300,42 @@ DEFINE_CLIENT_DEVICE_EVENT(rpcrdma_client_remove_one);
 DEFINE_CLIENT_DEVICE_EVENT(rpcrdma_client_wait_on);
 DEFINE_CLIENT_DEVICE_EVENT(rpcrdma_client_remove_one_done);
 
+DECLARE_EVENT_CLASS(rpcrdma_client_register_class,
+	TP_PROTO(
+		const struct ib_device *device,
+		const struct rpcrdma_notification *rn
+	),
+
+	TP_ARGS(device, rn),
+
+	TP_STRUCT__entry(
+		__string(name, device->name)
+		__field(void *, callback)
+		__field(u32, index)
+	),
+
+	TP_fast_assign(
+		__assign_str(name);
+		__entry->callback = rn->rn_done;
+		__entry->index = rn->rn_index;
+	),
+
+	TP_printk("device=%s index=%u done callback=%pS\n",
+		__get_str(name), __entry->index, __entry->callback
+	)
+);
+
+#define DEFINE_CLIENT_REGISTER_EVENT(name)				\
+	DEFINE_EVENT(rpcrdma_client_register_class, name,		\
+	TP_PROTO(							\
+		const struct ib_device *device,				\
+		const struct rpcrdma_notification *rn			\
+	),								\
+	TP_ARGS(device, rn))
+
+DEFINE_CLIENT_REGISTER_EVENT(rpcrdma_client_register);
+DEFINE_CLIENT_REGISTER_EVENT(rpcrdma_client_unregister);
+
 #endif /* _TRACE_RPCRDMA_H */
 
 #include <trace/define_trace.h>
diff --git a/net/sunrpc/xprtrdma/ib_client.c b/net/sunrpc/xprtrdma/ib_client.c
index 7913d7bad23d..8507cd4d8921 100644
--- a/net/sunrpc/xprtrdma/ib_client.c
+++ b/net/sunrpc/xprtrdma/ib_client.c
@@ -66,6 +66,7 @@ int rpcrdma_rn_register(struct ib_device *device,
 		return -ENOMEM;
 	kref_get(&rd->rd_kref);
 	rn->rn_done = done;
+	trace_rpcrdma_client_register(device, rn);
 	return 0;
 }
 
@@ -91,6 +92,7 @@ void rpcrdma_rn_unregister(struct ib_device *device,
 	if (!rd)
 		return;
 
+	trace_rpcrdma_client_unregister(device, rn);
 	xa_erase(&rd->rd_xa, rn->rn_index);
 	kref_put(&rd->rd_kref, rpcrdma_rn_release);
 }
-- 
2.45.1

Re: [PATCH 0/3] Fixes for NFS/RDMA device removal

[Anna Schumaker]

Hi Chuck,

On Mon, Aug 12, 2024 at 11:48 AM <cel@kernel.org> wrote:
>
> From: Chuck Lever <chuck.lever@oracle.com>
>
> Fix a handful of nits for the NFS/RDMA device removal code that
> went into v6.11-rc.

Are you intending these for 6.11-rc or 6.12?

Anna

>
> Chuck Lever (3):
>   rpcrdma: Device kref is over-incremented on error from xa_alloc
>   rpcrdma: Use XA_FLAGS_ALLOC instead of XA_FLAGS_ALLOC1
>   rpcrdma: Trace connection registration and unregistration
>
>  include/trace/events/rpcrdma.h  | 36 +++++++++++++++++++++++++++++++++
>  net/sunrpc/xprtrdma/ib_client.c |  6 ++++--
>  2 files changed, 40 insertions(+), 2 deletions(-)
>
> --
> 2.45.1
>

Re: [PATCH 0/3] Fixes for NFS/RDMA device removal

[Chuck Lever III]

> On Aug 21, 2024, at 3:31 PM, Anna Schumaker <anna@kernel.org> wrote:
> 
> Hi Chuck,
> 
> On Mon, Aug 12, 2024 at 11:48 AM <cel@kernel.org> wrote:
>> 
>> From: Chuck Lever <chuck.lever@oracle.com>
>> 
>> Fix a handful of nits for the NFS/RDMA device removal code that
>> went into v6.11-rc.
> 
> Are you intending these for 6.11-rc or 6.12?

IIRC the patches they fix went into 6.11-rc1, so I'd like to
see these go into a 6.11-rc release.


> Anna
> 
>> 
>> Chuck Lever (3):
>>  rpcrdma: Device kref is over-incremented on error from xa_alloc
>>  rpcrdma: Use XA_FLAGS_ALLOC instead of XA_FLAGS_ALLOC1
>>  rpcrdma: Trace connection registration and unregistration
>> 
>> include/trace/events/rpcrdma.h  | 36 +++++++++++++++++++++++++++++++++
>> net/sunrpc/xprtrdma/ib_client.c |  6 ++++--
>> 2 files changed, 40 insertions(+), 2 deletions(-)
>> 
>> --
>> 2.45.1
>> 

--
Chuck Lever

Re: [PATCH 0/3] Fixes for NFS/RDMA device removal

[Anna Schumaker]

On Wed, Aug 21, 2024 at 3:33 PM Chuck Lever III <chuck.lever@oracle.com> wrote:
>
>
>
> > On Aug 21, 2024, at 3:31 PM, Anna Schumaker <anna@kernel.org> wrote:
> >
> > Hi Chuck,
> >
> > On Mon, Aug 12, 2024 at 11:48 AM <cel@kernel.org> wrote:
> >>
> >> From: Chuck Lever <chuck.lever@oracle.com>
> >>
> >> Fix a handful of nits for the NFS/RDMA device removal code that
> >> went into v6.11-rc.
> >
> > Are you intending these for 6.11-rc or 6.12?
>
> IIRC the patches they fix went into 6.11-rc1, so I'd like to
> see these go into a 6.11-rc release.

Sounds good, thanks!

Anna

>
>
> > Anna
> >
> >>
> >> Chuck Lever (3):
> >>  rpcrdma: Device kref is over-incremented on error from xa_alloc
> >>  rpcrdma: Use XA_FLAGS_ALLOC instead of XA_FLAGS_ALLOC1
> >>  rpcrdma: Trace connection registration and unregistration
> >>
> >> include/trace/events/rpcrdma.h  | 36 +++++++++++++++++++++++++++++++++
> >> net/sunrpc/xprtrdma/ib_client.c |  6 ++++--
> >> 2 files changed, 40 insertions(+), 2 deletions(-)
> >>
> >> --
> >> 2.45.1
> >>
>
> --
> Chuck Lever
>
>