[Q] RCU usage in infiniband

[Q] RCU usage in infiniband

[Sebastian Andrzej Siewior]

Hi,

I've been randomly starting at infiniband code and noticed it uses
call_rcu() with a callback from its module. 

| drivers/infiniband/hw/mlx5/devx.c:         call_rcu(&event_sub->rcu, devx_free_subscription);
| drivers/infiniband/ulp/ipoib/ipoib_main.c: call_rcu(&neigh->rcu, ipoib_neigh_reclaim);

I don't see synchronize_rcu() and rcu_barrier() there so I've been
asking myself what ensures that the callback completes before module is
gone (via rmmod)? I would expect a rcu_barrier() in
ipoib_cleanup_module() for instance.

Is the unload path so "late" that the callbacks run before the module is
unmapped or is there something between the APIs that ensures this?

Sebastian

Re: [Q] RCU usage in infiniband

[Jason Gunthorpe]

On Wed, Jul 08, 2026 at 11:23:16AM +0200, Sebastian Andrzej Siewior wrote:
> Hi,
> 
> I've been randomly starting at infiniband code and noticed it uses
> call_rcu() with a callback from its module. 
> 
> | drivers/infiniband/hw/mlx5/devx.c:         call_rcu(&event_sub->rcu, devx_free_subscription);
> | drivers/infiniband/ulp/ipoib/ipoib_main.c: call_rcu(&neigh->rcu, ipoib_neigh_reclaim);
> 
> I don't see synchronize_rcu() and rcu_barrier() there so I've been
> asking myself what ensures that the callback completes before module is
> gone (via rmmod)? I would expect a rcu_barrier() in
> ipoib_cleanup_module() for instance.
> 
> Is the unload path so "late" that the callbacks run before the module is
> unmapped or is there something between the APIs that ensures this?

It wouldn't hurt to have a more explicit unload synchronize call, but
there are already some existing ones on the path to unload a module
that would cover anyhow

Jason

Re: [Q] RCU usage in infiniband

[Sebastian Andrzej Siewior]

On 2026-07-08 10:17:27 [-0300], Jason Gunthorpe wrote:
> On Wed, Jul 08, 2026 at 11:23:16AM +0200, Sebastian Andrzej Siewior wrote:
> > Hi,
> > 
> > I've been randomly starting at infiniband code and noticed it uses
> > call_rcu() with a callback from its module. 
> > 
> > | drivers/infiniband/hw/mlx5/devx.c:         call_rcu(&event_sub->rcu, devx_free_subscription);
> > | drivers/infiniband/ulp/ipoib/ipoib_main.c: call_rcu(&neigh->rcu, ipoib_neigh_reclaim);
> > 
> > I don't see synchronize_rcu() and rcu_barrier() there so I've been
> > asking myself what ensures that the callback completes before module is
> > gone (via rmmod)? I would expect a rcu_barrier() in
> > ipoib_cleanup_module() for instance.
> > 
> > Is the unload path so "late" that the callbacks run before the module is
> > unmapped or is there something between the APIs that ensures this?
> 
> It wouldn't hurt to have a more explicit unload synchronize call, but
> there are already some existing ones on the path to unload a module
> that would cover anyhow

Where? Within the module core or somewhere within infiniband?

> Jason

Sebastian

Re: [Q] RCU usage in infiniband

[Jason Gunthorpe]

On Wed, Jul 08, 2026 at 03:21:05PM +0200, Sebastian Andrzej Siewior wrote:
> On 2026-07-08 10:17:27 [-0300], Jason Gunthorpe wrote:
> > On Wed, Jul 08, 2026 at 11:23:16AM +0200, Sebastian Andrzej Siewior wrote:
> > > Hi,
> > > 
> > > I've been randomly starting at infiniband code and noticed it uses
> > > call_rcu() with a callback from its module. 
> > > 
> > > | drivers/infiniband/hw/mlx5/devx.c:         call_rcu(&event_sub->rcu, devx_free_subscription);
> > > | drivers/infiniband/ulp/ipoib/ipoib_main.c: call_rcu(&neigh->rcu, ipoib_neigh_reclaim);
> > > 
> > > I don't see synchronize_rcu() and rcu_barrier() there so I've been
> > > asking myself what ensures that the callback completes before module is
> > > gone (via rmmod)? I would expect a rcu_barrier() in
> > > ipoib_cleanup_module() for instance.
> > > 
> > > Is the unload path so "late" that the callbacks run before the module is
> > > unmapped or is there something between the APIs that ensures this?
> > 
> > It wouldn't hurt to have a more explicit unload synchronize call, but
> > there are already some existing ones on the path to unload a module
> > that would cover anyhow
> 
> Where? Within the module core or somewhere within infiniband?

Eg the uverbs path has a srcu synchronize IIRC before allowing the FDs
to close which holds the module as well. But it probably isn't wise to
rely on this

Jason

Re: [Q] RCU usage in infiniband

[Sebastian Andrzej Siewior]

On 2026-07-08 10:28:49 [-0300], Jason Gunthorpe wrote:
> > Where? Within the module core or somewhere within infiniband?
> 
> Eg the uverbs path has a srcu synchronize IIRC before allowing the FDs
> to close which holds the module as well. But it probably isn't wise to
> rely on this

But this is srcu. So this takes long enough so that we have a RCU grace
period, too? But there is also RCU_LAZY which could stretch this a bit
;)

> Jason

Sebastian

Re: [Q] RCU usage in infiniband

[Paul E. McKenney]

On Wed, Jul 08, 2026 at 03:34:10PM +0200, Sebastian Andrzej Siewior wrote:
> On 2026-07-08 10:28:49 [-0300], Jason Gunthorpe wrote:
> > > Where? Within the module core or somewhere within infiniband?
> > 
> > Eg the uverbs path has a srcu synchronize IIRC before allowing the FDs
> > to close which holds the module as well. But it probably isn't wise to
> > rely on this
> 
> But this is srcu. So this takes long enough so that we have a RCU grace
> period, too? But there is also RCU_LAZY which could stretch this a bit
> ;)

Just to emphasize Sebastian's point here...

If a module uses call_rcu(), there needs to either be an rcu_barrier()
or some explicit synchronization with the callbacks themselves before
module-unload time.  Any explicit synchronization must also include
a wait for existing softirq handlers to complete, so rcu_barrier()
is simpler and more maintainable.

Please note the rcu_barrier().  And that synchronize_rcu() is not a
legitimate substitute for that rcu_barrier().

Similarly, if that module uses call_srcu(), there needs to either be an
srcu_barrier() or explicit synchronization with the callbacks as mentioned
above.  Please note that neither rcu_barrier(), synchronize_rcu(),
nor synchronize_srcu() are legitimate substitutes for that srcu_barrier().

The difference is that while rcu_barrier() waits for the completion of the
callbacks queued by prior invocations of call_rcu(), synchronize_rcu()
instead waits only for prior RCU readers to complete.  So rcu_barrier()
does not care about RCU readers and synchronize_rcu() does not care
about RCU callbacks.  And the same holds for SRCU.

Also, RCU does not wait for SRCU and vice versa.

Does that help?

							Thanx, Paul