Re: [PATCH rdma-core 0/5] Common systemd/udev based boot support

[Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>]

On Tue, Jul 25, 2017 at 04:52:01PM -0500, Steve Wise wrote:

> > This sort of hotplug that cxbg4 does is quite strange, what happens
> > when 'ip link set X down' is done? Does it remove the RDMA device?
> > Does 'ip link set down' block until all users go away?
> 
> No.  iw_cxgb4 just triggers on the first 'up', to add the rdma provider instance
> for that device.  The Low Level Driver (LLD), cxgb4, passes the CXGB4_STATE_UP
> event to all registered upper level drivers (ULDs) when the first port is
> enabled (see cxgb_up).  Any rdma connections that are active when a link goes
> down still function, as any TCP connection would function if the interface was
> brought down;  eg: tcp retransmits if there is pending data until it gives up
> and aborts the connection.  So Netdev link down/up transitions are hidden from
> the rdma application.   

I think you should change this to create the RDMA device when the
module is installed and the hardware is present..

> > This is going to make it harder for cxgb users to get a reliably
> > bootup at this time, we need more kernel autoloading for things to be
> > reliable, and I'm sure iwpmd.service needs some dependency adjusting,
> > I just don't know enough about it to do it right. :\
> 
> I don't understand?

At the present moment udev will start running rules at the link up
time, which happens sometime around 'network.target'

However, systemd will continue processing unknowing what udev is
doing.

So, if you have a RDMA enabled daemon, and you make it start after the
RDMA device is plugged we have some races..

- udev is creating /dev/ nodes and telling systemd to start module loading
  units, and run iwpmd
- systemd may have already started loading the RDMA daemon before udev
  gets to any of this (racy) eg the /dev/ nodes may not exist yet, or
  the modules may still in process to be loaded
- systemd may have started iwpmd, but it is not yet ready and then
  starts the RDMA daemon (racy differently, this is helped with
  sd_notify)
- The RDMA daemon now needs explicit dependencies on the RDMA device
  to order properly, something simple like sysinit.target isn't going to work

Basically, it is very hard to start a RDMA daemon and not have it race
with something and randomly fail to start properly the more hotpluggy
things are.

The existing RDMA stuff largely relies on some sequentiality, eg
loading the RDMA module is enough to create the RDMA device, and that
more reliably happens before sysinit.target, so we can create some
predictable ordering in the system.

This is also why I have been so insistent that the only way to make
all of this work properly and reliably is to have robust kernel auto
loading.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html