RE: [PATCH rdma-core 0/5] Common systemd/udev based boot support

Steve Wise
2017-07-25T21:52:01Z

> -----Original Message-----
> From: Jason Gunthorpe [mailto:jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org]
> Sent: Tuesday, July 25, 2017 4:34 PM
> To: Steve Wise
> Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; 'Doug Ledford'; 'Ram Amrani'; 'Ira Weiny';
> 'Benjamin Drung'; 'Jarod Wilson'
> Subject: Re: [PATCH rdma-core 0/5] Common systemd/udev based boot support
>
> On Tue, Jul 25, 2017 at 01:49:37PM -0500, Steve Wise wrote:
>
> > After I rebooted, cxgb4 and iw_cxgb4 were loaded, but the services were not
> > started. I think that is probably due to the fact that iw_cxgb4 doesn't
> > register with the rdma core until the cxgb4 interfaces are brought
> > up.
>
> Yes, it is done on demand when the kernel creates its side of the
> interface.
>
> > I then brought up port 0 of the cxgb4 card:
> >
> > [root@stevo3 ~]# ip link set enp4s0f4 up
> > [root@stevo3 ~]# ip addr add 172.16.3.3/24 dev enp4s0f4
> >
> > And then the services were started:
>
> Yep, everything you sent me looks fine, thanks for taking a look at
> this.
>

Great! You can add a Tested-by: line for me.

> This sort of hotplug that cxbg4 does is quite strange, what happens
> when 'ip link set X down' is done? Does it remove the RDMA device?
> Does 'ip link set down' block until all users go away?
>

No. iw_cxgb4 just triggers on the first 'up', to add the rdma provider instance
for that device. The Low Level Driver (LLD), cxgb4, passes the CXGB4_STATE_UP
event to all registered upper level drivers (ULDs) when the first port is
enabled (see cxgb_up). Any rdma connections that are active when a link goes
down still function, as any TCP connection would function if the interface was
brought down; eg: tcp retransmits if there is pending data until it gives up
and aborts the connection. So Netdev link down/up transitions are hidden from
the rdma application.

> This is going to make it harder for cxgb users to get a reliably
> bootup at this time, we need more kernel autoloading for things to be
> reliable, and I'm sure iwpmd.service needs some dependency adjusting,
> I just don't know enough about it to do it right. :\

I don't understand?

Stevo