RE: [PATCH rdma-core 0/5] Common systemd/udev based boot support

Steve Wise
2017-07-26T16:29:15Z

> On Wed, Jul 26, 2017 at 09:05:03AM -0500, Steve Wise wrote:
>
> > > I think you should change this to create the RDMA device when the
> > > module is installed and the hardware is present..
> > >
> >
> > Not gonna happen. cxgb4 doesn't setup the queues, rss, irq mappings, etc,
until
> > an interface is brought up. So iw_cxgb4 cannot initialize and register with
the
> > rdma core until that happens.
>
> But it doesn't tear all that stuff down on ifdown?

hmm. I need to relook at this. I'm very sure the intent was offload
connections do not get torn down if the link goes down.

>
> > > Basically, it is very hard to start a RDMA daemon and not have it race
> > > with something and randomly fail to start properly the more hotpluggy
> > > things are.
> >
> > I think these races exist today, no? Or is this patch series introducing
the
> > races? The iwpmd does not need the providers registered at the time it
starts.
> > It will discover new iwarp providers as they initialize.
>
> There are races today, particularly with systemd, but I suspect they
> are slightly different.. That doesn't mean we should ignore them :)
>
> In particular the rdma.target approach loaded all the modules, even if
> there wasn't hardware support, so when the cxgb4 RDMA device is
> created everything is already to go. We still have races, but they are
> probably smaller
>
> Now, we don't even plug the modules until the RDMA device appears, so
> we have broader possibilities for racing..
>
> Overall it makes it hard to fit into the ideal system configuration
> where RDMA sets up before network-pre.target when a RDMA device
> doesn't even plug in until after network.target.
>
> > Can you think of other ways to address your concerns?
>
> Maybe we should have udev trigger loading iwarp services on the cxgb4
> ethernet device? At least the races then become more like what we had
> before. Are other roce/iwarp devices like this?
>

I'm not sure. I would be interested to hear what Intel does. +Chien + Mustafa

Steve.