Re: [PATCH] Add rdma service for kernel boot support

[Benjamin Drung <benjamin.drung-EIkl63zCoXaH+58JC4qpiA@public.gmane.org>]

Am Freitag, den 14.07.2017, 12:08 +0200 schrieb Benjamin Drung:
> Hi Jason,
> 
> Am Donnerstag, den 13.07.2017, 16:15 -0600 schrieb Jason Gunthorpe:
> > On Thu, Jul 13, 2017 at 06:33:07PM +0000, Bart Van Assche wrote:
> > > On Thu, 2017-07-13 at 12:20 -0600, Jason Gunthorpe wrote:
> > > > Here is an untested attempt to move module loading into
> > > > srp_daemon -
> > > > what do you think Bart?
> > > 
> > > Hello Jason,
> > > 
> > > Thanks for the patch. To me this looks like a nice simplification
> > > compared
> > > to the current approach. Do you want me to test this patch?
> > 
> > I did some more steps..
> > 
> > Benjamin/Bart let me know what you think, I'll send the patches to
> > the
> > list if you like the idea.
> > 
> > https://github.com/jgunthorpe/rdma-plumbing/tree/systemd
> > 
> > This replaces Benjamins rdma.conf based approach.
> > 
> > The basic approach is to directly use systemd to load the modules
> > and
> > determine what modules to load by using udev and explicit Requires
> > in
> > the units.
> 
> I like your approach very much. It's cleaner and simpler. So let's
> ditch my approach and use yours.
> 
> I tested your branch on a Debian 8 (jessie) system with a mlx4 card
> and
> found several points to fix/improve/discuss:
> 
> 1. The rdma-load-modules@ service needs to set
> DefaultDependencies=no.
> Otherwise it will end up in a dependency loop:
> 
> Found ordering cycle on rdma-load-modules-A9MMwDI1uB1TDjBF/Jpztg@public.gmane.org/start
> Found dependency on basic.target/start
> Found dependency on paths.target/start
> Found dependency on acpid.path/start
> Found dependency on sysinit.target/start
> Found dependency on rdma-load-modules-A9MMwDI1uB1TDjBF/Jpztg@public.gmane.org/start
> Breaking ordering cycle by deleting job paths.target/start
> Found ordering cycle on rdma-load-modules-A9MMwDI1uB1TDjBF/Jpztg@public.gmane.org/start
> Found dependency on paths.target/start
> Found dependency on acpid.path/start
> Found dependency on sysinit.target/start
> Found dependency on rdma-load-modules-A9MMwDI1uB1TDjBF/Jpztg@public.gmane.org/start
> Starting Load RDMA modules from /etc/rdma/modules/mlx4.conf...
> 
> 2. You can just specify "etc/rdma/modules" in debian/rdma-
> core.install
> instead of listing each .conf file individually.
> 
> 3. Default conf files for opa.conf and roce.conf are missing.
> 
> 4. Should rdma-load-modules@ *not* fail if the corresponding .conf is
> missing?
> 
> 5. How to handle build-in modules correctly? Our kernel has the i40e
> module built in (CONFIG_I40E=y) and rdma-load-modules-QqjR1VyAyFhTDjBF/Jpztg@public.gmane.org
> will
> be started, but the system does not have a i40e card and thus I don't
> want to have the module started. 

Forgot one point:

6. The ipoib module (loaded by rdma-load-modules@infiniband) needs to
loaded before the networking.service is running. The networking.service
brings up the network devices on Debian. It runs "ifup -a" which reads
/etc/network/interfaces which we use to configure our ipoib devices.
Here is the networking.service definition:

bdrung@server:~$ systemctl cat networking.service
# /run/systemd/generator.late/networking.service
# Automatically generated by systemd-sysv-generator

[Unit]
SourcePath=/etc/init.d/networking
Description=LSB: Raise network interfaces.
DefaultDependencies=no
Before=sysinit.target shutdown.target
After=mountkernfs.service local-fs.target urandom.service
Conflicts=shutdown.target

[Service]
Type=forking
Restart=no
TimeoutSec=0
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=yes
SysVStartPriority=13
ExecStart=/etc/init.d/networking start
ExecStop=/etc/init.d/networking stop
ExecReload=/etc/init.d/networking reload

# /run/systemd/generator/networking.service.d/50-insserv.conf-$network.conf
# Automatically generated by systemd-insserv-generator

[Unit]
Wants=network.target
Before=network.target

# /lib/systemd/system/networking.service.d/network-pre.conf
[Unit]
After=network-pre.target systemd-sysctl.service systemd-modules-load.service

-- 
Benjamin Drung
System Developer
Debian & Ubuntu Developer

ProfitBricks GmbH
Greifswalder Str. 207
D - 10405 Berlin

Email: benjamin.drung-EIkl63zCoXaH+58JC4qpiA@public.gmane.org
Web: https://www.profitbricks.com

Sitz der Gesellschaft: Berlin.
Registergericht: Amtsgericht Charlottenburg, HRB 125506B.
Geschäftsführer: Achim Weiss.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html