* [PATCH v2 rdma-core 1/6] Common infrastructure for auto loading rdma modules
[not found] ` <1501194685-9447-1-git-send-email-jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
@ 2017-07-27 22:31 ` Jason Gunthorpe
[not found] ` <1501194685-9447-2-git-send-email-jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2017-07-27 22:31 ` [PATCH v2 rdma-core 2/6] Move umad udev rules to rdma-core Jason Gunthorpe
` (6 subsequent siblings)
7 siblings, 1 reply; 24+ messages in thread
From: Jason Gunthorpe @ 2017-07-27 22:31 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA, Leon Romanovsky, Doug Ledford,
Ram Amrani, Steve Wise, Ira Weiny, Benjamin Drung, Jarod Wilson
This is inspired by the similar approach in the redhat directory but
takes a more general approach relying on udev and systemd to do the
actual work fully dynamically instead of a oneshot shell script.
Loading is split into two cases
1) Loading RDMA support modules when RDMA capable hardware is installed.
This is only needed for ethernet devices which do not load their RDMA
support modules via request_module in the kernel.
udev is used to detect when an ethernet device controlled by a specific
module is hot plugged and then udev directly loads the RDMA module
2) Loading RDMA ULP support when RDMA hardware is installed
This is done by having udev detect when RDMA hardware is installed and
udev causes systemd to load a list of modules from config files in
/etc/rdma/modules/
The user can customize these files to select which ULP modules should be
loaded.
This broadly replaces the redhat/rdma.conf scheme.
In all cases the users can prevent a module from being auto-loaded on their
system by blacking listing it in a file in /etc/modprobe.d/
Signed-off-by: Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
---
CMakeLists.txt | 1 +
Documentation/udev.md | 83 +++++++++++++++++++++++++++++++
debian/rdma-core.install | 9 ++++
kernel-boot/CMakeLists.txt | 24 +++++++++
kernel-boot/modules/infiniband.conf | 12 +++++
kernel-boot/modules/iwarp.conf | 2 +
kernel-boot/modules/opa.conf | 10 ++++
kernel-boot/modules/rdma.conf | 21 ++++++++
kernel-boot/modules/roce.conf | 2 +
kernel-boot/rdma-description.rules | 42 ++++++++++++++++
kernel-boot/rdma-hw-modules.rules | 39 +++++++++++++++
kernel-boot/rdma-load-modules@.service.in | 15 ++++++
kernel-boot/rdma-ulp-modules.rules | 11 ++++
rdma-core.spec | 1 +
redhat/rdma-core.spec | 11 +++-
15 files changed, 282 insertions(+), 1 deletion(-)
create mode 100644 Documentation/udev.md
create mode 100644 kernel-boot/CMakeLists.txt
create mode 100644 kernel-boot/modules/infiniband.conf
create mode 100644 kernel-boot/modules/iwarp.conf
create mode 100644 kernel-boot/modules/opa.conf
create mode 100644 kernel-boot/modules/rdma.conf
create mode 100644 kernel-boot/modules/roce.conf
create mode 100644 kernel-boot/rdma-description.rules
create mode 100644 kernel-boot/rdma-hw-modules.rules
create mode 100644 kernel-boot/rdma-load-modules@.service.in
create mode 100644 kernel-boot/rdma-ulp-modules.rules
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 16196205035f61..a03d8da31cbc5d 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -398,6 +398,7 @@ configure_file("${BUILDLIB}/config.h.in" "${BUILD_INCLUDE}/config.h" ESCAPE_QUOT
add_subdirectory(ccan)
add_subdirectory(util)
add_subdirectory(Documentation)
+add_subdirectory(kernel-boot)
# Libraries
add_subdirectory(libibumad)
add_subdirectory(libibumad/man)
diff --git a/Documentation/udev.md b/Documentation/udev.md
new file mode 100644
index 00000000000000..a6328d322c5194
--- /dev/null
+++ b/Documentation/udev.md
@@ -0,0 +1,83 @@
+# Kernel Module Loading
+
+The RDMA subsystem relies on the kernel, udev and systemd to load modules on
+demand when RDMA hardware is present. The RDMA subsystem is unique in that we
+do not load the optional RDMA hardware modules unless the system has the
+rdma-core package installed.
+
+This is to avoid exposing systems not using RDMA from having RDMA enabled, for
+instance if a system has a multi-protocol ethernet adapter, but is only using
+the net stack interface.
+
+## Boot ordering with systemd
+
+systemd assumes everything is hot pluggable and runs in an event driven
+manner. When working with RDMA devices we are firstly concerned with when the
+physical hardware its module loaded into the kernel.
+
+This can happen in several spots along the bootup:
+
+ - From the initrd or built into the kernel. If hardware modules are present
+ inthe initrd then they are loaded into the kernel before booting the
+ system. This is done largely synchronously with the boot process.
+
+ - From udev when it auto detects PCI hardware or otherwise.
+ This happens asynchronously in the boot process, systemd does not wait for
+ udev to finish loading modules before it continues on.
+
+ This path makes it very likely the system will experience a RDMA 'hot plug'
+ scenario.
+
+ - From systemd's fixed module loader systemd-modules-load.service, eg from
+ the list in /etc/modules-load.d/. In this case the modules load happens
+ synchronously within systemd and it will hold off sysinit.target until
+ modules are loaded
+
+Once the hardware module is loaded it may be necessary to load a protocol
+module, eg to enable RDMA support on an ethernet device.
+
+This is triggered automatically by udev rules that match the master devices
+and load the protocol module with udev's module loader. This happens
+asynchronously to the rest of the systemd startup.
+
+Once a RDMA device is created by the kernel then udev will cause systemd to
+schedule ULP module loading services (eg rdma-load-modules@.service) specific
+to the plugged hardware. If sysinit.target has not yet been passed then these
+loaders will defer sysinit.target until they complete, otherwise this is a hot
+plug event and things will load asynchronously to the boot up process.
+
+Finally udev will cause systemd to start RDMA specific daemons like
+srp_daemon, rdma-ndd and iwpmd. These starts are linked to the detection of
+the first RDMA hardware, and the daemons internally handle hot plug events for
+other hardware.
+
+## Hot Plug compatible services
+
+Services using RDMA need to have device specific systemd dependencies in their
+unit files, either created by hand by the admin or by using udev rules.
+
+For instance, a service that uses /dev/infiniband/umad0 requires:
+
+```
+After=dev-infiniband-umad0.device
+BindsTo=dev-infiniband-umad0.device
+```
+
+Which will ensure the service will not run until the required umad device
+appears.
+
+This is similar to how systemd handles mounting filesystems and configuring
+ethernet devices.
+
+## Interaction with legacy non-hotplug services
+
+Services that cannot handle hot plug must be ordered after
+systemd-udev-settle.service, which will wait for udev to complete loading
+modules and scheduling systemd services. This ensures that all RDMA hardware
+present at boot is setup before proceeding to run the legacy service.
+
+Admins using legacy services can also place their RDMA hardware modules (eg
+mlx4_ib) directly in /etc/modules-load.d/ or in their initrd which will cause
+systemd to defer passing to sysinit.target until all RDMA hardware is setup,
+this is usually sufficient for legacy services. This is probably the default
+behavior in many configurations.
diff --git a/debian/rdma-core.install b/debian/rdma-core.install
index 0b35ab6372fcbe..44437fcc83dbd8 100644
--- a/debian/rdma-core.install
+++ b/debian/rdma-core.install
@@ -1,7 +1,16 @@
etc/modprobe.d/mlx4.conf
etc/modprobe.d/truescale.conf
+etc/rdma/modules/infiniband.conf
+etc/rdma/modules/iwarp.conf
+etc/rdma/modules/opa.conf
+etc/rdma/modules/rdma.conf
+etc/rdma/modules/roce.conf
+lib/systemd/system/rdma-load-modules@.service
lib/systemd/system/rdma-ndd.service
lib/udev/rules.d/60-rdma-ndd.rules
+lib/udev/rules.d/75-rdma-description.rules
+lib/udev/rules.d/90-rdma-hw-modules.rules
+lib/udev/rules.d/90-rdma-ulp-modules.rules
usr/bin/rxe_cfg
usr/lib/truescale-serdes.cmds
usr/sbin/rdma-ndd
diff --git a/kernel-boot/CMakeLists.txt b/kernel-boot/CMakeLists.txt
new file mode 100644
index 00000000000000..0d4a2aec1c6a94
--- /dev/null
+++ b/kernel-boot/CMakeLists.txt
@@ -0,0 +1,24 @@
+rdma_subst_install(FILES rdma-load-modules@.service.in
+ DESTINATION "${CMAKE_INSTALL_SYSTEMD_SERVICEDIR}"
+ RENAME rdma-load-modules@.service
+ PERMISSIONS OWNER_WRITE OWNER_READ GROUP_READ WORLD_READ)
+
+install(FILES
+ modules/infiniband.conf
+ modules/iwarp.conf
+ modules/opa.conf
+ modules/rdma.conf
+ modules/roce.conf
+ DESTINATION "${CMAKE_INSTALL_SYSCONFDIR}/rdma/modules")
+
+install(FILES "rdma-description.rules"
+ RENAME "75-rdma-description.rules"
+ DESTINATION "${CMAKE_INSTALL_UDEV_RULESDIR}")
+
+install(FILES "rdma-hw-modules.rules"
+ RENAME "90-rdma-hw-modules.rules"
+ DESTINATION "${CMAKE_INSTALL_UDEV_RULESDIR}")
+
+install(FILES "rdma-ulp-modules.rules"
+ RENAME "90-rdma-ulp-modules.rules"
+ DESTINATION "${CMAKE_INSTALL_UDEV_RULESDIR}")
diff --git a/kernel-boot/modules/infiniband.conf b/kernel-boot/modules/infiniband.conf
new file mode 100644
index 00000000000000..99526e156fff40
--- /dev/null
+++ b/kernel-boot/modules/infiniband.conf
@@ -0,0 +1,12 @@
+# These modules are loaded by the system if any InfiniBand device is installed
+# InfiniBand over IP netdevice
+ib_ipoib
+
+# Access to fabric management SMPs and GMPs from userspace.
+ib_umad
+
+# SCSI Remote Protocol target support
+# ib_srpt
+
+# ib_ucm provides the obsolete /dev/infiniband/ucm0
+# ib_ucm
diff --git a/kernel-boot/modules/iwarp.conf b/kernel-boot/modules/iwarp.conf
new file mode 100644
index 00000000000000..882146e41ee2ba
--- /dev/null
+++ b/kernel-boot/modules/iwarp.conf
@@ -0,0 +1,2 @@
+# These modules are loaded by the system if any iWarp device is installed
+iw_cm
diff --git a/kernel-boot/modules/opa.conf b/kernel-boot/modules/opa.conf
new file mode 100644
index 00000000000000..b9bc9f1f0146af
--- /dev/null
+++ b/kernel-boot/modules/opa.conf
@@ -0,0 +1,10 @@
+# These modules are loaded by the system if any OmniPath Architecture device
+# is installed
+# Infiniband over IP netdevice
+ib_ipoib
+
+# Access to fabric management SMPs and GMPs from userspace.
+ib_umad
+
+# Omnipath Ethernet Virtual NIC netdevice
+opa_vnic
diff --git a/kernel-boot/modules/rdma.conf b/kernel-boot/modules/rdma.conf
new file mode 100644
index 00000000000000..2d342dd82f7db0
--- /dev/null
+++ b/kernel-boot/modules/rdma.conf
@@ -0,0 +1,21 @@
+# These modules are loaded by the system if any RDMA devices is installed
+# iSCSI over RDMA client support
+ib_iser
+
+# iSCSI over RDMA target support
+# ib_isert
+
+# User access to RDMA verbs (supports libibverbs)
+ib_uverbs
+
+# User access to RDMA connection management (supports librdmacm)
+rdma_ucm
+
+# RDS over RDMA support
+# rds_rdma
+
+# NFS over RDMA client support
+xprtrdma
+
+# NFS over RDMA server support
+svcrdma
diff --git a/kernel-boot/modules/roce.conf b/kernel-boot/modules/roce.conf
new file mode 100644
index 00000000000000..8e4927ce26f043
--- /dev/null
+++ b/kernel-boot/modules/roce.conf
@@ -0,0 +1,2 @@
+# These modules are loaded by the system if any RDMA over Converged Ethernet
+# device is installed
diff --git a/kernel-boot/rdma-description.rules b/kernel-boot/rdma-description.rules
new file mode 100644
index 00000000000000..4d7c5808401ac7
--- /dev/null
+++ b/kernel-boot/rdma-description.rules
@@ -0,0 +1,42 @@
+# This is a version of net-description.rules for /sys/class/infiniband devices
+
+ACTION=="remove", GOTO="rdma_description_end"
+SUBSYSTEM!="infiniband", GOTO="rdma_description_end"
+
+# NOTE: DRIVERS searches up the sysfs path to find the driver that is bound to
+# the PCI/etc device that the RDMA device is linked to. This is not the kernel
+# driver that is supplying the RDMA device (eg as seen in ID_NET_DRIVER)
+
+# FIXME: with kernel support we could actually detect the protocols the RDMA
+# driver itself supports, this is a work around for lack of that support.
+# In future we could do this with a udev IMPORT{program} helper program
+# that extracted the ID information from the RDMA netlink.
+
+# Hardware that supports InfiniBand
+DRIVERS=="mlx4_core", ENV{ID_RDMA_INFINIBAND}="1"
+DRIVERS=="mlx5_core", ENV{ID_RDMA_INFINIBAND}="1"
+DRIVERS=="qib", ENV{ID_RDMA_INFINIBAND}="1"
+
+# Hardware that supports OPA
+DRIVERS=="hfi1verbs", ENV{ID_RDMA_OPA}="1"
+
+# Hardware that supports iWarp
+DRIVERS=="cxgb3", ENV{ID_RDMA_IWARP}="1"
+DRIVERS=="cxgb4", ENV{ID_RDMA_IWARP}="1"
+
+# Hardware that supports RoCE
+DRIVERS=="be2net", ENV{ID_RDMA_ROCE}="1"
+DRIVERS=="bnxt_en", ENV{ID_RDMA_ROCE}="1"
+DRIVERS=="hns", ENV{ID_RDMA_ROCE}="1"
+DRIVERS=="i40e", ENV{ID_RDMA_ROCE}="1"
+DRIVERS=="mlx4_core", ENV{ID_RDMA_ROCE}="1"
+DRIVERS=="mlx5_core", ENV{ID_RDMA_ROCE}="1"
+DRIVERS=="qede", ENV{ID_RDMA_ROCE}="1"
+DEVPATH=="*/infiniband/rxe*", ATTR{parent}=="*", ENV{ID_RDMA_ROCE}="1"
+
+# Setup the usual ID information so that systemd will display a sane name for
+# the RDMA device units.
+SUBSYSTEMS=="pci", ENV{ID_BUS}="pci", ENV{ID_VENDOR_ID}="$attr{vendor}", ENV{ID_MODEL_ID}="$attr{device}"
+SUBSYSTEMS=="pci", IMPORT{builtin}="hwdb --subsystem=pci"
+
+LABEL="rdma_description_end"
diff --git a/kernel-boot/rdma-hw-modules.rules b/kernel-boot/rdma-hw-modules.rules
new file mode 100644
index 00000000000000..dde0ab8dacacab
--- /dev/null
+++ b/kernel-boot/rdma-hw-modules.rules
@@ -0,0 +1,39 @@
+ACTION=="remove", GOTO="rdma_hw_modules_end"
+SUBSYSTEM!="net", GOTO="rdma_hw_modules_end"
+
+# Automatically load RDMA specific kernel modules when a multi-function device is installed
+
+# These drivers autoload an ethernet driver based on hardware detection and
+# need userspace to load the module that has their RDMA component to turn on
+# RDMA.
+ENV{ID_NET_DRIVER}=="be2net", RUN{builtin}+="kmod load ocrdma"
+ENV{ID_NET_DRIVER}=="bnxt_en", RUN{builtin}+="kmod load bnxt_re"
+ENV{ID_NET_DRIVER}=="cxgb3", RUN{builtin}+="kmod load iw_cxgb3"
+ENV{ID_NET_DRIVER}=="cxgb4", RUN{builtin}+="kmod load iw_cxgb4"
+ENV{ID_NET_DRIVER}=="hns", RUN{builtin}+="kmod load hns_roce"
+ENV{ID_NET_DRIVER}=="i40e", RUN{builtin}+="kmod load i40iw"
+ENV{ID_NET_DRIVER}=="mlx4_en", RUN{builtin}+="kmod load mlx4_ib"
+ENV{ID_NET_DRIVER}=="mlx5_core", RUN{builtin}+="kmod load mlx5_ib"
+ENV{ID_NET_DRIVER}=="qede", RUN{builtin}+="kmod load qedr"
+
+# The user must explicitly load these modules via /etc/modules-load.d/ or otherwise
+# rxe
+
+# When in IB mode the kernel PCI core module autoloads the protocol modules
+# for these providers
+# mlx4
+# mlx5
+
+# enic no longer has a userspace verbs driver, this rule should probably be
+# owned by libfabric
+ENV{ID_NET_DRIVER}=="enic", RUN{builtin}+="kmod load usnic_verbs"
+
+# These providers are single function and autoload RDMA automatically based on
+# PCI probing
+# hfi1verbs
+# ipathverbs
+# mthca
+# vmw_pvrdma
+# nes
+
+LABEL="rdma_hw_modules_end"
diff --git a/kernel-boot/rdma-load-modules@.service.in b/kernel-boot/rdma-load-modules@.service.in
new file mode 100644
index 00000000000000..b35a493ebf230b
--- /dev/null
+++ b/kernel-boot/rdma-load-modules@.service.in
@@ -0,0 +1,15 @@
+[Unit]
+Description=Load RDMA modules from @CMAKE_INSTALL_SYSCONFDIR@/rdma/modules/%I.conf
+DefaultDependencies=no
+Conflicts=shutdown.target
+# network-pre.target is to support distro network setup scripts that run after
+# systemd-modules-load.service but before sysinit.target, eg a classic network
+# setup script.
+Before=sysinit.target shutdown.target network-pre.target
+ConditionCapability=CAP_SYS_MODULE
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+ExecStart=/lib/systemd/systemd-modules-load @CMAKE_INSTALL_SYSCONFDIR@/rdma/modules/%I.conf
+TimeoutSec=90s
diff --git a/kernel-boot/rdma-ulp-modules.rules b/kernel-boot/rdma-ulp-modules.rules
new file mode 100644
index 00000000000000..c090700c754b19
--- /dev/null
+++ b/kernel-boot/rdma-ulp-modules.rules
@@ -0,0 +1,11 @@
+ACTION=="remove", GOTO="rdma_ulp_modules_end"
+SUBSYSTEM!="infiniband", GOTO="rdma_ulp_modules_end"
+
+# Automatically load general RDMA ULP modules when RDMA hardware is installed
+TAG+="systemd", ENV{SYSTEMD_WANTS}+="rdma-load-modules-XSrBNAzizW1TDjBF/Jpztg@public.gmane.org"
+TAG+="systemd", ENV{ID_RDMA_INFINIBAND}=="1", ENV{SYSTEMD_WANTS}+="rdma-load-modules-WW8hglTQ5Ylgot6vXr2xLYq0XWbe/Iga@public.gmane.org"
+TAG+="systemd", ENV{ID_RDMA_IWARP}=="1", ENV{SYSTEMD_WANTS}+="rdma-load-modules-mPXOxrv7gkYc5CkaFG5UGg@public.gmane.org"
+TAG+="systemd", ENV{ID_RDMA_OPA}=="1", ENV{SYSTEMD_WANTS}+="rdma-load-modules-EhpGOaQAOk9DcWalurthmA@public.gmane.org"
+TAG+="systemd", ENV{ID_RDMA_ROCE}=="1", ENV{SYSTEMD_WANTS}+="rdma-load-modules-eW91h9tu8Y5TDjBF/Jpztg@public.gmane.org"
+
+LABEL="rdma_ulp_modules_end"
diff --git a/rdma-core.spec b/rdma-core.spec
index b923a5d7636548..0c721158d537b4 100644
--- a/rdma-core.spec
+++ b/rdma-core.spec
@@ -139,4 +139,5 @@ rm -rf %{buildroot}/%{my_unitdir}/
%config %{_sysconfdir}/iwpmd.conf
%config %{_sysconfdir}/srp_daemon.conf
%config %{_sysconfdir}/libibverbs.d/*
+%config %{_sysconfdir}/rdma/modules/*
%{_sysconfdir}/modprobe.d/*
diff --git a/redhat/rdma-core.spec b/redhat/rdma-core.spec
index 4413418ffc44cc..9892566a0333f5 100644
--- a/redhat/rdma-core.spec
+++ b/redhat/rdma-core.spec
@@ -321,17 +321,26 @@ rm -rf %{buildroot}/%{_sbindir}/srp_daemon.sh
%doc %{_docdir}/%{name}-%{version}/README.md
%doc %{_docdir}/%{name}-%{version}/rxe.md
%config(noreplace) %{_sysconfdir}/rdma/mlx4.conf
+%config(noreplace) %{_sysconfdir}/rdma/modules/infiniband.conf
+%config(noreplace) %{_sysconfdir}/rdma/modules/iwarp.conf
+%config(noreplace) %{_sysconfdir}/rdma/modules/opa.conf
+%config(noreplace) %{_sysconfdir}/rdma/modules/rdma.conf
+%config(noreplace) %{_sysconfdir}/rdma/modules/roce.conf
%config(noreplace) %{_sysconfdir}/rdma/rdma.conf
%config(noreplace) %{_sysconfdir}/rdma/sriov-vfs
%config(noreplace) %{_sysconfdir}/udev/rules.d/*
%config(noreplace) %{_sysconfdir}/modprobe.d/mlx4.conf
%config(noreplace) %{_sysconfdir}/modprobe.d/truescale.conf
%{_sysconfdir}/sysconfig/network-scripts/*
+%{_unitdir}/rdma-load-modules@.service
%{_unitdir}/rdma.service
%dir %{dracutlibdir}/modules.d/05rdma
%{dracutlibdir}/modules.d/05rdma/module-setup.sh
-%{_udevrulesdir}/98-rdma.rules
%{_udevrulesdir}/60-rdma-ndd.rules
+%{_udevrulesdir}/75-rdma-description.rules
+%{_udevrulesdir}/90-rdma-hw-modules.rules
+%{_udevrulesdir}/90-rdma-ulp-modules.rules
+%{_udevrulesdir}/98-rdma.rules
%{sysmodprobedir}/libmlx4.conf
%{sysmodprobedir}/cxgb3.conf
%{sysmodprobedir}/cxgb4.conf
--
2.7.4
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 24+ messages in thread* [PATCH v2 rdma-core 2/6] Move umad udev rules to rdma-core
[not found] ` <1501194685-9447-1-git-send-email-jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2017-07-27 22:31 ` [PATCH v2 rdma-core 1/6] Common infrastructure for auto loading rdma modules Jason Gunthorpe
@ 2017-07-27 22:31 ` Jason Gunthorpe
2017-07-27 22:31 ` [PATCH v2 rdma-core 3/6] debian: Cause udev to evaluate the installed rules after install Jason Gunthorpe
` (5 subsequent siblings)
7 siblings, 0 replies; 24+ messages in thread
From: Jason Gunthorpe @ 2017-07-27 22:31 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA, Leon Romanovsky, Doug Ledford,
Ram Amrani, Steve Wise, Ira Weiny, Benjamin Drung, Jarod Wilson
This works better from a packaging perspective, eg rdma-core
is already a matched dependency for srp_daemon in RedHat, and provides
one place to get all the udev meta data rules.
Also update debian's dependencies to rely on a new RDMA core
for srptools, as it now requires the rules files and
rdma-load-modules@.service.in from rdma-core.
Signed-off-by: Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
---
debian/control | 5 ++++-
debian/libibumad3.install | 1 -
debian/rdma-core.install | 1 +
kernel-boot/CMakeLists.txt | 4 ++++
libibumad/libibumad.udev-rules => kernel-boot/rdma-umad.rules | 0
libibumad/CMakeLists.txt | 5 -----
redhat/rdma-core.spec | 2 +-
7 files changed, 10 insertions(+), 8 deletions(-)
rename libibumad/libibumad.udev-rules => kernel-boot/rdma-umad.rules (100%)
diff --git a/debian/control b/debian/control
index 444b0eaa047ba8..e9d1f95bb7a5ca 100644
--- a/debian/control
+++ b/debian/control
@@ -339,7 +339,10 @@ Description: Examples for the librdmacm library
Package: srptools
Architecture: linux-any
-Depends: lsb-base (>= 3.2-14~), ${misc:Depends}, ${shlibs:Depends}
+Depends: lsb-base (>= 3.2-14~),
+ rdma-core (>= 15),
+ ${misc:Depends},
+ ${shlibs:Depends}
Description: Tools for Infiniband attached storage (SRP)
In conjunction with the kernel ib_srp driver, srptools allows you to
discover and use Infiniband attached storage devices which use the
diff --git a/debian/libibumad3.install b/debian/libibumad3.install
index 41339ebfa2f229..f6c298db52eb34 100644
--- a/debian/libibumad3.install
+++ b/debian/libibumad3.install
@@ -1,2 +1 @@
-lib/udev/rules.d/90-libibumad.rules
usr/lib/*/libibumad*.so.*
diff --git a/debian/rdma-core.install b/debian/rdma-core.install
index 44437fcc83dbd8..8021a0c0d9242b 100644
--- a/debian/rdma-core.install
+++ b/debian/rdma-core.install
@@ -11,6 +11,7 @@ lib/udev/rules.d/60-rdma-ndd.rules
lib/udev/rules.d/75-rdma-description.rules
lib/udev/rules.d/90-rdma-hw-modules.rules
lib/udev/rules.d/90-rdma-ulp-modules.rules
+lib/udev/rules.d/90-rdma-umad.rules
usr/bin/rxe_cfg
usr/lib/truescale-serdes.cmds
usr/sbin/rdma-ndd
diff --git a/kernel-boot/CMakeLists.txt b/kernel-boot/CMakeLists.txt
index 0d4a2aec1c6a94..fdb70117f5899c 100644
--- a/kernel-boot/CMakeLists.txt
+++ b/kernel-boot/CMakeLists.txt
@@ -22,3 +22,7 @@ install(FILES "rdma-hw-modules.rules"
install(FILES "rdma-ulp-modules.rules"
RENAME "90-rdma-ulp-modules.rules"
DESTINATION "${CMAKE_INSTALL_UDEV_RULESDIR}")
+
+install(FILES "rdma-umad.rules"
+ RENAME "90-rdma-umad.rules"
+ DESTINATION "${CMAKE_INSTALL_UDEV_RULESDIR}")
diff --git a/libibumad/libibumad.udev-rules b/kernel-boot/rdma-umad.rules
similarity index 100%
rename from libibumad/libibumad.udev-rules
rename to kernel-boot/rdma-umad.rules
diff --git a/libibumad/CMakeLists.txt b/libibumad/CMakeLists.txt
index 84673ee01271cb..ba2463b47ed82a 100644
--- a/libibumad/CMakeLists.txt
+++ b/libibumad/CMakeLists.txt
@@ -14,8 +14,3 @@ rdma_library(ibumad libibumad.map
umad.c
umad_str.c
)
-
-install(FILES libibumad.udev-rules
- RENAME 90-libibumad.rules
- DESTINATION "${CMAKE_INSTALL_UDEV_RULESDIR}")
-
diff --git a/redhat/rdma-core.spec b/redhat/rdma-core.spec
index 9892566a0333f5..f1895ea260ac52 100644
--- a/redhat/rdma-core.spec
+++ b/redhat/rdma-core.spec
@@ -340,6 +340,7 @@ rm -rf %{buildroot}/%{_sbindir}/srp_daemon.sh
%{_udevrulesdir}/75-rdma-description.rules
%{_udevrulesdir}/90-rdma-hw-modules.rules
%{_udevrulesdir}/90-rdma-ulp-modules.rules
+%{_udevrulesdir}/90-rdma-umad.rules
%{_udevrulesdir}/98-rdma.rules
%{sysmodprobedir}/libmlx4.conf
%{sysmodprobedir}/cxgb3.conf
@@ -414,7 +415,6 @@ rm -rf %{buildroot}/%{_sbindir}/srp_daemon.sh
%files -n libibumad
%{_libdir}/libibumad*.so.*
-%{_udevrulesdir}/90-libibumad.rules
%files -n librdmacm
%{_libdir}/librdmacm*.so.*
--
2.7.4
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 24+ messages in thread* [PATCH v2 rdma-core 3/6] debian: Cause udev to evaluate the installed rules after install
[not found] ` <1501194685-9447-1-git-send-email-jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2017-07-27 22:31 ` [PATCH v2 rdma-core 1/6] Common infrastructure for auto loading rdma modules Jason Gunthorpe
2017-07-27 22:31 ` [PATCH v2 rdma-core 2/6] Move umad udev rules to rdma-core Jason Gunthorpe
@ 2017-07-27 22:31 ` Jason Gunthorpe
2017-07-27 22:31 ` [PATCH v2 rdma-core 4/6] redhat: Remove cxgb3/4.modprobe files Jason Gunthorpe
` (4 subsequent siblings)
7 siblings, 0 replies; 24+ messages in thread
From: Jason Gunthorpe @ 2017-07-27 22:31 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA, Leon Romanovsky, Doug Ledford,
Ram Amrani, Steve Wise, Ira Weiny, Benjamin Drung, Jarod Wilson
Call uadevadm trigger with all the subsystems our rules touch.
Following the advice of
https://lists.ubuntu.com/archives/ubuntu-devel/2009-January/027260.html
And general consensus of other posinst files we We inject a change
message and ignore failure. This means the udev rules must accept
a change message and cannot just use ACTION=="add".
Signed-off-by: Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
---
debian/rdma-core.postinst | 13 +++++++++++++
debian/srptools.postinst | 11 +++++++++++
kernel-boot/rdma-umad.rules | 2 +-
srp_daemon/srp_daemon.rules | 2 +-
4 files changed, 26 insertions(+), 2 deletions(-)
create mode 100644 debian/rdma-core.postinst
create mode 100644 debian/srptools.postinst
diff --git a/debian/rdma-core.postinst b/debian/rdma-core.postinst
new file mode 100644
index 00000000000000..de13363f0ad7c7
--- /dev/null
+++ b/debian/rdma-core.postinst
@@ -0,0 +1,13 @@
+#!/bin/sh
+
+set -e
+
+#DEBHELPER#
+
+if [ "$1" = "configure" ]; then
+ # we ship udev rules, so trigger an update. This has to be done after
+ # DEBHELPER restarts systemd to get our new service files loaded.
+ udevadm trigger --subsystem-match=infiniband --action=change || true
+ udevadm trigger --subsystem-match=net --action=change || true
+ udevadm trigger --subsystem-match=infiniband_mad --action=change || true
+fi
diff --git a/debian/srptools.postinst b/debian/srptools.postinst
new file mode 100644
index 00000000000000..c5a3e86261b338
--- /dev/null
+++ b/debian/srptools.postinst
@@ -0,0 +1,11 @@
+#!/bin/sh
+
+set -e
+
+#DEBHELPER#
+
+if [ "$1" = "configure" ]; then
+ # we ship udev rules, so trigger an update. This has to be done after
+ # DEBHELPER restarts systemd to get our new service files loaded.
+ udevadm trigger --subsystem-match=infiniband_mad --action=change || true
+fi
diff --git a/kernel-boot/rdma-umad.rules b/kernel-boot/rdma-umad.rules
index cfb790e051ced7..ba7ee613815a7d 100644
--- a/kernel-boot/rdma-umad.rules
+++ b/kernel-boot/rdma-umad.rules
@@ -1 +1 @@
-ACTION=="add", SUBSYSTEM=="infiniband_mad", KERNEL=="*umad*", TAG+="systemd", ENV{SYSTEMD_ALIAS}="/sys/subsystem/rdma/devices/$attr{ibdev}:$attr{port}/umad"
+SUBSYSTEM=="infiniband_mad", KERNEL=="*umad*", TAG+="systemd", ENV{SYSTEMD_ALIAS}="/sys/subsystem/rdma/devices/$attr{ibdev}:$attr{port}/umad"
diff --git a/srp_daemon/srp_daemon.rules b/srp_daemon/srp_daemon.rules
index 4cfeabc762da2a..b6411dcec72432 100644
--- a/srp_daemon/srp_daemon.rules
+++ b/srp_daemon/srp_daemon.rules
@@ -1 +1 @@
-ACTION=="add", SUBSYSTEM=="infiniband_mad", KERNEL=="*umad*", PROGRAM=="/bin/systemctl show srp_daemon -p ActiveState", RESULT=="ActiveState=active", ENV{SYSTEMD_WANTS}+="srp_daemon_port@$attr{ibdev}:$attr{port}.service"
+SUBSYSTEM=="infiniband_mad", KERNEL=="*umad*", PROGRAM=="/bin/systemctl show srp_daemon -p ActiveState", RESULT=="ActiveState=active", ENV{SYSTEMD_WANTS}+="srp_daemon_port@$attr{ibdev}:$attr{port}.service"
--
2.7.4
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 24+ messages in thread* [PATCH v2 rdma-core 4/6] redhat: Remove cxgb3/4.modprobe files
[not found] ` <1501194685-9447-1-git-send-email-jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
` (2 preceding siblings ...)
2017-07-27 22:31 ` [PATCH v2 rdma-core 3/6] debian: Cause udev to evaluate the installed rules after install Jason Gunthorpe
@ 2017-07-27 22:31 ` Jason Gunthorpe
2017-07-27 22:31 ` [PATCH v2 rdma-core 5/6] srp_daemon: Autoload the SRP kernel module if required Jason Gunthorpe
` (3 subsequent siblings)
7 siblings, 0 replies; 24+ messages in thread
From: Jason Gunthorpe @ 2017-07-27 22:31 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA, Leon Romanovsky, Doug Ledford,
Ram Amrani, Steve Wise, Ira Weiny, Benjamin Drung, Jarod Wilson
This is now handled by the udev rules in kernel-boot
Signed-off-by: Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
---
redhat/rdma-core.spec | 4 ----
redhat/rdma.cxgb3.sys.modprobe | 1 -
redhat/rdma.cxgb4.sys.modprobe | 1 -
3 files changed, 6 deletions(-)
delete mode 100644 redhat/rdma.cxgb3.sys.modprobe
delete mode 100644 redhat/rdma.cxgb4.sys.modprobe
diff --git a/redhat/rdma-core.spec b/redhat/rdma-core.spec
index f1895ea260ac52..a11de8df3f45a7 100644
--- a/redhat/rdma-core.spec
+++ b/redhat/rdma-core.spec
@@ -260,8 +260,6 @@ install -D -m0644 redhat/rdma.udev-ipoib-naming.rules %{buildroot}%{_sysconfdir}
install -D -m0755 redhat/rdma.modules-setup.sh %{buildroot}%{dracutlibdir}/modules.d/05rdma/module-setup.sh
install -D -m0644 redhat/rdma.udev-rules %{buildroot}%{_udevrulesdir}/98-rdma.rules
install -D -m0644 redhat/rdma.mlx4.sys.modprobe %{buildroot}%{sysmodprobedir}/libmlx4.conf
-install -D -m0644 redhat/rdma.cxgb3.sys.modprobe %{buildroot}%{sysmodprobedir}/cxgb3.conf
-install -D -m0644 redhat/rdma.cxgb4.sys.modprobe %{buildroot}%{sysmodprobedir}/cxgb4.conf
install -D -m0755 redhat/rdma.kernel-init %{buildroot}%{_libexecdir}/rdma-init-kernel
install -D -m0755 redhat/rdma.sriov-init %{buildroot}%{_libexecdir}/rdma-set-sriov-vf
install -D -m0644 redhat/rdma.fixup-mtrr.awk %{buildroot}%{_libexecdir}/rdma-fixup-mtrr.awk
@@ -343,8 +341,6 @@ rm -rf %{buildroot}/%{_sbindir}/srp_daemon.sh
%{_udevrulesdir}/90-rdma-umad.rules
%{_udevrulesdir}/98-rdma.rules
%{sysmodprobedir}/libmlx4.conf
-%{sysmodprobedir}/cxgb3.conf
-%{sysmodprobedir}/cxgb4.conf
%{_libexecdir}/rdma-init-kernel
%{_libexecdir}/rdma-set-sriov-vf
%{_libexecdir}/rdma-fixup-mtrr.awk
diff --git a/redhat/rdma.cxgb3.sys.modprobe b/redhat/rdma.cxgb3.sys.modprobe
deleted file mode 100644
index d5925a715ccee1..00000000000000
diff --git a/redhat/rdma.cxgb4.sys.modprobe b/redhat/rdma.cxgb4.sys.modprobe
deleted file mode 100644
index 44163ab43cf649..00000000000000
--
2.7.4
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 24+ messages in thread* [PATCH v2 rdma-core 5/6] srp_daemon: Autoload the SRP kernel module if required
[not found] ` <1501194685-9447-1-git-send-email-jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
` (3 preceding siblings ...)
2017-07-27 22:31 ` [PATCH v2 rdma-core 4/6] redhat: Remove cxgb3/4.modprobe files Jason Gunthorpe
@ 2017-07-27 22:31 ` Jason Gunthorpe
2017-07-27 22:31 ` [PATCH v2 rdma-core 6/6] iwpmd: Autoload iwpmd " Jason Gunthorpe
` (2 subsequent siblings)
7 siblings, 0 replies; 24+ messages in thread
From: Jason Gunthorpe @ 2017-07-27 22:31 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA, Leon Romanovsky, Doug Ledford,
Ram Amrani, Steve Wise, Ira Weiny, Benjamin Drung, Jarod Wilson
We use the rdma-core module loading infrastructure to load a srp_daemon
specific module list before allowing srp_daemon to start.
This is in lieu of kernel module auto loading, which is hard to trigger
for sysfs based interfaces.
Signed-off-by: Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
---
debian/srptools.install | 1 +
redhat/rdma-core.spec | 1 +
srp_daemon/CMakeLists.txt | 4 ++++
srp_daemon/modules-srp_daemon.conf | 2 ++
srp_daemon/srp_daemon.service.in | 2 +-
srp_daemon/srp_daemon_port@.service.in | 3 ++-
6 files changed, 11 insertions(+), 2 deletions(-)
create mode 100644 srp_daemon/modules-srp_daemon.conf
diff --git a/debian/srptools.install b/debian/srptools.install
index f780d663c9113b..a30798cf088a03 100644
--- a/debian/srptools.install
+++ b/debian/srptools.install
@@ -1,3 +1,4 @@
+etc/rdma/modules/srp_daemon.conf
etc/srp_daemon.conf
lib/systemd/system/srp_daemon.service
lib/systemd/system/srp_daemon_port@.service
diff --git a/redhat/rdma-core.spec b/redhat/rdma-core.spec
index a11de8df3f45a7..f1b23669e7e601 100644
--- a/redhat/rdma-core.spec
+++ b/redhat/rdma-core.spec
@@ -449,6 +449,7 @@ rm -rf %{buildroot}/%{_sbindir}/srp_daemon.sh
%files -n srp_daemon
%config(noreplace) %{_sysconfdir}/srp_daemon.conf
+%config(noreplace) %{_sysconfdir}/rdma/modules/srp_daemon.conf
%{_libexecdir}/srp_daemon/start_on_all_ports
%{_unitdir}/srp_daemon.service
%{_unitdir}/srp_daemon_port@.service
diff --git a/srp_daemon/CMakeLists.txt b/srp_daemon/CMakeLists.txt
index 92a1d709364b0b..0e7de9594b6375 100644
--- a/srp_daemon/CMakeLists.txt
+++ b/srp_daemon/CMakeLists.txt
@@ -47,6 +47,10 @@ install(FILES "srp_daemon.rules"
RENAME "60-srp_daemon.rules"
DESTINATION "${CMAKE_INSTALL_UDEV_RULESDIR}")
+install(FILES modules-srp_daemon.conf
+ RENAME "srp_daemon.conf"
+ DESTINATION "${CMAKE_INSTALL_SYSCONFDIR}/rdma/modules")
+
# FIXME: The ib init.d file should really be included in rdma-core as well.
set(RDMA_SERVICE "openibd" CACHE STRING "init.d file service name to order srpd after")
# NOTE: These defaults are for CentOS, packagers should override.
diff --git a/srp_daemon/modules-srp_daemon.conf b/srp_daemon/modules-srp_daemon.conf
new file mode 100644
index 00000000000000..dbe434362f10a3
--- /dev/null
+++ b/srp_daemon/modules-srp_daemon.conf
@@ -0,0 +1,2 @@
+# These modules are loaded by the system if srp_daemon is to be run
+ib_srp
diff --git a/srp_daemon/srp_daemon.service.in b/srp_daemon/srp_daemon.service.in
index 33dddd5cb46fef..cca1fce9c99283 100644
--- a/srp_daemon/srp_daemon.service.in
+++ b/srp_daemon/srp_daemon.service.in
@@ -1,6 +1,6 @@
[Unit]
Description=Daemon that discovers and logs in to SRP target systems
-Documentation=man:srp_daemon file:/etc/rdma/rdma.conf file:/etc/srp_daemon.conf
+Documentation=man:srp_daemon file:/etc/srp_daemon.conf
DefaultDependencies=false
Conflicts=emergency.target emergency.service
Before=remote-fs-pre.target
diff --git a/srp_daemon/srp_daemon_port@.service.in b/srp_daemon/srp_daemon_port@.service.in
index 2475db8640a9bd..5c215cb935bc73 100644
--- a/srp_daemon/srp_daemon_port@.service.in
+++ b/srp_daemon/srp_daemon_port@.service.in
@@ -3,7 +3,8 @@ Description=SRP daemon that monitors port %i
Documentation=man:srp_daemon file:/etc/rdma/rdma.conf file:/etc/srp_daemon.conf
DefaultDependencies=false
Conflicts=emergency.target emergency.service
-After=srp_daemon.service sys-subsystem-rdma-devices-%i-umad.device network.target
+Requires=rdma-load-modules@srp_daemon.service
+After=srp_daemon.service rdma-load-modules@srp_daemon.service sys-subsystem-rdma-devices-%i-umad.device network.target
BindsTo=srp_daemon.service sys-subsystem-rdma-devices-%i-umad.device
Before=remote-fs-pre.target
--
2.7.4
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 24+ messages in thread* [PATCH v2 rdma-core 6/6] iwpmd: Autoload iwpmd if required
[not found] ` <1501194685-9447-1-git-send-email-jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
` (4 preceding siblings ...)
2017-07-27 22:31 ` [PATCH v2 rdma-core 5/6] srp_daemon: Autoload the SRP kernel module if required Jason Gunthorpe
@ 2017-07-27 22:31 ` Jason Gunthorpe
2017-07-28 13:54 ` [PATCH v2 rdma-core 0/6] Common udev/systemd based module auto loading Steve Wise
2017-07-28 15:42 ` Benjamin Drung
7 siblings, 0 replies; 24+ messages in thread
From: Jason Gunthorpe @ 2017-07-27 22:31 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA, Leon Romanovsky, Doug Ledford,
Ram Amrani, Steve Wise, Ira Weiny, Benjamin Drung, Jarod Wilson
Use udev to determine if an iWarp device is present, and if so then start
iwpmd and auto load the required module. iwpmd no longer starts if it is
simply installed.
This will be unnecessary on future kernels with the netlink module
auto-loading patches.
Signed-off-by: Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
---
debian/iwpmd.install | 2 ++
debian/iwpmd.postinst | 11 +++++++++++
iwpmd/CMakeLists.txt | 8 ++++++++
iwpmd/iwpmd.rules | 1 +
iwpmd/iwpmd.service.in | 6 +++---
iwpmd/modules-iwpmd.conf | 2 ++
kernel-boot/modules/iwarp.conf | 1 -
redhat/rdma-core.spec | 2 ++
8 files changed, 29 insertions(+), 4 deletions(-)
create mode 100644 debian/iwpmd.postinst
create mode 100644 iwpmd/iwpmd.rules
create mode 100644 iwpmd/modules-iwpmd.conf
diff --git a/debian/iwpmd.install b/debian/iwpmd.install
index cc26eccf18a57d..172e4105cc0a85 100644
--- a/debian/iwpmd.install
+++ b/debian/iwpmd.install
@@ -1,6 +1,8 @@
etc/init.d/iwpmd
etc/iwpmd.conf
+etc/rdma/modules/iwpmd.conf
lib/systemd/system/iwpmd.service
+lib/udev/rules.d/90-iwpmd.rules
usr/sbin/iwpmd
usr/share/man/man5/iwpmd.conf.5
usr/share/man/man8/iwpmd.8
diff --git a/debian/iwpmd.postinst b/debian/iwpmd.postinst
new file mode 100644
index 00000000000000..cabfe7c2d80ebe
--- /dev/null
+++ b/debian/iwpmd.postinst
@@ -0,0 +1,11 @@
+#!/bin/sh
+
+set -e
+
+#DEBHELPER#
+
+if [ "$1" = "configure" ]; then
+ # we ship udev rules, so trigger an update. This has to be done after
+ # DEBHELPER restarts systemd to get our new service files loaded.
+ udevadm trigger --subsystem-match=infiniband --action=change || true
+fi
diff --git a/iwpmd/CMakeLists.txt b/iwpmd/CMakeLists.txt
index 856e3886f9ea81..6fd36e47bbbc54 100644
--- a/iwpmd/CMakeLists.txt
+++ b/iwpmd/CMakeLists.txt
@@ -21,3 +21,11 @@ rdma_subst_install(FILES "iwpmd_init.in"
RENAME "iwpmd"
PERMISSIONS OWNER_WRITE OWNER_READ GROUP_READ WORLD_READ OWNER_EXECUTE GROUP_EXECUTE WORLD_EXECUTE)
install(FILES "iwpmd.conf" DESTINATION "${CMAKE_INSTALL_SYSCONFDIR}")
+
+install(FILES "iwpmd.rules"
+ RENAME "90-iwpmd.rules"
+ DESTINATION "${CMAKE_INSTALL_UDEV_RULESDIR}")
+
+install(FILES modules-iwpmd.conf
+ RENAME "iwpmd.conf"
+ DESTINATION "${CMAKE_INSTALL_SYSCONFDIR}/rdma/modules")
diff --git a/iwpmd/iwpmd.rules b/iwpmd/iwpmd.rules
new file mode 100644
index 00000000000000..5b22cceaf29f8c
--- /dev/null
+++ b/iwpmd/iwpmd.rules
@@ -0,0 +1 @@
+TAG+="systemd", ENV{ID_RDMA_IWARP}=="1", ENV{SYSTEMD_WANTS}+="iwpmd.service"
diff --git a/iwpmd/iwpmd.service.in b/iwpmd/iwpmd.service.in
index 25740ebdcf9a82..4e4b49738fa29d 100644
--- a/iwpmd/iwpmd.service.in
+++ b/iwpmd/iwpmd.service.in
@@ -1,11 +1,11 @@
[Unit]
Description=iWarp Port Mapper
Documentation=man:iwpmd file:/etc/iwpmd.conf
-After=network.target
+Requires=rdma-load-modules-r79FGtN6zq8c5CkaFG5UGg@public.gmane.org
+After=network.target rdma-load-modules-r79FGtN6zq8c5CkaFG5UGg@public.gmane.org
[Service]
ExecStart=@CMAKE_INSTALL_FULL_SBINDIR@/iwpmd --systemd
LimitNOFILE=102400
-[Install]
-WantedBy=multi-user.target
+# iwpmd is automatically started by udev when an iWarp RDMA device is present
diff --git a/iwpmd/modules-iwpmd.conf b/iwpmd/modules-iwpmd.conf
new file mode 100644
index 00000000000000..5544b359c08814
--- /dev/null
+++ b/iwpmd/modules-iwpmd.conf
@@ -0,0 +1,2 @@
+# These modules are loaded by the system if iwpmd is to be run
+iw_cm
diff --git a/kernel-boot/modules/iwarp.conf b/kernel-boot/modules/iwarp.conf
index 882146e41ee2ba..0cb831da5bb557 100644
--- a/kernel-boot/modules/iwarp.conf
+++ b/kernel-boot/modules/iwarp.conf
@@ -1,2 +1 @@
# These modules are loaded by the system if any iWarp device is installed
-iw_cm
diff --git a/redhat/rdma-core.spec b/redhat/rdma-core.spec
index f1b23669e7e601..773c98e49eb062 100644
--- a/redhat/rdma-core.spec
+++ b/redhat/rdma-core.spec
@@ -401,7 +401,9 @@ rm -rf %{buildroot}/%{_sbindir}/srp_daemon.sh
%files -n iwpmd
%{_sbindir}/iwpmd
%{_unitdir}/iwpmd.service
+%config(noreplace) %{_sysconfdir}/rdma/modules/iwpmd.conf
%config(noreplace) %{_sysconfdir}/iwpmd.conf
+%{_udevrulesdir}/90-iwpmd.rules
%{_mandir}/man8/iwpmd.*
%{_mandir}/man5/iwpmd.*
--
2.7.4
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 24+ messages in thread* RE: [PATCH v2 rdma-core 0/6] Common udev/systemd based module auto loading
[not found] ` <1501194685-9447-1-git-send-email-jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
` (5 preceding siblings ...)
2017-07-27 22:31 ` [PATCH v2 rdma-core 6/6] iwpmd: Autoload iwpmd " Jason Gunthorpe
@ 2017-07-28 13:54 ` Steve Wise
2017-07-28 16:11 ` Jason Gunthorpe
2017-07-28 15:42 ` Benjamin Drung
7 siblings, 1 reply; 24+ messages in thread
From: Steve Wise @ 2017-07-28 13:54 UTC (permalink / raw)
To: 'Jason Gunthorpe', linux-rdma-u79uwXL29TY76Z2rM5mHXA,
'Leon Romanovsky', 'Doug Ledford',
'Ram Amrani', 'Ira Weiny',
'Benjamin Drung', 'Jarod Wilson'
>
> Here is the next revision of the systemd module autoloading approach.
>
> All the comments have been collected and a few other things addressed.
>
It would be great to have a change list from the previous version.
> Notably, at least on debian, when rdma-core is installed for the first time it
> activates the new udev rules and moduels autoload immediately, no reboot
> required. Not sure about RH's policies, so leaving this to them to copy if
> desired.
>
> There have been positive test results now for rxe, mlx4 (IB), qedr and cxgb4
> on Debian derived.
My cxgb4 testing was on the RHEL7.3 distro.
Steve.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 24+ messages in thread* Re: [PATCH v2 rdma-core 0/6] Common udev/systemd based module auto loading
2017-07-28 13:54 ` [PATCH v2 rdma-core 0/6] Common udev/systemd based module auto loading Steve Wise
@ 2017-07-28 16:11 ` Jason Gunthorpe
0 siblings, 0 replies; 24+ messages in thread
From: Jason Gunthorpe @ 2017-07-28 16:11 UTC (permalink / raw)
To: Steve Wise
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, 'Leon Romanovsky',
'Doug Ledford', 'Ram Amrani', 'Ira Weiny',
'Benjamin Drung', 'Jarod Wilson'
On Fri, Jul 28, 2017 at 08:54:42AM -0500, Steve Wise wrote:
> > Here is the next revision of the systemd module autoloading approach.
> >
> > All the comments have been collected and a few other things addressed.
>
> It would be great to have a change list from the previous version.
Heh, it is getting hard, this started out as a little four patch
series and somehow expanded into about a 40 patch monster.
Broadly
- Add udevadm trigger's to the post install
- Udev ID_RDMA_ROCE triggers for rxe
- Various documentation improvements
- Packaging fixes (eg ./rdma-core.spec works)
- Fixing ID_RDMA_OPA for hif1verbs
> > There have been positive test results now for rxe, mlx4 (IB), qedr and cxgb4
> > on Debian derived.
>
> My cxgb4 testing was on the RHEL7.3 distro.
Right, I was tring to say my testing was on Debian
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH v2 rdma-core 0/6] Common udev/systemd based module auto loading
[not found] ` <1501194685-9447-1-git-send-email-jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
` (6 preceding siblings ...)
2017-07-28 13:54 ` [PATCH v2 rdma-core 0/6] Common udev/systemd based module auto loading Steve Wise
@ 2017-07-28 15:42 ` Benjamin Drung
[not found] ` <1501256525.4572.44.camel-EIkl63zCoXaH+58JC4qpiA@public.gmane.org>
7 siblings, 1 reply; 24+ messages in thread
From: Benjamin Drung @ 2017-07-28 15:42 UTC (permalink / raw)
To: Jason Gunthorpe, linux-rdma-u79uwXL29TY76Z2rM5mHXA,
Leon Romanovsky, Doug Ledford, Ram Amrani, Steve Wise, Ira Weiny,
Jarod Wilson
Hi Jason,
Am Donnerstag, den 27.07.2017, 16:31 -0600 schrieb Jason Gunthorpe:
> Here is the next revision of the systemd module autoloading approach.
>
> All the comments have been collected and a few other things
> addressed.
>
> Notably, at least on debian, when rdma-core is installed for the
> first time it
> activates the new udev rules and moduels autoload immediately, no
> reboot
> required. Not sure about RH's policies, so leaving this to them to
> copy if
> desired.
>
> There have been positive test results now for rxe, mlx4 (IB), qedr
> and cxgb4
> on Debian derived.
>
> This is now a PR:
>
> https://github.com/linux-rdma/rdma-core/pull/174/commits
thanks for your work. I reviewed the commits and commented them on
GitHub.
I tested the whole patch set on our Debian jessie system (with Mellanox
OFED kernel modules on kernel 4.4), but it failed to load properly.
The mlx4_core module was loaded, but not mlx4_ib. The directory
/sys/class/infiniband is empty.
$ sudo find /sys/ -name '*mlx*'
/sys/bus/pci/drivers/mlx4_core
/sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/mlx4_port1_mtu
/sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/mlx4_port2_mtu
/sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/mlx4_port1
/sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/mlx4_port2
$ journalctl | grep mlx
mlx4_core: Mellanox ConnectX core driver v3.3-1.0.4 (03 Jul 2016)
mlx4_core: Initializing 0000:02:00.0
mlx4_core: device is working in RoCE mode: Roce V1
mlx4_core: gid_type 1 for UD QPs is not supported by the devicegid_type 0 was chosen instead
mlx4_core: UD QP Gid type is: V1
mlx4_core 0000:02:00.0: PCIe link speed is 8.0GT/s, device supports 8.0GT/s
mlx4_core 0000:02:00.0: PCIe link width is x8, device supports x8
No clue how to further debug it.
--
Benjamin Drung
System Developer
Debian & Ubuntu Developer
ProfitBricks GmbH
Greifswalder Str. 207
D - 10405 Berlin
Email: benjamin.drung-EIkl63zCoXaH+58JC4qpiA@public.gmane.org
Web: https://www.profitbricks.com
Sitz der Gesellschaft: Berlin.
Registergericht: Amtsgericht Charlottenburg, HRB 125506B.
Geschäftsführer: Achim Weiss.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 24+ messages in thread