[PATCH v2 rdma-core 0/6] Common udev/systemd based module auto loading

[PATCH v2 rdma-core 0/6] Common udev/systemd based module auto loading

[Jason Gunthorpe]

Here is the next revision of the systemd module autoloading approach.

All the comments have been collected and a few other things addressed.

Notably, at least on debian, when rdma-core is installed for the first time it
activates the new udev rules and moduels autoload immediately, no reboot
required. Not sure about RH's policies, so leaving this to them to copy if
desired.

There have been positive test results now for rxe, mlx4 (IB), qedr and cxgb4
on Debian derived.

This is now a PR:

 https://github.com/linux-rdma/rdma-core/pull/174/commits

Jason Gunthorpe (6):
  Common infrastructure for auto loading rdma modules
  Move umad udev rules to rdma-core
  debian: Cause udev to evaluate the installed rules after install
  redhat: Remove cxgb3/4.modprobe files
  srp_daemon: Autoload the SRP kernel module if required
  iwpmd: Autoload iwpmd if required

 CMakeLists.txt                            |  1 +
 Documentation/udev.md                     | 83 +++++++++++++++++++++++++++++++
 debian/control                            |  5 +-
 debian/iwpmd.install                      |  2 +
 debian/iwpmd.postinst                     | 11 ++++
 debian/libibumad3.install                 |  1 -
 debian/rdma-core.install                  | 10 ++++
 debian/rdma-core.postinst                 | 13 +++++
 debian/srptools.install                   |  1 +
 debian/srptools.postinst                  | 11 ++++
 iwpmd/CMakeLists.txt                      |  8 +++
 iwpmd/iwpmd.rules                         |  1 +
 iwpmd/iwpmd.service.in                    |  6 +--
 iwpmd/modules-iwpmd.conf                  |  2 +
 kernel-boot/CMakeLists.txt                | 28 +++++++++++
 kernel-boot/modules/infiniband.conf       | 12 +++++
 kernel-boot/modules/iwarp.conf            |  1 +
 kernel-boot/modules/opa.conf              | 10 ++++
 kernel-boot/modules/rdma.conf             | 21 ++++++++
 kernel-boot/modules/roce.conf             |  2 +
 kernel-boot/rdma-description.rules        | 42 ++++++++++++++++
 kernel-boot/rdma-hw-modules.rules         | 39 +++++++++++++++
 kernel-boot/rdma-load-modules@.service.in | 15 ++++++
 kernel-boot/rdma-ulp-modules.rules        | 11 ++++
 kernel-boot/rdma-umad.rules               |  1 +
 libibumad/CMakeLists.txt                  |  5 --
 libibumad/libibumad.udev-rules            |  1 -
 rdma-core.spec                            |  1 +
 redhat/rdma-core.spec                     | 20 +++++---
 redhat/rdma.cxgb3.sys.modprobe            |  1 -
 redhat/rdma.cxgb4.sys.modprobe            |  1 -
 srp_daemon/CMakeLists.txt                 |  4 ++
 srp_daemon/modules-srp_daemon.conf        |  2 +
 srp_daemon/srp_daemon.rules               |  2 +-
 srp_daemon/srp_daemon.service.in          |  2 +-
 srp_daemon/srp_daemon_port@.service.in    |  3 +-
 36 files changed, 357 insertions(+), 22 deletions(-)
 create mode 100644 Documentation/udev.md
 create mode 100644 debian/iwpmd.postinst
 create mode 100644 debian/rdma-core.postinst
 create mode 100644 debian/srptools.postinst
 create mode 100644 iwpmd/iwpmd.rules
 create mode 100644 iwpmd/modules-iwpmd.conf
 create mode 100644 kernel-boot/CMakeLists.txt
 create mode 100644 kernel-boot/modules/infiniband.conf
 create mode 100644 kernel-boot/modules/iwarp.conf
 create mode 100644 kernel-boot/modules/opa.conf
 create mode 100644 kernel-boot/modules/rdma.conf
 create mode 100644 kernel-boot/modules/roce.conf
 create mode 100644 kernel-boot/rdma-description.rules
 create mode 100644 kernel-boot/rdma-hw-modules.rules
 create mode 100644 kernel-boot/rdma-load-modules@.service.in
 create mode 100644 kernel-boot/rdma-ulp-modules.rules
 create mode 100644 kernel-boot/rdma-umad.rules
 delete mode 100644 libibumad/libibumad.udev-rules
 delete mode 100644 redhat/rdma.cxgb3.sys.modprobe
 delete mode 100644 redhat/rdma.cxgb4.sys.modprobe
 create mode 100644 srp_daemon/modules-srp_daemon.conf

-- 
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 rdma-core 1/6] Common infrastructure for auto loading rdma modules

[Jason Gunthorpe]

This is inspired by the similar approach in the redhat directory but
takes a more general approach relying on udev and systemd to do the
actual work fully dynamically instead of a oneshot shell script.

Loading is split into two cases
 1) Loading RDMA support modules when RDMA capable hardware is installed.
    This is only needed for ethernet devices which do not load their RDMA
    support modules via request_module in the kernel.

    udev is used to detect when an ethernet device controlled by a specific
    module is hot plugged and then udev directly loads the RDMA module

 2) Loading RDMA ULP support when RDMA hardware is installed
    This is done by having udev detect when RDMA hardware is installed and
    udev causes systemd to load a list of modules from config files in
    /etc/rdma/modules/

    The user can customize these files to select which ULP modules should be
    loaded.

This broadly replaces the redhat/rdma.conf scheme.

In all cases the users can prevent a module from being auto-loaded on their
system by blacking listing it in a file in /etc/modprobe.d/

Signed-off-by: Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
---
 CMakeLists.txt                            |  1 +
 Documentation/udev.md                     | 83 +++++++++++++++++++++++++++++++
 debian/rdma-core.install                  |  9 ++++
 kernel-boot/CMakeLists.txt                | 24 +++++++++
 kernel-boot/modules/infiniband.conf       | 12 +++++
 kernel-boot/modules/iwarp.conf            |  2 +
 kernel-boot/modules/opa.conf              | 10 ++++
 kernel-boot/modules/rdma.conf             | 21 ++++++++
 kernel-boot/modules/roce.conf             |  2 +
 kernel-boot/rdma-description.rules        | 42 ++++++++++++++++
 kernel-boot/rdma-hw-modules.rules         | 39 +++++++++++++++
 kernel-boot/rdma-load-modules@.service.in | 15 ++++++
 kernel-boot/rdma-ulp-modules.rules        | 11 ++++
 rdma-core.spec                            |  1 +
 redhat/rdma-core.spec                     | 11 +++-
 15 files changed, 282 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/udev.md
 create mode 100644 kernel-boot/CMakeLists.txt
 create mode 100644 kernel-boot/modules/infiniband.conf
 create mode 100644 kernel-boot/modules/iwarp.conf
 create mode 100644 kernel-boot/modules/opa.conf
 create mode 100644 kernel-boot/modules/rdma.conf
 create mode 100644 kernel-boot/modules/roce.conf
 create mode 100644 kernel-boot/rdma-description.rules
 create mode 100644 kernel-boot/rdma-hw-modules.rules
 create mode 100644 kernel-boot/rdma-load-modules@.service.in
 create mode 100644 kernel-boot/rdma-ulp-modules.rules

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 16196205035f61..a03d8da31cbc5d 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -398,6 +398,7 @@ configure_file("${BUILDLIB}/config.h.in" "${BUILD_INCLUDE}/config.h" ESCAPE_QUOT
 add_subdirectory(ccan)
 add_subdirectory(util)
 add_subdirectory(Documentation)
+add_subdirectory(kernel-boot)
 # Libraries
 add_subdirectory(libibumad)
 add_subdirectory(libibumad/man)
diff --git a/Documentation/udev.md b/Documentation/udev.md
new file mode 100644
index 00000000000000..a6328d322c5194
--- /dev/null
+++ b/Documentation/udev.md
@@ -0,0 +1,83 @@
+# Kernel Module Loading
+
+The RDMA subsystem relies on the kernel, udev and systemd to load modules on
+demand when RDMA hardware is present. The RDMA subsystem is unique in that we
+do not load the optional RDMA hardware modules unless the system has the
+rdma-core package installed.
+
+This is to avoid exposing systems not using RDMA from having RDMA enabled, for
+instance if a system has a multi-protocol ethernet adapter, but is only using
+the net stack interface.
+
+## Boot ordering with systemd
+
+systemd assumes everything is hot pluggable and runs in an event driven
+manner.  When working with RDMA devices we are firstly concerned with when the
+physical hardware its module loaded into the kernel.
+
+This can happen in several spots along the bootup:
+
+ - From the initrd or built into the kernel. If hardware modules are present
+   inthe initrd then they are loaded into the kernel before booting the
+   system. This is done largely synchronously with the boot process.
+
+ - From udev when it auto detects PCI hardware or otherwise.
+   This happens asynchronously in the boot process, systemd does not wait for
+   udev to finish loading modules before it continues on.
+
+   This path makes it very likely the system will experience a RDMA 'hot plug'
+   scenario.
+
+ - From systemd's fixed module loader systemd-modules-load.service, eg from
+   the list in /etc/modules-load.d/. In this case the modules load happens
+   synchronously within systemd and it will hold off sysinit.target until
+   modules are loaded
+
+Once the hardware module is loaded it may be necessary to load a protocol
+module, eg to enable RDMA support on an ethernet device.
+
+This is triggered automatically by udev rules that match the master devices
+and load the protocol module with udev's module loader. This happens
+asynchronously to the rest of the systemd startup.
+
+Once a RDMA device is created by the kernel then udev will cause systemd to
+schedule ULP module loading services (eg rdma-load-modules@.service) specific
+to the plugged hardware. If sysinit.target has not yet been passed then these
+loaders will defer sysinit.target until they complete, otherwise this is a hot
+plug event and things will load asynchronously to the boot up process.
+
+Finally udev will cause systemd to start RDMA specific daemons like
+srp_daemon, rdma-ndd and iwpmd. These starts are linked to the detection of
+the first RDMA hardware, and the daemons internally handle hot plug events for
+other hardware.
+
+## Hot Plug compatible services
+
+Services using RDMA need to have device specific systemd dependencies in their
+unit files, either created by hand by the admin or by using udev rules.
+
+For instance, a service that uses /dev/infiniband/umad0 requires:
+
+```
+After=dev-infiniband-umad0.device
+BindsTo=dev-infiniband-umad0.device
+```
+
+Which will ensure the service will not run until the required umad device
+appears.
+
+This is similar to how systemd handles mounting filesystems and configuring
+ethernet devices.
+
+## Interaction with legacy non-hotplug services
+
+Services that cannot handle hot plug must be ordered after
+systemd-udev-settle.service, which will wait for udev to complete loading
+modules and scheduling systemd services. This ensures that all RDMA hardware
+present at boot is setup before proceeding to run the legacy service.
+
+Admins using legacy services can also place their RDMA hardware modules (eg
+mlx4_ib) directly in /etc/modules-load.d/ or in their initrd which will cause
+systemd to defer passing to sysinit.target until all RDMA hardware is setup,
+this is usually sufficient for legacy services. This is probably the default
+behavior in many configurations.
diff --git a/debian/rdma-core.install b/debian/rdma-core.install
index 0b35ab6372fcbe..44437fcc83dbd8 100644
--- a/debian/rdma-core.install
+++ b/debian/rdma-core.install
@@ -1,7 +1,16 @@
 etc/modprobe.d/mlx4.conf
 etc/modprobe.d/truescale.conf
+etc/rdma/modules/infiniband.conf
+etc/rdma/modules/iwarp.conf
+etc/rdma/modules/opa.conf
+etc/rdma/modules/rdma.conf
+etc/rdma/modules/roce.conf
+lib/systemd/system/rdma-load-modules@.service
 lib/systemd/system/rdma-ndd.service
 lib/udev/rules.d/60-rdma-ndd.rules
+lib/udev/rules.d/75-rdma-description.rules
+lib/udev/rules.d/90-rdma-hw-modules.rules
+lib/udev/rules.d/90-rdma-ulp-modules.rules
 usr/bin/rxe_cfg
 usr/lib/truescale-serdes.cmds
 usr/sbin/rdma-ndd
diff --git a/kernel-boot/CMakeLists.txt b/kernel-boot/CMakeLists.txt
new file mode 100644
index 00000000000000..0d4a2aec1c6a94
--- /dev/null
+++ b/kernel-boot/CMakeLists.txt
@@ -0,0 +1,24 @@
+rdma_subst_install(FILES rdma-load-modules@.service.in
+  DESTINATION "${CMAKE_INSTALL_SYSTEMD_SERVICEDIR}"
+  RENAME rdma-load-modules@.service
+  PERMISSIONS OWNER_WRITE OWNER_READ GROUP_READ WORLD_READ)
+
+install(FILES
+  modules/infiniband.conf
+  modules/iwarp.conf
+  modules/opa.conf
+  modules/rdma.conf
+  modules/roce.conf
+  DESTINATION "${CMAKE_INSTALL_SYSCONFDIR}/rdma/modules")
+
+install(FILES "rdma-description.rules"
+  RENAME "75-rdma-description.rules"
+  DESTINATION "${CMAKE_INSTALL_UDEV_RULESDIR}")
+
+install(FILES "rdma-hw-modules.rules"
+  RENAME "90-rdma-hw-modules.rules"
+  DESTINATION "${CMAKE_INSTALL_UDEV_RULESDIR}")
+
+install(FILES "rdma-ulp-modules.rules"
+  RENAME "90-rdma-ulp-modules.rules"
+  DESTINATION "${CMAKE_INSTALL_UDEV_RULESDIR}")
diff --git a/kernel-boot/modules/infiniband.conf b/kernel-boot/modules/infiniband.conf
new file mode 100644
index 00000000000000..99526e156fff40
--- /dev/null
+++ b/kernel-boot/modules/infiniband.conf
@@ -0,0 +1,12 @@
+# These modules are loaded by the system if any InfiniBand device is installed
+# InfiniBand over IP netdevice
+ib_ipoib
+
+# Access to fabric management SMPs and GMPs from userspace.
+ib_umad
+
+# SCSI Remote Protocol target support
+# ib_srpt
+
+# ib_ucm provides the obsolete /dev/infiniband/ucm0
+# ib_ucm
diff --git a/kernel-boot/modules/iwarp.conf b/kernel-boot/modules/iwarp.conf
new file mode 100644
index 00000000000000..882146e41ee2ba
--- /dev/null
+++ b/kernel-boot/modules/iwarp.conf
@@ -0,0 +1,2 @@
+# These modules are loaded by the system if any iWarp device is installed
+iw_cm
diff --git a/kernel-boot/modules/opa.conf b/kernel-boot/modules/opa.conf
new file mode 100644
index 00000000000000..b9bc9f1f0146af
--- /dev/null
+++ b/kernel-boot/modules/opa.conf
@@ -0,0 +1,10 @@
+# These modules are loaded by the system if any OmniPath Architecture device
+# is installed
+# Infiniband over IP netdevice
+ib_ipoib
+
+# Access to fabric management SMPs and GMPs from userspace.
+ib_umad
+
+# Omnipath Ethernet Virtual NIC netdevice
+opa_vnic
diff --git a/kernel-boot/modules/rdma.conf b/kernel-boot/modules/rdma.conf
new file mode 100644
index 00000000000000..2d342dd82f7db0
--- /dev/null
+++ b/kernel-boot/modules/rdma.conf
@@ -0,0 +1,21 @@
+# These modules are loaded by the system if any RDMA devices is installed
+# iSCSI over RDMA client support
+ib_iser
+
+# iSCSI over RDMA target support
+# ib_isert
+
+# User access to RDMA verbs (supports libibverbs)
+ib_uverbs
+
+# User access to RDMA connection management (supports librdmacm)
+rdma_ucm
+
+# RDS over RDMA support
+# rds_rdma
+
+# NFS over RDMA client support
+xprtrdma
+
+# NFS over RDMA server support
+svcrdma
diff --git a/kernel-boot/modules/roce.conf b/kernel-boot/modules/roce.conf
new file mode 100644
index 00000000000000..8e4927ce26f043
--- /dev/null
+++ b/kernel-boot/modules/roce.conf
@@ -0,0 +1,2 @@
+# These modules are loaded by the system if any RDMA over Converged Ethernet
+# device is installed
diff --git a/kernel-boot/rdma-description.rules b/kernel-boot/rdma-description.rules
new file mode 100644
index 00000000000000..4d7c5808401ac7
--- /dev/null
+++ b/kernel-boot/rdma-description.rules
@@ -0,0 +1,42 @@
+# This is a version of net-description.rules for /sys/class/infiniband devices
+
+ACTION=="remove", GOTO="rdma_description_end"
+SUBSYSTEM!="infiniband", GOTO="rdma_description_end"
+
+# NOTE: DRIVERS searches up the sysfs path to find the driver that is bound to
+# the PCI/etc device that the RDMA device is linked to. This is not the kernel
+# driver that is supplying the RDMA device (eg as seen in ID_NET_DRIVER)
+
+# FIXME: with kernel support we could actually detect the protocols the RDMA
+# driver itself supports, this is a work around for lack of that support.
+# In future we could do this with a udev IMPORT{program} helper program
+# that extracted the ID information from the RDMA netlink.
+
+# Hardware that supports InfiniBand
+DRIVERS=="mlx4_core", ENV{ID_RDMA_INFINIBAND}="1"
+DRIVERS=="mlx5_core", ENV{ID_RDMA_INFINIBAND}="1"
+DRIVERS=="qib", ENV{ID_RDMA_INFINIBAND}="1"
+
+# Hardware that supports OPA
+DRIVERS=="hfi1verbs", ENV{ID_RDMA_OPA}="1"
+
+# Hardware that supports iWarp
+DRIVERS=="cxgb3", ENV{ID_RDMA_IWARP}="1"
+DRIVERS=="cxgb4", ENV{ID_RDMA_IWARP}="1"
+
+# Hardware that supports RoCE
+DRIVERS=="be2net", ENV{ID_RDMA_ROCE}="1"
+DRIVERS=="bnxt_en", ENV{ID_RDMA_ROCE}="1"
+DRIVERS=="hns", ENV{ID_RDMA_ROCE}="1"
+DRIVERS=="i40e", ENV{ID_RDMA_ROCE}="1"
+DRIVERS=="mlx4_core", ENV{ID_RDMA_ROCE}="1"
+DRIVERS=="mlx5_core", ENV{ID_RDMA_ROCE}="1"
+DRIVERS=="qede", ENV{ID_RDMA_ROCE}="1"
+DEVPATH=="*/infiniband/rxe*", ATTR{parent}=="*", ENV{ID_RDMA_ROCE}="1"
+
+# Setup the usual ID information so that systemd will display a sane name for
+# the RDMA device units.
+SUBSYSTEMS=="pci", ENV{ID_BUS}="pci", ENV{ID_VENDOR_ID}="$attr{vendor}", ENV{ID_MODEL_ID}="$attr{device}"
+SUBSYSTEMS=="pci", IMPORT{builtin}="hwdb --subsystem=pci"
+
+LABEL="rdma_description_end"
diff --git a/kernel-boot/rdma-hw-modules.rules b/kernel-boot/rdma-hw-modules.rules
new file mode 100644
index 00000000000000..dde0ab8dacacab
--- /dev/null
+++ b/kernel-boot/rdma-hw-modules.rules
@@ -0,0 +1,39 @@
+ACTION=="remove", GOTO="rdma_hw_modules_end"
+SUBSYSTEM!="net", GOTO="rdma_hw_modules_end"
+
+# Automatically load RDMA specific kernel modules when a multi-function device is installed
+
+# These drivers autoload an ethernet driver based on hardware detection and
+# need userspace to load the module that has their RDMA component to turn on
+# RDMA.
+ENV{ID_NET_DRIVER}=="be2net", RUN{builtin}+="kmod load ocrdma"
+ENV{ID_NET_DRIVER}=="bnxt_en", RUN{builtin}+="kmod load bnxt_re"
+ENV{ID_NET_DRIVER}=="cxgb3", RUN{builtin}+="kmod load iw_cxgb3"
+ENV{ID_NET_DRIVER}=="cxgb4", RUN{builtin}+="kmod load iw_cxgb4"
+ENV{ID_NET_DRIVER}=="hns", RUN{builtin}+="kmod load hns_roce"
+ENV{ID_NET_DRIVER}=="i40e", RUN{builtin}+="kmod load i40iw"
+ENV{ID_NET_DRIVER}=="mlx4_en", RUN{builtin}+="kmod load mlx4_ib"
+ENV{ID_NET_DRIVER}=="mlx5_core", RUN{builtin}+="kmod load mlx5_ib"
+ENV{ID_NET_DRIVER}=="qede", RUN{builtin}+="kmod load qedr"
+
+# The user must explicitly load these modules via /etc/modules-load.d/ or otherwise
+# rxe
+
+# When in IB mode the kernel PCI core module autoloads the protocol modules
+# for these providers
+# mlx4
+# mlx5
+
+# enic no longer has a userspace verbs driver, this rule should probably be
+# owned by libfabric
+ENV{ID_NET_DRIVER}=="enic", RUN{builtin}+="kmod load usnic_verbs"
+
+# These providers are single function and autoload RDMA automatically based on
+# PCI probing
+# hfi1verbs
+# ipathverbs
+# mthca
+# vmw_pvrdma
+# nes
+
+LABEL="rdma_hw_modules_end"
diff --git a/kernel-boot/rdma-load-modules@.service.in b/kernel-boot/rdma-load-modules@.service.in
new file mode 100644
index 00000000000000..b35a493ebf230b
--- /dev/null
+++ b/kernel-boot/rdma-load-modules@.service.in
@@ -0,0 +1,15 @@
+[Unit]
+Description=Load RDMA modules from @CMAKE_INSTALL_SYSCONFDIR@/rdma/modules/%I.conf
+DefaultDependencies=no
+Conflicts=shutdown.target
+# network-pre.target is to support distro network setup scripts that run after
+# systemd-modules-load.service but before sysinit.target, eg a classic network
+# setup script.
+Before=sysinit.target shutdown.target network-pre.target
+ConditionCapability=CAP_SYS_MODULE
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+ExecStart=/lib/systemd/systemd-modules-load @CMAKE_INSTALL_SYSCONFDIR@/rdma/modules/%I.conf
+TimeoutSec=90s
diff --git a/kernel-boot/rdma-ulp-modules.rules b/kernel-boot/rdma-ulp-modules.rules
new file mode 100644
index 00000000000000..c090700c754b19
--- /dev/null
+++ b/kernel-boot/rdma-ulp-modules.rules
@@ -0,0 +1,11 @@
+ACTION=="remove", GOTO="rdma_ulp_modules_end"
+SUBSYSTEM!="infiniband", GOTO="rdma_ulp_modules_end"
+
+# Automatically load general RDMA ULP modules when RDMA hardware is installed
+TAG+="systemd", ENV{SYSTEMD_WANTS}+="rdma-load-modules-XSrBNAzizW1TDjBF/Jpztg@public.gmane.org"
+TAG+="systemd", ENV{ID_RDMA_INFINIBAND}=="1", ENV{SYSTEMD_WANTS}+="rdma-load-modules-WW8hglTQ5Ylgot6vXr2xLYq0XWbe/Iga@public.gmane.org"
+TAG+="systemd", ENV{ID_RDMA_IWARP}=="1", ENV{SYSTEMD_WANTS}+="rdma-load-modules-mPXOxrv7gkYc5CkaFG5UGg@public.gmane.org"
+TAG+="systemd", ENV{ID_RDMA_OPA}=="1", ENV{SYSTEMD_WANTS}+="rdma-load-modules-EhpGOaQAOk9DcWalurthmA@public.gmane.org"
+TAG+="systemd", ENV{ID_RDMA_ROCE}=="1", ENV{SYSTEMD_WANTS}+="rdma-load-modules-eW91h9tu8Y5TDjBF/Jpztg@public.gmane.org"
+
+LABEL="rdma_ulp_modules_end"
diff --git a/rdma-core.spec b/rdma-core.spec
index b923a5d7636548..0c721158d537b4 100644
--- a/rdma-core.spec
+++ b/rdma-core.spec
@@ -139,4 +139,5 @@ rm -rf %{buildroot}/%{my_unitdir}/
 %config %{_sysconfdir}/iwpmd.conf
 %config %{_sysconfdir}/srp_daemon.conf
 %config %{_sysconfdir}/libibverbs.d/*
+%config %{_sysconfdir}/rdma/modules/*
 %{_sysconfdir}/modprobe.d/*
diff --git a/redhat/rdma-core.spec b/redhat/rdma-core.spec
index 4413418ffc44cc..9892566a0333f5 100644
--- a/redhat/rdma-core.spec
+++ b/redhat/rdma-core.spec
@@ -321,17 +321,26 @@ rm -rf %{buildroot}/%{_sbindir}/srp_daemon.sh
 %doc %{_docdir}/%{name}-%{version}/README.md
 %doc %{_docdir}/%{name}-%{version}/rxe.md
 %config(noreplace) %{_sysconfdir}/rdma/mlx4.conf
+%config(noreplace) %{_sysconfdir}/rdma/modules/infiniband.conf
+%config(noreplace) %{_sysconfdir}/rdma/modules/iwarp.conf
+%config(noreplace) %{_sysconfdir}/rdma/modules/opa.conf
+%config(noreplace) %{_sysconfdir}/rdma/modules/rdma.conf
+%config(noreplace) %{_sysconfdir}/rdma/modules/roce.conf
 %config(noreplace) %{_sysconfdir}/rdma/rdma.conf
 %config(noreplace) %{_sysconfdir}/rdma/sriov-vfs
 %config(noreplace) %{_sysconfdir}/udev/rules.d/*
 %config(noreplace) %{_sysconfdir}/modprobe.d/mlx4.conf
 %config(noreplace) %{_sysconfdir}/modprobe.d/truescale.conf
 %{_sysconfdir}/sysconfig/network-scripts/*
+%{_unitdir}/rdma-load-modules@.service
 %{_unitdir}/rdma.service
 %dir %{dracutlibdir}/modules.d/05rdma
 %{dracutlibdir}/modules.d/05rdma/module-setup.sh
-%{_udevrulesdir}/98-rdma.rules
 %{_udevrulesdir}/60-rdma-ndd.rules
+%{_udevrulesdir}/75-rdma-description.rules
+%{_udevrulesdir}/90-rdma-hw-modules.rules
+%{_udevrulesdir}/90-rdma-ulp-modules.rules
+%{_udevrulesdir}/98-rdma.rules
 %{sysmodprobedir}/libmlx4.conf
 %{sysmodprobedir}/cxgb3.conf
 %{sysmodprobedir}/cxgb4.conf
-- 
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 rdma-core 2/6] Move umad udev rules to rdma-core

[Jason Gunthorpe]

This works better from a packaging perspective, eg rdma-core
is already a matched dependency for srp_daemon in RedHat, and provides
one place to get all the udev meta data rules.

Also update debian's dependencies to rely on a new RDMA core
for srptools, as it now requires the rules files and
rdma-load-modules@.service.in from rdma-core.

Signed-off-by: Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
---
 debian/control                                                | 5 ++++-
 debian/libibumad3.install                                     | 1 -
 debian/rdma-core.install                                      | 1 +
 kernel-boot/CMakeLists.txt                                    | 4 ++++
 libibumad/libibumad.udev-rules => kernel-boot/rdma-umad.rules | 0
 libibumad/CMakeLists.txt                                      | 5 -----
 redhat/rdma-core.spec                                         | 2 +-
 7 files changed, 10 insertions(+), 8 deletions(-)
 rename libibumad/libibumad.udev-rules => kernel-boot/rdma-umad.rules (100%)

diff --git a/debian/control b/debian/control
index 444b0eaa047ba8..e9d1f95bb7a5ca 100644
--- a/debian/control
+++ b/debian/control
@@ -339,7 +339,10 @@ Description: Examples for the librdmacm library
 
 Package: srptools
 Architecture: linux-any
-Depends: lsb-base (>= 3.2-14~), ${misc:Depends}, ${shlibs:Depends}
+Depends: lsb-base (>= 3.2-14~),
+         rdma-core (>= 15),
+         ${misc:Depends},
+         ${shlibs:Depends}
 Description: Tools for Infiniband attached storage (SRP)
  In conjunction with the kernel ib_srp driver, srptools allows you to
  discover and use Infiniband attached storage devices which use the
diff --git a/debian/libibumad3.install b/debian/libibumad3.install
index 41339ebfa2f229..f6c298db52eb34 100644
--- a/debian/libibumad3.install
+++ b/debian/libibumad3.install
@@ -1,2 +1 @@
-lib/udev/rules.d/90-libibumad.rules
 usr/lib/*/libibumad*.so.*
diff --git a/debian/rdma-core.install b/debian/rdma-core.install
index 44437fcc83dbd8..8021a0c0d9242b 100644
--- a/debian/rdma-core.install
+++ b/debian/rdma-core.install
@@ -11,6 +11,7 @@ lib/udev/rules.d/60-rdma-ndd.rules
 lib/udev/rules.d/75-rdma-description.rules
 lib/udev/rules.d/90-rdma-hw-modules.rules
 lib/udev/rules.d/90-rdma-ulp-modules.rules
+lib/udev/rules.d/90-rdma-umad.rules
 usr/bin/rxe_cfg
 usr/lib/truescale-serdes.cmds
 usr/sbin/rdma-ndd
diff --git a/kernel-boot/CMakeLists.txt b/kernel-boot/CMakeLists.txt
index 0d4a2aec1c6a94..fdb70117f5899c 100644
--- a/kernel-boot/CMakeLists.txt
+++ b/kernel-boot/CMakeLists.txt
@@ -22,3 +22,7 @@ install(FILES "rdma-hw-modules.rules"
 install(FILES "rdma-ulp-modules.rules"
   RENAME "90-rdma-ulp-modules.rules"
   DESTINATION "${CMAKE_INSTALL_UDEV_RULESDIR}")
+
+install(FILES "rdma-umad.rules"
+  RENAME "90-rdma-umad.rules"
+  DESTINATION "${CMAKE_INSTALL_UDEV_RULESDIR}")
diff --git a/libibumad/libibumad.udev-rules b/kernel-boot/rdma-umad.rules
similarity index 100%
rename from libibumad/libibumad.udev-rules
rename to kernel-boot/rdma-umad.rules
diff --git a/libibumad/CMakeLists.txt b/libibumad/CMakeLists.txt
index 84673ee01271cb..ba2463b47ed82a 100644
--- a/libibumad/CMakeLists.txt
+++ b/libibumad/CMakeLists.txt
@@ -14,8 +14,3 @@ rdma_library(ibumad libibumad.map
   umad.c
   umad_str.c
   )
-
-install(FILES libibumad.udev-rules
-  RENAME 90-libibumad.rules
-  DESTINATION "${CMAKE_INSTALL_UDEV_RULESDIR}")
-
diff --git a/redhat/rdma-core.spec b/redhat/rdma-core.spec
index 9892566a0333f5..f1895ea260ac52 100644
--- a/redhat/rdma-core.spec
+++ b/redhat/rdma-core.spec
@@ -340,6 +340,7 @@ rm -rf %{buildroot}/%{_sbindir}/srp_daemon.sh
 %{_udevrulesdir}/75-rdma-description.rules
 %{_udevrulesdir}/90-rdma-hw-modules.rules
 %{_udevrulesdir}/90-rdma-ulp-modules.rules
+%{_udevrulesdir}/90-rdma-umad.rules
 %{_udevrulesdir}/98-rdma.rules
 %{sysmodprobedir}/libmlx4.conf
 %{sysmodprobedir}/cxgb3.conf
@@ -414,7 +415,6 @@ rm -rf %{buildroot}/%{_sbindir}/srp_daemon.sh
 
 %files -n libibumad
 %{_libdir}/libibumad*.so.*
-%{_udevrulesdir}/90-libibumad.rules
 
 %files -n librdmacm
 %{_libdir}/librdmacm*.so.*
-- 
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 rdma-core 3/6] debian: Cause udev to evaluate the installed rules after install

[Jason Gunthorpe]

Call uadevadm trigger with all the subsystems our rules touch.

Following the advice of
 https://lists.ubuntu.com/archives/ubuntu-devel/2009-January/027260.html

And general consensus of other posinst files we We inject a change
message and ignore failure. This means the udev rules must accept
a change message and cannot just use ACTION=="add".

Signed-off-by: Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
---
 debian/rdma-core.postinst   | 13 +++++++++++++
 debian/srptools.postinst    | 11 +++++++++++
 kernel-boot/rdma-umad.rules |  2 +-
 srp_daemon/srp_daemon.rules |  2 +-
 4 files changed, 26 insertions(+), 2 deletions(-)
 create mode 100644 debian/rdma-core.postinst
 create mode 100644 debian/srptools.postinst

diff --git a/debian/rdma-core.postinst b/debian/rdma-core.postinst
new file mode 100644
index 00000000000000..de13363f0ad7c7
--- /dev/null
+++ b/debian/rdma-core.postinst
@@ -0,0 +1,13 @@
+#!/bin/sh
+
+set -e
+
+#DEBHELPER#
+
+if [ "$1" = "configure" ]; then
+    # we ship udev rules, so trigger an update. This has to be done after
+    # DEBHELPER restarts systemd to get our new service files loaded.
+    udevadm trigger --subsystem-match=infiniband --action=change || true
+    udevadm trigger --subsystem-match=net --action=change || true
+    udevadm trigger --subsystem-match=infiniband_mad --action=change || true
+fi
diff --git a/debian/srptools.postinst b/debian/srptools.postinst
new file mode 100644
index 00000000000000..c5a3e86261b338
--- /dev/null
+++ b/debian/srptools.postinst
@@ -0,0 +1,11 @@
+#!/bin/sh
+
+set -e
+
+#DEBHELPER#
+
+if [ "$1" = "configure" ]; then
+    # we ship udev rules, so trigger an update. This has to be done after
+    # DEBHELPER restarts systemd to get our new service files loaded.
+    udevadm trigger --subsystem-match=infiniband_mad --action=change || true
+fi
diff --git a/kernel-boot/rdma-umad.rules b/kernel-boot/rdma-umad.rules
index cfb790e051ced7..ba7ee613815a7d 100644
--- a/kernel-boot/rdma-umad.rules
+++ b/kernel-boot/rdma-umad.rules
@@ -1 +1 @@
-ACTION=="add", SUBSYSTEM=="infiniband_mad", KERNEL=="*umad*", TAG+="systemd", ENV{SYSTEMD_ALIAS}="/sys/subsystem/rdma/devices/$attr{ibdev}:$attr{port}/umad"
+SUBSYSTEM=="infiniband_mad", KERNEL=="*umad*", TAG+="systemd", ENV{SYSTEMD_ALIAS}="/sys/subsystem/rdma/devices/$attr{ibdev}:$attr{port}/umad"
diff --git a/srp_daemon/srp_daemon.rules b/srp_daemon/srp_daemon.rules
index 4cfeabc762da2a..b6411dcec72432 100644
--- a/srp_daemon/srp_daemon.rules
+++ b/srp_daemon/srp_daemon.rules
@@ -1 +1 @@
-ACTION=="add", SUBSYSTEM=="infiniband_mad", KERNEL=="*umad*", PROGRAM=="/bin/systemctl show srp_daemon -p ActiveState", RESULT=="ActiveState=active", ENV{SYSTEMD_WANTS}+="srp_daemon_port@$attr{ibdev}:$attr{port}.service"
+SUBSYSTEM=="infiniband_mad", KERNEL=="*umad*", PROGRAM=="/bin/systemctl show srp_daemon -p ActiveState", RESULT=="ActiveState=active", ENV{SYSTEMD_WANTS}+="srp_daemon_port@$attr{ibdev}:$attr{port}.service"
-- 
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 rdma-core 4/6] redhat: Remove cxgb3/4.modprobe files

[Jason Gunthorpe]

This is now handled by the udev rules in kernel-boot

Signed-off-by: Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
---
 redhat/rdma-core.spec          | 4 ----
 redhat/rdma.cxgb3.sys.modprobe | 1 -
 redhat/rdma.cxgb4.sys.modprobe | 1 -
 3 files changed, 6 deletions(-)
 delete mode 100644 redhat/rdma.cxgb3.sys.modprobe
 delete mode 100644 redhat/rdma.cxgb4.sys.modprobe

diff --git a/redhat/rdma-core.spec b/redhat/rdma-core.spec
index f1895ea260ac52..a11de8df3f45a7 100644
--- a/redhat/rdma-core.spec
+++ b/redhat/rdma-core.spec
@@ -260,8 +260,6 @@ install -D -m0644 redhat/rdma.udev-ipoib-naming.rules %{buildroot}%{_sysconfdir}
 install -D -m0755 redhat/rdma.modules-setup.sh %{buildroot}%{dracutlibdir}/modules.d/05rdma/module-setup.sh
 install -D -m0644 redhat/rdma.udev-rules %{buildroot}%{_udevrulesdir}/98-rdma.rules
 install -D -m0644 redhat/rdma.mlx4.sys.modprobe %{buildroot}%{sysmodprobedir}/libmlx4.conf
-install -D -m0644 redhat/rdma.cxgb3.sys.modprobe %{buildroot}%{sysmodprobedir}/cxgb3.conf
-install -D -m0644 redhat/rdma.cxgb4.sys.modprobe %{buildroot}%{sysmodprobedir}/cxgb4.conf
 install -D -m0755 redhat/rdma.kernel-init %{buildroot}%{_libexecdir}/rdma-init-kernel
 install -D -m0755 redhat/rdma.sriov-init %{buildroot}%{_libexecdir}/rdma-set-sriov-vf
 install -D -m0644 redhat/rdma.fixup-mtrr.awk %{buildroot}%{_libexecdir}/rdma-fixup-mtrr.awk
@@ -343,8 +341,6 @@ rm -rf %{buildroot}/%{_sbindir}/srp_daemon.sh
 %{_udevrulesdir}/90-rdma-umad.rules
 %{_udevrulesdir}/98-rdma.rules
 %{sysmodprobedir}/libmlx4.conf
-%{sysmodprobedir}/cxgb3.conf
-%{sysmodprobedir}/cxgb4.conf
 %{_libexecdir}/rdma-init-kernel
 %{_libexecdir}/rdma-set-sriov-vf
 %{_libexecdir}/rdma-fixup-mtrr.awk
diff --git a/redhat/rdma.cxgb3.sys.modprobe b/redhat/rdma.cxgb3.sys.modprobe
deleted file mode 100644
index d5925a715ccee1..00000000000000
diff --git a/redhat/rdma.cxgb4.sys.modprobe b/redhat/rdma.cxgb4.sys.modprobe
deleted file mode 100644
index 44163ab43cf649..00000000000000
-- 
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 rdma-core 5/6] srp_daemon: Autoload the SRP kernel module if required

[Jason Gunthorpe]

We use the rdma-core module loading infrastructure to load a srp_daemon
specific module list before allowing srp_daemon to start.

This is in lieu of kernel module auto loading, which is hard to trigger
for sysfs based interfaces.

Signed-off-by: Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
---
 debian/srptools.install                | 1 +
 redhat/rdma-core.spec                  | 1 +
 srp_daemon/CMakeLists.txt              | 4 ++++
 srp_daemon/modules-srp_daemon.conf     | 2 ++
 srp_daemon/srp_daemon.service.in       | 2 +-
 srp_daemon/srp_daemon_port@.service.in | 3 ++-
 6 files changed, 11 insertions(+), 2 deletions(-)
 create mode 100644 srp_daemon/modules-srp_daemon.conf

diff --git a/debian/srptools.install b/debian/srptools.install
index f780d663c9113b..a30798cf088a03 100644
--- a/debian/srptools.install
+++ b/debian/srptools.install
@@ -1,3 +1,4 @@
+etc/rdma/modules/srp_daemon.conf
 etc/srp_daemon.conf
 lib/systemd/system/srp_daemon.service
 lib/systemd/system/srp_daemon_port@.service
diff --git a/redhat/rdma-core.spec b/redhat/rdma-core.spec
index a11de8df3f45a7..f1b23669e7e601 100644
--- a/redhat/rdma-core.spec
+++ b/redhat/rdma-core.spec
@@ -449,6 +449,7 @@ rm -rf %{buildroot}/%{_sbindir}/srp_daemon.sh
 
 %files -n srp_daemon
 %config(noreplace) %{_sysconfdir}/srp_daemon.conf
+%config(noreplace) %{_sysconfdir}/rdma/modules/srp_daemon.conf
 %{_libexecdir}/srp_daemon/start_on_all_ports
 %{_unitdir}/srp_daemon.service
 %{_unitdir}/srp_daemon_port@.service
diff --git a/srp_daemon/CMakeLists.txt b/srp_daemon/CMakeLists.txt
index 92a1d709364b0b..0e7de9594b6375 100644
--- a/srp_daemon/CMakeLists.txt
+++ b/srp_daemon/CMakeLists.txt
@@ -47,6 +47,10 @@ install(FILES "srp_daemon.rules"
   RENAME "60-srp_daemon.rules"
   DESTINATION "${CMAKE_INSTALL_UDEV_RULESDIR}")
 
+install(FILES modules-srp_daemon.conf
+  RENAME "srp_daemon.conf"
+  DESTINATION "${CMAKE_INSTALL_SYSCONFDIR}/rdma/modules")
+
 # FIXME: The ib init.d file should really be included in rdma-core as well.
 set(RDMA_SERVICE "openibd" CACHE STRING "init.d file service name to order srpd after")
 # NOTE: These defaults are for CentOS, packagers should override.
diff --git a/srp_daemon/modules-srp_daemon.conf b/srp_daemon/modules-srp_daemon.conf
new file mode 100644
index 00000000000000..dbe434362f10a3
--- /dev/null
+++ b/srp_daemon/modules-srp_daemon.conf
@@ -0,0 +1,2 @@
+# These modules are loaded by the system if srp_daemon is to be run
+ib_srp
diff --git a/srp_daemon/srp_daemon.service.in b/srp_daemon/srp_daemon.service.in
index 33dddd5cb46fef..cca1fce9c99283 100644
--- a/srp_daemon/srp_daemon.service.in
+++ b/srp_daemon/srp_daemon.service.in
@@ -1,6 +1,6 @@
 [Unit]
 Description=Daemon that discovers and logs in to SRP target systems
-Documentation=man:srp_daemon file:/etc/rdma/rdma.conf file:/etc/srp_daemon.conf
+Documentation=man:srp_daemon file:/etc/srp_daemon.conf
 DefaultDependencies=false
 Conflicts=emergency.target emergency.service
 Before=remote-fs-pre.target
diff --git a/srp_daemon/srp_daemon_port@.service.in b/srp_daemon/srp_daemon_port@.service.in
index 2475db8640a9bd..5c215cb935bc73 100644
--- a/srp_daemon/srp_daemon_port@.service.in
+++ b/srp_daemon/srp_daemon_port@.service.in
@@ -3,7 +3,8 @@ Description=SRP daemon that monitors port %i
 Documentation=man:srp_daemon file:/etc/rdma/rdma.conf file:/etc/srp_daemon.conf
 DefaultDependencies=false
 Conflicts=emergency.target emergency.service
-After=srp_daemon.service sys-subsystem-rdma-devices-%i-umad.device network.target
+Requires=rdma-load-modules@srp_daemon.service
+After=srp_daemon.service rdma-load-modules@srp_daemon.service sys-subsystem-rdma-devices-%i-umad.device network.target
 BindsTo=srp_daemon.service sys-subsystem-rdma-devices-%i-umad.device
 Before=remote-fs-pre.target
 
-- 
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 rdma-core 6/6] iwpmd: Autoload iwpmd if required

[Jason Gunthorpe]

Use udev to determine if an iWarp device is present, and if so then start
iwpmd and auto load the required module. iwpmd no longer starts if it is
simply installed.

This will be unnecessary on future kernels with the netlink module
auto-loading patches.

Signed-off-by: Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
---
 debian/iwpmd.install           |  2 ++
 debian/iwpmd.postinst          | 11 +++++++++++
 iwpmd/CMakeLists.txt           |  8 ++++++++
 iwpmd/iwpmd.rules              |  1 +
 iwpmd/iwpmd.service.in         |  6 +++---
 iwpmd/modules-iwpmd.conf       |  2 ++
 kernel-boot/modules/iwarp.conf |  1 -
 redhat/rdma-core.spec          |  2 ++
 8 files changed, 29 insertions(+), 4 deletions(-)
 create mode 100644 debian/iwpmd.postinst
 create mode 100644 iwpmd/iwpmd.rules
 create mode 100644 iwpmd/modules-iwpmd.conf

diff --git a/debian/iwpmd.install b/debian/iwpmd.install
index cc26eccf18a57d..172e4105cc0a85 100644
--- a/debian/iwpmd.install
+++ b/debian/iwpmd.install
@@ -1,6 +1,8 @@
 etc/init.d/iwpmd
 etc/iwpmd.conf
+etc/rdma/modules/iwpmd.conf
 lib/systemd/system/iwpmd.service
+lib/udev/rules.d/90-iwpmd.rules
 usr/sbin/iwpmd
 usr/share/man/man5/iwpmd.conf.5
 usr/share/man/man8/iwpmd.8
diff --git a/debian/iwpmd.postinst b/debian/iwpmd.postinst
new file mode 100644
index 00000000000000..cabfe7c2d80ebe
--- /dev/null
+++ b/debian/iwpmd.postinst
@@ -0,0 +1,11 @@
+#!/bin/sh
+
+set -e
+
+#DEBHELPER#
+
+if [ "$1" = "configure" ]; then
+    # we ship udev rules, so trigger an update. This has to be done after
+    # DEBHELPER restarts systemd to get our new service files loaded.
+    udevadm trigger --subsystem-match=infiniband --action=change || true
+fi
diff --git a/iwpmd/CMakeLists.txt b/iwpmd/CMakeLists.txt
index 856e3886f9ea81..6fd36e47bbbc54 100644
--- a/iwpmd/CMakeLists.txt
+++ b/iwpmd/CMakeLists.txt
@@ -21,3 +21,11 @@ rdma_subst_install(FILES "iwpmd_init.in"
   RENAME "iwpmd"
   PERMISSIONS OWNER_WRITE OWNER_READ GROUP_READ WORLD_READ OWNER_EXECUTE GROUP_EXECUTE WORLD_EXECUTE)
 install(FILES "iwpmd.conf" DESTINATION "${CMAKE_INSTALL_SYSCONFDIR}")
+
+install(FILES "iwpmd.rules"
+  RENAME "90-iwpmd.rules"
+  DESTINATION "${CMAKE_INSTALL_UDEV_RULESDIR}")
+
+install(FILES modules-iwpmd.conf
+  RENAME "iwpmd.conf"
+  DESTINATION "${CMAKE_INSTALL_SYSCONFDIR}/rdma/modules")
diff --git a/iwpmd/iwpmd.rules b/iwpmd/iwpmd.rules
new file mode 100644
index 00000000000000..5b22cceaf29f8c
--- /dev/null
+++ b/iwpmd/iwpmd.rules
@@ -0,0 +1 @@
+TAG+="systemd", ENV{ID_RDMA_IWARP}=="1", ENV{SYSTEMD_WANTS}+="iwpmd.service"
diff --git a/iwpmd/iwpmd.service.in b/iwpmd/iwpmd.service.in
index 25740ebdcf9a82..4e4b49738fa29d 100644
--- a/iwpmd/iwpmd.service.in
+++ b/iwpmd/iwpmd.service.in
@@ -1,11 +1,11 @@
 [Unit]
 Description=iWarp Port Mapper
 Documentation=man:iwpmd file:/etc/iwpmd.conf
-After=network.target
+Requires=rdma-load-modules-r79FGtN6zq8c5CkaFG5UGg@public.gmane.org
+After=network.target rdma-load-modules-r79FGtN6zq8c5CkaFG5UGg@public.gmane.org
 
 [Service]
 ExecStart=@CMAKE_INSTALL_FULL_SBINDIR@/iwpmd --systemd
 LimitNOFILE=102400
 
-[Install]
-WantedBy=multi-user.target
+# iwpmd is automatically started by udev when an iWarp RDMA device is present
diff --git a/iwpmd/modules-iwpmd.conf b/iwpmd/modules-iwpmd.conf
new file mode 100644
index 00000000000000..5544b359c08814
--- /dev/null
+++ b/iwpmd/modules-iwpmd.conf
@@ -0,0 +1,2 @@
+# These modules are loaded by the system if iwpmd is to be run
+iw_cm
diff --git a/kernel-boot/modules/iwarp.conf b/kernel-boot/modules/iwarp.conf
index 882146e41ee2ba..0cb831da5bb557 100644
--- a/kernel-boot/modules/iwarp.conf
+++ b/kernel-boot/modules/iwarp.conf
@@ -1,2 +1 @@
 # These modules are loaded by the system if any iWarp device is installed
-iw_cm
diff --git a/redhat/rdma-core.spec b/redhat/rdma-core.spec
index f1b23669e7e601..773c98e49eb062 100644
--- a/redhat/rdma-core.spec
+++ b/redhat/rdma-core.spec
@@ -401,7 +401,9 @@ rm -rf %{buildroot}/%{_sbindir}/srp_daemon.sh
 %files -n iwpmd
 %{_sbindir}/iwpmd
 %{_unitdir}/iwpmd.service
+%config(noreplace) %{_sysconfdir}/rdma/modules/iwpmd.conf
 %config(noreplace) %{_sysconfdir}/iwpmd.conf
+%{_udevrulesdir}/90-iwpmd.rules
 %{_mandir}/man8/iwpmd.*
 %{_mandir}/man5/iwpmd.*
 
-- 
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH v2 rdma-core 0/6] Common udev/systemd based module auto loading

[Steve Wise]

> 
> Here is the next revision of the systemd module autoloading approach.
> 
> All the comments have been collected and a few other things addressed.
> 

It would be great to have a change list from the previous version.

> Notably, at least on debian, when rdma-core is installed for the first time it
> activates the new udev rules and moduels autoload immediately, no reboot
> required. Not sure about RH's policies, so leaving this to them to copy if
> desired.
> 
> There have been positive test results now for rxe, mlx4 (IB), qedr and cxgb4
> on Debian derived.

My cxgb4 testing was on the RHEL7.3 distro.

Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 rdma-core 0/6] Common udev/systemd based module auto loading

[Benjamin Drung]

Hi Jason,

Am Donnerstag, den 27.07.2017, 16:31 -0600 schrieb Jason Gunthorpe:
> Here is the next revision of the systemd module autoloading approach.
> 
> All the comments have been collected and a few other things
> addressed.
> 
> Notably, at least on debian, when rdma-core is installed for the
> first time it
> activates the new udev rules and moduels autoload immediately, no
> reboot
> required. Not sure about RH's policies, so leaving this to them to
> copy if
> desired.
> 
> There have been positive test results now for rxe, mlx4 (IB), qedr
> and cxgb4
> on Debian derived.
> 
> This is now a PR:
> 
>  https://github.com/linux-rdma/rdma-core/pull/174/commits

thanks for your work. I reviewed the commits and commented them on
GitHub.

I tested the whole patch set on our Debian jessie system (with Mellanox
OFED kernel modules on kernel 4.4), but it failed to load properly.

The mlx4_core module was loaded, but not mlx4_ib. The directory
/sys/class/infiniband is empty.

$ sudo find /sys/ -name '*mlx*'
/sys/bus/pci/drivers/mlx4_core
/sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/mlx4_port1_mtu
/sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/mlx4_port2_mtu
/sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/mlx4_port1
/sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/mlx4_port2

$ journalctl | grep mlx
mlx4_core: Mellanox ConnectX core driver v3.3-1.0.4 (03 Jul 2016)
mlx4_core: Initializing 0000:02:00.0
mlx4_core: device is working in RoCE mode: Roce V1
mlx4_core: gid_type 1 for UD QPs is not supported by the devicegid_type 0 was chosen instead
mlx4_core: UD QP Gid type is: V1
mlx4_core 0000:02:00.0: PCIe link speed is 8.0GT/s, device supports 8.0GT/s
mlx4_core 0000:02:00.0: PCIe link width is x8, device supports x8

No clue how to further debug it.

-- 
Benjamin Drung
System Developer
Debian & Ubuntu Developer

ProfitBricks GmbH
Greifswalder Str. 207
D - 10405 Berlin

Email: benjamin.drung-EIkl63zCoXaH+58JC4qpiA@public.gmane.org
Web: https://www.profitbricks.com

Sitz der Gesellschaft: Berlin.
Registergericht: Amtsgericht Charlottenburg, HRB 125506B.
Geschäftsführer: Achim Weiss.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 rdma-core 0/6] Common udev/systemd based module auto loading

[Jason Gunthorpe]

On Fri, Jul 28, 2017 at 05:42:05PM +0200, Benjamin Drung wrote:

> $ journalctl | grep mlx
> mlx4_core: Mellanox ConnectX core driver v3.3-1.0.4 (03 Jul 2016)
> mlx4_core: Initializing 0000:02:00.0
> mlx4_core: device is working in RoCE mode: Roce V1

So you are running purely in roce mode?

What does this say:

$ udevadm info /sys/class/net/XXX

Where 'XXX' is the ethernet device for the mlx4?

> No clue how to further debug it.

This is the line that is expected to load mlx4_ib:

ENV{ID_NET_DRIVER}=="mlx4_en", RUN{builtin}+="kmod load mlx4_ib"

So we should see ID_NET_DRIVER=mlx4_en in the above udevadm print.

>From there run

$ udevadm test /sys/class/net/XXX

And confirm it tries to load the module..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 rdma-core 0/6] Common udev/systemd based module auto loading

[Jason Gunthorpe]

On Fri, Jul 28, 2017 at 08:54:42AM -0500, Steve Wise wrote:
> > Here is the next revision of the systemd module autoloading approach.
> > 
> > All the comments have been collected and a few other things addressed.
> 
> It would be great to have a change list from the previous version.

Heh, it is getting hard, this started out as a little four patch
series and somehow expanded into about a 40 patch monster.

Broadly
 - Add udevadm trigger's to the post install
 - Udev ID_RDMA_ROCE triggers for rxe
 - Various documentation improvements
 - Packaging fixes (eg ./rdma-core.spec works)
 - Fixing ID_RDMA_OPA for hif1verbs

> > There have been positive test results now for rxe, mlx4 (IB), qedr and cxgb4
> > on Debian derived.
> 
> My cxgb4 testing was on the RHEL7.3 distro.

Right, I was tring to say my testing was on Debian

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 rdma-core 0/6] Common udev/systemd based module auto loading

[Benjamin Drung]

Am Freitag, den 28.07.2017, 09:47 -0600 schrieb Jason Gunthorpe:
> On Fri, Jul 28, 2017 at 05:42:05PM +0200, Benjamin Drung wrote:
> 
> > $ journalctl | grep mlx
> > mlx4_core: Mellanox ConnectX core driver v3.3-1.0.4 (03 Jul 2016)
> > mlx4_core: Initializing 0000:02:00.0
> > mlx4_core: device is working in RoCE mode: Roce V1
> 
> So you are running purely in roce mode?

I want to use mlx4_ib and IPoIB. The output is the same for a properly
working system, but these system will load mlx4_ib and then more logs
will appear.

> What does this say:
> 
> $ udevadm info /sys/class/net/XXX
> 
> Where 'XXX' is the ethernet device for the mlx4?

The net device is missing. When I load mlx4_ib manually, they will
appear and show:

$ sudo udevadm info /sys/class/net/ib0
P: /devices/pci0000:00/0000:00:02.0/0000:02:00.0/net/ib0
E: DEVPATH=/devices/pci0000:00/0000:00:02.0/0000:02:00.0/net/ib0
E: ID_BUS=pci
E: ID_MODEL_FROM_DATABASE=MT27500 Family [ConnectX-3]
E: ID_MODEL_ID=0x1003
E: ID_NET_DRIVER=ib_ipoib
E: ID_PATH=pci-0000:02:00.0
E: ID_PATH_TAG=pci-0000_02_00_0
E: ID_PCI_CLASS_FROM_DATABASE=Network controller
E: ID_PCI_SUBCLASS_FROM_DATABASE=Network controller
E: ID_VENDOR_FROM_DATABASE=Mellanox Technologies
E: ID_VENDOR_ID=0x15b3
E: IFINDEX=6
E: INTERFACE=ib0
E: SUBSYSTEM=net
E: SYSTEMD_ALIAS=/sys/subsystem/net/devices/ib0
E: TAGS=:systemd:
E: USEC_INITIALIZED=65619465

> > No clue how to further debug it.
> 
> This is the line that is expected to load mlx4_ib:
> 
> ENV{ID_NET_DRIVER}=="mlx4_en", RUN{builtin}+="kmod load mlx4_ib"
> 
> So we should see ID_NET_DRIVER=mlx4_en in the above udevadm print.

Strange. It is ib_ipoib instead of mlx4_en.

-- 
Benjamin Drung
System Developer
Debian & Ubuntu Developer

ProfitBricks GmbH
Greifswalder Str. 207
D - 10405 Berlin

Email: benjamin.drung-EIkl63zCoXaH+58JC4qpiA@public.gmane.org
Web: https://www.profitbricks.com

Sitz der Gesellschaft: Berlin.
Registergericht: Amtsgericht Charlottenburg, HRB 125506B.
Geschäftsführer: Achim Weiss.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 rdma-core 0/6] Common udev/systemd based module auto loading

[Jason Gunthorpe]

On Fri, Jul 28, 2017 at 06:41:05PM +0200, Benjamin Drung wrote:
> Am Freitag, den 28.07.2017, 09:47 -0600 schrieb Jason Gunthorpe:
> > On Fri, Jul 28, 2017 at 05:42:05PM +0200, Benjamin Drung wrote:
> > 
> > > $ journalctl | grep mlx
> > > mlx4_core: Mellanox ConnectX core driver v3.3-1.0.4 (03 Jul 2016)
> > > mlx4_core: Initializing 0000:02:00.0
> > > mlx4_core: device is working in RoCE mode: Roce V1
> > 
> > So you are running purely in roce mode?
> 
> I want to use mlx4_ib and IPoIB. The output is the same for a properly
> working system, but these system will load mlx4_ib and then more logs
> will appear.

This approach relies on this patch in upstream:

commit b046ffe54dc13ff8ae918c83bedb71aa7919d63b
Author: Eyal Perry <eyalpe-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Date:   Tue Oct 15 16:55:24 2013 +0200

    net/mlx4_core: Load higher level modules according to ports type
    
    Mellanox ConnectX architecture is:  mlx4_core is the lower level
    PCI driver which register on the PCI id, and protocol specific drivers
    are depended on it: mlx4_en - for Ethernet and mlx4_ib for Infiniband.
    NIC could have multiple ports which can change their type dynamically.
    We use the request_module() call to load the relevant protocol driver
    when needed: on loading time or at port type change event.

$ git describe --contains b046ffe54dc13ff8ae918c83bedb71aa7919d63b
v3.13-rc1~105^2~189^2

And it works fine on my 4.13 stock systems with mlx4.

Are you using an initrd? If you have mlx4_core in the initrd then you
must also have mlx4_ib and mlx4_en as well, otherwise the kernel
triggered autoload will fail to find the module in the initrd and it
will not be retried.

Try rmmod mlx4_core; modprobe mlx4_core right after booting, if that
triggers all the module loads then the above is likely the problem.

> > What does this say:
> > 
> > $ udevadm info /sys/class/net/XXX
> > 
> > Where 'XXX' is the ethernet device for the mlx4?
> 
> The net device is missing. When I load mlx4_ib manually, they will
> appear and show:

Right, you are using IB mode not ROCE mode, so there is no ethernet
device expected. I didn't mean the ipoib device..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 rdma-core 1/6] Common infrastructure for auto loading rdma modules

[Chien Tin Tung]

On Thu, Jul 27, 2017 at 04:31:20PM -0600, Jason Gunthorpe wrote:

[...]

Sorry Jason, I can't test this till next week.  On inspection I found a couple of issues
below.

> diff --git a/kernel-boot/rdma-description.rules b/kernel-boot/rdma-description.rules
> new file mode 100644
> index 00000000000000..4d7c5808401ac7
> --- /dev/null
> +++ b/kernel-boot/rdma-description.rules
> @@ -0,0 +1,42 @@
> +# This is a version of net-description.rules for /sys/class/infiniband devices
> +
> +ACTION=="remove", GOTO="rdma_description_end"
> +SUBSYSTEM!="infiniband", GOTO="rdma_description_end"
> +
> +# NOTE: DRIVERS searches up the sysfs path to find the driver that is bound to
> +# the PCI/etc device that the RDMA device is linked to. This is not the kernel
> +# driver that is supplying the RDMA device (eg as seen in ID_NET_DRIVER)
> +
> +# FIXME: with kernel support we could actually detect the protocols the RDMA
> +# driver itself supports, this is a work around for lack of that support.
> +# In future we could do this with a udev IMPORT{program} helper program
> +# that extracted the ID information from the RDMA netlink.
> +
> +# Hardware that supports InfiniBand
> +DRIVERS=="mlx4_core", ENV{ID_RDMA_INFINIBAND}="1"
> +DRIVERS=="mlx5_core", ENV{ID_RDMA_INFINIBAND}="1"
> +DRIVERS=="qib", ENV{ID_RDMA_INFINIBAND}="1"
> +
> +# Hardware that supports OPA
> +DRIVERS=="hfi1verbs", ENV{ID_RDMA_OPA}="1"
> +
> +# Hardware that supports iWarp
> +DRIVERS=="cxgb3", ENV{ID_RDMA_IWARP}="1"
> +DRIVERS=="cxgb4", ENV{ID_RDMA_IWARP}="1"

Need to add nes and i40e here under iWARP.
This is enough to autoload iwpmd for nes and i40e/i40iw, correct?

> +
> +# Hardware that supports RoCE
> +DRIVERS=="be2net", ENV{ID_RDMA_ROCE}="1"
> +DRIVERS=="bnxt_en", ENV{ID_RDMA_ROCE}="1"
> +DRIVERS=="hns", ENV{ID_RDMA_ROCE}="1"
> +DRIVERS=="i40e", ENV{ID_RDMA_ROCE}="1"

i40e does not support RoCE, see above.

Chien
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 rdma-core 1/6] Common infrastructure for auto loading rdma modules

[Jason Gunthorpe]

On Fri, Jul 28, 2017 at 04:48:55PM -0500, Chien Tin Tung wrote:

> Sorry Jason, I can't test this till next week.  On inspection I found a couple of issues
> below.

Sure, thanks!

> > +# Hardware that supports InfiniBand
> > +DRIVERS=="mlx4_core", ENV{ID_RDMA_INFINIBAND}="1"
> > +DRIVERS=="mlx5_core", ENV{ID_RDMA_INFINIBAND}="1"
> > +DRIVERS=="qib", ENV{ID_RDMA_INFINIBAND}="1"
> > +
> > +# Hardware that supports OPA
> > +DRIVERS=="hfi1verbs", ENV{ID_RDMA_OPA}="1"
> > +
> > +# Hardware that supports iWarp
> > +DRIVERS=="cxgb3", ENV{ID_RDMA_IWARP}="1"
> > +DRIVERS=="cxgb4", ENV{ID_RDMA_IWARP}="1"
> 
> Need to add nes and i40e here under iWARP.

Okay, yes.. nes was missed because I was not able to guess the
ethernet driver name, I will add it and move i40e

> This is enough to autoload iwpmd for nes and i40e/i40iw, correct?

Yes, ID_RDMA_IWARP in the udev database does that and whatever else in
future we need.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 rdma-core 0/6] Common udev/systemd based module auto loading

[Benjamin Drung]

Am Freitag, den 28.07.2017, 10:56 -0600 schrieb Jason Gunthorpe:
> On Fri, Jul 28, 2017 at 06:41:05PM +0200, Benjamin Drung wrote:
> > Am Freitag, den 28.07.2017, 09:47 -0600 schrieb Jason Gunthorpe:
> > > On Fri, Jul 28, 2017 at 05:42:05PM +0200, Benjamin Drung wrote:
> > > 
> > > > $ journalctl | grep mlx
> > > > mlx4_core: Mellanox ConnectX core driver v3.3-1.0.4 (03 Jul
> > > > 2016)
> > > > mlx4_core: Initializing 0000:02:00.0
> > > > mlx4_core: device is working in RoCE mode: Roce V1
> > > 
> > > So you are running purely in roce mode?
> > 
> > I want to use mlx4_ib and IPoIB. The output is the same for a
> > properly
> > working system, but these system will load mlx4_ib and then more
> > logs
> > will appear.
> 
> This approach relies on this patch in upstream:
> 
> $ git describe --contains b046ffe54dc13ff8ae918c83bedb71aa7919d63b
> v3.13-rc1~105^2~189^2
> 
> And it works fine on my 4.13 stock systems with mlx4.

Yes, that commit is part of our 4.4 kernel.

> Are you using an initrd? If you have mlx4_core in the initrd then you
> must also have mlx4_ib and mlx4_en as well, otherwise the kernel
> triggered autoload will fail to find the module in the initrd and it
> will not be retried.

Yes. That was the issue. I am using an initrd (from initramfs-tools).
The mlx4_core module was included in the initrd, but mlx4_ib was
missing. I tweaked initramfs-tools to include mlx4_ib and then it was
autoloaded. Thanks.

Remaining problem: The IPoIB InfiniBand partition does not come up
correctly:

$ ip a
[...]
9: ib1.d00b@ib1: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 65520
qdisc mq state UP group default qlen 1024
[...]

The corresponding config in /etc/network/interfaces:

auto ib0.d00b
iface ib0.d00b inet6 static
    address <redacted>
    netmask 64
    pre-up echo connected > /sys/class/net/$IFACE/mode
    dad-attempts 300

I tried bring up the device manually:

$ sudo ip link set dev ib0.d00b down
$ sudo ifup ib0.d00b
$ sudo ifdown ib0.d00b
$ sudo ifup ib0.d00b

But the result is still the same. It still has the M-DOWN state and
pinging other host does not work.

I haven't found useful information on the web regarding the M-DOWN
state.

-- 
Benjamin Drung
System Developer
Debian & Ubuntu Developer

ProfitBricks GmbH
Greifswalder Str. 207
D - 10405 Berlin

Email: benjamin.drung-EIkl63zCoXaH+58JC4qpiA@public.gmane.org
Web: https://www.profitbricks.com

Sitz der Gesellschaft: Berlin.
Registergericht: Amtsgericht Charlottenburg, HRB 125506B.
Geschäftsführer: Achim Weiss.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 rdma-core 0/6] Common udev/systemd based module auto loading

[Jason Gunthorpe]

On Thu, Aug 03, 2017 at 02:00:59PM +0200, Benjamin Drung wrote:

> Yes. That was the issue. I am using an initrd (from initramfs-tools).
> The mlx4_core module was included in the initrd, but mlx4_ib was
> missing. I tweaked initramfs-tools to include mlx4_ib and then it was
> autoloaded. Thanks.

Do you know how to fix this generally for Debian? Is it possible?

> Remaining problem: The IPoIB InfiniBand partition does not come up
> correctly:

AFAIK, M-DOWN means 'master-down', it means that the interface 'ib1'
is down. 'ip link set up dev ib1'

Something wrong with ordering of the legacy network setup script?

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 rdma-core 0/6] Common udev/systemd based module auto loading

[Benjamin Drung]

Am Donnerstag, den 03.08.2017, 11:52 -0600 schrieb Jason Gunthorpe:
> On Thu, Aug 03, 2017 at 02:00:59PM +0200, Benjamin Drung wrote:
> 
> > Yes. That was the issue. I am using an initrd (from initramfs-
> > tools).
> > The mlx4_core module was included in the initrd, but mlx4_ib was
> > missing. I tweaked initramfs-tools to include mlx4_ib and then it
> > was
> > autoloaded. Thanks.
> 
> Do you know how to fix this generally for Debian? Is it possible?

Analysis for Debian jessie/stretch:

$ sudo update-initramfs -uv
[...]
Copying module directory kernel/drivers/net
(excluding appletalk arcnet bonding can dummy.ko hamradio hippi ifb.ko
irda macvlan.ko macvtap.ko pcmcia sb1000.ko team tokenring tun.ko usb
veth.ko wan wimax wireless xen-netback.ko)
[...]
Adding module /lib/modules/$version/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko
Adding module /lib/modules/$version/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko

update-initramfs calls auto_add_modules() in
/usr/share/initramfs-tools/hook-functions which includes 'net' as
default. The 'net' option calls 
"copy_modules_dir kernel/drivers/net [...]" which copies the
mlx5_core.ko and mlx4_core.ko modules.

but the mlx4_ib.ko module is in the infiniband directory:

$ find kernel/drivers/infiniband ! -type d
kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko
kernel/drivers/infiniband/hw/mthca/ib_mthca.ko
kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko
kernel/drivers/infiniband/core/ib_sa.ko
kernel/drivers/infiniband/core/ib_umad.ko
kernel/drivers/infiniband/core/ib_netlink.ko
kernel/drivers/infiniband/core/ib_core.ko
kernel/drivers/infiniband/core/rdma_cm.ko
kernel/drivers/infiniband/core/ib_ucm.ko
kernel/drivers/infiniband/core/ib_addr.ko
kernel/drivers/infiniband/core/rdma_ucm.ko
kernel/drivers/infiniband/core/ib_cm.ko
kernel/drivers/infiniband/core/ib_uverbs.ko
kernel/drivers/infiniband/core/iw_cm.ko
kernel/drivers/infiniband/core/ib_mad.ko
kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko
kernel/drivers/infiniband/ulp/srp/ib_srp.ko

By default, nothing from the infiniband directory is included. The
question is: What is the right thing to do? Should everything from
kernel/drivers/infiniband/hw be included in the initrd by default?

> > Remaining problem: The IPoIB InfiniBand partition does not come up
> > correctly:
> 
> AFAIK, M-DOWN means 'master-down', it means that the interface 'ib1'
> is down. 'ip link set up dev ib1'
> 
> Something wrong with ordering of the legacy network setup script?

Thanks. The master-down is correct, since we just use only the
InifiniBand partition and not the master device. Even the old
configuration had M-DOWN.

My ping tests were incorrect. I compared the IP configuration with the
previous configuration and it was identical. Then I discovered that the
ping target was not reachable even with the old configuration. I took a
different host that was pingable with the old config and this one was
pingable with rdma-core too. So rdma-core works as designed.

-- 
Benjamin Drung
System Developer
Debian & Ubuntu Developer

ProfitBricks GmbH
Greifswalder Str. 207
D - 10405 Berlin

Email: benjamin.drung-EIkl63zCoXaH+58JC4qpiA@public.gmane.org
Web: https://www.profitbricks.com

Sitz der Gesellschaft: Berlin.
Registergericht: Amtsgericht Charlottenburg, HRB 125506B.
Geschäftsführer: Achim Weiss.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 rdma-core 0/6] Common udev/systemd based module auto loading

[Jason Gunthorpe]

On Fri, Aug 04, 2017 at 06:22:31PM +0200, Benjamin Drung wrote:

> By default, nothing from the infiniband directory is included. The
> question is: What is the right thing to do? Should everything from
> kernel/drivers/infiniband/hw be included in the initrd by default?

No, only modules that can be request_module'd from the initrd should
be included in the initrd.

AFAIK only the two mellanox drivers do request module, so only those
two or three modules (mlx4_ib, mlx4_en, mlx5_ib?) should be included.

That sounds like something that should be fixed in the initd package?
Or should rdma-core be dropping some kind of hook in a .d directory?

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 rdma-core 0/6] Common udev/systemd based module auto loading

[Benjamin Drung]

Am Dienstag, den 08.08.2017, 16:38 -0600 schrieb Jason Gunthorpe:
> On Fri, Aug 04, 2017 at 06:22:31PM +0200, Benjamin Drung wrote:
> 
> > By default, nothing from the infiniband directory is included. The
> > question is: What is the right thing to do? Should everything from
> > kernel/drivers/infiniband/hw be included in the initrd by default?
> 
> No, only modules that can be request_module'd from the initrd should
> be included in the initrd.
> 
> AFAIK only the two mellanox drivers do request module, so only those
> two or three modules (mlx4_ib, mlx4_en, mlx5_ib?) should be included.
> 
> That sounds like something that should be fixed in the initd package?
> Or should rdma-core be dropping some kind of hook in a .d directory?

Yes. I opened a bug report against initramfs-tools with a patch
attached: https://bugs.debian.org/871595

-- 
Benjamin Drung
System Developer
Debian & Ubuntu Developer

ProfitBricks GmbH
Greifswalder Str. 207
D - 10405 Berlin

Email: benjamin.drung-EIkl63zCoXaH+58JC4qpiA@public.gmane.org
Web: https://www.profitbricks.com

Sitz der Gesellschaft: Berlin.
Registergericht: Amtsgericht Charlottenburg, HRB 125506B.
Geschäftsführer: Achim Weiss.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 rdma-core 0/6] Common udev/systemd based module auto loading

[Doug Ledford]

[-- Attachment #1.1: Type: text/plain, Size: 3623 bytes --]

On 8/4/2017 12:22 PM, Benjamin Drung wrote:
> Am Donnerstag, den 03.08.2017, 11:52 -0600 schrieb Jason Gunthorpe:
>> On Thu, Aug 03, 2017 at 02:00:59PM +0200, Benjamin Drung wrote:
>>
>>> Yes. That was the issue. I am using an initrd (from initramfs-
>>> tools).
>>> The mlx4_core module was included in the initrd, but mlx4_ib was
>>> missing. I tweaked initramfs-tools to include mlx4_ib and then it
>>> was
>>> autoloaded. Thanks.
>>
>> Do you know how to fix this generally for Debian? Is it possible?
> 
> Analysis for Debian jessie/stretch:
> 
> $ sudo update-initramfs -uv
> [...]
> Copying module directory kernel/drivers/net
> (excluding appletalk arcnet bonding can dummy.ko hamradio hippi ifb.ko
> irda macvlan.ko macvtap.ko pcmcia sb1000.ko team tokenring tun.ko usb
> veth.ko wan wimax wireless xen-netback.ko)
> [...]
> Adding module /lib/modules/$version/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko
> Adding module /lib/modules/$version/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko
> 
> update-initramfs calls auto_add_modules() in
> /usr/share/initramfs-tools/hook-functions which includes 'net' as
> default. The 'net' option calls 
> "copy_modules_dir kernel/drivers/net [...]" which copies the
> mlx5_core.ko and mlx4_core.ko modules.
> 
> but the mlx4_ib.ko module is in the infiniband directory:
> 
> $ find kernel/drivers/infiniband ! -type d
> kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko
> kernel/drivers/infiniband/hw/mthca/ib_mthca.ko
> kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko
> kernel/drivers/infiniband/core/ib_sa.ko
> kernel/drivers/infiniband/core/ib_umad.ko
> kernel/drivers/infiniband/core/ib_netlink.ko
> kernel/drivers/infiniband/core/ib_core.ko
> kernel/drivers/infiniband/core/rdma_cm.ko
> kernel/drivers/infiniband/core/ib_ucm.ko
> kernel/drivers/infiniband/core/ib_addr.ko
> kernel/drivers/infiniband/core/rdma_ucm.ko
> kernel/drivers/infiniband/core/ib_cm.ko
> kernel/drivers/infiniband/core/ib_uverbs.ko
> kernel/drivers/infiniband/core/iw_cm.ko
> kernel/drivers/infiniband/core/ib_mad.ko
> kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko
> kernel/drivers/infiniband/ulp/srp/ib_srp.ko
> 
> By default, nothing from the infiniband directory is included. The
> question is: What is the right thing to do? Should everything from
> kernel/drivers/infiniband/hw be included in the initrd by default?

That largely depends on what you want your initramfs to be capable of.
In dracut speak, an initramfs is either generic or host_only.  A generic
initramfs should be able to start any system and is suitable for things
like an installer image.  In that case, the answer is yes, you want
everything (you need the whole kit and caboodle to support root over
iSER, SRP, or NFSoRDMA, or to use an IPoIB interface as your install
source).  If you want a host_only initramfs, then you have to sort
through what's required to get to your rootfs.  If you have a network
rootfs, then the requirements jump up considerably.  If you don't, and
you can mount locally, then you only need network drivers if you want a
backup in case your rootfs craps out on you and you need to be able to
ssh in/out or perform other extraordinary recovery measures.  But if you
want to support those extraordinary recovery measures, then you are back
to a good portion of the stack because you might need IPoIB to get
anything done.


-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
    GPG Key ID: B826A3330E572FDD
    Key fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]

Re: [PATCH v2 rdma-core 0/6] Common udev/systemd based module auto loading

[Jason Gunthorpe]

On Wed, Aug 09, 2017 at 07:43:50PM +0200, Benjamin Drung wrote:
> Am Dienstag, den 08.08.2017, 16:38 -0600 schrieb Jason Gunthorpe:
> > On Fri, Aug 04, 2017 at 06:22:31PM +0200, Benjamin Drung wrote:
> > 
> > > By default, nothing from the infiniband directory is included. The
> > > question is: What is the right thing to do? Should everything from
> > > kernel/drivers/infiniband/hw be included in the initrd by default?
> > 
> > No, only modules that can be request_module'd from the initrd should
> > be included in the initrd.
> > 
> > AFAIK only the two mellanox drivers do request module, so only those
> > two or three modules (mlx4_ib, mlx4_en, mlx5_ib?) should be included.
> > 
> > That sounds like something that should be fixed in the initd package?
> > Or should rdma-core be dropping some kind of hook in a .d directory?
> 
> Yes. I opened a bug report against initramfs-tools with a patch
> attached: https://bugs.debian.org/871595

Looks fine to me, thanks

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 rdma-core 0/6] Common udev/systemd based module auto loading

[Jason Gunthorpe]

On Wed, Aug 09, 2017 at 01:54:30PM -0600, Jason Gunthorpe wrote:
> On Wed, Aug 09, 2017 at 07:43:50PM +0200, Benjamin Drung wrote:
> > Am Dienstag, den 08.08.2017, 16:38 -0600 schrieb Jason Gunthorpe:
> > > On Fri, Aug 04, 2017 at 06:22:31PM +0200, Benjamin Drung wrote:
> > > 
> > > > By default, nothing from the infiniband directory is included. The
> > > > question is: What is the right thing to do? Should everything from
> > > > kernel/drivers/infiniband/hw be included in the initrd by default?
> > > 
> > > No, only modules that can be request_module'd from the initrd should
> > > be included in the initrd.
> > > 
> > > AFAIK only the two mellanox drivers do request module, so only those
> > > two or three modules (mlx4_ib, mlx4_en, mlx5_ib?) should be included.
> > > 
> > > That sounds like something that should be fixed in the initd package?
> > > Or should rdma-core be dropping some kind of hook in a .d directory?
> > 
> > Yes. I opened a bug report against initramfs-tools with a patch
> > attached: https://bugs.debian.org/871595
> 
> Looks fine to me, thanks

Actually, does it somehow get mlx4_en too?

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 rdma-core 0/6] Common udev/systemd based module auto loading

[Benjamin Drung]

Am Mittwoch, den 09.08.2017, 14:00 -0600 schrieb Jason Gunthorpe:
> On Wed, Aug 09, 2017 at 01:54:30PM -0600, Jason Gunthorpe wrote:
> > On Wed, Aug 09, 2017 at 07:43:50PM +0200, Benjamin Drung wrote:
> > > Am Dienstag, den 08.08.2017, 16:38 -0600 schrieb Jason Gunthorpe:
> > > > On Fri, Aug 04, 2017 at 06:22:31PM +0200, Benjamin Drung wrote:
> > > > 
> > > > > By default, nothing from the infiniband directory is
> > > > > included. The
> > > > > question is: What is the right thing to do? Should everything
> > > > > from
> > > > > kernel/drivers/infiniband/hw be included in the initrd by
> > > > > default?
> > > > 
> > > > No, only modules that can be request_module'd from the initrd
> > > > should
> > > > be included in the initrd.
> > > > 
> > > > AFAIK only the two mellanox drivers do request module, so only
> > > > those
> > > > two or three modules (mlx4_ib, mlx4_en, mlx5_ib?) should be
> > > > included.
> > > > 
> > > > That sounds like something that should be fixed in the initd
> > > > package?
> > > > Or should rdma-core be dropping some kind of hook in a .d
> > > > directory?
> > > 
> > > Yes. I opened a bug report against initramfs-tools with a patch
> > > attached: https://bugs.debian.org/871595
> > 
> > Looks fine to me, thanks
> 
> Actually, does it somehow get mlx4_en too?

Yes. mlx4_en.ko is in kernel/drivers/net/ethernet/mellanox/mlx4/ and
the whole kernel/drivers/net is included in the initrd (with some
specific excludes).

-- 
Benjamin Drung
System Developer
Debian & Ubuntu Developer

ProfitBricks GmbH
Greifswalder Str. 207
D - 10405 Berlin

Email: benjamin.drung-EIkl63zCoXaH+58JC4qpiA@public.gmane.org
Web: https://www.profitbricks.com

Sitz der Gesellschaft: Berlin.
Registergericht: Amtsgericht Charlottenburg, HRB 125506B.
Geschäftsführer: Achim Weiss.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html