Netdev List
 help / color / mirror / Atom feed
* [PATCH iwl-net 0/3] ice: fix RTNL vs auxiliary device lock inversion
@ 2026-10-19  6:52 Petr Oros
  2026-10-19  6:52 ` [PATCH iwl-net 1/3] ice: avoid RTNL deadlock when notifying the RDMA auxiliary driver Petr Oros
                   ` (2 more replies)
  0 siblings, 3 replies; 4+ messages in thread
From: Petr Oros @ 2026-10-19  6:52 UTC (permalink / raw)
  To: netdev
  Cc: Petr Oros, Tony Nguyen, Przemek Kitszel, Andrew Lunn,
	David S . Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
	Dave Ertman, Shiraz Saleem, Leon Romanovsky, Kamil Maziarz,
	Rafal Rogalski, Aleksandr Loktionov, Michal Schmidt, linux-rdma,
	intel-wired-lan, linux-kernel, Sebastian Andrzej Siewior,
	Clark Williams, Steven Rostedt, Mateusz Palczewski,
	linux-rt-devel

ice takes the auxiliary device lock of its RDMA auxiliary device from
three places that run with the RTNL lock held. The RDMA auxiliary probe
takes the same two locks in the opposite order, because __driver_attach()
holds the auxiliary device lock across ib_register_device(), which needs
the RTNL lock. ib_query_port() reaches ib_get_eth_speed() first from the
port immutable setup and again from ib_cache_setup_one(). The result is
a hard deadlock that leaves the whole RTNL-serialised part of the stack
stuck, so the boot never completes:

  systemd-udevd   state:D
    really_probe                  <- holds device_lock(&adev->dev)
    icrdma_probe
    irdma_ib_register_device
    ib_register_device
    setup_device
    irdma_roce_port_immutable
    ib_query_port
    irdma_query_port
    ib_get_eth_speed              <- blocks on rtnl_lock()

  lldpad          state:D
    rtnetlink_rcv_msg             <- holds rtnl_lock()
    dcb_doit
    dcbnl_ieee_set
    ice_dcbnl_setets
    ice_pf_dcb_cfg
    ice_send_event_to_aux         <- blocks on device_lock(&adev->dev)

linkwatch, addrconf, NetworkManager and every ethtool ioctl then pile up
behind lldpad. On a PREEMPT_RT kernel with CONFIG_DEBUG_RT_MUTEXES the
rtmutex chain walker reports "rtmutex deadlock detected" from whichever
task closes the cycle instead of hanging silently. Plain lockdep cannot
flag it, because device_initialize() marks dev->mutex with
lockdep_set_novalidate_class().

The probe side cannot be changed, so the three ice callers stop blocking
and use device_trylock() instead. Contention is only possible while the
driver core is binding, unbinding or suspending the auxiliary device,
since all three sites already serialise against each other under
pf->adev_mutex, and ice_unplug_aux_dev() clears cdev->adev under that
mutex before calling auxiliary_device_delete(). ice_unplug_aux_dev()
itself is never reached with the RTNL lock held, so there is no fourth
site to convert.

Patch 1 then handles a contended lock per event, because the right
answer differs. BEFORE_TC_CHANGE has to arrive before the traffic
classes change, so the DCB reconfiguration is refused with -EBUSY. The
MTU change and critical error events are sent from the service task,
which consumes their pending bits before sending, so they are re-armed
for the next run. AFTER_TC_CHANGE keeps its existing fire and forget
behaviour, and the commit message explains why deferring that one is
not safe. Patches 2 and 3 only turn the lock into a trylock, where a
contended lock means the operation is refused with -EBUSY and can be
retried, which is what the existing check already does when the RDMA
driver is bound.

This keeps the fix small enough to backport. It does not address the
underlying layering problem, which is that ice reaches for the driver
core lock of a foreign device at all. i40e and mlx5 both solve the same
need with provider-owned registration instead, and converting IIDC to
that model would make the inversion impossible by construction. That is
a net-next change across ice, irdma and iidc_rdma.h, and I am happy to
follow up with it if there is interest.

Tested on an E810-XXV with two PFs, irdma bound to both auxiliary
devices, on net tip commit 191aa41548b2d2 ("MAINTAINERS: add Bobby
Eshleman as vsock reviewer") with and without this series, same config
otherwise. A reproducer binds the irdma auxiliary driver while driving
one of the three RTNL-held callers into the probe window:

                        without the series   with the series
  ethtool set_channels  deadlock 10/10       0/10, window entered 300x
  ndo_setup_tc mqprio   deadlock             0/10, window entered 50x
  dcbnl ets set         deadlock             0/10, window entered 30x

A window entry is a contended device_trylock(), that is, the exact
moment really_probe() holds the auxiliary device lock and waits for the
RTNL lock, so each one is an occasion that used to close the cycle.
Without the series the console shows both halves and the box only
returns through hung_task_panic. 25 reboots on the patched kernel were
clean.

Petr Oros (3):
  ice: avoid RTNL deadlock when notifying the RDMA auxiliary driver
  ice: do not block on the aux device lock in ethtool set_channels
  ice: do not block on the aux device lock in ndo_setup_tc

 drivers/net/ethernet/intel/ice/ice_dcb_lib.c |  9 +++++++-
 drivers/net/ethernet/intel/ice/ice_ethtool.c |  6 +++++-
 drivers/net/ethernet/intel/ice/ice_idc.c     | 20 ++++++++++++++----
 drivers/net/ethernet/intel/ice/ice_idc_int.h |  2 +-
 drivers/net/ethernet/intel/ice/ice_main.c    | 22 +++++++++++++++-----
 5 files changed, 47 insertions(+), 12 deletions(-)

-- 
2.56.0


^ permalink raw reply	[flat|nested] 4+ messages in thread
end of thread, other threads:[~2026-10-19  6:53 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-10-19  6:52 [PATCH iwl-net 0/3] ice: fix RTNL vs auxiliary device lock inversion Petr Oros
2026-10-19  6:52 ` [PATCH iwl-net 1/3] ice: avoid RTNL deadlock when notifying the RDMA auxiliary driver Petr Oros
2026-10-19  6:53 ` [PATCH iwl-net 2/3] ice: do not block on the aux device lock in ethtool set_channels Petr Oros
2026-10-19  6:53 ` [PATCH iwl-net 3/3] ice: do not block on the aux device lock in ndo_setup_tc Petr Oros

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox