From: Ivan Vecera <ivecera@redhat.com>
To: Jakub Kicinski <kuba@kernel.org>
Cc: netdev@vger.kernel.org,
Vadim Fedorenko <vadim.fedorenko@linux.dev>,
Arkadiusz Kubalewski <arkadiusz.kubalewski@intel.com>,
Jiri Pirko <jiri@resnulli.us>, Jonathan Corbet <corbet@lwn.net>,
Shuah Khan <skhan@linuxfoundation.org>,
Randy Dunlap <rdunlap@infradead.org>,
Min Li <min.li@microchip.com>,
Tony Nguyen <anthony.l.nguyen@intel.com>,
Przemek Kitszel <przemyslaw.kitszel@intel.com>,
Andrew Lunn <andrew+netdev@lunn.ch>,
"David S. Miller" <davem@davemloft.net>,
Eric Dumazet <edumazet@kernel.org>,
Paolo Abeni <pabeni@redhat.com>,
Richard Cochran <richardcochran@gmail.com>,
linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org,
intel-wired-lan@lists.osuosl.org,
Grzegorz Nitka <grzegorz.nitka@intel.com>,
Karol Kolacinski <karol.kolacinski@intel.com>,
Saeed Mahameed <saeedm@nvidia.com>,
Tariq Toukan <tariqt@nvidia.com>, Petr Oros <poros@redhat.com>
Subject: Re: [PATCH net-next v2 0/9] dpll: do not serialize driver callbacks under global dpll_lock
Date: Wed, 18 Nov 2026 16:29:23 +0100 [thread overview]
Message-ID: <fe1dcdd5-ef75-4231-bf9f-03bf8536c2aa@redhat.com> (raw)
In-Reply-To: <20261116181612.107a4b57@kernel.org>
On 11/17/26 03:16, Jakub Kicinski wrote:
> On Wed, 11 Nov 2026 17:40:11 +0100 Ivan Vecera wrote:
>> dpll_lock protects topology and object lifetime, but it also serializes
>> all driver callbacks in the system, so one slow device stalls every other
>> DPLL device, whichever driver it belongs to. mlx5 callbacks issue firmware
>> commands with a 60 s default timeout, and zl3073x callbacks can wait
>> seconds on a driver lock held across ToD writes and phase steps.
>
> Can you provide some real life data of how much this actually helps?
> Real device topology + scenario you hit?
The system is GNR-D with a dual complex E825 and a ZL30732 on the
board. The zl3073x has its own PHC, takes the recovered clocks of the
E825 ports and feeds the TX and time reference clocks of the E825. Each
ice PF connects to the zl3073x like this:
+------------------------- zl3073x --------------------------+
| EEC DPLL, PPS DPLL |
| |
| input pin output pin |
| (recovered clock of E825) 156M25_NAC0_CLKREF_SYNC |
+-------^---------------------------------|------------------+
| parent pin | registered with
| dpll_pin_on_pin_register() | dpll_pin_register()
+-------|---------------------------------v------------------+
| RCLK pin TXC DPLL |
+------------------------- ice PF ---------------------------+
The PF owning the source timer also registers the zl3073x output pin
156M25_NAC0_TIMEREF_SYNC with its TSPLL DPLL.
A step of the zl3073x PHC, which ptp4l and ts2phc do at start and after
a large offset, waits for the next 1 PPS edge, up to 1 s, with the
channel lock held. A phase step can hold it for up to 3 s while it
waits for the previous one. A dpll callback of that channel waits for
the lock under dpll_lock, and every other DPLL request and notification
in the system waits behind it. With ToD writes running, requests to an
ice DPLL took 1448 ms in median and 3.9 s at most. With the series they
took 0.2 ms and 0.3 ms, the same as idle.
The bigger reason are zl3073x callbacks we are working on, which have
to wait for the hardware themselves:
- a device reset with resync waits for the fast lock, up to 1 s with a
1 PPS input
- a glitchless stop or start of an output completes on an output edge,
so it takes up to one period, 1 s for a 1 PPS output
Holding the global dpll_lock for that long would stall every DPLL
device in the system. For the output state we tried an asynchronous
implementation, but then the callback returns success before it knows
the result. A failure can only be reported by a later notification, and
the next request gets -EBUSY until the change completes. With a lock
per chip the callback can simply wait, and one chip cannot do these
operations in parallel anyway.
> For the cross-domain pins - shouldn't we stretch the domain to cover
> all interconnected objects on the system instead of hoping people get
> lock ordering right? You documented some rules in the doc, admittedly,
> but not sure how enforceable and practical they are.
Stretching the domains would not improve anything here. The only
in-tree registrations across domains are the ones in the diagram above,
and every PF connects to the zl3073x, so the zl3073x and all ice PFs
would end up in one domain. The system above would then stall the same
way as with the global lock.
There is no lock ordering to get right. Only one domain lock is ever
held and all of them share one lockdep class, so any nesting is
reported. __dpll_{device,pin}_change_ntf() assert the domain lock of
the object, so a callback notifying an object of another domain is
reported as well. The rule that a callback must not call functions
taking dpll_lock holds on main already. What lockdep cannot check is
that a driver registering a pin of another driver protects the state
these callbacks share with its own ones. ice is the only such driver
and uses pf->dplls.lock for it.
Thanks,
Ivan
prev parent reply other threads:[~2026-11-18 15:29 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-11 16:40 [PATCH net-next v2 0/9] dpll: do not serialize driver callbacks under global dpll_lock Ivan Vecera
2026-11-11 16:40 ` [PATCH net-next v2 1/9] ice: dpll: notify TSPLL lock status from the periodic work only Ivan Vecera
2026-11-11 16:40 ` [PATCH net-next v2 2/9] dpll: add domain object Ivan Vecera
2026-11-11 16:40 ` [PATCH net-next v2 3/9] dpll: take domain locks around driver callbacks Ivan Vecera
2026-11-11 16:40 ` [PATCH net-next v2 4/9] dpll: convert dpll_lock to rwsem Ivan Vecera
2026-11-16 8:21 ` Ivan Vecera
2026-11-11 16:40 ` [PATCH net-next v2 5/9] dpll: add KUnit tests for core locking Ivan Vecera
2026-11-11 16:40 ` [PATCH net-next v2 6/9] dpll: reject measured_freq_get for foreign and on-pin registrations Ivan Vecera
2026-11-11 16:40 ` [PATCH net-next v2 7/9] dpll: zl3073x: update locking comments Ivan Vecera
2026-11-11 16:40 ` [PATCH net-next v2 8/9] ice: dpll: " Ivan Vecera
2026-11-16 8:22 ` Ivan Vecera
2026-11-11 16:40 ` [PATCH net-next v2 9/9] dpll: document locking rules Ivan Vecera
2026-11-16 8:23 ` Ivan Vecera
2026-11-16 8:24 ` [PATCH net-next v2 0/9] dpll: do not serialize driver callbacks under global dpll_lock Ivan Vecera
2026-11-17 2:17 ` Jakub Kicinski
2026-11-18 15:30 ` Ivan Vecera
2026-11-17 2:16 ` Jakub Kicinski
2026-11-18 15:29 ` Ivan Vecera [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=fe1dcdd5-ef75-4231-bf9f-03bf8536c2aa@redhat.com \
--to=ivecera@redhat.com \
--cc=andrew+netdev@lunn.ch \
--cc=anthony.l.nguyen@intel.com \
--cc=arkadiusz.kubalewski@intel.com \
--cc=corbet@lwn.net \
--cc=davem@davemloft.net \
--cc=edumazet@kernel.org \
--cc=grzegorz.nitka@intel.com \
--cc=intel-wired-lan@lists.osuosl.org \
--cc=jiri@resnulli.us \
--cc=karol.kolacinski@intel.com \
--cc=kuba@kernel.org \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=min.li@microchip.com \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=poros@redhat.com \
--cc=przemyslaw.kitszel@intel.com \
--cc=rdunlap@infradead.org \
--cc=richardcochran@gmail.com \
--cc=saeedm@nvidia.com \
--cc=skhan@linuxfoundation.org \
--cc=tariqt@nvidia.com \
--cc=vadim.fedorenko@linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox