* [PATCH v5 00/12] fix several inconsistencies with sysfs configuration in etmX
From: Yeoreum Yun @ 2026-04-15 16:55 UTC (permalink / raw)
To: coresight, linux-arm-kernel, linux-kernel
Cc: suzuki.poulose, mike.leach, james.clark, alexander.shishkin,
leo.yan, jie.gan, Yeoreum Yun
The current ETMx configuration via sysfs can lead to the following
inconsistencies:
- If a configuration is modified via sysfs while a perf session is
active, the running configuration may differ between before
a sched-out and after a subsequent sched-in.
- If a perf session and sysfs session tries to enable concurrently,
configuration from configfs could be corrupted (etm4).
- There is chance to corrupt drvdata->config if perf session tries
to enabled among handling cscfg_csdev_disable_active_config()
in etm4_disable_sysfs() (etm4).
To resolve these inconsistencies, the configuration should be separated into:
- active_config, which is applied configuration for the current session
- config, which stores the settings configured via sysfs.
and apply configuration from configfs after taking a mode.
Also, This patch set includes some small fixes:
- missing trace id release in etm4x.
- underflow issue for nrseqstate.
- wrong check in etm4x_sspcicrn_present().
This patch based on v7.0
Patch History
=============
from v4 to v5:
- add rb-tag.
- fix underflow issue for nrseqstate.
- fix wrong check in etm4_sspcicrn_present().
- remove redundant fields on etmv4_save_state.
- rename caps->ss_status to ss_cmp.
- fix wrong location of etm4_release_trace_id.
- https://lore.kernel.org/all/20260413142003.3549310-1-yeoreum.yun@arm.com/
from v3 to v4:
- change etm_drvdata->spinlock type to raw_spin_lock_t
- remove redundant call etmX_enable_hw() with starting_cpu() callsback.
- fix missing trace id release.
- add missing docs.
- https://lore.kernel.org/all/20260412175506.412301-1-yeoreum.yun@arm.com/
from v2 to v3:
- fix build error for etm3x.
- fix checkpatch warning.
- https://lore.kernel.org/all/20260410074310.2693385-1-yeoreum.yun@arm.com/
from v1 to v2
- rebased to v7.0-rc7.
- introduce etmX_caps structure to save etmX's capabilities.
- remove ss_status from etmv4_config.
- modify active_config after taking a mode (perf/sysfs).
- https://lore.kernel.org/all/20260317181705.2456271-1-yeoreum.yun@arm.com/
Yeoreum Yun (12):
coresight: etm4x: fix wrong check of etm4x_sspcicrn_present()
coresight: etm4x: fix underflow for nrseqstate
coresight: etm4x: introduce struct etm4_caps
coresight: etm4x: exclude ss_status from drvdata->config
coresight: etm4x: remove redundant fields in etmv4_save_state
coresight: etm4x: fix leaked trace id
coresight: etm4x: fix inconsistencies with sysfs configuration
coresight: etm4x: remove redundant call etm4_enable_hw() with hotplug
coresight: etm3x: change drvdata->spinlock type to raw_spin_lock_t
coresight: etm3x: introduce struct etm_caps
coresight: etm3x: fix inconsistencies with sysfs configuration
coresight: etm3x: remove redundant call etm_enable_hw() with hotplug
drivers/hwtracing/coresight/coresight-etm.h | 46 ++-
.../coresight/coresight-etm3x-core.c | 100 ++---
.../coresight/coresight-etm3x-sysfs.c | 159 ++++----
.../hwtracing/coresight/coresight-etm4x-cfg.c | 3 +-
.../coresight/coresight-etm4x-core.c | 374 ++++++++++--------
.../coresight/coresight-etm4x-sysfs.c | 199 ++++++----
drivers/hwtracing/coresight/coresight-etm4x.h | 190 ++++-----
7 files changed, 588 insertions(+), 483 deletions(-)
base-commit: 028ef9c96e96197026887c0f092424679298aae8
--
LEVI:{C3F47F37-75D8-414A-A8BA-3980EC8A46D7}
^ permalink raw reply
* [RFC PATCH v3 2/7] hq-spinlock: implement inner logic
From: Fedorov Nikita @ 2026-04-15 16:44 UTC (permalink / raw)
To: Catalin Marinas, Will Deacon, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, Dave Hansen, x86, hpa, Juergen Gross, Ajay Kaher,
Alexey Makhalov, bcm-kernel-feedback-list, Arnd Bergmann,
Peter Zijlstra, Boqun Feng, Waiman Long, Darren Hart,
Davidlohr Bueso, andrealmeid, Andrew Morton, David Hildenbrand,
Zi Yan, Matthew Brost, Joshua Hahn, Rakie Kim, byungchul,
Gregory Price, Ying Huang, Alistair Popple, Anatoly Stepanov
Cc: Nikita Fedorov, linux-arm-kernel, linux-kernel, virtualization,
linux-arch, linux-mm, guohanjun, wangkefeng.wang, weiyongjun1,
yusongping, leijitang, artem.kuzin, kang.sun, chenjieping3
In-Reply-To: <20260415164459.2904963-1-fedorov.nikita@h-partners.com>
The design might be considered as a combination of cohort-locking
by Dave Dice and Linux kernel queued spinlock.
The contenders are organized into 2-level scheme where each NUMA-node
has it's own FIFO contender queue.
NUMA-queues are linked into single-linked list alike structure, while
maintaining FIFO order between them.
When there're no contenders left in a NUMA-queue, the queue is removed from
the list.
Contenders try to enqueue to local NUMA-queue first and if there's
no such queue - link it to the list.
As for "qspinlock" only the first contender is spinning globally,
all others - MCS-spinning.
"Handoff" operation becomes two-staged:
- local handoff: between contenders in a single NUMA-queue
- remote handoff: between different NUMA-queues.
If "remote handoff" reaches the end of the NUMA-queue linked list it goes to
the list head.
To avoid potential starvation issues, we use handoff thresholds.
Key challenge here was keeping the same "qspinlock" structure size
and "bind" a given lock to related NUMA-queues.
We came up with dynamic lock metadata concept, where we can
dynamically "bind" a given lock to it's NUMA-related metadata, and
then "unbind" when the lock is released.
This approach allows to avoid metadata reservation for every lock in advance,
thus giving the upperbound of metadata instance number to ~ (NR_CPUS x nr_contexts / 2).
Which corresponds to maximum amount of different locks falling into the slowpath.
HQ-lock supports switching between "default qspinlock" mode to "NUMA-aware lock" mode and backwards.
If for some reason "NUMA-aware" mode cannot be enabled we fallback to default qspinlock mode.
Functions that will be used in generic qspinlock code are started with "hqlock_"
(`hqlock_xchg_tail`, `hqlock_clear_pending`, `hqlock_clear_pending_set_locked`,
`hqlock_try_clear_tail`, `hqlock_handoff`).
Testing with locktorture module shows that this design reduces overhead
on both x86 and arm64 NUMA systems. On AMD EPYC 9654, throughput gains reach
up to 186% in the evaluated NPS=12 configuration. On Kunpeng 920,
throughput gains range from 93% to 158% across the tested thread
counts. Fairness on AMD EPYC 9654 remained in the 0.51-0.61 range in
the evaluated configurations.
Co-developed-by: Anatoly Stepanov <stepanov.anatoly@huawei.com>
Signed-off-by: Anatoly Stepanov <stepanov.anatoly@huawei.com>
Co-developed-by: Nikita Fedorov <fedorov.nikita@h-partners.com>
Signed-off-by: Nikita Fedorov <fedorov.nikita@h-partners.com>
---
The full benchmark tables are included in the cover letter.
---
kernel/locking/hqlock_core.h | 715 +++++++++++++++++++++++++++++++++++
kernel/locking/hqlock_meta.h | 467 +++++++++++++++++++++++
2 files changed, 1182 insertions(+)
create mode 100644 kernel/locking/hqlock_core.h
create mode 100644 kernel/locking/hqlock_meta.h
diff --git a/kernel/locking/hqlock_core.h b/kernel/locking/hqlock_core.h
new file mode 100644
index 0000000000..7322199228
--- /dev/null
+++ b/kernel/locking/hqlock_core.h
@@ -0,0 +1,715 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _GEN_HQ_SPINLOCK_SLOWPATH
+#error "Do not include this file!"
+#endif
+
+#include <linux/nodemask.h>
+#include <linux/topology.h>
+#include <linux/sched/clock.h>
+#include <linux/moduleparam.h>
+#include <linux/sched/rt.h>
+#include <linux/random.h>
+#include <linux/mm.h>
+#include <linux/memblock.h>
+#include <linux/sysctl.h>
+#include <linux/types.h>
+#include <linux/percpu.h>
+#include <linux/slab.h>
+#include <linux/panic.h>
+#include <linux/vmalloc.h>
+#include <linux/slab.h>
+#include <linux/syscalls.h>
+#include <linux/sprintf.h>
+#include <linux/proc_fs.h>
+#include <linux/uaccess.h>
+#include <linux/swab.h>
+#include <linux/hash.h>
+
+/* Contains queues for all possible lock id */
+static struct numa_queue *queue_table[MAX_NUMNODES];
+
+#include "hqlock_types.h"
+#include "hqlock_proc.h"
+
+/* Gets node_id (1..N) */
+static inline struct numa_queue *
+get_queue(u16 lock_id, u16 node_id)
+{
+ return &queue_table[node_id - 1][lock_id];
+}
+
+static inline struct numa_queue *
+get_local_queue(struct numa_qnode *qnode)
+{
+ return get_queue(qnode->lock_id, qnode->numa_node);
+}
+
+static inline void init_queue_link(struct numa_queue *queue)
+{
+ queue->prev_node = 0;
+ queue->next_node = 0;
+}
+
+static inline void init_queue(struct numa_qnode *qnode)
+{
+ struct numa_queue *queue = get_local_queue(qnode);
+
+ queue->head = qnode;
+ queue->handoffs_not_head = 0;
+ init_queue_link(queue);
+}
+
+static void set_next_queue(u16 lock_id, u16 prev_node_id, u16 node_id)
+{
+ struct numa_queue *local_queue = get_queue(lock_id, node_id);
+ struct numa_queue *prev_queue =
+ get_queue(lock_id, prev_node_id);
+
+ WRITE_ONCE(local_queue->prev_node, prev_node_id);
+ /*
+ * Needs to be guaranteed the following:
+ * when appending "local_queue", if "prev_queue->next_node" link
+ * is observed then "local_queue->prev_node" is also observed.
+ *
+ * We need this to guarantee correctness of concurrent
+ * "unlink_node_queue" for the "prev_queue", if "prev_queue" is the first in the list.
+ * [prev_queue] <-> [local_queue]
+ *
+ * In this case "unlink_node_queue" would be setting "local_queue->prev_node = 0", thus
+ * w/o the smp-barrier, it might race with "set_next_queue", if
+ * "local_queue->prev_node = prev_node_id" happens afterwards, leading to corrupted list.
+ */
+ smp_wmb();
+ WRITE_ONCE(prev_queue->next_node, node_id);
+}
+
+static inline struct lock_metadata *get_meta(u16 lock_id);
+
+/**
+ * Put new node's queue into global NUMA-level queue
+ */
+static inline u16 append_node_queue(u16 lock_id, u16 node_id)
+{
+ struct lock_metadata *lock_meta = get_meta(lock_id);
+ u16 prev_node_id = xchg(&lock_meta->tail_node, node_id);
+
+ if (prev_node_id)
+ set_next_queue(lock_id, prev_node_id, node_id);
+ else
+ WRITE_ONCE(lock_meta->head_node, node_id);
+ return prev_node_id;
+}
+
+#include "hqlock_meta.h"
+
+/**
+ * Update tail
+ *
+ * Call proper function depending on lock's mode
+ * until successful queuing
+ */
+static inline u32 hqlock_xchg_tail(struct qspinlock *lock, u32 tail,
+ struct mcs_spinlock *node, bool *numa_awareness_on)
+{
+ struct numa_qnode *qnode = (struct numa_qnode *)node;
+
+ u16 lock_id;
+ u32 old_tail;
+ u32 next_tail = tail;
+
+ /*
+ * Key lock's mode switches questions:
+ * - After init lock is in LOCK_MODE_QSPINLOCK
+ * - If many contenders have come while lock was in LOCK_MODE_QSPINLOCK,
+ * we want this lock to use NUMA awareness next time,
+ * so we clean LOCK_MODE_QSPINLOCK, see 'low_contention_try_clear_tail'
+ * - During next lock's usages we try to go through NUMA-aware path.
+ * We can fail here, because we use shared metadata
+ * and can have a conflict with another lock, see 'hqlock_meta.h' for details.
+ * In this case we fallback to generic qspinlock approach.
+ *
+ * In other words, lock can be in 3 mode states:
+ *
+ * 1. LOCK_MODE_QSPINLOCK - there was low contention or not at all earlier,
+ * or (unlikely) a conflict in metadata
+ * 2. LOCK_NO_MODE - there was a contention on a lock earlier,
+ * now there are no contenders in the queue (we are likely the first)
+ * and we need to try using NUMA awareness
+ * 3. LOCK_MODE_HQLOCK - lock is currently under contention
+ * and using NUMA awareness.
+ */
+
+ /*
+ * numa_awareness_on == false means we saw LOCK_MODE_QSPINLOCK (1st state)
+ * before starting slowpath, see 'queued_spin_lock_slowpath'
+ */
+ if (*numa_awareness_on == false &&
+ try_update_tail_qspinlock_mode(lock, tail, &old_tail, &next_tail))
+ return old_tail;
+
+ /* Calculate the lock_id hash here once */
+ qnode->lock_id = lock_id = hash_ptr(lock, LOCK_ID_BITS);
+
+try_again:
+ /*
+ * Lock is in state 2 or 3 - go through NUMA-aware path
+ */
+ if (try_update_tail_hqlock_mode(lock, lock_id, qnode, tail, &next_tail, &old_tail)) {
+ *numa_awareness_on = true;
+ return old_tail;
+ }
+
+ /*
+ * We have failed (conflict in metadata), now lock is in LOCK_MODE_QSPINLOCK again
+ */
+ if (try_update_tail_qspinlock_mode(lock, tail, &old_tail, &next_tail)) {
+ *numa_awareness_on = false;
+ return old_tail;
+ }
+
+ /*
+ * We were slow and clear_tail after high contention has already happened
+ * (very unlikely situation)
+ */
+ goto try_again;
+}
+
+static inline void hqlock_clear_pending(struct qspinlock *lock, u32 old_val)
+{
+ WRITE_ONCE(lock->pending, (old_val & _Q_LOCK_TYPE_MODE_MASK) >> _Q_PENDING_OFFSET);
+}
+
+static inline void hqlock_clear_pending_set_locked(struct qspinlock *lock, u32 old_val)
+{
+ WRITE_ONCE(lock->locked_pending,
+ _Q_LOCKED_VAL | (old_val & _Q_LOCK_TYPE_MODE_MASK));
+}
+
+static inline void unlink_node_queue(u16 lock_id,
+ u16 prev_node_id,
+ u16 next_node_id)
+{
+ struct numa_queue *prev_queue =
+ prev_node_id ? get_queue(lock_id, prev_node_id) : NULL;
+ struct numa_queue *next_queue = get_queue(lock_id, next_node_id);
+
+ if (prev_queue)
+ WRITE_ONCE(prev_queue->next_node, next_node_id);
+ /*
+ * This is guaranteed to be ordered "after" next_node_id observation
+ * by implicit full-barrier in the caller-code.
+ */
+ WRITE_ONCE(next_queue->prev_node, prev_node_id);
+}
+
+static inline bool try_clear_queue_tail(struct numa_queue *queue, u32 tail)
+{
+ /*
+ * We need full ordering here to:
+ * - ensure all prior operations with global tail and prev_queue
+ * are observed before clearing local tail
+ * - guarantee all subsequent operations
+ * with metadata release, unlink etc will be observed after clearing local tail
+ */
+ return cmpxchg(&queue->tail, tail, 0) == tail;
+}
+
+/*
+ * Determine if we have another local and global contenders.
+ * Try clear local and global tail, understand handoff type we need to perform.
+ * In case we are the last, free lock's metadata
+ */
+static inline bool hqlock_try_clear_tail(struct qspinlock *lock, u32 val,
+ u32 tail, struct mcs_spinlock *node,
+ int *p_next_node)
+{
+ bool ret = false;
+ struct numa_qnode *qnode = (void *)node;
+
+ u16 lock_id = qnode->lock_id;
+ u16 local_node = qnode->numa_node;
+ struct numa_queue *queue = get_queue(lock_id, qnode->numa_node);
+
+ struct lock_metadata *lock_meta = get_meta(lock_id);
+
+ u16 prev_node = 0, next_node = 0;
+ u16 node_tail;
+
+ u32 old_val;
+
+ bool lock_tail_updated = false;
+ bool lock_tail_cleared = false;
+
+ /* Do we have *next node* arrived */
+ bool pending_next_node = false;
+
+ tail >>= _Q_TAIL_OFFSET;
+
+ /* Do we have other CPUs in the node queue ? */
+ if (READ_ONCE(queue->tail) != tail) {
+ *p_next_node = HQLOCK_HANDOFF_LOCAL;
+ goto out;
+ }
+
+ /*
+ * Key observations and actions:
+ * 1) next queue isn't observed:
+ * a) if prev queue is observed, try to unpublish local queue
+ * b) if prev queue is not observed, try to clean global tail
+ * Anyway, perform these operations before clearing local tail.
+ *
+ * Such trick is essential to safely unlink the local queue,
+ * otherwise we could race with upcomming local contenders,
+ * which will perform 'append_node_queue' while our unlink is not properly done.
+ *
+ * 2) next queue is observed:
+ * safely perform 'try_clear_queue_tail' and unlink local node if succeeded.
+ */
+
+ prev_node = READ_ONCE(queue->prev_node);
+ pending_next_node = READ_ONCE(lock_meta->tail_node) != local_node;
+
+ /*
+ * Tail case:
+ * [prev_node] -> [local_node], lock_meta->tail_node == local_node
+ *
+ * There're no nodes after us at the moment, try updating the "lock_meta->tail_node"
+ */
+ if (!pending_next_node && prev_node) {
+ struct numa_queue *prev_queue =
+ get_queue(lock_id, prev_node);
+
+ /* Reset next_node, in case no one will come after */
+ WRITE_ONCE(prev_queue->next_node, 0);
+
+ /*
+ * release to publish prev_queue->next_node = 0
+ * and to ensure ordering with 'READ_ONCE(queue->tail) != tail'
+ */
+ if (cmpxchg_release(&lock_meta->tail_node, local_node, prev_node) == local_node) {
+ lock_tail_updated = true;
+
+ queue->next_node = 0;
+ queue->prev_node = 0;
+ next_node = 0;
+ } else {
+ /* If some node came after the local meanwhile, reset next_node back */
+ WRITE_ONCE(prev_queue->next_node, local_node);
+
+ /* We either observing updated "queue->next" or it equals zero */
+ next_node = READ_ONCE(queue->next_node);
+ }
+ }
+
+ node_tail = READ_ONCE(lock_meta->tail_node);
+
+ /* If nobody else is waiting, try clean global tail */
+ if (node_tail == local_node && !prev_node) {
+ old_val = (((u32)local_node) | (((u32)local_node) << 16));
+ /* release to ensure ordering with 'READ_ONCE(queue->tail) != tail' */
+ lock_tail_cleared = try_cmpxchg_release(&lock_meta->nodes_tail, &old_val, 0);
+ }
+
+ /*
+ * lock_meta->tail_node was not updated and cleared,
+ * so we have at least single non-empty node after us
+ */
+ if (!lock_tail_updated && !lock_tail_cleared) {
+ /*
+ * If there's a node came before clearing node queue - wait for it to link properly.
+ * We need this for correct upcoming *unlink*, otherwise the *unlink* might race with parallel set_next_node()
+ */
+ if (!next_node) {
+ next_node =
+ smp_cond_load_relaxed(&queue->next_node, (VAL));
+ }
+ }
+
+ /* if we're the last one in the queue - clear the queue tail */
+ if (try_clear_queue_tail(queue, tail)) {
+ /*
+ * "lock_tail_cleared == true"
+ * It means: we cleared "lock_meta->tail_node" and "lock_meta->head_node".
+ *
+ * First new contender will do "global spin" anyway, so no handoff needed
+ * "ret == true"
+ */
+ if (lock_tail_cleared) {
+ ret = true;
+
+ /*
+ * If someone has arrived in the meanwhile,
+ * don't try to free the metadata.
+ */
+ old_val = READ_ONCE(lock_meta->nodes_tail);
+ if (!old_val) {
+ /*
+ * We are probably the last contender,
+ * so, need to free lock's metadata.
+ */
+ release_lock_meta(lock, lock_meta, qnode);
+ }
+ goto out;
+ }
+
+ /*
+ * "lock_tail_updated == true" (implies "lock_tail_cleared == false")
+ * It means we have at least "prev_node" and unlinked "local node"
+ *
+ * As we unlinked "local node", we only need to guarantee correct
+ * remote handoff, thus we have:
+ * "ret == false"
+ * "next_node == HQLOCK_HANDOFF_REMOTE_HEAD"
+ */
+ if (lock_tail_updated) {
+ *p_next_node = HQLOCK_HANDOFF_REMOTE_HEAD;
+ goto out;
+ }
+
+ /*
+ * "!lock_tail_cleared && !lock_tail_updated"
+ * It means we have at least single node after us.
+ *
+ * remote handoff and corect "local node" unlink are needed.
+ *
+ * "next_node" visibility guarantees that we observe
+ * correctly additon of "next_node", so the following unlink
+ * is safe and correct.
+ *
+ * "next_node > 0"
+ * "ret == false"
+ */
+ unlink_node_queue(lock_id, prev_node, next_node);
+
+ /*
+ * If at the head - update one.
+ *
+ * Another place, where "lock_meta->head_node" is updated is "append_node_queue"
+ * But we're safe, as that happens only with the first node on empty "node list".
+ */
+ if (!prev_node)
+ WRITE_ONCE(lock_meta->head_node, next_node);
+
+ *p_next_node = next_node;
+ } else {
+ /*
+ * local queue has other contenders.
+ *
+ * 1) "lock_tail_updated == true":
+ * It means we have at least "prev_node" and unlinked "local node"
+ * Also, some new nodes can arrive and link after "prev_node".
+ * We should just re-add "local node": (prev_node) => ... => (local_node)
+ * and perform local handoff, as other CPUs from the local node do "mcs spin"
+ *
+ * 2) "lock_tail_cleared == true"
+ * It means we cleared "lock_meta->tail_node" and "lock->head_node".
+ * We need to re-add "local node" and move "local_queue->head" to the next "mcs-node",
+ * which is in the progress of linking after the current "mcs-node"
+ * (that's why we couldn't clear the "local_queue->tail").
+ *
+ * Meanwhile other nodes can arrive: (new_node) => (...)
+ * That "new_node" will spin in "global spin" mode.
+ * In this case no handoff needed.
+ *
+ * 3) "!lock_tail_cleared && !lock_tail_updated"
+ * It means we had at least one node after us before 'try_clear_queue_tail'
+ * and only need to perform local handoff
+ */
+
+ /* Cases 1) and 2) */
+ if (lock_tail_updated || lock_tail_cleared) {
+ u16 prev_node_id;
+
+ init_queue_link(queue);
+ prev_node_id =
+ append_node_queue(lock_id, local_node);
+
+ if (prev_node_id && lock_tail_cleared) {
+ /* Case 2) */
+ ret = true;
+ WRITE_ONCE(queue->head,
+ (void *) smp_cond_load_relaxed(&node->next, (VAL)));
+ goto out;
+ }
+ }
+
+ /* Cases 1) and 3) */
+ *p_next_node = HQLOCK_HANDOFF_LOCAL;
+ ret = false;
+ }
+out:
+ /*
+ * Either handoff for current node,
+ * or remote handoff if the quota is expired
+ */
+ return ret;
+}
+
+static inline void hqlock_handoff(struct qspinlock *lock,
+ struct mcs_spinlock *node,
+ struct mcs_spinlock *next, u32 tail,
+ int handoff_info);
+
+
+/*
+ * Chech if contention has risen and if we need to set NUMA-aware mode
+ */
+static __always_inline bool determine_contention_qspinlock_mode(struct mcs_spinlock *node)
+{
+ struct numa_qnode *qnode = (void *)node;
+
+ if (qnode->general_handoffs > READ_ONCE(hqlock_general_handoffs_turn_numa))
+ return true;
+ return false;
+}
+
+static __always_inline bool low_contention_try_clear_tail(struct qspinlock *lock,
+ u32 val,
+ struct mcs_spinlock *node)
+{
+ u32 update_val = _Q_LOCKED_VAL | _Q_LOCKTYPE_HQ;
+
+ bool high_contention = determine_contention_qspinlock_mode(node);
+
+ /*
+ * If we have high contention, we set _Q_LOCK_INVALID_TAIL
+ * to notify upcomming contenders, which have seen QSPINLOCK mode,
+ * that performing generic 'xchg_tail' is wrong.
+ *
+ * We cannot also set HQLOCK mode here,
+ * because first contender in updated mode
+ * should check if lock's metadata is free
+ */
+ if (!high_contention)
+ update_val |= _Q_LOCK_MODE_QSPINLOCK_VAL;
+ else
+ update_val |= _Q_LOCK_INVALID_TAIL;
+
+ return atomic_try_cmpxchg_relaxed(&lock->val, &val, update_val);
+}
+
+static __always_inline void low_contention_mcs_lock_handoff(struct mcs_spinlock *node,
+ struct mcs_spinlock *next, struct mcs_spinlock *prev)
+{
+ struct numa_qnode *qnode = (void *)node;
+ struct numa_qnode *qnext = (void *)next;
+
+ static u16 max_u16 = (u16)(-1);
+
+ u16 general_handoffs = qnode->general_handoffs;
+
+ if (next != prev && likely(general_handoffs + 1 != max_u16))
+ general_handoffs++;
+
+ qnext->general_handoffs = general_handoffs;
+ arch_mcs_spin_unlock_contended(&next->locked);
+}
+
+static inline void hqlock_clear_tail_handoff(struct qspinlock *lock, u32 val,
+ u32 tail,
+ struct mcs_spinlock *node,
+ struct mcs_spinlock *next,
+ struct mcs_spinlock *prev,
+ bool is_numa_lock)
+{
+ int handoff_info;
+ struct numa_qnode *qnode = (void *)node;
+
+ /*
+ * qnode->wrong_fallback_tail means we have queued globally
+ * in 'try_update_tail_qspinlock_mode' after another contender,
+ * but lock's mode was not QSPINLOCK in that moment.
+ *
+ * First confused contender has restored _Q_LOCK_INVALID_TAIL in global tail
+ * and set us in his local queue.
+ */
+ if (is_numa_lock || qnode->wrong_fallback_tail) {
+ /*
+ * Because of splitting generic tail and NUMA tail we must set locked before clearing tail,
+ * otherwise double lock is possible
+ */
+ set_locked(lock);
+
+ if (hqlock_try_clear_tail(lock, val, tail, node, &handoff_info))
+ return;
+
+ hqlock_handoff(lock, node, next, tail, handoff_info);
+ } else {
+ if ((val & _Q_TAIL_MASK) == tail) {
+ if (low_contention_try_clear_tail(lock, val, node))
+ return;
+ }
+
+ set_locked(lock);
+
+ if (!next)
+ next = smp_cond_load_relaxed(&node->next, (VAL));
+
+ low_contention_mcs_lock_handoff(node, next, prev);
+ }
+}
+
+static inline void hqlock_init_node(struct mcs_spinlock *node)
+{
+ struct numa_qnode *qnode = (void *)node;
+
+ qnode->general_handoffs = 0;
+ qnode->numa_node = numa_node_id() + 1;
+ qnode->lock_id = 0;
+ qnode->wrong_fallback_tail = 0;
+}
+
+static inline void reset_handoff_counter(struct numa_qnode *qnode)
+{
+ qnode->general_handoffs = 0;
+}
+
+static inline void handoff_local(struct mcs_spinlock *node,
+ struct mcs_spinlock *next,
+ u32 tail)
+{
+ static u16 max_u16 = (u16)(-1);
+
+ struct numa_qnode *qnode = (struct numa_qnode *)node;
+ struct numa_qnode *qnext = (struct numa_qnode *)next;
+
+ u16 general_handoffs = qnode->general_handoffs;
+
+ if (likely(general_handoffs + 1 != max_u16))
+ general_handoffs++;
+
+ qnext->general_handoffs = general_handoffs;
+
+ u16 wrong_fallback_tail = qnode->wrong_fallback_tail;
+
+ if (wrong_fallback_tail != 0 && wrong_fallback_tail != (tail >> _Q_TAIL_OFFSET)) {
+ qnext->numa_node = qnode->numa_node;
+ qnext->wrong_fallback_tail = wrong_fallback_tail;
+ qnext->lock_id = qnode->lock_id;
+ }
+
+ arch_mcs_spin_unlock_contended(&next->locked);
+}
+
+static inline void handoff_remote(struct qspinlock *lock,
+ struct numa_qnode *qnode,
+ u32 tail, int handoff_info)
+{
+ struct numa_queue *next_queue = NULL;
+ struct mcs_spinlock *mcs_head = NULL;
+ struct numa_qnode *qhead = NULL;
+ u16 lock_id = qnode->lock_id;
+
+ struct lock_metadata *lock_meta = get_meta(lock_id);
+ struct numa_queue *queue = get_local_queue(qnode);
+
+ u16 next_node_id;
+ u16 node_head, node_tail;
+
+ node_tail = READ_ONCE(lock_meta->tail_node);
+ node_head = READ_ONCE(lock_meta->head_node);
+
+ /*
+ * 'handoffs_not_head > 0' means at the head of NUMA-level queue we have a node
+ * which is heavily loaded and has performed a remote handoff upon reaching the threshold.
+ *
+ * Perform handoff to the head instead of next node in the NUMA-level queue,
+ * if handoffs_not_head >= nr_online_nodes
+ * (It means other contended nodes have been taking the lock at least once after the head one)
+ */
+ u16 handoffs_not_head = READ_ONCE(queue->handoffs_not_head);
+
+ if (handoff_info > 0 && (handoffs_not_head < nr_online_nodes)) {
+ next_node_id = handoff_info;
+ if (node_head != qnode->numa_node)
+ handoffs_not_head++;
+ } else {
+ if (!node_head) {
+ /* If we're here - we have defintely other node-contenders, let's wait */
+ next_node_id = smp_cond_load_relaxed(&lock_meta->head_node, (VAL));
+ } else {
+ next_node_id = node_head;
+ }
+
+ handoffs_not_head = 0;
+ }
+
+ next_queue = get_queue(lock_id, next_node_id);
+ WRITE_ONCE(next_queue->handoffs_not_head, handoffs_not_head);
+
+ qhead = READ_ONCE(next_queue->head);
+
+ mcs_head = (void *) qhead;
+
+ /* arch_mcs_spin_unlock_contended implies smp-barrier */
+ arch_mcs_spin_unlock_contended(&mcs_head->locked);
+}
+
+static inline bool has_other_nodes(struct qspinlock *lock,
+ struct numa_qnode *qnode)
+{
+ struct lock_metadata *lock_meta = get_meta(qnode->lock_id);
+
+ return lock_meta->tail_node != qnode->numa_node;
+}
+
+static inline bool is_node_threshold_reached(struct numa_qnode *qnode)
+{
+ return qnode->general_handoffs > hqlock_fairness_threshold;
+}
+
+static inline void hqlock_handoff(struct qspinlock *lock,
+ struct mcs_spinlock *node,
+ struct mcs_spinlock *next, u32 tail,
+ int handoff_info)
+{
+ struct numa_qnode *qnode = (void *)node;
+ u16 lock_id = qnode->lock_id;
+ struct lock_metadata *lock_meta = get_meta(lock_id);
+ struct numa_queue *queue = get_local_queue(qnode);
+
+ if (handoff_info == HQLOCK_HANDOFF_LOCAL) {
+ if (!next)
+ next = smp_cond_load_relaxed(&node->next, (VAL));
+ WRITE_ONCE(queue->head, (void *) next);
+
+ bool threshold_expired = is_node_threshold_reached(qnode);
+
+ if (!threshold_expired || qnode->wrong_fallback_tail) {
+ handoff_local(node, next, tail);
+ return;
+ }
+
+ u16 queue_next = READ_ONCE(queue->next_node);
+ bool has_others = has_other_nodes(lock, qnode);
+
+ /*
+ * This check is racy, but it's ok,
+ * because we fallback to local node in the worst case
+ * and do not call reset_handoff_counter.
+ * Next local contender will perform remote handoff
+ * after next queue is properly linked
+ */
+ if (has_others) {
+ handoff_info =
+ queue_next > 0 ? queue_next : HQLOCK_HANDOFF_LOCAL;
+ } else {
+ handoff_info = HQLOCK_HANDOFF_REMOTE_HEAD;
+ }
+
+ if (handoff_info == HQLOCK_HANDOFF_LOCAL ||
+ (handoff_info == HQLOCK_HANDOFF_REMOTE_HEAD &&
+ READ_ONCE(lock_meta->head_node) == qnode->numa_node)) {
+ /*
+ * No other nodes have come yet, so we can clean fairness counter
+ */
+ if (handoff_info == HQLOCK_HANDOFF_REMOTE_HEAD)
+ reset_handoff_counter(qnode);
+ handoff_local(node, next, tail);
+ return;
+ }
+ }
+
+ handoff_remote(lock, qnode, tail, handoff_info);
+ reset_handoff_counter(qnode);
+}
diff --git a/kernel/locking/hqlock_meta.h b/kernel/locking/hqlock_meta.h
new file mode 100644
index 0000000000..5b54801326
--- /dev/null
+++ b/kernel/locking/hqlock_meta.h
@@ -0,0 +1,467 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _GEN_HQ_SPINLOCK_SLOWPATH
+#error "Do not include this file!"
+#endif
+
+/* Lock metadata pool */
+static struct lock_metadata *meta_pool;
+
+static inline struct lock_metadata *get_meta(u16 lock_id)
+{
+ return &meta_pool[lock_id];
+}
+
+static inline hqlock_mode_t set_lock_mode(struct qspinlock *lock, int __val, u16 lock_id)
+{
+ u32 val = (u32)__val;
+ u32 new_val = 0;
+ u32 lock_mode = encode_lock_mode(lock_id);
+
+ while (!(val & _Q_LOCK_MODE_MASK)) {
+ /*
+ * We need wait until pending is gone.
+ * Otherwise, clearing pending can erase a NUMA mode we will set here
+ */
+ if (val & _Q_PENDING_VAL) {
+ val = atomic_cond_read_relaxed(&lock->val, !(VAL & _Q_PENDING_VAL));
+
+ if (val & _Q_LOCK_MODE_MASK)
+ return LOCK_NO_MODE;
+ }
+
+ /*
+ * If we are enabling NUMA-awareness, we should keep previous value in lock->tail
+ * in case of having contenders seen LOCK_MODE_QSPINLOCK and set their tails via xchg_tail
+ * (They will restore it to _Q_LOCK_INVALID_TAIL later).
+ * If we are setting LOCK_MODE_QSPINLOCK, remove _Q_LOCK_INVALID_TAIL
+ */
+ if (lock_id != LOCK_ID_NONE)
+ new_val = val | lock_mode;
+ else
+ new_val = (val & ~_Q_LOCK_INVALID_TAIL) | lock_mode;
+
+ /*
+ * If we're setting LOCK_MODE_HQLOCK, make sure all "seq_counter"
+ * updates (per-queue, lock_meta) are observed before lock mode update.
+ * Paired with smp_rmb() in setup_lock_mode().
+ */
+ if (lock_id != LOCK_ID_NONE)
+ smp_wmb();
+
+ bool updated = atomic_try_cmpxchg_relaxed(&lock->val, &val, new_val);
+
+ if (updated) {
+ return (lock_id == LOCK_ID_NONE) ?
+ LOCK_MODE_QSPINLOCK : LOCK_MODE_HQLOCK;
+ }
+ }
+
+ return LOCK_NO_MODE;
+}
+
+static inline hqlock_mode_t set_mode_hqlock(struct qspinlock *lock, int val, u16 lock_id)
+{
+ return set_lock_mode(lock, val, lock_id);
+}
+
+static inline hqlock_mode_t set_mode_qspinlock(struct qspinlock *lock, int val)
+{
+ return set_lock_mode(lock, val, LOCK_ID_NONE);
+}
+
+/* Dynamic lock-mode conditions */
+static inline bool is_mode_hqlock(int val)
+{
+ return decode_lock_mode(val) == LOCK_MODE_HQLOCK;
+}
+
+static inline bool is_mode_qspinlock(int val)
+{
+ return decode_lock_mode(val) == LOCK_MODE_QSPINLOCK;
+}
+
+enum meta_status {
+ META_CONFLICT = 0,
+ META_GRABBED,
+ META_SHARED,
+};
+
+static inline enum meta_status grab_lock_meta(struct qspinlock *lock, u32 lock_id, u32 *seq)
+{
+ int nid, seq_counter;
+ struct numa_queue *queue;
+ struct qspinlock *old = READ_ONCE(meta_pool[lock_id].lock_ptr);
+
+ if (old && old != lock)
+ return META_CONFLICT;
+
+ if (old && old == lock)
+ return META_SHARED;
+
+ old = cmpxchg_acquire(&meta_pool[lock_id].lock_ptr, NULL, lock);
+ if (!old)
+ goto init_meta;
+
+ /* Hash-conflict */
+ if (old != lock)
+ return META_CONFLICT;
+
+ return META_SHARED;
+init_meta:
+ /*
+ * Update allocations counter and set it to per-NUMA queues
+ * to prevent upcomming contenders from parking on deallocated queues
+ */
+ seq_counter = atomic_inc_return_relaxed(&meta_pool[lock_id].seq_counter);
+
+ /* Very unlikely we can overflow */
+ if (unlikely(seq_counter == 0))
+ seq_counter = atomic_inc_return_relaxed(&meta_pool[lock_id].seq_counter);
+
+ for_each_online_node(nid) {
+ queue = &queue_table[nid][lock_id];
+ WRITE_ONCE(queue->seq_counter, (u32)seq_counter);
+ }
+
+ *seq = seq_counter;
+ return META_GRABBED;
+}
+
+/*
+ * Try to setup current lock mode:
+ *
+ * LOCK_MODE_HQLOCK or fallback to default LOCK_MODE_QSPINLOCK
+ * if there's hash conflict with another lock in the system.
+ *
+ * In general the setup consists of grabbing lock-related metadata and
+ * publishing the mode in the global lock variable.
+ *
+ * For quick meta-lookup the pointer hashing is used.
+ *
+ * To identify "occupied/free" metadata record, we use "meta->lock_ptr"
+ * which is set to corresponding spinlock lock pointer or "NULL".
+ *
+ * The action sequence from initial state is the following:
+ *
+ * "Find lock-meta by hash" => "Occupy lock-meta" => publish "LOCK_MODE_HQLOCK" in
+ * global lock variable.
+ *
+ */
+static inline
+hqlock_mode_t setup_lock_mode(struct qspinlock *lock, u16 lock_id, u32 *meta_seq_counter)
+{
+ hqlock_mode_t mode;
+
+ do {
+ enum meta_status status;
+ int val = atomic_read(&lock->val);
+
+ if (is_mode_hqlock(val)) {
+ struct lock_metadata *lock_meta = get_meta(lock_id);
+ /*
+ * The lock is currently in LOCK_MODE_HQLOCK, we need to make sure the
+ * associated metadata isn't used by another lock.
+ *
+ * In the meanwhile several situations can occur:
+ *
+ * [Case 1] Another lock using the meta (hash-conflict)
+ *
+ * If "release + reallocate" of the meta happenned in the meanwhile,
+ * we're guaranteed to observe lock-mode change in the "lock->val",
+ * due to the following event ordering:
+ *
+ * [release_lock_meta]
+ * Clear lock mode in "lock->val", so we wouldn't
+ * observe LOCK_MODE_HQLOCK mode.
+ * =>
+ * [setup_lock_mode]
+ * Update lock->seq_counter
+ *
+ * [Case 2] For exact same lock, some contender did "release + reallocate" of the meta
+ *
+ * Either We'll get newly set "seq_counter", or in the worst case, we'll get
+ * outdated "seq_counter" fail in the CAS(queue) in the caller function.
+ *
+ * [Case 3] Meta is free, nobody using it
+ * [Case 4] The lock mode is changed to LOCK_MODE_QSPINLOCK.
+ */
+ int seq_counter = atomic_read(&lock_meta->seq_counter);
+
+ /*
+ * "seq_counter" and "lock->val" should be read in program order.
+ * Otherwise we might observe "seq_counter" updated on-behalf another lock.
+ * Paired with smp_wmb() in set_lock_mode().
+ */
+ smp_rmb();
+ val = atomic_read(&lock->val);
+
+ if (is_mode_hqlock(val)) {
+ *meta_seq_counter = (u32)seq_counter;
+ return LOCK_MODE_HQLOCK;
+ }
+ /*
+ * [else] Here it can be 2 options:
+ *
+ * 1. Lock-meta is free, and nobody using it.
+ * In this case, we need to try occupying the meta and
+ * publish lock-mode LOCK_MODE_HQLOCK again.
+ *
+ * 2. Lock mode transitioned to LOCK_MODE_QSPINLOCK mode.
+ */
+ continue;
+ } else if (is_mode_qspinlock(val)) {
+ return LOCK_MODE_QSPINLOCK;
+ }
+
+ /*
+ * Trying to get temporary metadata "weak" ownership,
+ * Three situations might happen:
+ *
+ * 1. Metadata isn't used by anyone
+ * Just take the ownership.
+ *
+ * 2. Metadata is already grabbed by one of the lock contenders.
+ *
+ * 3. Hash conflict: metadata is owned by another lock
+ * Give up, fallback to LOCK_MODE_QSPINLOCK.
+ */
+ status = grab_lock_meta(lock, lock_id, meta_seq_counter);
+ if (status == META_SHARED) {
+ /*
+ * Someone started publishing lock_id for us:
+ * 1. We can catch the "LOCK_MODE_HQLOCK" mode quickly
+ * 2. We can loop several times before we'll see "LOCK_MODE_HQLOCK" mode set.
+ * (lightweight check)
+ * 3. Another contender might be able to relase lock meta meanwhile.
+ * Either we catch it in above "seq_counter" check, or we'll grab
+ * lock meta first and try publishing lock_id.
+ */
+ continue;
+ }
+
+ /* Setup the lock-mode */
+ if (status == META_GRABBED)
+ mode = set_mode_hqlock(lock, val, lock_id);
+ else if (status == META_CONFLICT)
+ mode = set_mode_qspinlock(lock, val);
+ else
+ BUG_ON(1);
+ /*
+ * If we grabbed the meta but were unable to publish LOCK_MODE_HQLOCK
+ * release it, just by resetting the pointer.
+ */
+ if (status == META_GRABBED && mode != LOCK_MODE_HQLOCK) {
+ smp_store_release(&meta_pool[lock_id].lock_ptr, NULL);
+ }
+ } while (mode == LOCK_NO_MODE);
+
+ return mode;
+}
+
+static inline void release_lock_meta(struct qspinlock *lock,
+ struct lock_metadata *meta,
+ struct numa_qnode *qnode)
+{
+ int nid;
+ struct numa_queue *queue;
+ bool cleared = false;
+ u32 upd_val = _Q_LOCKTYPE_HQ | _Q_LOCKED_VAL;
+ u16 lock_id = qnode->lock_id;
+ int seq_counter = atomic_read(&meta->seq_counter);
+
+ /*
+ * Firstly, go across per-NUMA queues and set seq counter to 0,
+ * it will prevent possible contenders, which haven't even queued locally,
+ * from using already deoccupied metadata.
+ *
+ * We need to perform counter reset with CAS,
+ * because local contenders (we didn't see them while try_clear_lock_tail and try_clear_queue_tail)
+ * may have appeared while we were coming that point.
+ *
+ * If any CAS is not successful, it means someone has already queued locally,
+ * in that case we should restore usability of all local queues
+ * and return seq counter to every per-NUMA queue.
+ *
+ * If all CASes are successful, nobody will queue on this metadata's queues,
+ * and we can free it and allow other locks to use it.
+ */
+
+ /*
+ * Before metadata release read every queue tail,
+ * if we have at least one contender, don't do CASes and leave
+ * (Reads are much faster and also prefetch local queue's cachelines)
+ */
+ for_each_online_node(nid) {
+ struct numa_queue *queue = get_queue(lock_id, nid + 1);
+
+ if (READ_ONCE(queue->tail) != 0)
+ return;
+ }
+
+ for_each_online_node(nid) {
+ struct numa_queue *queue = get_queue(lock_id, nid + 1);
+
+ if (cmpxchg_relaxed(&queue->seq_counter_tail, encode_tc(0, seq_counter), 0)
+ != encode_tc(0, seq_counter))
+ /* Some contender arrived - rollback */
+ goto do_rollback;
+ }
+
+ /*
+ * We need wait until pending is gone.
+ * Otherwise, clearing pending can erase a mode we will set here
+ */
+ while (!cleared) {
+ u32 old_lock_val = atomic_cond_read_relaxed(&lock->val, !(VAL & _Q_PENDING_VAL));
+
+ cleared = atomic_try_cmpxchg_relaxed(&lock->val,
+ &old_lock_val, upd_val | (old_lock_val & _Q_TAIL_MASK));
+ }
+
+ /*
+ * guarantee current seq counter is erased from every local queue
+ * and lock mode has been updated before another lock can use metadata
+ */
+ smp_store_release(&meta_pool[qnode->lock_id].lock_ptr, NULL);
+ return;
+
+do_rollback:
+ for_each_online_node(nid) {
+ queue = get_queue(lock_id, nid + 1);
+ WRITE_ONCE(queue->seq_counter, seq_counter);
+ }
+}
+
+/*
+ * Call it if we observe LOCK_MODE_QSPINLOCK.
+ *
+ * We can do generic xchg_tail in this case,
+ * if lock's mode has already been changed, we will get _Q_LOCK_INVALID_TAIL.
+ *
+ * If we have such a situation, we perform CAS cycle
+ * to restore _Q_LOCK_INVALID_TAIL or wait until lock's mode is LOCK_MODE_QSPINLOCK.
+ *
+ * All upcomming confused contenders will see valid tail.
+ * We will remember the last one before successful CAS and put its tail in local queue.
+ * During handoff we will notify them about mode change via qnext->wrong_fallback_tail
+ */
+static inline bool try_update_tail_qspinlock_mode(struct qspinlock *lock, u32 tail, u32 *old_tail, u32 *next_tail)
+{
+ /*
+ * next_tail may be tail or last cpu from previous unsuccessful call
+ * (highly unlikely, but still)
+ */
+ u32 xchged_tail = xchg_tail(lock, *next_tail);
+
+ if (likely(xchged_tail != _Q_LOCK_INVALID_TAIL)) {
+ *old_tail = xchged_tail;
+ return true;
+ }
+
+ /*
+ * If we got _Q_LOCK_INVALID_TAIL, it means lock was not in LOCK_MODE_QSPINLOCK.
+ * In this case we should restore _Q_LOCK_INVALID_TAIL
+ * and remember next contenders that got confused.
+ * Later we will update lock's or local queue's tail to the last contender seen here.
+ */
+ u32 val = atomic_read(&lock->val);
+
+ bool fixed = false;
+
+ while (!fixed) {
+ if (decode_lock_mode(val) == LOCK_MODE_QSPINLOCK) {
+ *old_tail = 0;
+ return true;
+ }
+
+ /*
+ * CAS is needed here to catch possible lock mode change
+ * from LOCK_MODE_HQLOCK to LOCK_MODE_QSPINLOCK in the meanwhile.
+ * Thus preventing from publishing _Q_LOCK_INVALID_TAIL
+ * when LOCK_MODE_QSPINLOCK is enabled.
+ */
+ fixed = atomic_try_cmpxchg_relaxed(&lock->val, &val, _Q_LOCK_INVALID_TAIL |
+ (val & (_Q_LOCKED_PENDING_MASK | _Q_LOCK_TYPE_MODE_MASK)));
+ }
+
+ if ((val & _Q_TAIL_MASK) != tail)
+ *next_tail = val & _Q_TAIL_MASK;
+
+ return false;
+}
+
+/*
+ * Call it if we observe LOCK_MODE_HQLOCK or LOCK_NO_MODE in the lock.
+ *
+ * Actions performed:
+ * - Call setup_lock_mode to set or read lock's mode,
+ * read metadata's sequential counter for valid local queueing
+ * - CAS on union of local tail and meta_seq_counter
+ * to guarantee metadata usage correctness.
+ * Repeat from the beginning if fail.
+ * - If we are the first local contender,
+ * update global tail with our NUMA node
+ */
+static inline bool try_update_tail_hqlock_mode(struct qspinlock *lock, u16 lock_id,
+ struct numa_qnode *qnode, u32 tail, u32 *next_tail, u32 *old_tail)
+{
+ u32 meta_seq_counter;
+ hqlock_mode_t mode;
+
+ struct numa_queue *queue;
+ u64 old_counter_tail;
+ bool updated_queue_tail = false;
+
+re_setup:
+ mode = setup_lock_mode(lock, lock_id, &meta_seq_counter);
+
+ if (mode == LOCK_MODE_QSPINLOCK)
+ return false;
+
+ queue = get_local_queue(qnode);
+
+ /*
+ * While queueing locally, perform CAS cycle
+ * on union of tail and meta_seq_counter.
+ *
+ * meta_seq_counter is taken from the lock metadata while allocation,
+ * it's updated every time it's used by a next lock.
+ * It shows that queue is used correctly
+ * and metadata hasn't been deoccupied before we queued locally.
+ */
+ old_counter_tail = READ_ONCE(queue->seq_counter_tail);
+
+ while (!updated_queue_tail &&
+ decode_tc_counter(old_counter_tail) == meta_seq_counter) {
+ updated_queue_tail =
+ try_cmpxchg_relaxed(&queue->seq_counter_tail, &old_counter_tail,
+ encode_tc((*next_tail) >> _Q_TAIL_OFFSET, meta_seq_counter));
+ }
+
+ /* queue->seq_counter changed */
+ if (!updated_queue_tail)
+ goto re_setup;
+
+ /*
+ * The condition means we tried to perform generic tail update in try_update_tail_qspinlock_mode,
+ * but before we did it, lock type was changed.
+ * Moreover, some contenders have come after us in LOCK_MODE_QSPINLOCK,
+ * during handoff we must notify them that they are set in LOCK_MODE_HQLOCK in our node's local queue
+ */
+ if (unlikely(*next_tail != tail))
+ qnode->wrong_fallback_tail = *next_tail >> _Q_TAIL_OFFSET;
+
+ *old_tail = decode_tc_tail(old_counter_tail);
+
+ if (!(*old_tail)) {
+ u16 prev_node_id;
+
+ init_queue(qnode);
+ prev_node_id = append_node_queue(lock_id, qnode->numa_node);
+ *old_tail = prev_node_id ? Q_NEW_NODE_QUEUE : 0;
+ } else {
+ *old_tail <<= _Q_TAIL_OFFSET;
+ }
+
+ return true;
+}
--
2.34.1
^ permalink raw reply related
* [RFC PATCH v3 5/7] kernel: introduce general hq-spinlock support
From: Fedorov Nikita @ 2026-04-15 16:44 UTC (permalink / raw)
To: Catalin Marinas, Will Deacon, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, Dave Hansen, x86, hpa, Juergen Gross, Ajay Kaher,
Alexey Makhalov, bcm-kernel-feedback-list, Arnd Bergmann,
Peter Zijlstra, Boqun Feng, Waiman Long, Darren Hart,
Davidlohr Bueso, andrealmeid, Andrew Morton, David Hildenbrand,
Zi Yan, Matthew Brost, Joshua Hahn, Rakie Kim, byungchul,
Gregory Price, Ying Huang, Alistair Popple, Anatoly Stepanov
Cc: Nikita Fedorov, linux-arm-kernel, linux-kernel, virtualization,
linux-arch, linux-mm, guohanjun, wangkefeng.wang, weiyongjun1,
yusongping, leijitang, artem.kuzin, kang.sun, chenjieping3
In-Reply-To: <20260415164459.2904963-1-fedorov.nikita@h-partners.com>
Integrate HQ spinlocks into the generic queued spinlock code.
It includes:
- Kernel config
- Required macros
- PV-locks support
- HQ-lock type/mode masking support in generic code
- Integration hq-spinlock into queued spinlock slowpath
Now hq-spinlock can be enabled with:
- `spin_lock_init_hq()` for dynamic locks
- `DEFINE_SPINLOCK_HQ()` for static locks
Co-developed-by: Anatoly Stepanov <stepanov.anatoly@huawei.com>
Signed-off-by: Anatoly Stepanov <stepanov.anatoly@huawei.com>
Co-developed-by: Nikita Fedorov <fedorov.nikita@h-partners.com>
Signed-off-by: Nikita Fedorov <fedorov.nikita@h-partners.com>
---
arch/arm64/include/asm/qspinlock.h | 37 ++++++++++++
arch/x86/include/asm/hq-spinlock.h | 34 +++++++++++
arch/x86/include/asm/paravirt-spinlock.h | 3 +-
arch/x86/include/asm/qspinlock.h | 6 +-
include/asm-generic/qspinlock.h | 23 ++++---
include/linux/spinlock.h | 26 ++++++++
include/linux/spinlock_types.h | 26 ++++++++
include/linux/spinlock_types_raw.h | 20 ++++++
kernel/Kconfig.locks | 29 +++++++++
kernel/locking/hqlock_core.h | 77 ++++++++++++++++++++++++
kernel/locking/qspinlock.c | 65 +++++++++++++++++---
kernel/locking/qspinlock.h | 4 +-
kernel/locking/spinlock_debug.c | 20 ++++++
mm/mempolicy.c | 4 ++
14 files changed, 352 insertions(+), 22 deletions(-)
create mode 100644 arch/arm64/include/asm/qspinlock.h
create mode 100644 arch/x86/include/asm/hq-spinlock.h
diff --git a/arch/arm64/include/asm/qspinlock.h b/arch/arm64/include/asm/qspinlock.h
new file mode 100644
index 0000000000..5b8b1ca0f4
--- /dev/null
+++ b/arch/arm64/include/asm/qspinlock.h
@@ -0,0 +1,37 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_ARM64_QSPINLOCK_H
+#define _ASM_ARM64_QSPINLOCK_H
+
+#ifdef CONFIG_HQSPINLOCKS
+
+extern void hq_configure_spin_lock_slowpath(void);
+
+extern void (*hq_queued_spin_lock_slowpath)(struct qspinlock *lock, u32 val);
+extern void native_queued_spin_lock_slowpath(struct qspinlock *lock, u32 val);
+
+#define queued_spin_unlock queued_spin_unlock
+/**
+ * queued_spin_unlock - release a queued spinlock
+ * @lock : Pointer to queued spinlock structure
+ *
+ * A smp_store_release() on the least-significant byte.
+ */
+static inline void native_queued_spin_unlock(struct qspinlock *lock)
+{
+ smp_store_release(&lock->locked, 0);
+}
+
+static inline void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val)
+{
+ hq_queued_spin_lock_slowpath(lock, val);
+}
+
+static inline void queued_spin_unlock(struct qspinlock *lock)
+{
+ native_queued_spin_unlock(lock);
+}
+#endif
+
+#include <asm-generic/qspinlock.h>
+
+#endif /* _ASM_ARM64_QSPINLOCK_H */
diff --git a/arch/x86/include/asm/hq-spinlock.h b/arch/x86/include/asm/hq-spinlock.h
new file mode 100644
index 0000000000..f4b088164b
--- /dev/null
+++ b/arch/x86/include/asm/hq-spinlock.h
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef _ASM_X86_HQ_SPINLOCK_H
+#define _ASM_X86_HQ_SPINLOCK_H
+
+extern void hq_configure_spin_lock_slowpath(void);
+
+#ifndef CONFIG_PARAVIRT_SPINLOCKS
+extern void (*hq_queued_spin_lock_slowpath)(struct qspinlock *lock, u32 val);
+extern void native_queued_spin_lock_slowpath(struct qspinlock *lock, u32 val);
+
+#define queued_spin_unlock queued_spin_unlock
+/**
+ * queued_spin_unlock - release a queued spinlock
+ * @lock : Pointer to queued spinlock structure
+ *
+ * A smp_store_release() on the least-significant byte.
+ */
+static inline void native_queued_spin_unlock(struct qspinlock *lock)
+{
+ smp_store_release(&lock->locked, 0);
+}
+
+static inline void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val)
+{
+ hq_queued_spin_lock_slowpath(lock, val);
+}
+
+static inline void queued_spin_unlock(struct qspinlock *lock)
+{
+ native_queued_spin_unlock(lock);
+}
+#endif // !CONFIG_PARAVIRT_SPINLOCKS
+
+#endif // _ASM_X86_HQ_SPINLOCK_H
diff --git a/arch/x86/include/asm/paravirt-spinlock.h b/arch/x86/include/asm/paravirt-spinlock.h
index 7beffcb08e..9c37d6b47b 100644
--- a/arch/x86/include/asm/paravirt-spinlock.h
+++ b/arch/x86/include/asm/paravirt-spinlock.h
@@ -134,7 +134,8 @@ static inline bool virt_spin_lock(struct qspinlock *lock)
__retry:
val = atomic_read(&lock->val);
- if (val || !atomic_try_cmpxchg(&lock->val, &val, _Q_LOCKED_VAL)) {
+ if ((val & ~_Q_SERVICE_MASK) ||
+ !atomic_try_cmpxchg(&lock->val, &val, _Q_LOCKED_VAL | (val & _Q_SERVICE_MASK))) {
cpu_relax();
goto __retry;
}
diff --git a/arch/x86/include/asm/qspinlock.h b/arch/x86/include/asm/qspinlock.h
index 25a1919542..13950221eb 100644
--- a/arch/x86/include/asm/qspinlock.h
+++ b/arch/x86/include/asm/qspinlock.h
@@ -11,6 +11,10 @@
#include <asm/paravirt-spinlock.h>
#endif
+#ifdef CONFIG_HQSPINLOCKS
+#include <asm/hq-spinlock.h>
+#endif
+
#define _Q_PENDING_LOOPS (1 << 9)
#define queued_fetch_set_pending_acquire queued_fetch_set_pending_acquire
@@ -25,7 +29,7 @@ static __always_inline u32 queued_fetch_set_pending_acquire(struct qspinlock *lo
*/
val = GEN_BINARY_RMWcc(LOCK_PREFIX "btsl", lock->val.counter, c,
"I", _Q_PENDING_OFFSET) * _Q_PENDING_VAL;
- val |= atomic_read(&lock->val) & ~_Q_PENDING_MASK;
+ val |= atomic_read(&lock->val) & ~_Q_PENDING_VAL;
return val;
}
diff --git a/include/asm-generic/qspinlock.h b/include/asm-generic/qspinlock.h
index bf47cca2c3..af3ab0286e 100644
--- a/include/asm-generic/qspinlock.h
+++ b/include/asm-generic/qspinlock.h
@@ -54,7 +54,7 @@ static __always_inline int queued_spin_is_locked(struct qspinlock *lock)
* Any !0 state indicates it is locked, even if _Q_LOCKED_VAL
* isn't immediately observable.
*/
- return atomic_read(&lock->val);
+ return atomic_read(&lock->val) & ~_Q_SERVICE_MASK;
}
#endif
@@ -70,7 +70,7 @@ static __always_inline int queued_spin_is_locked(struct qspinlock *lock)
*/
static __always_inline int queued_spin_value_unlocked(struct qspinlock lock)
{
- return !lock.val.counter;
+ return !(lock.val.counter & ~_Q_SERVICE_MASK);
}
/**
@@ -80,8 +80,10 @@ static __always_inline int queued_spin_value_unlocked(struct qspinlock lock)
*/
static __always_inline int queued_spin_is_contended(struct qspinlock *lock)
{
- return atomic_read(&lock->val) & ~_Q_LOCKED_MASK;
+ return atomic_read(&lock->val) & ~(_Q_LOCKED_MASK | _Q_SERVICE_MASK);
}
+
+#ifndef queued_spin_trylock
/**
* queued_spin_trylock - try to acquire the queued spinlock
* @lock : Pointer to queued spinlock structure
@@ -91,11 +93,12 @@ static __always_inline int queued_spin_trylock(struct qspinlock *lock)
{
int val = atomic_read(&lock->val);
- if (unlikely(val))
+ if (unlikely(val & ~_Q_SERVICE_MASK))
return 0;
- return likely(atomic_try_cmpxchg_acquire(&lock->val, &val, _Q_LOCKED_VAL));
+ return likely(atomic_try_cmpxchg_acquire(&lock->val, &val, val | _Q_LOCKED_VAL));
}
+#endif
extern void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val);
@@ -106,14 +109,16 @@ extern void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val);
*/
static __always_inline void queued_spin_lock(struct qspinlock *lock)
{
- int val = 0;
+ int val = atomic_read(&lock->val);
- if (likely(atomic_try_cmpxchg_acquire(&lock->val, &val, _Q_LOCKED_VAL)))
- return;
+ if (likely(!(val & ~_Q_SERVICE_MASK))) {
+ if (likely(atomic_try_cmpxchg_acquire(&lock->val, &val, val | _Q_LOCKED_VAL)))
+ return;
+ }
queued_spin_lock_slowpath(lock, val);
}
-#endif
+#endif // queued_spin_lock
#ifndef queued_spin_unlock
/**
diff --git a/include/linux/spinlock.h b/include/linux/spinlock.h
index e1e2f144af..e953018934 100644
--- a/include/linux/spinlock.h
+++ b/include/linux/spinlock.h
@@ -100,6 +100,8 @@
#ifdef CONFIG_DEBUG_SPINLOCK
extern void __raw_spin_lock_init(raw_spinlock_t *lock, const char *name,
struct lock_class_key *key, short inner);
+ extern void __raw_spin_lock_init_hq(raw_spinlock_t *lock, const char *name,
+ struct lock_class_key *key, short inner);
# define raw_spin_lock_init(lock) \
do { \
@@ -108,9 +110,19 @@ do { \
__raw_spin_lock_init((lock), #lock, &__key, LD_WAIT_SPIN); \
} while (0)
+# define raw_spin_lock_init_hq(lock) \
+do { \
+ static struct lock_class_key __key; \
+ \
+ __raw_spin_lock_init_hq((lock), #lock, &__key, LD_WAIT_SPIN); \
+} while (0)
+
#else
# define raw_spin_lock_init(lock) \
do { *(lock) = __RAW_SPIN_LOCK_UNLOCKED(lock); } while (0)
+
+# define raw_spin_lock_init_hq(lock) \
+ do { *(lock) = __RAW_SPIN_LOCK_UNLOCKED_HQ(lock); } while (0)
#endif
#define raw_spin_is_locked(lock) arch_spin_is_locked(&(lock)->raw_lock)
@@ -325,6 +337,14 @@ do { \
#lock, &__key, LD_WAIT_CONFIG); \
} while (0)
+# define spin_lock_init_hq(lock) \
+do { \
+ static struct lock_class_key __key; \
+ \
+ __raw_spin_lock_init_hq(spinlock_check(lock), \
+ #lock, &__key, LD_WAIT_CONFIG); \
+} while (0)
+
#else
# define spin_lock_init(_lock) \
@@ -333,6 +353,12 @@ do { \
*(_lock) = __SPIN_LOCK_UNLOCKED(_lock); \
} while (0)
+# define spin_lock_init_hq(_lock) \
+do { \
+ spinlock_check(_lock); \
+ *(_lock) = __SPIN_LOCK_UNLOCKED_HQ(_lock); \
+} while (0)
+
#endif
static __always_inline void spin_lock(spinlock_t *lock)
diff --git a/include/linux/spinlock_types.h b/include/linux/spinlock_types.h
index b65bb6e445..ad68f6ad8d 100644
--- a/include/linux/spinlock_types.h
+++ b/include/linux/spinlock_types.h
@@ -43,6 +43,29 @@ typedef struct spinlock spinlock_t;
#define DEFINE_SPINLOCK(x) spinlock_t x = __SPIN_LOCK_UNLOCKED(x)
+#ifdef __ARCH_SPIN_LOCK_UNLOCKED_HQ
+#define ___SPIN_LOCK_INITIALIZER_HQ(lockname) \
+ { \
+ .raw_lock = __ARCH_SPIN_LOCK_UNLOCKED_HQ, \
+ SPIN_DEBUG_INIT(lockname) \
+ SPIN_DEP_MAP_INIT(lockname) }
+
+#else
+#define ___SPIN_LOCK_INITIALIZER_HQ(lockname) \
+ { \
+ .raw_lock = __ARCH_SPIN_LOCK_UNLOCKED, \
+ SPIN_DEBUG_INIT(lockname) \
+ SPIN_DEP_MAP_INIT(lockname) }
+#endif
+
+#define __SPIN_LOCK_INITIALIZER_HQ(lockname) \
+ { { .rlock = ___SPIN_LOCK_INITIALIZER_HQ(lockname) } }
+
+#define __SPIN_LOCK_UNLOCKED_HQ(lockname) \
+ ((spinlock_t) __SPIN_LOCK_INITIALIZER_HQ(lockname))
+
+#define DEFINE_SPINLOCK_HQ(x) spinlock_t x = __SPIN_LOCK_UNLOCKED_HQ(x)
+
#else /* !CONFIG_PREEMPT_RT */
/* PREEMPT_RT kernels map spinlock to rt_mutex */
@@ -71,6 +94,9 @@ typedef struct spinlock spinlock_t;
#define DEFINE_SPINLOCK(name) \
spinlock_t name = __SPIN_LOCK_UNLOCKED(name)
+#define DEFINE_SPINLOCK_HQ(name) \
+ spinlock_t name = __SPIN_LOCK_UNLOCKED(name)
+
#endif /* CONFIG_PREEMPT_RT */
#include <linux/rwlock_types.h>
diff --git a/include/linux/spinlock_types_raw.h b/include/linux/spinlock_types_raw.h
index e5644ab216..0e4126a23b 100644
--- a/include/linux/spinlock_types_raw.h
+++ b/include/linux/spinlock_types_raw.h
@@ -71,4 +71,24 @@ typedef struct raw_spinlock raw_spinlock_t;
#define DEFINE_RAW_SPINLOCK(x) raw_spinlock_t x = __RAW_SPIN_LOCK_UNLOCKED(x)
+#ifdef __ARCH_SPIN_LOCK_UNLOCKED_HQ
+#define __RAW_SPIN_LOCK_INITIALIZER_HQ(lockname) \
+{ \
+ .raw_lock = __ARCH_SPIN_LOCK_UNLOCKED_HQ, \
+ SPIN_DEBUG_INIT(lockname) \
+ RAW_SPIN_DEP_MAP_INIT(lockname) }
+
+#else
+#define __RAW_SPIN_LOCK_INITIALIZER_HQ(lockname) \
+{ \
+ .raw_lock = __ARCH_SPIN_LOCK_UNLOCKED, \
+ SPIN_DEBUG_INIT(lockname) \
+ RAW_SPIN_DEP_MAP_INIT(lockname) }
+#endif
+
+#define __RAW_SPIN_LOCK_UNLOCKED_HQ(lockname) \
+ ((raw_spinlock_t) __RAW_SPIN_LOCK_INITIALIZER_HQ(lockname))
+
+#define DEFINE_RAW_SPINLOCK_HQ(x) raw_spinlock_t x = __RAW_SPIN_LOCK_UNLOCKED_HQ(x)
+
#endif /* __LINUX_SPINLOCK_TYPES_RAW_H */
diff --git a/kernel/Kconfig.locks b/kernel/Kconfig.locks
index 4198f0273e..c96f3a4551 100644
--- a/kernel/Kconfig.locks
+++ b/kernel/Kconfig.locks
@@ -243,6 +243,35 @@ config QUEUED_SPINLOCKS
def_bool y if ARCH_USE_QUEUED_SPINLOCKS
depends on SMP
+config HQSPINLOCKS
+ bool "(NUMA-aware) Hierarchical Queued spinlock"
+ depends on NUMA
+ depends on QUEUED_SPINLOCKS
+ depends on NR_CPUS < 16384
+ depends on !CPU_BIG_ENDIAN
+ help
+ Introduce NUMA (Non Uniform Memory Access) awareness into
+ the slow path of kernel's spinlocks.
+
+ We preallocate 'LOCK_ID_MAX' lock_metadata structures with corresponing per-NUMA queues,
+ and the first queueing contender finds metadata corresponding to its lock by lock's hash and occupies it.
+ If the metadata is already occupied, perform fallback to default qspinlock approach.
+ Upcomming contenders then exchange the tail of per-NUMA queue instead of global tail,
+ and until threshold is reached, we pass the lock to the condenters from the local queue.
+ At the global tail we keep NUMA nodes to maintain FIFO on per-node level.
+
+ That approach helps to reduce cross-numa accesses and thus improve lock's performance
+ in high-load scenarios on multi-NUMA node machines.
+
+ Say N, if you want absolute first come first serve fairness.
+
+config HQSPINLOCKS_DEBUG
+ bool "Enable debug output for numa-aware spinlocks"
+ depends on HQSPINLOCKS
+ default n
+ help
+ This option enables statistics for dynamic metadata allocation for HQspinlock
+
config BPF_ARCH_SPINLOCK
bool
diff --git a/kernel/locking/hqlock_core.h b/kernel/locking/hqlock_core.h
index b7681915b4..74f92ac3d8 100644
--- a/kernel/locking/hqlock_core.h
+++ b/kernel/locking/hqlock_core.h
@@ -771,3 +771,80 @@ static inline void hqlock_handoff(struct qspinlock *lock,
handoff_remote(lock, qnode, tail, handoff_info);
reset_handoff_counter(qnode);
}
+
+static void __init hqlock_alloc_global_queues(void)
+{
+ int nid;
+ unsigned long meta_pool_size, queues_size;
+
+ meta_pool_size = ALIGN(sizeof(struct lock_metadata), L1_CACHE_BYTES) * LOCK_ID_MAX;
+
+ pr_info("Init HQspinlock lock_metadata info: size = 0x%lx B\n",
+ meta_pool_size);
+
+ meta_pool = kvzalloc(meta_pool_size, GFP_KERNEL);
+
+ if (!meta_pool)
+ panic("HQspinlock lock_metadata metadata info: allocation failure.\n");
+
+ for (int i = 0; i < LOCK_ID_MAX; i++)
+ atomic_set(&meta_pool[i].seq_counter, 0);
+
+ queues_size = LOCK_ID_MAX * ALIGN(sizeof(struct numa_queue), L1_CACHE_BYTES);
+
+ pr_info("Init HQspinlock per-NUMA metadata (per-node size = 0x%lx B)\n",
+ queues_size);
+
+ for_each_node(nid) {
+ queue_table[nid] = kvzalloc_node(queues_size, GFP_KERNEL, nid);
+
+ if (!queue_table[nid])
+ panic("HQspinlock per-NUMA metadata: allocation failure for node %d.\n", nid);
+ }
+}
+
+#ifdef CONFIG_PARAVIRT_SPINLOCKS
+#define hq_queued_spin_lock_slowpath pv_ops_lock.queued_spin_lock_slowpath
+#else
+void (*hq_queued_spin_lock_slowpath)(struct qspinlock *lock, u32 val) =
+ native_queued_spin_lock_slowpath;
+EXPORT_SYMBOL(hq_queued_spin_lock_slowpath);
+#endif
+
+static int numa_spinlock_flag;
+
+static int __init numa_spinlock_setup(char *str)
+{
+ if (!strcmp(str, "auto")) {
+ numa_spinlock_flag = 0;
+ return 1;
+ } else if (!strcmp(str, "on")) {
+ numa_spinlock_flag = 1;
+ return 1;
+ } else if (!strcmp(str, "off")) {
+ numa_spinlock_flag = -1;
+ return 1;
+ }
+
+ return 0;
+}
+__setup("numa_spinlock=", numa_spinlock_setup);
+
+void __hq_queued_spin_lock_slowpath(struct qspinlock *lock, u32 val);
+
+void __init hq_configure_spin_lock_slowpath(void)
+{
+ if (numa_spinlock_flag < 0)
+ return;
+
+ if (numa_spinlock_flag == 0 && (nr_node_ids < 2 ||
+ hq_queued_spin_lock_slowpath !=
+ native_queued_spin_lock_slowpath)) {
+ return;
+ }
+
+ numa_spinlock_flag = 1;
+ hq_queued_spin_lock_slowpath = __hq_queued_spin_lock_slowpath;
+ pr_info("Enabling HQspinlock\n");
+ hqlock_alloc_global_queues();
+}
diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c
index af8d122bb6..7feab0046e 100644
--- a/kernel/locking/qspinlock.c
+++ b/kernel/locking/qspinlock.c
@@ -11,7 +11,7 @@
* Peter Zijlstra <peterz@infradead.org>
*/
-#ifndef _GEN_PV_LOCK_SLOWPATH
+#if !defined(_GEN_PV_LOCK_SLOWPATH) && !defined(_GEN_HQ_SPINLOCK_SLOWPATH)
#include <linux/smp.h>
#include <linux/bug.h>
@@ -100,7 +100,7 @@ static __always_inline u32 __pv_wait_head_or_lock(struct qspinlock *lock,
#define pv_kick_node __pv_kick_node
#define pv_wait_head_or_lock __pv_wait_head_or_lock
-#ifdef CONFIG_PARAVIRT_SPINLOCKS
+#if defined(CONFIG_PARAVIRT_SPINLOCKS) || defined(CONFIG_HQSPINLOCKS)
#define queued_spin_lock_slowpath native_queued_spin_lock_slowpath
#endif
@@ -133,6 +133,11 @@ void __lockfunc queued_spin_lock_slowpath(struct qspinlock *lock, u32 val)
u32 old, tail;
int idx;
+#if defined(_GEN_HQ_SPINLOCK_SLOWPATH) && !defined(_GEN_PV_LOCK_SLOWPATH)
+ bool is_numa_lock = val & _Q_LOCKTYPE_MASK;
+ bool numa_awareness_on = is_numa_lock && !(val & _Q_LOCK_MODE_QSPINLOCK_VAL);
+#endif
+
BUILD_BUG_ON(CONFIG_NR_CPUS >= (1U << _Q_TAIL_CPU_BITS));
if (pv_enabled())
@@ -147,16 +152,16 @@ void __lockfunc queued_spin_lock_slowpath(struct qspinlock *lock, u32 val)
*
* 0,1,0 -> 0,0,1
*/
- if (val == _Q_PENDING_VAL) {
+ if ((val & ~_Q_SERVICE_MASK) == _Q_PENDING_VAL) {
int cnt = _Q_PENDING_LOOPS;
val = atomic_cond_read_relaxed(&lock->val,
- (VAL != _Q_PENDING_VAL) || !cnt--);
+ ((VAL & ~_Q_SERVICE_MASK) != _Q_PENDING_VAL) || !cnt--);
}
/*
* If we observe any contention; queue.
*/
- if (val & ~_Q_LOCKED_MASK)
+ if (val & ~(_Q_LOCKED_MASK | _Q_SERVICE_MASK))
goto queue;
/*
@@ -173,11 +178,15 @@ void __lockfunc queued_spin_lock_slowpath(struct qspinlock *lock, u32 val)
* n,0,0 -> 0,0,0 transition fail and it will now be waiting
* on @next to become !NULL.
*/
- if (unlikely(val & ~_Q_LOCKED_MASK)) {
-
+ if (unlikely(val & ~(_Q_LOCKED_MASK | _Q_SERVICE_MASK))) {
/* Undo PENDING if we set it. */
- if (!(val & _Q_PENDING_MASK))
+ if (!(val & _Q_PENDING_VAL)) {
+#if defined(_GEN_HQ_SPINLOCK_SLOWPATH) && !defined(_GEN_PV_LOCK_SLOWPATH)
+ hqlock_clear_pending(lock, val);
+#else
clear_pending(lock);
+#endif
+ }
goto queue;
}
@@ -194,14 +203,18 @@ void __lockfunc queued_spin_lock_slowpath(struct qspinlock *lock, u32 val)
* barriers.
*/
if (val & _Q_LOCKED_MASK)
- smp_cond_load_acquire(&lock->locked, !VAL);
+ smp_cond_load_acquire(&lock->locked_pending, !(VAL & _Q_LOCKED_MASK));
/*
* take ownership and clear the pending bit.
*
* 0,1,0 -> 0,0,1
*/
+#if defined(_GEN_HQ_SPINLOCK_SLOWPATH) && !defined(_GEN_PV_LOCK_SLOWPATH)
+ hqlock_clear_pending_set_locked(lock, val);
+#else
clear_pending_set_locked(lock);
+#endif
lockevent_inc(lock_pending);
return;
@@ -274,7 +287,17 @@ void __lockfunc queued_spin_lock_slowpath(struct qspinlock *lock, u32 val)
*
* p,*,* -> n,*,*
*/
+#if defined(_GEN_HQ_SPINLOCK_SLOWPATH) && !defined(_GEN_PV_LOCK_SLOWPATH)
+ if (is_numa_lock)
+ old = hqlock_xchg_tail(lock, tail, node, &numa_awareness_on);
+ else
+ old = xchg_tail(lock, tail);
+
+ if (numa_awareness_on && old == Q_NEW_NODE_QUEUE)
+ goto mcs_spin;
+#else
old = xchg_tail(lock, tail);
+#endif
next = NULL;
/*
@@ -288,6 +311,9 @@ void __lockfunc queued_spin_lock_slowpath(struct qspinlock *lock, u32 val)
WRITE_ONCE(prev->next, node);
pv_wait_node(node, prev);
+#if defined(_GEN_HQ_SPINLOCK_SLOWPATH) && !defined(_GEN_PV_LOCK_SLOWPATH)
+mcs_spin:
+#endif
arch_mcs_spin_lock_contended(&node->locked);
/*
@@ -349,6 +375,12 @@ void __lockfunc queued_spin_lock_slowpath(struct qspinlock *lock, u32 val)
* above wait condition, therefore any concurrent setting of
* PENDING will make the uncontended transition fail.
*/
+#if defined(_GEN_HQ_SPINLOCK_SLOWPATH) && !defined(_GEN_PV_LOCK_SLOWPATH)
+ if (is_numa_lock) {
+ hqlock_clear_tail_handoff(lock, val, tail, node, next, prev, numa_awareness_on);
+ goto release;
+ }
+#endif
if ((val & _Q_TAIL_MASK) == tail) {
if (atomic_try_cmpxchg_relaxed(&lock->val, &val, _Q_LOCKED_VAL))
goto release; /* No contention */
@@ -380,6 +412,21 @@ void __lockfunc queued_spin_lock_slowpath(struct qspinlock *lock, u32 val)
}
EXPORT_SYMBOL(queued_spin_lock_slowpath);
+/* Generate the code for NUMA-aware qspinlock (HQspinlock) */
+#if !defined(_GEN_HQ_SPINLOCK_SLOWPATH) && defined(CONFIG_HQSPINLOCKS)
+#define _GEN_HQ_SPINLOCK_SLOWPATH
+
+#undef pv_init_node
+#define pv_init_node hqlock_init_node
+
+#undef queued_spin_lock_slowpath
+#include "hqlock_core.h"
+#define queued_spin_lock_slowpath __hq_queued_spin_lock_slowpath
+
+#include "qspinlock.c"
+
+#endif
+
/*
* Generate the paravirt code for queued_spin_unlock_slowpath().
*/
diff --git a/kernel/locking/qspinlock.h b/kernel/locking/qspinlock.h
index d69958a844..9933e0e666 100644
--- a/kernel/locking/qspinlock.h
+++ b/kernel/locking/qspinlock.h
@@ -39,7 +39,7 @@
*/
struct qnode {
struct mcs_spinlock mcs;
-#ifdef CONFIG_PARAVIRT_SPINLOCKS
+#if defined(CONFIG_PARAVIRT_SPINLOCKS) || defined(CONFIG_HQSPINLOCKS)
long reserved[2];
#endif
};
@@ -74,7 +74,7 @@ struct mcs_spinlock *grab_mcs_node(struct mcs_spinlock *base, int idx)
return &((struct qnode *)base + idx)->mcs;
}
-#define _Q_LOCKED_PENDING_MASK (_Q_LOCKED_MASK | _Q_PENDING_MASK)
+#define _Q_LOCKED_PENDING_MASK (_Q_LOCKED_MASK | _Q_PENDING_VAL)
#if _Q_PENDING_BITS == 8
/**
diff --git a/kernel/locking/spinlock_debug.c b/kernel/locking/spinlock_debug.c
index 2338b3adfb..f31d13cbc5 100644
--- a/kernel/locking/spinlock_debug.c
+++ b/kernel/locking/spinlock_debug.c
@@ -30,6 +30,26 @@ void __raw_spin_lock_init(raw_spinlock_t *lock, const char *name,
lock->owner_cpu = -1;
}
+void __raw_spin_lock_init_hq(raw_spinlock_t *lock, const char *name,
+ struct lock_class_key *key, short inner)
+{
+#ifdef CONFIG_DEBUG_LOCK_ALLOC
+ /*
+ * Make sure we are not reinitializing a held lock:
+ */
+ debug_check_no_locks_freed((void *)lock, sizeof(*lock));
+ lockdep_init_map_wait(&lock->dep_map, name, key, 0, inner);
+#endif
+#ifdef __ARCH_SPIN_LOCK_UNLOCKED_HQ
+ lock->raw_lock = (arch_spinlock_t)__ARCH_SPIN_LOCK_UNLOCKED_HQ;
+#else
+ lock->raw_lock = (arch_spinlock_t)__ARCH_SPIN_LOCK_UNLOCKED;
+#endif
+ lock->magic = SPINLOCK_MAGIC;
+ lock->owner = SPINLOCK_OWNER_INIT;
+ lock->owner_cpu = -1;
+}
+
EXPORT_SYMBOL(__raw_spin_lock_init);
#ifndef CONFIG_PREEMPT_RT
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index cf92bd6a82..e2bc6646ce 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -3383,6 +3383,10 @@ void __init numa_policy_init(void)
pr_err("%s: interleaving failed\n", __func__);
check_numabalancing_enable();
+
+#ifdef CONFIG_HQSPINLOCKS
+ hq_configure_spin_lock_slowpath();
+#endif
}
/* Reset policy of current process to default */
--
2.34.1
^ permalink raw reply related
* [RFC PATCH v3 7/7] futex: use hq-spinlock for hash buckets
From: Fedorov Nikita @ 2026-04-15 16:44 UTC (permalink / raw)
To: Catalin Marinas, Will Deacon, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, Dave Hansen, x86, hpa, Juergen Gross, Ajay Kaher,
Alexey Makhalov, bcm-kernel-feedback-list, Arnd Bergmann,
Peter Zijlstra, Boqun Feng, Waiman Long, Darren Hart,
Davidlohr Bueso, andrealmeid, Andrew Morton, David Hildenbrand,
Zi Yan, Matthew Brost, Joshua Hahn, Rakie Kim, byungchul,
Gregory Price, Ying Huang, Alistair Popple, Anatoly Stepanov
Cc: Nikita Fedorov, linux-arm-kernel, linux-kernel, virtualization,
linux-arch, linux-mm, guohanjun, wangkefeng.wang, weiyongjun1,
yusongping, leijitang, artem.kuzin, kang.sun, chenjieping3
In-Reply-To: <20260415164459.2904963-1-fedorov.nikita@h-partners.com>
Example of hq-spinlock enabled for futex hash-table bucket locks
(used in memcached testing scenario)
In the evaluated memcached workloads, this improved throughput by up to
10% on AMD EPYC 9654 and by up to 8% on Kunpeng 920, with corresponding
latency reductions.
Co-developed-by: Anatoly Stepanov <stepanov.anatoly@huawei.com>
Signed-off-by: Anatoly Stepanov <stepanov.anatoly@huawei.com>
Co-developed-by: Nikita Fedorov <fedorov.nikita@h-partners.com>
Signed-off-by: Nikita Fedorov <fedorov.nikita@h-partners.com>
---
kernel/futex/core.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/futex/core.c b/kernel/futex/core.c
index 31e83a0978..042cdc7f75 100644
--- a/kernel/futex/core.c
+++ b/kernel/futex/core.c
@@ -1521,7 +1521,7 @@ static void futex_hash_bucket_init(struct futex_hash_bucket *fhb,
#endif
atomic_set(&fhb->waiters, 0);
plist_head_init(&fhb->chain);
- spin_lock_init(&fhb->lock);
+ spin_lock_init_hq(&fhb->lock);
}
#define FH_CUSTOM 0x01
--
2.34.1
^ permalink raw reply related
* [RFC PATCH v3 3/7] hq-spinlock: add contention detection
From: Fedorov Nikita @ 2026-04-15 16:44 UTC (permalink / raw)
To: Catalin Marinas, Will Deacon, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, Dave Hansen, x86, hpa, Juergen Gross, Ajay Kaher,
Alexey Makhalov, bcm-kernel-feedback-list, Arnd Bergmann,
Peter Zijlstra, Boqun Feng, Waiman Long, Darren Hart,
Davidlohr Bueso, andrealmeid, Andrew Morton, David Hildenbrand,
Zi Yan, Matthew Brost, Joshua Hahn, Rakie Kim, byungchul,
Gregory Price, Ying Huang, Alistair Popple, Anatoly Stepanov
Cc: Nikita Fedorov, linux-arm-kernel, linux-kernel, virtualization,
linux-arch, linux-mm, guohanjun, wangkefeng.wang, weiyongjun1,
yusongping, leijitang, artem.kuzin, kang.sun, chenjieping3
In-Reply-To: <20260415164459.2904963-1-fedorov.nikita@h-partners.com>
The hierarchical slowpath is needed for locks that experience
sustained cross-node contention. Enabling it unconditionally is
undesirable for lightly contended locks and may decrease the performance.
Add a simple contention detection scheme that tracks remote handoffs
separately from overall handoff activity and enables HQ mode only when
the observed handoff pattern indicates that cross-node contention is
high enough to benefit from NUMA-aware queueing.
HQ lock type can be turned on if remote_handoffs exceeds `hqlock_remote_handoffs_turn_numa`.
Lock can be turned back into QSPINLOCK mode if in HQ mode
amount of remote handoffs will not exceed `hqlock_remote_handoffs_keep_numa`.
Remote handoffs counter will increase only if the amount of local handoffs after previous increase
is not less than `hqlock_local_handoffs_to_increase_remotes`.
Additional locktorture reruns showed no degradation
in low-contention configurations after adding contention-based switching
while maintaining practically the same performance improvement in high contention cases.
Co-developed-by: Anatoly Stepanov <stepanov.anatoly@huawei.com>
Signed-off-by: Anatoly Stepanov <stepanov.anatoly@huawei.com>
Co-developed-by: Nikita Fedorov <fedorov.nikita@h-partners.com>
Signed-off-by: Nikita Fedorov <fedorov.nikita@h-partners.com>
---
kernel/locking/hqlock_core.h | 57 +++++++++++++++++++++++++++++++++--
kernel/locking/hqlock_meta.h | 4 +++
kernel/locking/hqlock_types.h | 8 +++--
3 files changed, 65 insertions(+), 4 deletions(-)
diff --git a/kernel/locking/hqlock_core.h b/kernel/locking/hqlock_core.h
index 7322199228..e2ba09d758 100644
--- a/kernel/locking/hqlock_core.h
+++ b/kernel/locking/hqlock_core.h
@@ -450,6 +450,23 @@ static inline void hqlock_handoff(struct qspinlock *lock,
struct mcs_spinlock *next, u32 tail,
int handoff_info);
+/*
+ * In low_contention_mcs_lock_handoff we wanted to help processor optimise writes
+ * and avoid extra reading of our cpu cacheline (read our qnode->numa_node),
+ * so previous contender has saved his numa node in our prev_numa_node,
+ * and now we need to update remote_handoffs counter by ourself
+ */
+static __always_inline void update_counters_qspinlock(struct numa_qnode *qnode)
+{
+ if (qnode->numa_node != qnode->prev_numa_node) {
+ if ((qnode->general_handoffs - qnode->prev_general_handoffs)
+ > hqlock_local_handoffs_to_increase_remotes) {
+ qnode->remote_handoffs++;
+ }
+
+ qnode->prev_general_handoffs = qnode->general_handoffs;
+ }
+}
/*
* Chech if contention has risen and if we need to set NUMA-aware mode
@@ -458,8 +475,13 @@ static __always_inline bool determine_contention_qspinlock_mode(struct mcs_spinl
{
struct numa_qnode *qnode = (void *)node;
- if (qnode->general_handoffs > READ_ONCE(hqlock_general_handoffs_turn_numa))
+ unsigned long general_handoffs = (unsigned long) qnode->general_handoffs;
+ unsigned long remote_handoffs = (unsigned long) qnode->remote_handoffs;
+
+ if ((general_handoffs > hqlock_general_handoffs_turn_numa) &&
+ (remote_handoffs > hqlock_remote_handoffs_turn_numa))
return true;
+
return false;
}
@@ -485,7 +507,14 @@ static __always_inline bool low_contention_try_clear_tail(struct qspinlock *lock
else
update_val |= _Q_LOCK_INVALID_TAIL;
- return atomic_try_cmpxchg_relaxed(&lock->val, &val, update_val);
+ bool ret = atomic_try_cmpxchg_relaxed(&lock->val, &val, update_val);
+
+#ifdef CONFIG_HQSPINLOCKS_DEBUG
+ if (ret && high_contention)
+ atomic_inc(&transitions_from_qspinlock_to_hq);
+#endif
+
+ return ret;
}
static __always_inline void low_contention_mcs_lock_handoff(struct mcs_spinlock *node,
@@ -502,6 +531,17 @@ static __always_inline void low_contention_mcs_lock_handoff(struct mcs_spinlock
general_handoffs++;
qnext->general_handoffs = general_handoffs;
+ qnext->remote_handoffs = qnode->remote_handoffs;
+ qnext->prev_general_handoffs = qnode->prev_general_handoffs;
+
+ /*
+ * Show next contender our numa node and assume
+ * he will update remote_handoffs counter in update_counters_qspinlock by himself
+ * instead of reading his numa_node and updating remote_handoffs here
+ * to avoid extra cacheline transferring and help processor optimise several writes here
+ */
+ qnext->prev_numa_node = qnode->numa_node;
+
arch_mcs_spin_unlock_contended(&next->locked);
}
@@ -557,6 +597,10 @@ static inline void hqlock_init_node(struct mcs_spinlock *node)
qnode->numa_node = numa_node_id() + 1;
qnode->lock_id = 0;
qnode->wrong_fallback_tail = 0;
+
+ qnode->remote_handoffs = 0;
+ qnode->prev_numa_node = 0;
+ qnode->prev_general_handoffs = 0;
}
static inline void reset_handoff_counter(struct numa_qnode *qnode)
@@ -580,6 +624,8 @@ static inline void handoff_local(struct mcs_spinlock *node,
qnext->general_handoffs = general_handoffs;
+ qnext->remote_handoffs = qnode->remote_handoffs;
+
u16 wrong_fallback_tail = qnode->wrong_fallback_tail;
if (wrong_fallback_tail != 0 && wrong_fallback_tail != (tail >> _Q_TAIL_OFFSET)) {
@@ -641,6 +687,13 @@ static inline void handoff_remote(struct qspinlock *lock,
mcs_head = (void *) qhead;
+ u16 remote_handoffs = qnode->remote_handoffs;
+
+ if (qnode->general_handoffs > hqlock_local_handoffs_to_increase_remotes)
+ remote_handoffs++;
+
+ qhead->remote_handoffs = remote_handoffs;
+
/* arch_mcs_spin_unlock_contended implies smp-barrier */
arch_mcs_spin_unlock_contended(&mcs_head->locked);
}
diff --git a/kernel/locking/hqlock_meta.h b/kernel/locking/hqlock_meta.h
index 5b54801326..561d5a5fd0 100644
--- a/kernel/locking/hqlock_meta.h
+++ b/kernel/locking/hqlock_meta.h
@@ -307,6 +307,10 @@ static inline void release_lock_meta(struct qspinlock *lock,
goto do_rollback;
}
+ if (qnode->remote_handoffs < hqlock_remote_handoffs_keep_numa) {
+ upd_val |= _Q_LOCK_MODE_QSPINLOCK_VAL;
+ }
+
/*
* We need wait until pending is gone.
* Otherwise, clearing pending can erase a mode we will set here
diff --git a/kernel/locking/hqlock_types.h b/kernel/locking/hqlock_types.h
index 32d06f2755..40061f11a1 100644
--- a/kernel/locking/hqlock_types.h
+++ b/kernel/locking/hqlock_types.h
@@ -37,9 +37,13 @@ struct numa_qnode {
u16 lock_id;
u16 wrong_fallback_tail;
- u16 general_handoffs;
-
u16 numa_node;
+
+
+ u16 general_handoffs;
+ u16 remote_handoffs;
+ u16 prev_general_handoffs;
+ u16 prev_numa_node;
};
struct numa_queue {
--
2.34.1
^ permalink raw reply related
* [RFC PATCH v3 6/7] lockref: use hq-spinlock
From: Fedorov Nikita @ 2026-04-15 16:44 UTC (permalink / raw)
To: Catalin Marinas, Will Deacon, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, Dave Hansen, x86, hpa, Juergen Gross, Ajay Kaher,
Alexey Makhalov, bcm-kernel-feedback-list, Arnd Bergmann,
Peter Zijlstra, Boqun Feng, Waiman Long, Darren Hart,
Davidlohr Bueso, andrealmeid, Andrew Morton, David Hildenbrand,
Zi Yan, Matthew Brost, Joshua Hahn, Rakie Kim, byungchul,
Gregory Price, Ying Huang, Alistair Popple, Anatoly Stepanov
Cc: Nikita Fedorov, linux-arm-kernel, linux-kernel, virtualization,
linux-arch, linux-mm, guohanjun, wangkefeng.wang, weiyongjun1,
yusongping, leijitang, artem.kuzin, kang.sun, chenjieping3
In-Reply-To: <20260415164459.2904963-1-fedorov.nikita@h-partners.com>
Example of hq-spinlock enabled for dentry->lockref spinlock
(used in nginx testing scenario)
In the evaluated nginx single-file workload on Kunpeng 920, throughput
gains reached 68-78% at 64-96 workers.
Co-developed-by: Anatoly Stepanov <stepanov.anatoly@huawei.com>
Signed-off-by: Anatoly Stepanov <stepanov.anatoly@huawei.com>
Co-developed-by: Nikita Fedorov <fedorov.nikita@h-partners.com>
Signed-off-by: Nikita Fedorov <fedorov.nikita@h-partners.com>
---
include/linux/lockref.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/include/linux/lockref.h b/include/linux/lockref.h
index 6ded24cdb4..19a1c3823c 100644
--- a/include/linux/lockref.h
+++ b/include/linux/lockref.h
@@ -42,7 +42,7 @@ struct lockref {
*/
static inline void lockref_init(struct lockref *lockref)
{
- spin_lock_init(&lockref->lock);
+ spin_lock_init_hq(&lockref->lock);
lockref->count = 1;
}
--
2.34.1
^ permalink raw reply related
* [RFC PATCH v3 1/7] kernel: add hq-spinlock types
From: Fedorov Nikita @ 2026-04-15 16:44 UTC (permalink / raw)
To: Catalin Marinas, Will Deacon, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, Dave Hansen, x86, hpa, Juergen Gross, Ajay Kaher,
Alexey Makhalov, bcm-kernel-feedback-list, Arnd Bergmann,
Peter Zijlstra, Boqun Feng, Waiman Long, Darren Hart,
Davidlohr Bueso, andrealmeid, Andrew Morton, David Hildenbrand,
Zi Yan, Matthew Brost, Joshua Hahn, Rakie Kim, byungchul,
Gregory Price, Ying Huang, Alistair Popple, Anatoly Stepanov
Cc: Nikita Fedorov, linux-arm-kernel, linux-kernel, virtualization,
linux-arch, linux-mm, guohanjun, wangkefeng.wang, weiyongjun1,
yusongping, leijitang, artem.kuzin, kang.sun, chenjieping3
In-Reply-To: <20260415164459.2904963-1-fedorov.nikita@h-partners.com>
Introduce the type definitions used by Hierarchical Queued spinlocks.
Add the HQ-specific types definitions and extend qspinlock so
that a lock can be marked as using the HQ slowpath
while preserving the existing qspinlock layout
Co-developed-by: Anatoly Stepanov <stepanov.anatoly@huawei.com>
Signed-off-by: Anatoly Stepanov <stepanov.anatoly@huawei.com>
Co-developed-by: Nikita Fedorov <fedorov.nikita@h-partners.com>
Signed-off-by: Nikita Fedorov <fedorov.nikita@h-partners.com>
---
include/asm-generic/qspinlock_types.h | 44 ++++++++--
kernel/locking/hqlock_types.h | 118 ++++++++++++++++++++++++++
2 files changed, 157 insertions(+), 5 deletions(-)
create mode 100644 kernel/locking/hqlock_types.h
diff --git a/include/asm-generic/qspinlock_types.h b/include/asm-generic/qspinlock_types.h
index 2fd1fb89ec..a97387ae48 100644
--- a/include/asm-generic/qspinlock_types.h
+++ b/include/asm-generic/qspinlock_types.h
@@ -43,11 +43,6 @@ typedef struct qspinlock {
};
} arch_spinlock_t;
-/*
- * Initializier
- */
-#define __ARCH_SPIN_LOCK_UNLOCKED { { .val = ATOMIC_INIT(0) } }
-
/*
* Bitfields in the atomic value:
*
@@ -76,6 +71,26 @@ typedef struct qspinlock {
#else
#define _Q_PENDING_BITS 1
#endif
+
+#ifdef CONFIG_HQSPINLOCKS
+/* For locks with HQ-mode we always use single pending bit */
+#define _Q_PENDING_HQLOCK_BITS 1
+#define _Q_PENDING_HQLOCK_OFFSET (_Q_LOCKED_OFFSET + _Q_LOCKED_BITS)
+#define _Q_PENDING_HQLOCK_MASK _Q_SET_MASK(PENDING_HQLOCK)
+
+#define _Q_LOCKTYPE_OFFSET (_Q_PENDING_HQLOCK_OFFSET + _Q_PENDING_HQLOCK_BITS)
+#define _Q_LOCKTYPE_BITS 1
+#define _Q_LOCKTYPE_MASK _Q_SET_MASK(LOCKTYPE)
+#define _Q_LOCKTYPE_HQ (1U << _Q_LOCKTYPE_OFFSET)
+
+#define _Q_LOCK_MODE_OFFSET (_Q_LOCKTYPE_OFFSET + _Q_LOCKTYPE_BITS)
+#define _Q_LOCK_MODE_BITS 2
+#define _Q_LOCK_MODE_MASK _Q_SET_MASK(LOCK_MODE)
+#define _Q_LOCK_MODE_QSPINLOCK_VAL (1U << _Q_LOCK_MODE_OFFSET)
+
+#define _Q_LOCK_TYPE_MODE_MASK (_Q_LOCKTYPE_MASK | _Q_LOCK_MODE_MASK)
+#endif //CONFIG_HQSPINLOCKS
+
#define _Q_PENDING_MASK _Q_SET_MASK(PENDING)
#define _Q_TAIL_IDX_OFFSET (_Q_PENDING_OFFSET + _Q_PENDING_BITS)
@@ -92,4 +107,23 @@ typedef struct qspinlock {
#define _Q_LOCKED_VAL (1U << _Q_LOCKED_OFFSET)
#define _Q_PENDING_VAL (1U << _Q_PENDING_OFFSET)
+#ifdef CONFIG_HQSPINLOCKS
+#define _Q_LOCK_INVALID_TAIL (_Q_TAIL_IDX_MASK)
+
+#define _Q_SERVICE_MASK (_Q_LOCKTYPE_MASK | _Q_LOCK_MODE_QSPINLOCK_VAL | _Q_LOCK_INVALID_TAIL)
+#else // CONFIG_HQSPINLOCKS
+#define _Q_SERVICE_MASK 0
+#endif
+
+/*
+ * Initializier
+ */
+#define __ARCH_SPIN_LOCK_UNLOCKED { { .val = ATOMIC_INIT(0) } }
+
+#ifdef CONFIG_HQSPINLOCKS
+#define __ARCH_SPIN_LOCK_UNLOCKED_HQ { { .val = ATOMIC_INIT(_Q_LOCKTYPE_HQ | _Q_LOCK_MODE_QSPINLOCK_VAL) } }
+#else
+#define __ARCH_SPIN_LOCK_UNLOCKED_HQ { { .val = ATOMIC_INIT(0) } }
+#endif
+
#endif /* __ASM_GENERIC_QSPINLOCK_TYPES_H */
diff --git a/kernel/locking/hqlock_types.h b/kernel/locking/hqlock_types.h
new file mode 100644
index 0000000000..32d06f2755
--- /dev/null
+++ b/kernel/locking/hqlock_types.h
@@ -0,0 +1,118 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _GEN_HQ_SPINLOCK_SLOWPATH
+#error "Do not include this file!"
+#endif
+
+#define IRQ_NODE (MAX_NUMNODES + 1)
+#define Q_NEW_NODE_QUEUE 1
+#define LOCK_ID_BITS (12)
+
+#define LOCK_ID_MAX (1 << LOCK_ID_BITS)
+#define LOCK_ID_NONE (LOCK_ID_MAX + 1)
+
+#define _QUEUE_TAIL_MASK (((1ULL << 32) - 1) << 0)
+#define _QUEUE_SEQ_COUNTER_MASK (((1ULL << 32) - 1) << 32)
+
+/*
+ * Output code for handoff-logic:
+ *
+ * == 0 (HQLOCK_HANDOFF_LOCAL) - has local nodes to handoff
+ * > 0 - has remote node to handoff, id is visible
+ * == -1 (HQLOCK_HANDOFF_REMOTE_HEAD) - has remote node to handoff,
+ * id isn't visible yet, will be in the *lock_meta->head_node*
+ */
+enum {
+ HQLOCK_HANDOFF_LOCAL = 0,
+ HQLOCK_HANDOFF_REMOTE_HEAD = -1
+};
+
+typedef enum {
+ LOCK_NO_MODE = 0,
+ LOCK_MODE_QSPINLOCK = 1,
+ LOCK_MODE_HQLOCK = 2,
+} hqlock_mode_t;
+
+struct numa_qnode {
+ struct mcs_spinlock mcs;
+
+ u16 lock_id;
+ u16 wrong_fallback_tail;
+ u16 general_handoffs;
+
+ u16 numa_node;
+};
+
+struct numa_queue {
+ struct numa_qnode *head;
+ union {
+ u64 seq_counter_tail;
+ struct {
+ u32 tail;
+ u32 seq_counter;
+ };
+ };
+
+ u16 next_node;
+ u16 prev_node;
+
+ u16 handoffs_not_head;
+} ____cacheline_aligned;
+
+/**
+ * Lock metadata
+ * "allocated"/"freed" on demand.
+ *
+ * Used to dynamically bind numa_queue to a lock,
+ * maintain FIFO-order for the NUMA-queues.
+ *
+ * seq_counter is needed to distinguish metadata usage
+ * by different locks, preventing local contenders
+ * from queueing in the wrong per-NUMA queue
+ *
+ * @see set_bucket
+ * @see numa_xchg_tail
+ */
+struct lock_metadata {
+ atomic_t seq_counter;
+ struct qspinlock *lock_ptr;
+
+ /* NUMA-queues of contenders ae kept in FIFO order */
+ union {
+ u32 nodes_tail;
+ struct {
+ u16 tail_node;
+ u16 head_node;
+ };
+ };
+};
+
+static inline int decode_lock_mode(u32 lock_val)
+{
+ return (lock_val & _Q_LOCK_MODE_MASK) >> _Q_LOCK_MODE_OFFSET;
+}
+
+static inline u32 encode_lock_mode(u16 lock_id)
+{
+ if (lock_id == LOCK_ID_NONE)
+ return LOCK_MODE_QSPINLOCK << _Q_LOCK_MODE_OFFSET;
+
+ return LOCK_MODE_HQLOCK << _Q_LOCK_MODE_OFFSET;
+}
+
+static inline u64 encode_tc(u32 tail, u32 counter)
+{
+ u64 __tail = (u64)tail;
+ u64 __counter = (u64)counter;
+
+ return __tail | (__counter << 32);
+}
+
+static inline u32 decode_tc_tail(u64 tail_counter)
+{
+ return (u32)(tail_counter & _QUEUE_TAIL_MASK);
+}
+
+static inline u32 decode_tc_counter(u64 tail_counter)
+{
+ return (u32)((tail_counter & _QUEUE_SEQ_COUNTER_MASK) >> 32);
+}
--
2.34.1
^ permalink raw reply related
* [RFC PATCH v3 0/7]
From: Fedorov Nikita @ 2026-04-15 16:44 UTC (permalink / raw)
To: Catalin Marinas, Will Deacon, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, Dave Hansen, x86, hpa, Juergen Gross, Ajay Kaher,
Alexey Makhalov, bcm-kernel-feedback-list, Arnd Bergmann,
Peter Zijlstra, Boqun Feng, Waiman Long, Darren Hart,
Davidlohr Bueso, andrealmeid, Andrew Morton, David Hildenbrand,
Zi Yan, Matthew Brost, Joshua Hahn, Rakie Kim, byungchul,
Gregory Price, Ying Huang, Alistair Popple, Anatoly Stepanov
Cc: Nikita Fedorov, linux-arm-kernel, linux-kernel, virtualization,
linux-arch, linux-mm, guohanjun, wangkefeng.wang, weiyongjun1,
yusongping, leijitang, artem.kuzin, kang.sun, chenjieping3
Changes since v2:
- split the patch into a smaller patch series,
though it is still hard to split the hqlock internal logic itself
- remove some unused code
- rebased onto Linux v7.0
- allocate hq-spinlock metadata with kvmalloc instead of memblock
- added contention detection and verified we have no performance degradation in low contention scenario
[Motivation]
In a high contention case, existing Linux kernel spinlock implementations can become
inefficient on modern NUMA systems due to frequent and expensive
cross-NUMA cacheline transfers.
This might happen due to following reasons:
- on "contender enqueue" each lock contender updates a shared lock structure
- on "MCS handoff" cross-NUMA cache-line transfer occurs when
two contenders are from different NUMA nodes.
Previous work regarding NUMA-aware spinlock in Linux kernel is CNA-lock:
https://lore.kernel.org/lkml/20210514200743.3026725-1-alex.kogan@oracle.com/
It reduces cross-NUMA cacheline traffic during handoff, but does not reduce it during enqueuing.
CNA design also requires the first contender to do additional work during global spinning
and keeps threads from all nodes other than the first one in the single secondary queue.
In our measurements, we only saw benefits from using it on Kunpeng;
on x86 platforms, CNA behaved the same as a regular qspinlock.
Thus, there is still quite a lot of potential for optimization.
HQ-lock has completely different design concept: kind of cohort-lock and
queued-spinlock hybrid.
If someone wants to try the HQ-lock in some subsystem, just
change lock initialization code from `spin_lock_init()` to `spin_lock_init_hq()`,
or change `DEFINE_SPINLOCK()` macro to `DEFINE_SPINLOCK_HQ()` if the lock is static.
The dedicated bit in the lock structure is used to distiguish between the two lock types.
[Performance measurements]
Performance measurements were done on x86 (AMD EPYC) and arm64 (Kunpeng 920)
platforms with the following scenarious:
- Locktorture benchmark
- Memcached + memtier benchmark
- Ngnix + Wrk benchmark
[Locktorture]
NPS stands for "Nodes per socket"
+------------------------------+-----------------------+-------+-------+--------+
| AMD EPYC 9654 |
+------------------------------+-----------------------+-------+-------+--------+
| 192 cores (x2 hyper-threads) | | | | |
| 2 sockets | | | | |
| Locktorture 60 sec. | NUMA nodes per-socket | | | |
| Average gain (single lock) | 1 NPS | 2 NPS | 4 NPS | 12 NPS |
| Total contender threads | | | | |
| 8 | 19% | 21% | 12% | 12% |
| 16 | 13% | 18% | 34% | 75% |
| 32 | 8% | 14% | 25% | 112% |
| 64 | 11% | 12% | 30% | 152% |
| 128 | 9% | 17% | 37% | 163% |
| 256 | 2% | 16% | 40% | 168% |
| 384 | -1% | 14% | 44% | 186% |
+------------------------------+-----------------------+-------+-------+--------+
+-----------------+-------+-------+-------+--------+
| Fairness factor | 1 NPS | 2 NPS | 4 NPS | 12 NPS |
+-----------------+-------+-------+-------+--------+
| 8 | 0.54 | 0.57 | 0.57 | 0.55 |
| 16 | 0.52 | 0.53 | 0.60 | 0.58 |
| 32 | 0.53 | 0.53 | 0.53 | 0.61 |
| 64 | 0.52 | 0.56 | 0.54 | 0.56 |
| 128 | 0.51 | 0.54 | 0.54 | 0.53 |
| 256 | 0.52 | 0.52 | 0.52 | 0.52 |
| 384 | 0.51 | 0.51 | 0.51 | 0.51 |
+-----------------+-------+-------+-------+--------+
+-------------------------+--------------+
| Kunpeng 920 (arm64) | |
+-------------------------+--------------+
| 96 cores (no MT) | |
| 2 sockets, 4 NUMA nodes | |
| Locktorture 60 sec. | |
| | |
| Total contender threads | Average gain |
| 8 | 93% |
| 16 | 142% |
| 32 | 129% |
| 64 | 152% |
| 96 | 158% |
+-------------------------+--------------+
[Memcached]
+---------------------------------+-----------------+-------------------+
| AMD EPYC 9654 | | |
+---------------------------------+-----------------+-------------------+
| 192 cores (x2 hyper-threads) | | |
| 2 sockets, NPS=4 | | |
| | | |
| Memtier+memcached 1:1 R/W ratio | | |
| Workers | Throughput gain | Latency reduction |
| 32 | 1% | -1% |
| 64 | 1% | -1% |
| 128 | 3% | -4% |
| 256 | 7% | -6% |
| 384 | 10% | -8% |
+---------------------------------+-----------------+-------------------+
+---------------------------------+-----------------+-------------------+
| Kunpeng 920 (arm64) | | |
+---------------------------------+-----------------+-------------------+
| 96 cores (no MT) | | |
| 2 sockets, 4 NUMA nodes | | |
| | | |
| Memtier+memcached 1:1 R/W ratio | | |
| Workers | Throughput gain | Latency reduction |
| 32 | 4% | -3% |
| 64 | 6% | -6% |
| 80 | 8% | -7% |
| 96 | 8% | -8% |
+---------------------------------+-----------------+-------------------+
[Nginx]
+-----------------------------------------------------------------------+-----------------+
| Kunpeng 920 (arm64) | |
+-----------------------------------------------------------------------+-----------------+
| 96 cores (no MT) | |
| 2 sockets, 4 NUMA nodes | |
| | |
| Nginx + WRK benchmark, single file (lockref spinlock contention case) | |
| Workers | Throughput gain |
| 32 | 1% |
| 64 | 68% |
| 80 | 72% |
| 96 | 78% |
+-----------------------------------------------------------------------+-----------------+
Despite, the test is a single-file test, it can be related to real-life cases, when some
html-pages are accessed much more frequently than others (index.html, etc.)
[Low contention remarks]
After adding contention detection scheme, we do not see performance degradation in low contention scenario (< 8 threads),
throughput of HQspinlock is equal to qspinlock,
while still having practically the same improvement in high contention case as mentioned above.
Previous version:
https://lore.kernel.org/lkml/20251206062106.2109014-1-stepanov.anatoly@huawei.com/
Anatoly Stepanov (7):
kernel: add hq-spinlock types
hq-spinlock: implement inner logic
hq-spinlock: add contention detection
hq-spinlock: add hq-spinlock tunables and debug statistics
kernel: introduce general hq-spinlock support
lockref: use hq-spinlock
futex: use hq-spinlock for hash buckets
arch/arm64/include/asm/qspinlock.h | 37 +
arch/x86/include/asm/hq-spinlock.h | 34 +
arch/x86/include/asm/paravirt-spinlock.h | 3 +-
arch/x86/include/asm/qspinlock.h | 6 +-
include/asm-generic/qspinlock.h | 23 +-
include/asm-generic/qspinlock_types.h | 44 +-
include/linux/lockref.h | 2 +-
include/linux/spinlock.h | 26 +
include/linux/spinlock_types.h | 26 +
include/linux/spinlock_types_raw.h | 20 +
kernel/Kconfig.locks | 29 +
kernel/futex/core.c | 2 +-
kernel/locking/hqlock_core.h | 850 +++++++++++++++++++++++
kernel/locking/hqlock_meta.h | 487 +++++++++++++
kernel/locking/hqlock_proc.h | 164 +++++
kernel/locking/hqlock_types.h | 122 ++++
kernel/locking/qspinlock.c | 65 +-
kernel/locking/qspinlock.h | 4 +-
kernel/locking/spinlock_debug.c | 20 +
mm/mempolicy.c | 4 +
20 files changed, 1939 insertions(+), 29 deletions(-)
create mode 100644 arch/arm64/include/asm/qspinlock.h
create mode 100644 arch/x86/include/asm/hq-spinlock.h
create mode 100644 kernel/locking/hqlock_core.h
create mode 100644 kernel/locking/hqlock_meta.h
create mode 100644 kernel/locking/hqlock_proc.h
create mode 100644 kernel/locking/hqlock_types.h
--
2.34.1
^ permalink raw reply
* [RFC PATCH v3 4/7] hq-spinlock: add hq-spinlock tunables and debug statistics
From: Fedorov Nikita @ 2026-04-15 16:44 UTC (permalink / raw)
To: Catalin Marinas, Will Deacon, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, Dave Hansen, x86, hpa, Juergen Gross, Ajay Kaher,
Alexey Makhalov, bcm-kernel-feedback-list, Arnd Bergmann,
Peter Zijlstra, Boqun Feng, Waiman Long, Darren Hart,
Davidlohr Bueso, andrealmeid, Andrew Morton, David Hildenbrand,
Zi Yan, Matthew Brost, Joshua Hahn, Rakie Kim, byungchul,
Gregory Price, Ying Huang, Alistair Popple, Anatoly Stepanov
Cc: Nikita Fedorov, linux-arm-kernel, linux-kernel, virtualization,
linux-arch, linux-mm, guohanjun, wangkefeng.wang, weiyongjun1,
yusongping, leijitang, artem.kuzin, kang.sun, chenjieping3
In-Reply-To: <20260415164459.2904963-1-fedorov.nikita@h-partners.com>
The HQ slowpath and contention-based mode switching depend on several
parameters that affect when NUMA-aware mode becomes active and how long
local handoff can continue. Expose these parameters through procfs so
that the behaviour can be inspected and tuned without rebuilding the
kernel.
Also add debug statistics that make it possible to observe HQ lock
activation, handoff behaviour, and mode switching decisions during
testing and evaluation.
These controls are intended to simplify validation and analysis of HQ
lock behaviour on different systems and workloads.
Co-developed-by: Anatoly Stepanov <stepanov.anatoly@huawei.com>
Signed-off-by: Anatoly Stepanov <stepanov.anatoly@huawei.com>
Co-developed-by: Nikita Fedorov <fedorov.nikita@h-partners.com>
Signed-off-by: Nikita Fedorov <fedorov.nikita@h-partners.com>
---
kernel/locking/hqlock_core.h | 5 ++
kernel/locking/hqlock_meta.h | 16 ++++
kernel/locking/hqlock_proc.h | 164 +++++++++++++++++++++++++++++++++++
3 files changed, 185 insertions(+)
create mode 100644 kernel/locking/hqlock_proc.h
diff --git a/kernel/locking/hqlock_core.h b/kernel/locking/hqlock_core.h
index e2ba09d758..b7681915b4 100644
--- a/kernel/locking/hqlock_core.h
+++ b/kernel/locking/hqlock_core.h
@@ -530,6 +530,11 @@ static __always_inline void low_contention_mcs_lock_handoff(struct mcs_spinlock
if (next != prev && likely(general_handoffs + 1 != max_u16))
general_handoffs++;
+#ifdef CONFIG_HQSPINLOCKS_DEBUG
+ if (READ_ONCE(max_general_handoffs) < general_handoffs)
+ WRITE_ONCE(max_general_handoffs, general_handoffs);
+#endif
+
qnext->general_handoffs = general_handoffs;
qnext->remote_handoffs = qnode->remote_handoffs;
qnext->prev_general_handoffs = qnode->prev_general_handoffs;
diff --git a/kernel/locking/hqlock_meta.h b/kernel/locking/hqlock_meta.h
index 561d5a5fd0..1c69df536b 100644
--- a/kernel/locking/hqlock_meta.h
+++ b/kernel/locking/hqlock_meta.h
@@ -124,6 +124,12 @@ static inline enum meta_status grab_lock_meta(struct qspinlock *lock, u32 lock_i
}
*seq = seq_counter;
+#ifdef CONFIG_HQSPINLOCKS_DEBUG
+ int current_used = atomic_inc_return_relaxed(&cur_buckets_in_use);
+
+ if (READ_ONCE(max_buckets_in_use) < current_used)
+ WRITE_ONCE(max_buckets_in_use, current_used);
+#endif
return META_GRABBED;
}
@@ -252,6 +258,9 @@ hqlock_mode_t setup_lock_mode(struct qspinlock *lock, u16 lock_id, u32 *meta_seq
*/
if (status == META_GRABBED && mode != LOCK_MODE_HQLOCK) {
smp_store_release(&meta_pool[lock_id].lock_ptr, NULL);
+#ifdef CONFIG_HQSPINLOCKS_DEBUG
+ atomic_dec(&cur_buckets_in_use);
+#endif
}
} while (mode == LOCK_NO_MODE);
@@ -307,8 +316,15 @@ static inline void release_lock_meta(struct qspinlock *lock,
goto do_rollback;
}
+#ifdef CONFIG_HQSPINLOCKS_DEBUG
+ atomic_dec(&cur_buckets_in_use);
+#endif
+
if (qnode->remote_handoffs < hqlock_remote_handoffs_keep_numa) {
upd_val |= _Q_LOCK_MODE_QSPINLOCK_VAL;
+#ifdef CONFIG_HQSPINLOCKS_DEBUG
+ atomic_inc(&transitions_from_hq_to_qspinlock);
+#endif
}
/*
diff --git a/kernel/locking/hqlock_proc.h b/kernel/locking/hqlock_proc.h
new file mode 100644
index 0000000000..ea68635851
--- /dev/null
+++ b/kernel/locking/hqlock_proc.h
@@ -0,0 +1,164 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _GEN_HQ_SPINLOCK_SLOWPATH
+#error "Do not include this file!"
+#endif
+
+#include <linux/sysctl.h>
+
+/*
+ * Local handoffs threshold to maintain global fairness,
+ * perform remote handoff if it's reached
+ */
+unsigned long hqlock_fairness_threshold = 1000;
+
+/*
+ * Minimal amount of handoffs in LOCK_MODE_QSPINLOCK
+ * to enable NUMA-awareness
+ */
+unsigned long hqlock_general_handoffs_turn_numa = 50;
+
+/*
+ * Minimal amount of remote handoffs in LOCK_MODE_QSPINLOCK
+ * to enable NUMA-awareness.
+ *
+ * counter is increased if local handoffs >= hqlock_local_handoffs_to_increase_remotes
+ */
+unsigned long hqlock_remote_handoffs_turn_numa = 2;
+
+/*
+ * How many remote handoffs are needed
+ * to keep NUMA-awareness on
+ */
+unsigned long hqlock_remote_handoffs_keep_numa = 1;
+
+/*
+ * How many local handoffs are needed
+ * to increase remote handoffs counter.
+ *
+ * That is needed to avoid using LOCK_MODE_HQLOCK mode
+ * with 1-2 threads from several NUMA nodes,
+ * in this case HQlock will give more overhead then benefit
+ */
+unsigned long hqlock_local_handoffs_to_increase_remotes = 2;
+
+unsigned long hqlock_probability_of_force_stay_numa = 5000;
+
+static unsigned long long_zero;
+static unsigned long long_max = LONG_MAX;
+static unsigned long long_hundred_percent = 10000;
+
+static const struct ctl_table hqlock_settings[] = {
+ {
+ .procname = "hqlock_fairness_threshold",
+ .data = &hqlock_fairness_threshold,
+ .maxlen = sizeof(hqlock_fairness_threshold),
+ .mode = 0644,
+ .proc_handler = proc_doulongvec_minmax
+ },
+ {
+ .procname = "hqlock_general_handoffs_turn_numa",
+ .data = &hqlock_general_handoffs_turn_numa,
+ .maxlen = sizeof(hqlock_general_handoffs_turn_numa),
+ .mode = 0644,
+ .proc_handler = proc_doulongvec_minmax,
+ .extra1 = &long_zero,
+ .extra2 = &long_max,
+ },
+ {
+ .procname = "hqlock_probability_of_force_stay_numa",
+ .data = &hqlock_probability_of_force_stay_numa,
+ .maxlen = sizeof(hqlock_probability_of_force_stay_numa),
+ .mode = 0644,
+ .proc_handler = proc_doulongvec_minmax,
+ .extra1 = &long_zero,
+ .extra2 = &long_hundred_percent,
+ },
+ {
+ .procname = "hqlock_remote_handoffs_turn_numa",
+ .data = &hqlock_remote_handoffs_turn_numa,
+ .maxlen = sizeof(hqlock_remote_handoffs_turn_numa),
+ .mode = 0644,
+ .proc_handler = proc_doulongvec_minmax,
+ .extra1 = &long_zero,
+ .extra2 = &long_max,
+ },
+ {
+ .procname = "hqlock_remote_handoffs_keep_numa",
+ .data = &hqlock_remote_handoffs_keep_numa,
+ .maxlen = sizeof(hqlock_remote_handoffs_keep_numa),
+ .mode = 0644,
+ .proc_handler = proc_doulongvec_minmax,
+ .extra1 = &long_zero,
+ .extra2 = &long_max,
+ },
+ {
+ .procname = "hqlock_local_handoffs_to_increase_remotes",
+ .data = &hqlock_local_handoffs_to_increase_remotes,
+ .maxlen = sizeof(hqlock_local_handoffs_to_increase_remotes),
+ .mode = 0644,
+ .proc_handler = proc_doulongvec_minmax,
+ .extra1 = &long_zero,
+ .extra2 = &long_max,
+ },
+};
+static int __init init_numa_spinlock_sysctl(void)
+{
+ if (!register_sysctl("kernel", hqlock_settings))
+ return -EINVAL;
+ return 0;
+}
+core_initcall(init_numa_spinlock_sysctl);
+
+
+#ifdef CONFIG_HQSPINLOCKS_DEBUG
+static int max_buckets_in_use;
+static int max_general_handoffs;
+static atomic_t cur_buckets_in_use = ATOMIC_INIT(0);
+
+static atomic_t transitions_from_qspinlock_to_hq = ATOMIC_INIT(0);
+static atomic_t transitions_from_hq_to_qspinlock = ATOMIC_INIT(0);
+
+
+static int print_hqlock_stats(struct seq_file *file, void *v)
+{
+ seq_printf(file, "Max dynamic metada in use after previous print: %d\n",
+ READ_ONCE(max_buckets_in_use));
+ WRITE_ONCE(max_buckets_in_use, 0);
+
+ seq_printf(file, "Currently in use: %d\n",
+ atomic_read(&cur_buckets_in_use));
+
+ seq_printf(file, "Max MCS handoffs after previous print: %d\n",
+ READ_ONCE(max_general_handoffs));
+ WRITE_ONCE(max_general_handoffs, 0);
+
+ seq_printf(file, "Transitions from qspinlock to HQ mode after previous print: %d\n",
+ atomic_xchg_relaxed(&transitions_from_qspinlock_to_hq, 0));
+
+ seq_printf(file, "Transitions from HQ to qspinlock mode after previous print: %d\n",
+ atomic_xchg_relaxed(&transitions_from_hq_to_qspinlock, 0));
+
+ return 0;
+}
+
+
+static int stats_open(struct inode *inode, struct file *file)
+{
+ return single_open(file, print_hqlock_stats, NULL);
+}
+
+static const struct proc_ops stats_ops = {
+ .proc_open = stats_open,
+ .proc_read = seq_read,
+ .proc_lseek = seq_lseek,
+};
+
+static int __init stats_init(void)
+{
+ proc_create("hqlock_stats", 0444, NULL, &stats_ops);
+ return 0;
+}
+
+core_initcall(stats_init);
+
+#endif // HQSPINLOCKS_DEBUG
--
2.34.1
^ permalink raw reply related
* Re: [PATCH net v2] net: airoha: Add missing bits in airoha_qdma_cleanup_tx_queue()
From: Simon Horman @ 2026-04-15 16:46 UTC (permalink / raw)
To: Lorenzo Bianconi
Cc: Andrew Lunn, David S. Miller, Eric Dumazet, Jakub Kicinski,
Paolo Abeni, linux-arm-kernel, linux-mediatek, netdev
In-Reply-To: <20260414-airoha_qdma_cleanup_tx_queue-fix-net-v2-1-875de57cc022@kernel.org>
On Tue, Apr 14, 2026 at 08:50:52AM +0200, Lorenzo Bianconi wrote:
> Similar to airoha_qdma_cleanup_rx_queue(), reset DMA TX descriptors in
> airoha_qdma_cleanup_tx_queue routine. Moreover, reset TX_DMA_IDX to
> TX_CPU_IDX to notify the NIC the QDMA TX ring is empty.
>
> Fixes: 23020f0493270 ("net: airoha: Introduce ethernet support for EN7581 SoC")
> Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
> ---
> Changes in v2:
> - Move q->ndesc initialization at end of airoha_qdma_init_tx routine in
> order to avoid any possible NULL pointer dereference in
> airoha_qdma_cleanup_tx_queue()
This seems to be a separate issue.
If so, I think it should be split out into a separate patch.
> - Check if q->tx_list is empty in airoha_qdma_cleanup_tx_queue()
> - Link to v1: https://lore.kernel.org/r/20260410-airoha_qdma_cleanup_tx_queue-fix-net-v1-1-b7171c8f1e78@kernel.org
I think it was covered in the review Jakub forwarded for v1. But FTR,
Sashiko has some feedback on this patch in the form of an existing bug
(that should almost certainly be handled separately from this patch).
> ---
> drivers/net/ethernet/airoha/airoha_eth.c | 41 ++++++++++++++++++++++++++------
> 1 file changed, 34 insertions(+), 7 deletions(-)
>
> diff --git a/drivers/net/ethernet/airoha/airoha_eth.c b/drivers/net/ethernet/airoha/airoha_eth.c
> index 9e995094c32a..3c1a2bc68c42 100644
> --- a/drivers/net/ethernet/airoha/airoha_eth.c
> +++ b/drivers/net/ethernet/airoha/airoha_eth.c
> @@ -966,27 +966,27 @@ static int airoha_qdma_init_tx_queue(struct airoha_queue *q,
> dma_addr_t dma_addr;
>
> spin_lock_init(&q->lock);
> - q->ndesc = size;
> q->qdma = qdma;
> q->free_thr = 1 + MAX_SKB_FRAGS;
> INIT_LIST_HEAD(&q->tx_list);
>
> - q->entry = devm_kzalloc(eth->dev, q->ndesc * sizeof(*q->entry),
> + q->entry = devm_kzalloc(eth->dev, size * sizeof(*q->entry),
> GFP_KERNEL);
> if (!q->entry)
> return -ENOMEM;
>
> - q->desc = dmam_alloc_coherent(eth->dev, q->ndesc * sizeof(*q->desc),
> + q->desc = dmam_alloc_coherent(eth->dev, size * sizeof(*q->desc),
> &dma_addr, GFP_KERNEL);
> if (!q->desc)
> return -ENOMEM;
>
> - for (i = 0; i < q->ndesc; i++) {
> + for (i = 0; i < size; i++) {
> u32 val = FIELD_PREP(QDMA_DESC_DONE_MASK, 1);
>
> list_add_tail(&q->entry[i].list, &q->tx_list);
> WRITE_ONCE(q->desc[i].ctrl, cpu_to_le32(val));
> }
> + q->ndesc = size;
>
> /* xmit ring drop default setting */
> airoha_qdma_set(qdma, REG_TX_RING_BLOCKING(qid),
> @@ -1051,13 +1051,17 @@ static int airoha_qdma_init_tx(struct airoha_qdma *qdma)
>
> static void airoha_qdma_cleanup_tx_queue(struct airoha_queue *q)
> {
> - struct airoha_eth *eth = q->qdma->eth;
> - int i;
> + struct airoha_qdma *qdma = q->qdma;
> + struct airoha_eth *eth = qdma->eth;
> + int i, qid = q - &qdma->q_tx[0];
> + struct airoha_queue_entry *e;
> + u16 index = 0;
>
> spin_lock_bh(&q->lock);
> for (i = 0; i < q->ndesc; i++) {
> - struct airoha_queue_entry *e = &q->entry[i];
super nit: In v2 e is always used within a block (here and in the hunk below).
So I would lean towards declaring e in the blocks where it is
used.
No need to repost just for this!
> + struct airoha_qdma_desc *desc = &q->desc[i];
>
> + e = &q->entry[i];
> if (!e->dma_addr)
> continue;
>
> @@ -1067,8 +1071,31 @@ static void airoha_qdma_cleanup_tx_queue(struct airoha_queue *q)
> e->dma_addr = 0;
> e->skb = NULL;
> list_add_tail(&e->list, &q->tx_list);
> +
> + /* Reset DMA descriptor */
> + WRITE_ONCE(desc->ctrl, 0);
> + WRITE_ONCE(desc->addr, 0);
> + WRITE_ONCE(desc->data, 0);
> + WRITE_ONCE(desc->msg0, 0);
> + WRITE_ONCE(desc->msg1, 0);
> + WRITE_ONCE(desc->msg2, 0);
> +
> q->queued--;
> }
> +
> + if (!list_empty(&q->tx_list)) {
> + e = list_first_entry(&q->tx_list, struct airoha_queue_entry,
> + list);
> + index = e - q->entry;
> + }
> + /* Set TX_DMA_IDX to TX_CPU_IDX to notify the hw the QDMA TX ring is
> + * empty.
> + */
> + airoha_qdma_rmw(qdma, REG_TX_CPU_IDX(qid), TX_RING_CPU_IDX_MASK,
> + FIELD_PREP(TX_RING_CPU_IDX_MASK, index));
> + airoha_qdma_rmw(qdma, REG_TX_DMA_IDX(qid), TX_RING_DMA_IDX_MASK,
> + FIELD_PREP(TX_RING_DMA_IDX_MASK, index));
> +
> spin_unlock_bh(&q->lock);
> }
>
>
> ---
> base-commit: 2cd7e6971fc2787408ceef17906ea152791448cf
> change-id: 20260410-airoha_qdma_cleanup_tx_queue-fix-net-93375f5ee80f
>
> Best regards,
> --
> Lorenzo Bianconi <lorenzo@kernel.org>
>
^ permalink raw reply
* Re: [PATCH v4 7/9] coresight: etm3x: introduce struct etm_caps
From: Yeoreum Yun @ 2026-04-15 16:45 UTC (permalink / raw)
To: Jie Gan
Cc: coresight, linux-arm-kernel, linux-kernel, suzuki.poulose,
mike.leach, james.clark, alexander.shishkin, leo.yan
In-Reply-To: <585280ea-8395-482a-8a3e-7527fce20539@oss.qualcomm.com>
[...]
> > Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
> > ---
> > drivers/hwtracing/coresight/coresight-etm.h | 42 ++++++++++++-------
> > .../coresight/coresight-etm3x-core.c | 39 ++++++++++-------
> > .../coresight/coresight-etm3x-sysfs.c | 29 ++++++++-----
> > 3 files changed, 67 insertions(+), 43 deletions(-)
> >
> > diff --git a/drivers/hwtracing/coresight/coresight-etm.h b/drivers/hwtracing/coresight/coresight-etm.h
> > index 40f20daded4f..8d1a1079b008 100644
> > --- a/drivers/hwtracing/coresight/coresight-etm.h
> > +++ b/drivers/hwtracing/coresight/coresight-etm.h
> > @@ -140,6 +140,30 @@
> > ETM_ADD_COMP_0 | \
> > ETM_EVENT_NOT_A)
> > +/**
> > + * struct etmv_caps - specifics ETM capabilities
>
> s/etmv_caps/etm_caps
Sorry. I'll fix it. Thanks!
[...]
--
Sincerely,
Yeoreum Yun
^ permalink raw reply
* Re: [PATCH net v5] net: stmmac: Prevent NULL deref when RX memory exhausted
From: Russell King (Oracle) @ 2026-04-15 16:28 UTC (permalink / raw)
To: Sam Edwards
Cc: Andrew Lunn, David S. Miller, Eric Dumazet, Jakub Kicinski,
Paolo Abeni, Maxime Coquelin, Alexandre Torgue, Maxime Chevallier,
Ovidiu Panait, Vladimir Oltean, Baruch Siach, Serge Semin,
Giuseppe Cavallaro, netdev, linux-stm32, linux-arm-kernel,
linux-kernel, stable
In-Reply-To: <ad-LAB08-_rpmMzK@shell.armlinux.org.uk>
On Wed, Apr 15, 2026 at 01:56:32PM +0100, Russell King (Oracle) wrote:
> On Tue, Apr 14, 2026 at 07:39:47PM -0700, Sam Edwards wrote:
> > The CPU receives frames from the MAC through conventional DMA: the CPU
> > allocates buffers for the MAC, then the MAC fills them and returns
> > ownership to the CPU. For each hardware RX queue, the CPU and MAC
> > coordinate through a shared ring array of DMA descriptors: one
> > descriptor per DMA buffer. Each descriptor includes the buffer's
> > physical address and a status flag ("OWN") indicating which side owns
> > the buffer: OWN=0 for CPU, OWN=1 for MAC. The CPU is only allowed to set
> > the flag and the MAC is only allowed to clear it, and both must move
> > through the ring in sequence: thus the ring is used for both
> > "submissions" and "completions."
> >
> > In the stmmac driver, stmmac_rx() bookmarks its position in the ring
> > with the `cur_rx` index. The main receive loop in that function checks
> > for rx_descs[cur_rx].own=0, gives the corresponding buffer to the
> > network stack (NULLing the pointer), and increments `cur_rx` modulo the
> > ring size. After the loop exits, stmmac_rx_refill(), which bookmarks its
> > position with `dirty_rx`, allocates fresh buffers and rearms the
> > descriptors (setting OWN=1). If it fails any allocation, it simply stops
> > early (leaving OWN=0) and will retry where it left off when next called.
> >
> > This means descriptors have a three-stage lifecycle (terms my own):
> > - `empty` (OWN=1, buffer valid)
> > - `full` (OWN=0, buffer valid and populated)
> > - `dirty` (OWN=0, buffer NULL)
> >
> > But because stmmac_rx() only checks OWN, it confuses `full`/`dirty`. In
> > the past (see 'Fixes:'), there was a bug where the loop could cycle
> > `cur_rx` all the way back to the first descriptor it dirtied, resulting
> > in a NULL dereference when mistaken for `full`. The aforementioned
> > commit resolved that *specific* failure by capping the loop's iteration
> > limit at `dma_rx_size - 1`, but this is only a partial fix: if the
> > previous stmmac_rx_refill() didn't complete, then there are leftover
> > `dirty` descriptors that the loop might encounter without needing to
> > cycle fully around. The current code therefore panics (see 'Closes:')
> > when stmmac_rx_refill() is memory-starved long enough for `cur_rx` to
> > catch up to `dirty_rx`.
> >
> > Fix this by further tightening the clamp from `dma_rx_size - 1` to
> > `dma_rx_size - stmmac_rx_dirty() - 1`, subtracting any remnant dirty
> > entries and limiting the loop so that `cur_rx` cannot catch back up to
> > `dirty_rx`. This carries no risk of arithmetic underflow: since the
> > maximum possible return value of stmmac_rx_dirty() is `dma_rx_size - 1`,
> > the worst the clamp can do is prevent the loop from running at all.
> >
> > Fixes: b6cb4541853c7 ("net: stmmac: avoid rx queue overrun")
> > Closes: https://bugzilla.kernel.org/show_bug.cgi?id=221010
> > Cc: stable@vger.kernel.org
> > Signed-off-by: Sam Edwards <CFSworks@gmail.com>
>
> Locally, while debugging my issues, I used this to prevent cur_rx
> catching up with dirty_rx:
>
> status = stmmac_rx_status(priv, &priv->xstats, p);
> /* check if managed by the DMA otherwise go ahead */
> if (unlikely(status & dma_own))
> break;
>
> next_entry = STMMAC_NEXT_ENTRY(rx_q->cur_rx,
> priv->dma_conf.dma_rx_size);
> if (unlikely(next_entry == rx_q->dirty_rx))
> break;
>
> rx_q->cur_rx = next_entry;
>
> If we care about the cost of reloading rx_q->dirty_rx on every
> iteration, then I'd suggest that the cost we already incur reading and
> writing rx_q->cur_rx is something that should be addressed, and
> eliminating that would counter the cost of reading rx_q->dirty_rx. I
> suspect, however, that the cost is minimal, as cur_tx and dirty_rx are
> likely in the same cache line.
>
> It looks like any fix to stmmac_rx() will also need a corresponding
> fix for stmmac_rx_zc().
I have some further information, but a new curveball has just been
chucked... and I've no idea what this will mean at this stage. Just
take it that I won't be responding for a while.
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last!
^ permalink raw reply
* Re: [PATCH RFC v2 02/11] ASoC: meson: aiu-encoder-i2s: use gx_iface and gx_stream structures
From: Mark Brown @ 2026-04-15 16:26 UTC (permalink / raw)
To: Jerome Brunet
Cc: Valerio Setti, Liam Girdwood, Jaroslav Kysela, Takashi Iwai,
Neil Armstrong, Kevin Hilman, Martin Blumenstingl, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, linux-kernel, linux-sound,
linux-arm-kernel, linux-amlogic, devicetree
In-Reply-To: <1jy0ios3f9.fsf@starbuckisacylon.baylibre.com>
[-- Attachment #1: Type: text/plain, Size: 298 bytes --]
On Wed, Apr 15, 2026 at 04:28:58PM +0200, Jerome Brunet wrote:
> Valerio maybe you could keep function above just to set the rate, but
> enabling the clocks through a DAPM supply widget ? This is kind of what
> the AXG is doing.
> what do you think ?
FWIW this seems like a sensible plan to me.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply
* Re: [PATCH RFC v2 00/11] Add support for AUDIN driver in Amlogic GXBB
From: Krzysztof Kozlowski @ 2026-04-15 15:53 UTC (permalink / raw)
To: Jerome Brunet, Valerio Setti
Cc: Liam Girdwood, Mark Brown, Jaroslav Kysela, Takashi Iwai,
Neil Armstrong, Kevin Hilman, Martin Blumenstingl, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, linux-kernel, linux-sound,
linux-arm-kernel, linux-amlogic, devicetree
In-Reply-To: <1jh5pcs2gw.fsf@starbuckisacylon.baylibre.com>
On 15/04/2026 16:49, Jerome Brunet wrote:
> On sam. 11 avril 2026 at 16:57, Valerio Setti <vsetti@baylibre.com> wrote:
>
>> This series adds support for I2S audio input (AUDIN) on the Amlogic GXBB
>> platform.
>>
>> It has been largely reshaped compared to what proposed in v1. Instead of
>> adding an HACK commit to allow AIU to export its clock so that also
>> AUDIN can control it, now the design closely follows what was implemented
>> in the Meson AXG platform. "aiu-encoder-i2s" becomes the shared interface
>> for playback/capture and it controls pins and clocks; data formatting
>> is implemented in formatters which are named "aiu-formatter-i2s" and
>> "audin-decoder-i2s" [1].
>> Formatters are DAPM widgets which are dynamically attached/detached to
>> the streams when the latters starts/stop, respectively.
>>
>> As of now only I2S input is supported, because it's the only one
>> I could physically test in my setup, but other input sources (ex: SPDIF)
>> are also allowed according to the SOC's manual and can be added in the
>> future.
>> This series was tested on an OdroidC2 board (Amlogic S905 SOC) with an
>> NXP SGTL5000 codec connected to its I2S input port.
>>
>> Since this work brings GX platform very close to the AXG one, once this
>> series is accepted, follow up work will be done in order to unify
>> GX and AXG formatters so as to minimize the number of implementations.
>>
>> The series a bit long and it includes changes to drivers, dt-bindings and
>> device-tree. Of course this only happens because this is an RFC and I
>> wanted to give a full overview of what will be the final design. If no
>> objection is raised, this patch series will be split into 3: one for
>> reshaping AIU and introducing formatters, one to add AUDIN driver and its
>> dt-bindings, one for the device-tree changes.
>>
>> [1]: Different naming for the aiu part is related to the fact that
>> "aiu-encoder-i2s" is already used for the interface and the goal
>> of this series was to introduce the minimum amount of changes that allow
>> I2S capture to work. Renaming can be implemented in the future as follow up
>> activity.
>
> Thanks a lot for this awesome work Valerio. I know this was a lot of
> effort. With Mark and Krzysztof comments addressed
>
My comments are still unanswered. One of the devices looks like
artificially split from some other, because one word register is not a
device.
Best regards,
Krzysztof
^ permalink raw reply
* Re: [PATCH v6 01/30] mm: Introduce kpkeys
From: Kevin Brodsky @ 2026-04-15 15:50 UTC (permalink / raw)
To: David Hildenbrand (Arm), linux-hardening
Cc: linux-kernel, Andrew Morton, Andy Lutomirski, Catalin Marinas,
Dave Hansen, Ira Weiny, Jann Horn, Jeff Xu, Joey Gouly, Kees Cook,
Linus Walleij, Lorenzo Stoakes, Marc Zyngier, Mark Brown,
Matthew Wilcox, Maxwell Bland, Mike Rapoport (IBM),
Peter Zijlstra, Pierre Langlois, Quentin Perret, Rick Edgecombe,
Ryan Roberts, Thomas Gleixner, Vlastimil Babka, Will Deacon,
Yang Shi, Yeoreum Yun, linux-arm-kernel, linux-mm, x86
In-Reply-To: <eaca640b-5006-489b-8b2d-148a3c4073da@kernel.org>
On 15/04/2026 15:00, David Hildenbrand (Arm) wrote:
> On 2/27/26 18:54, Kevin Brodsky wrote:
>> kpkeys is a simple framework to enable the use of protection keys
>> (pkeys) to harden the kernel itself. This patch introduces the basic
>> API in <linux/kpkeys.h>: a couple of functions to set and restore
>> the pkey register and macros to define guard objects.
>>
>> kpkeys introduces a new concept on top of pkeys: the kpkeys level.
>> Each level is associated to a set of permissions for the pkeys
>> managed by the kpkeys framework. kpkeys_set_level(lvl) sets those
>> permissions according to lvl, and returns the original pkey
>> register, to be later restored by kpkeys_restore_pkey_reg(). To
>> start with, only KPKEYS_LVL_DEFAULT is available, which is meant
>> to grant RW access to KPKEYS_PKEY_DEFAULT (i.e. all memory since
>> this is the only available pkey for now).
>>
>> Because each architecture implementing pkeys uses a different
>> representation for the pkey register, and may reserve certain pkeys
>> for specific uses, support for kpkeys must be explicitly indicated
>> by selecting ARCH_HAS_KPKEYS and defining the following functions in
>> <asm/kpkeys.h>, in addition to the macros provided in
>> <asm-generic/kpkeys.h>:
> I don't quite understand the reason for using levels. Levels sounds like
> it would all be in some ordered fashion, where higher levels have access
> to lower levels.
That was originally the idea indeed, but in practice I don't expect
levels to have a strict ordering, as it's not practical for composing
features.
> Think of that as a key that can unlock all "lower" locks, not just a
> single lock.
>
> Then, the question is about the ordering once we introduce new
> keys/locks. With two, it obviously doesn't matter :)
>
> So naturally I wonder whether levels is really the right abstraction
> here, and why we are not simply using "distinct" keys, like
>
> KPKEY_DEFAULT
> KPKEY_PGTABLE
> KPKEY_SUPER_SECRET1
> KPKEY_SUPER_SECRET2
>
> Is it because you want KPKEY_PGTABLE also be able to write to KPKEY_DEFAULT?
Right, and in general a given level may be able to write to any number
of pkeys. That's why I don't want to conflate pkeys and levels. Agreed
that "level" might not be the clearest term though, since there's no
strict ordering.
> But how would you handle KPKEY_SUPER_SECRET1 and KPKEY_SUPER_SECRET2 then?
Presumably those would also have access to KPKEY_DEFAULT. However, if
you consider the reverse situation where the level is less privileged
than the default (say an eBPF program), then write access to
KPKEY_DEFAULT would not be granted.
Also worth noting on the notion of level that POE2 will bring further
per-level restrictions, besides which pkeys can be accessed. For
instance, we could prevent an unprivileged level from executing certain
instructions. This isn't in scope for this series, but this is a
consideration in the design of the kpkeys abstractions.
- Kevin
^ permalink raw reply
* Re: [PATCH v6 00/30] pkeys-based page table hardening
From: Kevin Brodsky @ 2026-04-15 15:48 UTC (permalink / raw)
To: David Hildenbrand (Arm), linux-hardening
Cc: linux-kernel, Andrew Morton, Andy Lutomirski, Catalin Marinas,
Dave Hansen, Ira Weiny, Jann Horn, Jeff Xu, Joey Gouly, Kees Cook,
Linus Walleij, Lorenzo Stoakes, Marc Zyngier, Mark Brown,
Matthew Wilcox, Maxwell Bland, Mike Rapoport (IBM),
Peter Zijlstra, Pierre Langlois, Quentin Perret, Rick Edgecombe,
Ryan Roberts, Thomas Gleixner, Vlastimil Babka, Will Deacon,
Yang Shi, Yeoreum Yun, linux-arm-kernel, linux-mm, x86
In-Reply-To: <1c8e2cd6-4b50-4891-8a2d-6a45623e805f@kernel.org>
On 15/04/2026 14:48, David Hildenbrand (Arm) wrote:
> On 2/27/26 18:54, Kevin Brodsky wrote:
>> NEW in v6: support for large block mappings through a dedicated page table
>> allocator (patch 14-17)
> Heh, I had to read till the very end to realize that this is an RFC, and
> then saw your other mail.
>
> I can recommend using b4 for patch management, where you can configure a
> sticky prefix through
>
> b4 prep --set-prefixes RFC
>
> And using "b4 send" to automate all the rest.
I certainly should... sorry for the confusion!
>> Threat model
>> ============
>>
>> The proposed scheme aims at mitigating data-only attacks (e.g.
>> use-after-free/cross-cache attacks). In other words, it is assumed that
>> control flow is not corrupted, and that the attacker does not achieve
>> arbitrary code execution. Nothing prevents the pkey register from being
>> set to its most permissive state - the assumption is that the register
>> is only modified on legitimate code paths.
>>
>> A few related notes:
>>
>> - Functions that set the pkey register are all implemented inline.
>> Besides performance considerations, this is meant to avoid creating
>> a function that can be used as a straightforward gadget to set the
>> pkey register to an arbitrary value.
>>
>> - kpkeys_set_level() only accepts a compile-time constant as argument,
>> as a variable could be manipulated by an attacker. This could be
>> relaxed but it seems unlikely that a variable kpkeys level would be
>> needed in practice.
>>
> I see a lot of value for that also as a debugging mechanism. I hear that
> other people had private patches that would attempt to only map leaf
> pages in the direct map in pte_offset_map_lock() and friends. I assume
> there are some tricky bits to that (concurrent access to page tables).
Indeed, this should be a much better solution, not only because it means
a lot fewer TLBIs, but also because it is truly per-thread (so
concurrency is not a concern).
> What's the general take regarding the thread model you describe vs. MTE?
I'd say quite similar, although corrupting pointers (specifically the
tag bits) remains possible in a data-only attack, while corrupting the
POR_EL1 register would require some control flow hijack (only constant
values are written to POR_EL1).
> Regarding use-after-free, I'd assume KASAN would achieve something
> similar. And with MTE "reasonably" fast. Or what is the biggest
> difference you see, there?
For use-after-free specifically, yes that sounds about right.
> I'd assume that one difference would be, that not even match-all
> pointers could accidentally modify page tables.
Yep that's pretty much what I tried to say above - with pkeys you have
to corrupt a system register to bypass the protection.
> In the future, would you think that both mechanisms (pkey PT table
> protection + KASAN) would be active at the same time, or wouldn't there
> really be a lot of value in having both enabled?
I think these are fairly orthogonal, KASAN gives you probabilistic
spatial+temporal safety for most allocations, while kpkeys restricts
access to key data to a small set of functions. I don't think one
reduces the usefulness of the other. Of course KASAN makes it harder to
use an arbitrary pointer to write to page tables, but kpkeys gives a
clear guarantee (assuming CFI is preserved).
> [...]
>
>>
>> Open questions
>> ==============
>>
>> A few aspects in this RFC that are debatable and/or worth discussing:
>>
>> - Can the pkeys block allocator be abstracted into something more
>> generic? This seems desirable considering other use-cases for changing
>> attributes of regions of the linear map, but the handling of page
>> tables while splitting may be difficult to integrate in a generic
>> allocator.
>>
>> - There is currently no restriction on how kpkeys levels map to pkeys
>> permissions. A typical approach is to allocate one pkey per level and
>> make it writable at that level only. As the number of levels
>> increases, we may however run out of pkeys, especially on arm64 (just
>> 8 pkeys with POE). Depending on the use-cases, it may be acceptable to
>> use the same pkey for the data associated to multiple levels.
>>
>>
>> Any comment or feedback is highly appreciated, be it on the high-level
>> approach or implementation choices!
> How crucial would the dedicated page table allocator be for a first up
> streamed version?
>
> Assuming we introduce this as a debugging feature first, it would be
> perfectly reasonable to just disallow large block mappings in the direct
> map when enabled.
>
> That means, we could merge basic support first and think about how to
> deal with page tables in a different way with most of the pkey details
> out of the picture.
I think that makes perfect sense, at least on arm64 where it's just a
matter of configuring force_pte_mapping() appropriately. I'm not sure
whether there is such an option on x86, though.
- Kevin
^ permalink raw reply
* Re: Highmem on AXM and K2
From: Arnd Bergmann @ 2026-04-15 15:32 UTC (permalink / raw)
To: Stefan Wiehler; +Cc: linux-kernel, linux-arm-kernel
In-Reply-To: <64ceb81d-68e0-498c-bcf8-1ebb83ac2d23@nokia.com>
On Wed, Apr 15, 2026, at 16:48, Stefan Wiehler wrote:
> Hi Arnd,
>
> I would like to inform you that after some internal discussion, we
> decided not to continue with my experiments to transition from
> VMSPLIT_3G towards VMSPLIT_2G{_OPT} on the K2. The K2 is used in a
> legacy product after all, and the more senior colleagues were quite
> concerned about making such a major change due to various hacks that
> might fall apart along the way.
>
> Therefore, we will go the CIP path by staying with the last SLTS kernel
> with full highmem support. Therefore, we would be very glad if this is
> the case for at least the next SLTS kernel, supposedly to be released in
> 2027, as we should then be able (despite barely) to meet our projected
> EOL of 2037.
Hi Stefan,
The CIP kernel from 2027 should certainly still support highmem,
though I am in the process of removing highmem usage from drivers
that rarely use a lot of memory, which means you may see a
slightly higher memory consumption, and changing the default to
CONFIG_VMSPLIT_2G_OPT/CONFIG_VMSPLIT_2G means that your configuration
will see less testing in the future.
I don't think there is an immediate problem for your product
using the 2027 LTS kernel, or (most likely) the 2029 version,
but I'm still worried about the boot regression you reported
with CONFIG_VMSPLIT_2G, which means we still have to make sure
K2 works with the new default.
Arnd
^ permalink raw reply
* Re: [PATCH v2] Bluetooth: Add Broadcom channel priority commands
From: Sasha Finkelstein @ 2026-04-15 15:31 UTC (permalink / raw)
To: Luiz Augusto von Dentz
Cc: Sven Peter, Janne Grunau, Neal Gompa, Marcel Holtmann,
David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Simon Horman, linux-kernel, asahi, linux-arm-kernel,
linux-bluetooth, netdev
In-Reply-To: <CABBYNZLNR8hYS9jLLKeB=M9XVvtSFtf1wi4DmcJKBbQVvHTPaw@mail.gmail.com>
On Wed, 15 Apr 2026 at 17:19, Luiz Augusto von Dentz
<luiz.dentz@gmail.com> wrote:
> Ok, then maybe we should decrease the priority, so it can only go up.
> That said, in a multiple connection scenario, we cannot really tell
> what should be prioritized if we cannot momentarily decrease the
> priority.
I believe that the priority is only per-connection and is not designed
to be used per-packet. On Android they change priority when an
A2DP stream starts or stops, by sending the commands from
userspace and are accepting that other things using the same hci
connection will also have high priority.
^ permalink raw reply
* [PATCH 9/9] ARM: dts: mediatek: mt7623n-bananapi-bpi-r2: add HDMI audio machine node
From: Daniel Golle @ 2026-04-15 15:24 UTC (permalink / raw)
To: Liam Girdwood, Mark Brown, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Matthias Brugger, AngeloGioacchino Del Regno,
Jaroslav Kysela, Takashi Iwai, Cyril Chao, Arnd Bergmann,
Kuninori Morimoto, Daniel Golle, Nícolas F. R. A. Prado,
Eugen Hristev, linux-sound, devicetree, linux-kernel,
linux-arm-kernel, linux-mediatek
In-Reply-To: <cover.1776265610.git.daniel@makrotopia.org>
Instantiate the mediatek,mt2701-hdmi-audio machine on the
BananaPi BPI-R2, binding the AFE HDMI playback path to the
on-chip HDMI transmitter acting as the generic HDMI codec.
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
---
arch/arm/boot/dts/mediatek/mt7623n-bananapi-bpi-r2.dts | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/arch/arm/boot/dts/mediatek/mt7623n-bananapi-bpi-r2.dts b/arch/arm/boot/dts/mediatek/mt7623n-bananapi-bpi-r2.dts
index a37f3fa223c72..139a76764faa0 100644
--- a/arch/arm/boot/dts/mediatek/mt7623n-bananapi-bpi-r2.dts
+++ b/arch/arm/boot/dts/mediatek/mt7623n-bananapi-bpi-r2.dts
@@ -132,6 +132,13 @@ memory@80000000 {
device_type = "memory";
reg = <0 0x80000000 0 0x80000000>;
};
+
+ sound-hdmi {
+ compatible = "mediatek,mt7623n-hdmi-audio",
+ "mediatek,mt2701-hdmi-audio";
+ mediatek,platform = <&afe>;
+ mediatek,audio-codec = <&hdmi0>;
+ };
};
&bls {
--
2.53.0
^ permalink raw reply related
* [PATCH 8/9] ARM: dts: mediatek: mt7623: wire HDMI audio path clocks into AFE
From: Daniel Golle @ 2026-04-15 15:24 UTC (permalink / raw)
To: Liam Girdwood, Mark Brown, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Matthias Brugger, AngeloGioacchino Del Regno,
Jaroslav Kysela, Takashi Iwai, Cyril Chao, Arnd Bergmann,
Kuninori Morimoto, Daniel Golle, Nícolas F. R. A. Prado,
Eugen Hristev, linux-sound, devicetree, linux-kernel,
linux-arm-kernel, linux-mediatek
In-Reply-To: <cover.1776265610.git.daniel@makrotopia.org>
Mirror the MT2701 change for the MT7623 SoC dtsi: add HADDS2PLL,
audio_hdmi, audio_spdf and audio_apll to the AFE clocks list and
reparent the AUDPLL mux to HADDS2PLL_98M. Required for HDMI audio
on MT7623N boards via the shared mt2701 AFE driver.
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
---
arch/arm/boot/dts/mediatek/mt7623.dtsi | 21 ++++++++++++++++-----
1 file changed, 16 insertions(+), 5 deletions(-)
diff --git a/arch/arm/boot/dts/mediatek/mt7623.dtsi b/arch/arm/boot/dts/mediatek/mt7623.dtsi
index 71ac2b94c6ba3..4eb028ffee6f5 100644
--- a/arch/arm/boot/dts/mediatek/mt7623.dtsi
+++ b/arch/arm/boot/dts/mediatek/mt7623.dtsi
@@ -665,7 +665,11 @@ afe: audio-controller {
<&audsys CLK_AUD_AFE_CONN>,
<&audsys CLK_AUD_A1SYS>,
<&audsys CLK_AUD_A2SYS>,
- <&audsys CLK_AUD_AFE_MRGIF>;
+ <&audsys CLK_AUD_AFE_MRGIF>,
+ <&topckgen CLK_TOP_HADDS2PLL_294M>,
+ <&audsys CLK_AUD_HDMI>,
+ <&audsys CLK_AUD_SPDF>,
+ <&audsys CLK_AUD_APLL>;
clock-names = "infra_sys_audio_clk",
"top_audio_mux1_sel",
@@ -700,15 +704,22 @@ afe: audio-controller {
"audio_afe_conn_pd",
"audio_a1sys_pd",
"audio_a2sys_pd",
- "audio_mrgif_pd";
+ "audio_mrgif_pd",
+ "hadds2pll_294m",
+ "audio_hdmi_pd",
+ "audio_spdf_pd",
+ "audio_apll_pd";
assigned-clocks = <&topckgen CLK_TOP_AUD_MUX1_SEL>,
<&topckgen CLK_TOP_AUD_MUX2_SEL>,
<&topckgen CLK_TOP_AUD_MUX1_DIV>,
- <&topckgen CLK_TOP_AUD_MUX2_DIV>;
+ <&topckgen CLK_TOP_AUD_MUX2_DIV>,
+ <&topckgen CLK_TOP_AUDPLL_MUX_SEL>;
assigned-clock-parents = <&topckgen CLK_TOP_AUD1PLL_98M>,
- <&topckgen CLK_TOP_AUD2PLL_90M>;
- assigned-clock-rates = <0>, <0>, <49152000>, <45158400>;
+ <&topckgen CLK_TOP_AUD2PLL_90M>,
+ <0>, <0>,
+ <&topckgen CLK_TOP_HADDS2PLL_98M>;
+ assigned-clock-rates = <0>, <0>, <49152000>, <45158400>, <0>;
};
};
--
2.53.0
^ permalink raw reply related
* [PATCH 7/9] ARM: dts: mediatek: mt2701: wire HDMI audio path clocks into AFE
From: Daniel Golle @ 2026-04-15 15:24 UTC (permalink / raw)
To: Liam Girdwood, Mark Brown, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Matthias Brugger, AngeloGioacchino Del Regno,
Jaroslav Kysela, Takashi Iwai, Cyril Chao, Arnd Bergmann,
Kuninori Morimoto, Daniel Golle, Nícolas F. R. A. Prado,
Eugen Hristev, linux-sound, devicetree, linux-kernel,
linux-arm-kernel, linux-mediatek
In-Reply-To: <cover.1776265610.git.daniel@makrotopia.org>
Add the HADDS2 PLL 294 MHz root, the audio_hdmi and audio_spdf
interface gates and the audio_apll gate to the MT2701 AFE node,
and reparent the AUDPLL mux to HADDS2PLL_98M so the HDMI audio
serial clock path has a stable 294.912 MHz source. The clock
names match the updated mediatek,mt2701-audio binding.
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
---
arch/arm/boot/dts/mediatek/mt2701.dtsi | 21 ++++++++++++++++-----
1 file changed, 16 insertions(+), 5 deletions(-)
diff --git a/arch/arm/boot/dts/mediatek/mt2701.dtsi b/arch/arm/boot/dts/mediatek/mt2701.dtsi
index 128b87229f3d5..80c8c7e6a422a 100644
--- a/arch/arm/boot/dts/mediatek/mt2701.dtsi
+++ b/arch/arm/boot/dts/mediatek/mt2701.dtsi
@@ -464,7 +464,11 @@ afe: audio-controller {
<&audsys CLK_AUD_AFE_CONN>,
<&audsys CLK_AUD_A1SYS>,
<&audsys CLK_AUD_A2SYS>,
- <&audsys CLK_AUD_AFE_MRGIF>;
+ <&audsys CLK_AUD_AFE_MRGIF>,
+ <&topckgen CLK_TOP_HADDS2PLL_294M>,
+ <&audsys CLK_AUD_HDMI>,
+ <&audsys CLK_AUD_SPDF>,
+ <&audsys CLK_AUD_APLL>;
clock-names = "infra_sys_audio_clk",
"top_audio_mux1_sel",
@@ -499,15 +503,22 @@ afe: audio-controller {
"audio_afe_conn_pd",
"audio_a1sys_pd",
"audio_a2sys_pd",
- "audio_mrgif_pd";
+ "audio_mrgif_pd",
+ "hadds2pll_294m",
+ "audio_hdmi_pd",
+ "audio_spdf_pd",
+ "audio_apll_pd";
assigned-clocks = <&topckgen CLK_TOP_AUD_MUX1_SEL>,
<&topckgen CLK_TOP_AUD_MUX2_SEL>,
<&topckgen CLK_TOP_AUD_MUX1_DIV>,
- <&topckgen CLK_TOP_AUD_MUX2_DIV>;
+ <&topckgen CLK_TOP_AUD_MUX2_DIV>,
+ <&topckgen CLK_TOP_AUDPLL_MUX_SEL>;
assigned-clock-parents = <&topckgen CLK_TOP_AUD1PLL_98M>,
- <&topckgen CLK_TOP_AUD2PLL_90M>;
- assigned-clock-rates = <0>, <0>, <49152000>, <45158400>;
+ <&topckgen CLK_TOP_AUD2PLL_90M>,
+ <0>, <0>,
+ <&topckgen CLK_TOP_HADDS2PLL_98M>;
+ assigned-clock-rates = <0>, <0>, <49152000>, <45158400>, <0>;
};
};
--
2.53.0
^ permalink raw reply related
* [PATCH 6/9] ASoC: mediatek: mt2701: add machine driver for on-chip HDMI codec
From: Daniel Golle @ 2026-04-15 15:24 UTC (permalink / raw)
To: Liam Girdwood, Mark Brown, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Matthias Brugger, AngeloGioacchino Del Regno,
Jaroslav Kysela, Takashi Iwai, Cyril Chao, Arnd Bergmann,
Kuninori Morimoto, Daniel Golle, Nícolas F. R. A. Prado,
Eugen Hristev, linux-sound, devicetree, linux-kernel,
linux-arm-kernel, linux-mediatek
In-Reply-To: <cover.1776265610.git.daniel@makrotopia.org>
Add a simple ASoC machine driver that wires the MT2701/MT7623N
AFE HDMI playback path to the on-chip HDMI transmitter exposed
as a generic hdmi-audio-codec "i2s-hifi" DAI.
The driver binds to "mediatek,mt2701-hdmi-audio". MT7623N device
trees carry "mediatek,mt7623n-hdmi-audio" as a board-specific
fallback, matching the dt-binding.
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
---
sound/soc/mediatek/Kconfig | 10 +++
sound/soc/mediatek/mt2701/Makefile | 1 +
sound/soc/mediatek/mt2701/mt2701-hdmi.c | 114 ++++++++++++++++++++++++
3 files changed, 125 insertions(+)
create mode 100644 sound/soc/mediatek/mt2701/mt2701-hdmi.c
diff --git a/sound/soc/mediatek/Kconfig b/sound/soc/mediatek/Kconfig
index 3a1e1fa3fe5cc..fa076e7854adc 100644
--- a/sound/soc/mediatek/Kconfig
+++ b/sound/soc/mediatek/Kconfig
@@ -26,6 +26,16 @@ config SND_SOC_MT2701_CS42448
Select Y if you have such device.
If unsure select "N".
+config SND_SOC_MT2701_HDMI
+ tristate "ASoC Audio driver for MT2701 with on-chip HDMI codec"
+ depends on SND_SOC_MT2701
+ select SND_SOC_HDMI_CODEC
+ help
+ This adds the ASoC machine driver for MediaTek MT2701 and
+ MT7623N boards routing the AFE I2S back-end to the on-chip
+ HDMI transmitter via the generic HDMI codec.
+ If unsure select "N".
+
config SND_SOC_MT2701_WM8960
tristate "ASoc Audio driver for MT2701 with WM8960 codec"
depends on SND_SOC_MT2701 && I2C
diff --git a/sound/soc/mediatek/mt2701/Makefile b/sound/soc/mediatek/mt2701/Makefile
index 507fa26c39452..59623d3d3a038 100644
--- a/sound/soc/mediatek/mt2701/Makefile
+++ b/sound/soc/mediatek/mt2701/Makefile
@@ -5,4 +5,5 @@ obj-$(CONFIG_SND_SOC_MT2701) += snd-soc-mt2701-afe.o
# machine driver
obj-$(CONFIG_SND_SOC_MT2701_CS42448) += mt2701-cs42448.o
+obj-$(CONFIG_SND_SOC_MT2701_HDMI) += mt2701-hdmi.o
obj-$(CONFIG_SND_SOC_MT2701_WM8960) += mt2701-wm8960.o
diff --git a/sound/soc/mediatek/mt2701/mt2701-hdmi.c b/sound/soc/mediatek/mt2701/mt2701-hdmi.c
new file mode 100644
index 0000000000000..a84907879c04e
--- /dev/null
+++ b/sound/soc/mediatek/mt2701/mt2701-hdmi.c
@@ -0,0 +1,114 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * mt2701-hdmi.c -- MT2701 HDMI ALSA SoC machine driver
+ *
+ * Copyright (c) 2026 Daniel Golle <daniel@makrotopia.org>
+ *
+ * Based on mt2701-cs42448.c
+ */
+
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/platform_device.h>
+#include <sound/soc.h>
+
+enum {
+ DAI_LINK_FE_HDMI_OUT,
+ DAI_LINK_BE_HDMI_I2S,
+};
+
+SND_SOC_DAILINK_DEFS(fe_hdmi_out,
+ DAILINK_COMP_ARRAY(COMP_CPU("PCM_HDMI")),
+ DAILINK_COMP_ARRAY(COMP_DUMMY()),
+ DAILINK_COMP_ARRAY(COMP_EMPTY()));
+
+SND_SOC_DAILINK_DEFS(be_hdmi_i2s,
+ DAILINK_COMP_ARRAY(COMP_CPU("HDMI I2S")),
+ DAILINK_COMP_ARRAY(COMP_CODEC(NULL, "i2s-hifi")),
+ DAILINK_COMP_ARRAY(COMP_EMPTY()));
+
+static struct snd_soc_dai_link mt2701_hdmi_dai_links[] = {
+ [DAI_LINK_FE_HDMI_OUT] = {
+ .name = "HDMI Playback",
+ .stream_name = "HDMI Playback",
+ .trigger = { SND_SOC_DPCM_TRIGGER_POST,
+ SND_SOC_DPCM_TRIGGER_POST },
+ .dynamic = 1,
+ .playback_only = 1,
+ SND_SOC_DAILINK_REG(fe_hdmi_out),
+ },
+ [DAI_LINK_BE_HDMI_I2S] = {
+ .name = "HDMI BE",
+ .no_pcm = 1,
+ .playback_only = 1,
+ .dai_fmt = SND_SOC_DAIFMT_I2S | SND_SOC_DAIFMT_NB_NF |
+ SND_SOC_DAIFMT_CBC_CFC,
+ SND_SOC_DAILINK_REG(be_hdmi_i2s),
+ },
+};
+
+static struct snd_soc_card mt2701_hdmi_soc_card = {
+ .name = "mt2701-hdmi",
+ .owner = THIS_MODULE,
+ .dai_link = mt2701_hdmi_dai_links,
+ .num_links = ARRAY_SIZE(mt2701_hdmi_dai_links),
+};
+
+static int mt2701_hdmi_machine_probe(struct platform_device *pdev)
+{
+ struct snd_soc_card *card = &mt2701_hdmi_soc_card;
+ struct device *dev = &pdev->dev;
+ struct device_node *platform_node;
+ struct device_node *codec_node;
+ struct snd_soc_dai_link *dai_link;
+ int ret;
+ int i;
+
+ platform_node = of_parse_phandle(dev->of_node, "mediatek,platform", 0);
+ if (!platform_node)
+ return dev_err_probe(dev, -EINVAL,
+ "Property 'mediatek,platform' missing\n");
+
+ for_each_card_prelinks(card, i, dai_link) {
+ if (dai_link->platforms->name)
+ continue;
+ dai_link->platforms->of_node = platform_node;
+ }
+
+ codec_node = of_parse_phandle(dev->of_node, "mediatek,audio-codec", 0);
+ if (!codec_node) {
+ of_node_put(platform_node);
+ return dev_err_probe(dev, -EINVAL,
+ "Property 'mediatek,audio-codec' missing\n");
+ }
+ mt2701_hdmi_dai_links[DAI_LINK_BE_HDMI_I2S].codecs->of_node = codec_node;
+
+ card->dev = dev;
+
+ ret = devm_snd_soc_register_card(dev, card);
+
+ of_node_put(platform_node);
+ of_node_put(codec_node);
+ return ret;
+}
+
+static const struct of_device_id mt2701_hdmi_machine_dt_match[] = {
+ { .compatible = "mediatek,mt2701-hdmi-audio" },
+ { .compatible = "mediatek,mt7623n-hdmi-audio" },
+ {}
+};
+MODULE_DEVICE_TABLE(of, mt2701_hdmi_machine_dt_match);
+
+static struct platform_driver mt2701_hdmi_machine = {
+ .driver = {
+ .name = "mt2701-hdmi",
+ .of_match_table = mt2701_hdmi_machine_dt_match,
+ },
+ .probe = mt2701_hdmi_machine_probe,
+};
+module_platform_driver(mt2701_hdmi_machine);
+
+MODULE_DESCRIPTION("MT2701 HDMI ALSA SoC machine driver");
+MODULE_AUTHOR("Daniel Golle <daniel@makrotopia.org>");
+MODULE_LICENSE("GPL");
+MODULE_ALIAS("platform:mt2701-hdmi");
--
2.53.0
^ permalink raw reply related
* [PATCH 5/9] ASoC: mediatek: mt2701: add HDMI audio memif, FE and BE DAIs
From: Daniel Golle @ 2026-04-15 15:23 UTC (permalink / raw)
To: Liam Girdwood, Mark Brown, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Matthias Brugger, AngeloGioacchino Del Regno,
Jaroslav Kysela, Takashi Iwai, Cyril Chao, Arnd Bergmann,
Kuninori Morimoto, Daniel Golle, Nícolas F. R. A. Prado,
Eugen Hristev, linux-sound, devicetree, linux-kernel,
linux-arm-kernel, linux-mediatek
In-Reply-To: <cover.1776265610.git.daniel@makrotopia.org>
Extend the MT2701/MT7623N AFE driver with the HDMI playback path:
- a new HDMI DMA memif (MT2701_MEMIF_HDMI) mapped to the
AFE_HDMI_OUT_{CON0,BASE,CUR,END} registers;
- a PCM_HDMI front-end DAI (S16_LE only, 44.1k/48k) which feeds
the memif via DPCM;
- an HDMI BE DAI wrapping the AFE_8CH_I2S_OUT_CON engine that
serialises L/R samples towards the on-chip HDMI transmitter.
Sample-rate programming uses the empirically determined
HDMI_BCK_DIV = 45 * 48000 / rate - 1 formula in AUDIO_TOP_CON3,
which covers 44.1 kHz and 48 kHz within the 6-bit divider range.
The AFE_HDMI_CONN0 interconnect is programmed to route memif
output pairs to the serializer inputs with L/R in the right order
for hdmi-audio-codec.
The existing I2S engine helpers (mt2701_mclk_configuration,
mt2701_i2s_path_enable, mt2701_afe_i2s_path_disable) are reused
for the HDMI BE so that MCLK at 128*fs and the ASYS I2S3 FS field
are programmed and cleanly released across open/close cycles.
Only S16_LE and 44.1k/48k are exposed to userspace. Other rates
fall outside the 6-bit BCK divider range, and wider sample
formats require DMA BIT_WIDTH programming that the current memif
setup does not handle. These limits match what the MT8173 AFE
driver exposes for its HDMI path.
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
---
sound/soc/mediatek/mt2701/mt2701-afe-common.h | 2 +
sound/soc/mediatek/mt2701/mt2701-afe-pcm.c | 281 +++++++++++++++++-
2 files changed, 282 insertions(+), 1 deletion(-)
diff --git a/sound/soc/mediatek/mt2701/mt2701-afe-common.h b/sound/soc/mediatek/mt2701/mt2701-afe-common.h
index 7b15283d6351e..8b6f3a200048a 100644
--- a/sound/soc/mediatek/mt2701/mt2701-afe-common.h
+++ b/sound/soc/mediatek/mt2701/mt2701-afe-common.h
@@ -33,6 +33,7 @@ enum {
MT2701_MEMIF_UL5,
MT2701_MEMIF_DLBT,
MT2701_MEMIF_ULBT,
+ MT2701_MEMIF_HDMI,
MT2701_MEMIF_NUM,
MT2701_IO_I2S = MT2701_MEMIF_NUM,
MT2701_IO_2ND_I2S,
@@ -41,6 +42,7 @@ enum {
MT2701_IO_5TH_I2S,
MT2701_IO_6TH_I2S,
MT2701_IO_MRG,
+ MT2701_IO_HDMI,
};
enum {
diff --git a/sound/soc/mediatek/mt2701/mt2701-afe-pcm.c b/sound/soc/mediatek/mt2701/mt2701-afe-pcm.c
index fcae38135d93f..61b4fb512be35 100644
--- a/sound/soc/mediatek/mt2701/mt2701-afe-pcm.c
+++ b/sound/soc/mediatek/mt2701/mt2701-afe-pcm.c
@@ -13,6 +13,7 @@
#include <linux/mfd/syscon.h>
#include <linux/of.h>
#include <linux/pm_runtime.h>
+#include <sound/pcm_params.h>
#include "mt2701-afe-common.h"
#include "mt2701-afe-clock-ctrl.h"
@@ -542,6 +543,221 @@ static const struct snd_soc_dai_ops mt2701_btmrg_ops = {
.hw_params = mt2701_btmrg_hw_params,
};
+/*
+ * HDMI BE DAI -- drives the on-SoC 8-channel I2S engine whose output
+ * feeds the HDMI transmitter audio port.
+ *
+ * The HDMI audio hardware path is:
+ * HDMI memif DMA (AFE_HDMI_OUT_*) -> interconnect mux (AFE_HDMI_CONN0)
+ * -> 8-channel I2S engine (AFE_8CH_I2S_OUT_CON) -> HDMI TX audio port
+ *
+ * The I2S3 clock tree provides the bit/master clocks; we set its
+ * mclk_rate to 128*fs (matching HDMI_AUD_MCLK_128FS) and let
+ * mt2701_mclk_configuration program the PLL/divider path.
+ */
+#define MT2701_HDMI_I2S_PATH 3
+
+static int mt2701_afe_hdmi_startup(struct snd_pcm_substream *substream,
+ struct snd_soc_dai *dai)
+{
+ struct mtk_base_afe *afe = snd_soc_dai_get_drvdata(dai);
+ struct mt2701_afe_private *afe_priv = afe->platform_priv;
+ int ret;
+
+ if (!afe_priv->hadds2pll_ck || !afe_priv->audio_hdmi_ck) {
+ dev_err(afe->dev, "HDMI audio clocks not available\n");
+ return -ENODEV;
+ }
+
+ ret = clk_prepare_enable(afe_priv->hadds2pll_ck);
+ if (ret)
+ return ret;
+
+ ret = clk_prepare_enable(afe_priv->audio_hdmi_ck);
+ if (ret)
+ goto err_hdmi;
+
+ if (afe_priv->audio_spdf_ck) {
+ ret = clk_prepare_enable(afe_priv->audio_spdf_ck);
+ if (ret)
+ goto err_spdf;
+ }
+
+ if (afe_priv->audio_apll_ck) {
+ ret = clk_prepare_enable(afe_priv->audio_apll_ck);
+ if (ret)
+ goto err_apll;
+ }
+
+ ret = mt2701_afe_enable_mclk(afe, MT2701_HDMI_I2S_PATH);
+ if (ret)
+ goto err_mclk;
+
+ return 0;
+
+err_mclk:
+ if (afe_priv->audio_apll_ck)
+ clk_disable_unprepare(afe_priv->audio_apll_ck);
+err_apll:
+ if (afe_priv->audio_spdf_ck)
+ clk_disable_unprepare(afe_priv->audio_spdf_ck);
+err_spdf:
+ clk_disable_unprepare(afe_priv->audio_hdmi_ck);
+err_hdmi:
+ clk_disable_unprepare(afe_priv->hadds2pll_ck);
+ return ret;
+}
+
+static void mt2701_afe_hdmi_shutdown(struct snd_pcm_substream *substream,
+ struct snd_soc_dai *dai)
+{
+ struct mtk_base_afe *afe = snd_soc_dai_get_drvdata(dai);
+ struct mt2701_afe_private *afe_priv = afe->platform_priv;
+
+ mt2701_afe_disable_mclk(afe, MT2701_HDMI_I2S_PATH);
+ if (afe_priv->audio_apll_ck)
+ clk_disable_unprepare(afe_priv->audio_apll_ck);
+ if (afe_priv->audio_spdf_ck)
+ clk_disable_unprepare(afe_priv->audio_spdf_ck);
+ clk_disable_unprepare(afe_priv->audio_hdmi_ck);
+ clk_disable_unprepare(afe_priv->hadds2pll_ck);
+}
+
+static int mt2701_afe_hdmi_hw_params(struct snd_pcm_substream *substream,
+ struct snd_pcm_hw_params *params,
+ struct snd_soc_dai *dai)
+{
+ struct mtk_base_afe *afe = snd_soc_dai_get_drvdata(dai);
+ struct mt2701_afe_private *afe_priv = afe->platform_priv;
+ unsigned int channels = params_channels(params);
+ unsigned int rate = params_rate(params);
+ unsigned int divp1;
+ unsigned int val;
+ unsigned int i;
+ int ret;
+
+ /*
+ * Compute AUDIO_TOP_CON3.HDMI_BCK_DIV up front. The divider
+ * drives an internal reference for the HDMI transmitter's
+ * audio packet engine; it must scale with the sample rate so
+ * that the packet engine's timing matches the data flowing in
+ * from the AFE memif/I2S3 side. Empirically, with audpll_sel
+ * parented to hadds2pll_98m (98.304 MHz), the correct value at
+ * 48 kHz is div = 44 (i.e. (div+1) = 45), giving 1.0923 MHz.
+ * Scaling inversely with rate: (div + 1) = 45 * 48000 / rate.
+ * Integer rounding introduces small (<1%) errors at 32 kHz;
+ * 44.1 kHz is nearly exact via round-to-nearest. Reject rates
+ * that fall outside the 6-bit divider range before touching
+ * any hardware so no side effects are left behind on error.
+ */
+ divp1 = (45U * 48000U + rate / 2) / rate;
+ if (divp1 == 0 || divp1 > 64)
+ return -EINVAL;
+
+ /*
+ * Park the I2S3 clock tree at 128*fs -- this is the MCLK that
+ * the ASYS I2S3 engine uses to derive its BCK/LRCK. The engine
+ * outputs BCK = 64*fs (stereo, 32-bit word length).
+ */
+ afe_priv->i2s_path[MT2701_HDMI_I2S_PATH].mclk_rate = rate * 128;
+ ret = mt2701_mclk_configuration(afe, MT2701_HDMI_I2S_PATH);
+ if (ret)
+ return ret;
+
+ /* Program and start the ASYS I2S3 engine (FS, I2S mode, enable). */
+ mt2701_i2s_path_enable(afe,
+ &afe_priv->i2s_path[MT2701_HDMI_I2S_PATH],
+ SNDRV_PCM_STREAM_PLAYBACK, rate);
+
+ regmap_update_bits(afe->regmap, AUDIO_TOP_CON3,
+ AUDIO_TOP_CON3_HDMI_BCK_DIV_MASK,
+ AUDIO_TOP_CON3_HDMI_BCK_DIV(divp1 - 1));
+
+ /* Channel count into the HDMI output memif (bits [7:4]). */
+ regmap_update_bits(afe->regmap, AFE_HDMI_OUT_CON0,
+ 0x000000f0, channels << 4);
+
+ /*
+ * Interconnect mux -- map DMA input slots to HDMI output slots.
+ * Each output takes a 3-bit field at shift (i*3). Swap the first
+ * two inputs so that the DMA's interleaved L/R pair lands on the
+ * correct HDMI L/R output slots. Remaining slots are identity.
+ */
+ val = (1 << 0) | (0 << 3); /* O20 <- I21, O21 <- I20 */
+ for (i = 2; i < 8; i++)
+ val |= ((i & 0x7) << (i * 3));
+ regmap_write(afe->regmap, AFE_HDMI_CONN0, val);
+
+ /*
+ * 8-channel I2S framing: standard I2S, 32-bit slots,
+ * LRCK/BCK inverted. The wire protocol is fixed.
+ */
+ regmap_update_bits(afe->regmap, AFE_8CH_I2S_OUT_CON,
+ AFE_8CH_I2S_OUT_CON_WLEN_MASK |
+ AFE_8CH_I2S_OUT_CON_I2S_DELAY |
+ AFE_8CH_I2S_OUT_CON_LRCK_INV |
+ AFE_8CH_I2S_OUT_CON_BCK_INV,
+ AFE_8CH_I2S_OUT_CON_WLEN_32BIT |
+ AFE_8CH_I2S_OUT_CON_I2S_DELAY |
+ AFE_8CH_I2S_OUT_CON_LRCK_INV |
+ AFE_8CH_I2S_OUT_CON_BCK_INV);
+ return 0;
+}
+
+static int mt2701_afe_hdmi_trigger(struct snd_pcm_substream *substream, int cmd,
+ struct snd_soc_dai *dai)
+{
+ struct mtk_base_afe *afe = snd_soc_dai_get_drvdata(dai);
+
+ switch (cmd) {
+ case SNDRV_PCM_TRIGGER_START:
+ case SNDRV_PCM_TRIGGER_RESUME:
+ /* Ungate HDMI and SPDIF power islands. */
+ regmap_update_bits(afe->regmap, AUDIO_TOP_CON0,
+ AUDIO_TOP_CON0_PDN_HDMI_CK |
+ AUDIO_TOP_CON0_PDN_SPDIF_CK, 0);
+ /* Enable HDMI output memif. */
+ regmap_update_bits(afe->regmap, AFE_HDMI_OUT_CON0, 0x1, 0x1);
+ /* Enable 8-channel I2S engine. */
+ regmap_update_bits(afe->regmap, AFE_8CH_I2S_OUT_CON,
+ AFE_8CH_I2S_OUT_CON_EN,
+ AFE_8CH_I2S_OUT_CON_EN);
+ return 0;
+ case SNDRV_PCM_TRIGGER_STOP:
+ case SNDRV_PCM_TRIGGER_SUSPEND:
+ regmap_update_bits(afe->regmap, AFE_8CH_I2S_OUT_CON,
+ AFE_8CH_I2S_OUT_CON_EN, 0);
+ regmap_update_bits(afe->regmap, AFE_HDMI_OUT_CON0, 0x1, 0);
+ regmap_update_bits(afe->regmap, AUDIO_TOP_CON0,
+ AUDIO_TOP_CON0_PDN_HDMI_CK |
+ AUDIO_TOP_CON0_PDN_SPDIF_CK,
+ AUDIO_TOP_CON0_PDN_HDMI_CK |
+ AUDIO_TOP_CON0_PDN_SPDIF_CK);
+ return 0;
+ }
+ return -EINVAL;
+}
+
+static int mt2701_afe_hdmi_hw_free(struct snd_pcm_substream *substream,
+ struct snd_soc_dai *dai)
+{
+ struct mtk_base_afe *afe = snd_soc_dai_get_drvdata(dai);
+ struct mt2701_afe_private *afe_priv = afe->platform_priv;
+
+ mt2701_afe_i2s_path_disable(afe,
+ &afe_priv->i2s_path[MT2701_HDMI_I2S_PATH],
+ SNDRV_PCM_STREAM_PLAYBACK);
+ return 0;
+}
+
+static const struct snd_soc_dai_ops mt2701_afe_hdmi_ops = {
+ .startup = mt2701_afe_hdmi_startup,
+ .shutdown = mt2701_afe_hdmi_shutdown,
+ .hw_params = mt2701_afe_hdmi_hw_params,
+ .hw_free = mt2701_afe_hdmi_hw_free,
+ .trigger = mt2701_afe_hdmi_trigger,
+};
+
static struct snd_soc_dai_driver mt2701_afe_pcm_dais[] = {
/* FE DAIs: memory intefaces to CPU */
{
@@ -628,6 +844,19 @@ static struct snd_soc_dai_driver mt2701_afe_pcm_dais[] = {
},
.ops = &mt2701_single_memif_dai_ops,
},
+ {
+ .name = "PCM_HDMI",
+ .id = MT2701_MEMIF_HDMI,
+ .playback = {
+ .stream_name = "HDMI Multich",
+ .channels_min = 2,
+ .channels_max = 8,
+ .rates = (SNDRV_PCM_RATE_44100 |
+ SNDRV_PCM_RATE_48000),
+ .formats = SNDRV_PCM_FMTBIT_S16_LE,
+ },
+ .ops = &mt2701_single_memif_dai_ops,
+ },
/* BE DAIs */
{
.name = "I2S0",
@@ -748,7 +977,20 @@ static struct snd_soc_dai_driver mt2701_afe_pcm_dais[] = {
},
.ops = &mt2701_btmrg_ops,
.symmetric_rate = 1,
- }
+ },
+ {
+ .name = "HDMI I2S",
+ .id = MT2701_IO_HDMI,
+ .playback = {
+ .stream_name = "HDMI 8CH I2S Playback",
+ .channels_min = 2,
+ .channels_max = 8,
+ .rates = (SNDRV_PCM_RATE_44100 |
+ SNDRV_PCM_RATE_48000),
+ .formats = SNDRV_PCM_FMTBIT_S16_LE,
+ },
+ .ops = &mt2701_afe_hdmi_ops,
+ },
};
static const struct snd_kcontrol_new mt2701_afe_o00_mix[] = {
@@ -927,6 +1169,14 @@ static const struct snd_soc_dapm_route mt2701_afe_pcm_routes[] = {
{"I16I17", "Multich I2S2 Out Switch", "DLM"},
{"I18I19", "Multich I2S3 Out Switch", "DLM"},
+ /*
+ * HDMI FE -> BE direct route. The HDMI memif has its own DMA
+ * path that feeds the 8-channel internal I2S straight into the
+ * HDMI transmitter; no mixer/interconnect selection is exposed
+ * to the user.
+ */
+ {"HDMI 8CH I2S Playback", NULL, "HDMI Multich"},
+
{ "I12", NULL, "I12I13" },
{ "I13", NULL, "I12I13" },
{ "I14", NULL, "I14I15" },
@@ -1207,6 +1457,35 @@ static const struct mtk_base_memif_data memif_data_array[MT2701_MEMIF_NUM] = {
.agent_disable_shift = 16,
.msb_reg = -1,
},
+ {
+ /*
+ * HDMI memif feeds the on-SoC 8-channel internal I2S that
+ * drives the HDMI transmitter audio port. Unlike the
+ * standard memifs, the enable bit, channel count and bit
+ * width all live in AFE_HDMI_OUT_CON0, so mono/fs/hd/agent
+ * fields are left at -1 and programmed from the BE DAI ops
+ * instead.
+ */
+ .name = "HDMI",
+ .id = MT2701_MEMIF_HDMI,
+ .reg_ofs_base = AFE_HDMI_OUT_BASE,
+ .reg_ofs_cur = AFE_HDMI_OUT_CUR,
+ .reg_ofs_end = AFE_HDMI_OUT_END,
+ .fs_reg = -1,
+ .fs_shift = -1,
+ .fs_maskbit = 0,
+ .mono_reg = -1,
+ .mono_shift = -1,
+ .enable_reg = AFE_HDMI_OUT_CON0,
+ .enable_shift = 0,
+ .hd_reg = -1,
+ .hd_shift = -1,
+ .hd_align_reg = -1,
+ .hd_align_mshift = 0,
+ .agent_disable_reg = -1,
+ .agent_disable_shift = 0,
+ .msb_reg = -1,
+ },
};
static const struct mtk_base_irq_data irq_data[MT2701_IRQ_ASYS_END] = {
--
2.53.0
^ permalink raw reply related
* [PATCH 4/9] ASoC: mediatek: mt2701: add optional HDMI audio path clocks
From: Daniel Golle @ 2026-04-15 15:23 UTC (permalink / raw)
To: Liam Girdwood, Mark Brown, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Matthias Brugger, AngeloGioacchino Del Regno,
Jaroslav Kysela, Takashi Iwai, Cyril Chao, Arnd Bergmann,
Kuninori Morimoto, Daniel Golle, Nícolas F. R. A. Prado,
Eugen Hristev, linux-sound, devicetree, linux-kernel,
linux-arm-kernel, linux-mediatek
In-Reply-To: <cover.1776265610.git.daniel@makrotopia.org>
The HDMI audio output path on MT2701/MT7623N is rooted in HADDS2PLL
and gated by the audio_hdmi, audio_spdf and audio_apll power gates.
Acquire these four clocks from device tree using devm_clk_get_optional
so that existing platforms which do not wire up HDMI audio keep
probing unchanged. Actual clock enable/prepare is deferred to the
upcoming HDMI DAI startup path.
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
---
.../mediatek/mt2701/mt2701-afe-clock-ctrl.c | 22 +++++++++++++++++++
sound/soc/mediatek/mt2701/mt2701-afe-common.h | 4 ++++
2 files changed, 26 insertions(+)
diff --git a/sound/soc/mediatek/mt2701/mt2701-afe-clock-ctrl.c b/sound/soc/mediatek/mt2701/mt2701-afe-clock-ctrl.c
index ae620890bb3ac..5a2bcf027b4fb 100644
--- a/sound/soc/mediatek/mt2701/mt2701-afe-clock-ctrl.c
+++ b/sound/soc/mediatek/mt2701/mt2701-afe-clock-ctrl.c
@@ -95,6 +95,28 @@ int mt2701_init_clock(struct mtk_base_afe *afe)
afe_priv->mrgif_ck = NULL;
}
+ /*
+ * Optional HDMI audio clocks. Platforms that do not wire up the
+ * HDMI output (e.g. MT2701 devkits using only the I2S BE DAIs)
+ * may omit these; in that case the HDMI BE DAI simply cannot be
+ * enabled, but the rest of the AFE still probes.
+ */
+ afe_priv->hadds2pll_ck = devm_clk_get_optional(afe->dev, "hadds2pll_294m");
+ if (IS_ERR(afe_priv->hadds2pll_ck))
+ return PTR_ERR(afe_priv->hadds2pll_ck);
+
+ afe_priv->audio_hdmi_ck = devm_clk_get_optional(afe->dev, "audio_hdmi_pd");
+ if (IS_ERR(afe_priv->audio_hdmi_ck))
+ return PTR_ERR(afe_priv->audio_hdmi_ck);
+
+ afe_priv->audio_spdf_ck = devm_clk_get_optional(afe->dev, "audio_spdf_pd");
+ if (IS_ERR(afe_priv->audio_spdf_ck))
+ return PTR_ERR(afe_priv->audio_spdf_ck);
+
+ afe_priv->audio_apll_ck = devm_clk_get_optional(afe->dev, "audio_apll_pd");
+ if (IS_ERR(afe_priv->audio_apll_ck))
+ return PTR_ERR(afe_priv->audio_apll_ck);
+
return 0;
}
diff --git a/sound/soc/mediatek/mt2701/mt2701-afe-common.h b/sound/soc/mediatek/mt2701/mt2701-afe-common.h
index 32bef5e2a56d9..7b15283d6351e 100644
--- a/sound/soc/mediatek/mt2701/mt2701-afe-common.h
+++ b/sound/soc/mediatek/mt2701/mt2701-afe-common.h
@@ -90,6 +90,10 @@ struct mt2701_afe_private {
struct mt2701_i2s_path *i2s_path;
struct clk *base_ck[MT2701_BASE_CLK_NUM];
struct clk *mrgif_ck;
+ struct clk *hadds2pll_ck;
+ struct clk *audio_hdmi_ck;
+ struct clk *audio_spdf_ck;
+ struct clk *audio_apll_ck;
bool mrg_enable[MTK_STREAM_NUM];
const struct mt2701_soc_variants *soc;
--
2.53.0
^ permalink raw reply related
* [PATCH 3/9] ASoC: mediatek: mt2701: add AFE HDMI register definitions
From: Daniel Golle @ 2026-04-15 15:23 UTC (permalink / raw)
To: Liam Girdwood, Mark Brown, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Matthias Brugger, AngeloGioacchino Del Regno,
Jaroslav Kysela, Takashi Iwai, Cyril Chao, Arnd Bergmann,
Kuninori Morimoto, Daniel Golle, Nícolas F. R. A. Prado,
Eugen Hristev, linux-sound, devicetree, linux-kernel,
linux-arm-kernel, linux-mediatek
In-Reply-To: <cover.1776265610.git.daniel@makrotopia.org>
Add register offsets and bit defines for the MT2701/MT7623N AFE
HDMI audio output path: the HDMI BCK divider in AUDIO_TOP_CON3,
the HDMI output memif control and descriptor registers, the 8-bit
AFE_HDMI_CONN0 interconnect, and the AFE_8CH_I2S_OUT_CON engine
that drives the HDMI TX serial link. These are a prerequisite for
adding an HDMI playback path to the mt2701 AFE driver and have no
behavioural effect on their own.
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
---
sound/soc/mediatek/mt2701/mt2701-reg.h | 35 ++++++++++++++++++++++++++
1 file changed, 35 insertions(+)
diff --git a/sound/soc/mediatek/mt2701/mt2701-reg.h b/sound/soc/mediatek/mt2701/mt2701-reg.h
index c84d14cdd7ae8..b7a25bfb58662 100644
--- a/sound/soc/mediatek/mt2701/mt2701-reg.h
+++ b/sound/soc/mediatek/mt2701/mt2701-reg.h
@@ -10,10 +10,17 @@
#define _MT2701_REG_H_
#define AUDIO_TOP_CON0 0x0000
+#define AUDIO_TOP_CON3 0x000c
#define AUDIO_TOP_CON4 0x0010
#define AUDIO_TOP_CON5 0x0014
#define AFE_DAIBT_CON0 0x001c
#define AFE_MRGIF_CON 0x003c
+#define AFE_HDMI_OUT_CON0 0x0370
+#define AFE_HDMI_OUT_BASE 0x0374
+#define AFE_HDMI_OUT_CUR 0x0378
+#define AFE_HDMI_OUT_END 0x037c
+#define AFE_HDMI_CONN0 0x0390
+#define AFE_8CH_I2S_OUT_CON 0x0394
#define ASMI_TIMING_CON1 0x0100
#define ASMO_TIMING_CON1 0x0104
#define PWR1_ASM_CON1 0x0108
@@ -125,6 +132,34 @@
#define AFE_MEMIF_PBUF_SIZE_DLM_BYTE_MASK (0x3 << 12)
#define AFE_MEMIF_PBUF_SIZE_DLM_32BYTES (0x1 << 12)
+/* AUDIO_TOP_CON0 (0x0000) -- HDMI audio clock gating */
+#define AUDIO_TOP_CON0_PDN_HDMI_CK (0x1 << 20)
+#define AUDIO_TOP_CON0_PDN_SPDIF_CK (0x1 << 21)
+#define AUDIO_TOP_CON0_PDN_SPDIF2_CK (0x1 << 22)
+#define AUDIO_TOP_CON0_PDN_APLL_CK (0x1 << 23)
+
+/* AUDIO_TOP_CON3 (0x000c) -- HDMI BCK divider */
+#define AUDIO_TOP_CON3_HDMI_BCK_DIV_MASK (0x3f << 8)
+#define AUDIO_TOP_CON3_HDMI_BCK_DIV(x) (((x) & 0x3f) << 8)
+
+/* AFE_HDMI_OUT_CON0 (0x0370) */
+#define AFE_HDMI_OUT_CON0_OUT_ON (0x1 << 0)
+#define AFE_HDMI_OUT_CON0_BIT_WIDTH_MASK (0x1 << 1)
+#define AFE_HDMI_OUT_CON0_BIT_WIDTH_16 (0x0 << 1)
+#define AFE_HDMI_OUT_CON0_BIT_WIDTH_32 (0x1 << 1)
+#define AFE_HDMI_OUT_CON0_CH_NUM_MASK (0xf << 4)
+#define AFE_HDMI_OUT_CON0_CH_NUM(x) (((x) & 0xf) << 4)
+
+/* AFE_8CH_I2S_OUT_CON (0x0394) -- on-SoC 8-channel I2S that feeds HDMI TX */
+#define AFE_8CH_I2S_OUT_CON_EN (0x1 << 0)
+#define AFE_8CH_I2S_OUT_CON_BCK_INV (0x1 << 1)
+#define AFE_8CH_I2S_OUT_CON_LRCK_INV (0x1 << 2)
+#define AFE_8CH_I2S_OUT_CON_I2S_DELAY (0x1 << 3)
+#define AFE_8CH_I2S_OUT_CON_WLEN_MASK (0x3 << 4)
+#define AFE_8CH_I2S_OUT_CON_WLEN_16BIT (0x1 << 4)
+#define AFE_8CH_I2S_OUT_CON_WLEN_24BIT (0x2 << 4)
+#define AFE_8CH_I2S_OUT_CON_WLEN_32BIT (0x3 << 4)
+
/* I2S in/out register bit control */
#define ASYS_I2S_CON_FS (0x1f << 8)
#define ASYS_I2S_CON_FS_SET(x) ((x) << 8)
--
2.53.0
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox