* [RFC PATCH 1/6] lib/sbitmap: add ranged allocation, bounded wakeup relay, and ranged weight
2026-07-06 17:34 [RFC PATCH 0/6] sbitmap enforced fairness for blk-mq Keith Busch
@ 2026-07-06 17:34 ` Keith Busch
2026-07-06 17:34 ` [RFC PATCH 2/6] blk-mq: replace shared-tag fairness counter with allocation windows Keith Busch
` (5 subsequent siblings)
6 siblings, 0 replies; 9+ messages in thread
From: Keith Busch @ 2026-07-06 17:34 UTC (permalink / raw)
To: linux-block; +Cc: linux-scsi, axboe, hch, bvanassche, sumit.saxena, Keith Busch
From: Keith Busch <kbusch@kernel.org>
Add some helpers that let a caller carve a shared bitmap into per-user
windows.
* sbitmap_queue_get_range() allocates a free bit only within requested range.
* sbitmap_queue_wake_up_relay() hands a wakeup to another waiter when the
woken one cannot use the freed bit (it fell outside that waiter's window).
* sbitmap_weight_range() counts set (and not cleared) bits in a sub-range.
No callers yet. The intention is for blk-mq users for shared tag
fairness without requiring per-io atomic accounting operations.
Signed-off-by: Keith Busch <kbusch@kernel.org>
---
include/linux/sbitmap.h | 53 +++++++++++
lib/sbitmap.c | 200 ++++++++++++++++++++++++++++++++++++++++
2 files changed, 253 insertions(+)
diff --git a/include/linux/sbitmap.h b/include/linux/sbitmap.h
index cc7ad189caa5c..9fceda5d9de51 100644
--- a/include/linux/sbitmap.h
+++ b/include/linux/sbitmap.h
@@ -86,6 +86,7 @@ struct sbitmap {
#define SBQ_WAIT_QUEUES 8
#define SBQ_WAKE_BATCH 8
+#define SBQ_RELAY_BUDGET SBQ_WAKE_BATCH
/**
* struct sbq_wait_state - Wait queue in a &struct sbitmap_queue.
@@ -149,6 +150,14 @@ struct sbitmap_queue {
* @wakeup_cnt: Number of thread wake ups issued.
*/
atomic_t wakeup_cnt;
+
+ /**
+ * @relay_credits: Remaining range-relay wakeups allowed before the next
+ * genuine completion. Refilled by sbitmap_queue_wake_up() and consumed
+ * by sbitmap_queue_wake_up_relay(); bounds the relay chain so it cannot
+ * cycle when no waiter can use a freed bit.
+ */
+ atomic_t relay_credits;
};
/**
@@ -375,6 +384,18 @@ void sbitmap_show(struct sbitmap *sb, struct seq_file *m);
*/
unsigned int sbitmap_weight(const struct sbitmap *sb);
+/**
+ * sbitmap_weight_range() - Return how many set and not cleared bits in a
+ * sub-range of a &struct sbitmap.
+ * @sb: Bitmap to check.
+ * @off: First bit of the range.
+ * @len: Number of bits in the range.
+ *
+ * Return: How many bits in [@off, @off + @len) are set and not cleared.
+ */
+unsigned int sbitmap_weight_range(const struct sbitmap *sb, unsigned int off,
+ unsigned int len);
+
/**
* sbitmap_bitmap_show() - Write a hex dump of a &struct sbitmap to a &struct
* seq_file.
@@ -472,6 +493,22 @@ unsigned long __sbitmap_queue_get_batch(struct sbitmap_queue *sbq, int nr_tags,
int sbitmap_queue_get_shallow(struct sbitmap_queue *sbq,
unsigned int shallow_depth);
+/**
+ * sbitmap_queue_get_range() - Try to allocate a free bit from a restricted
+ * range of a &struct sbitmap_queue, with preemption already disabled.
+ * @sbq: Bitmap queue to allocate from.
+ * @min: First bit (inclusive) the caller is allowed to allocate.
+ * @max: Last bit (exclusive) the caller is allowed to allocate.
+ *
+ * Allocation is confined to the [@min, @max) window. This lets several users
+ * share one bitmap while each is bounded to its own (possibly overlapping)
+ * window, providing fairness without a separate per-user accounting counter.
+ *
+ * Return: Non-negative allocated bit number if successful, -1 otherwise.
+ */
+int sbitmap_queue_get_range(struct sbitmap_queue *sbq, unsigned int min,
+ unsigned int max);
+
/**
* sbitmap_queue_get() - Try to allocate a free bit from a &struct
* sbitmap_queue.
@@ -575,6 +612,22 @@ void sbitmap_queue_wake_all(struct sbitmap_queue *sbq);
*/
void sbitmap_queue_wake_up(struct sbitmap_queue *sbq, int nr);
+/**
+ * sbitmap_queue_wake_up_relay() - Relay a wakeup to the next waiter.
+ * @sbq: Bitmap queue to wake up.
+ * @cur: The relaying waiter's own wait queue, skipped so it does not wake
+ * itself (it is about to sleep again); may be NULL.
+ *
+ * Wake a single waiter on the next active wait queue without touching the
+ * wake_batch accounting. Intended for range-limited allocation: a waiter that
+ * was woken but could not use the freed bit (it fell outside the waiter's
+ * allowed range) calls this to pass the wakeup along to another waiter that
+ * may be able to use it. The relay is bounded by a credit budget refilled
+ * only by genuine completions, so it cannot cycle indefinitely.
+ */
+void sbitmap_queue_wake_up_relay(struct sbitmap_queue *sbq,
+ struct sbq_wait_state *cur);
+
/**
* sbitmap_queue_show() - Dump &struct sbitmap_queue information to a &struct
* seq_file.
diff --git a/lib/sbitmap.c b/lib/sbitmap.c
index 4d188d05db153..3264085b4d533 100644
--- a/lib/sbitmap.c
+++ b/lib/sbitmap.c
@@ -338,6 +338,130 @@ static int sbitmap_get_shallow(struct sbitmap *sb, unsigned long shallow_depth)
return nr;
}
+/*
+ * Find and set a free bit in the [low, high) sub-range of a single word.
+ * Wraps back to @low once so a free bit below @hint is still found.
+ */
+static int __sbitmap_get_word_range(unsigned long *word, unsigned int low,
+ unsigned int high, unsigned int hint)
+{
+ bool wrap;
+ int nr;
+
+ if (hint < low || hint >= high)
+ hint = low;
+ /* don't wrap if already starting from the bottom of the range */
+ wrap = hint > low;
+
+ while (1) {
+ nr = find_next_zero_bit(word, high, hint);
+ if (unlikely(nr >= high)) {
+ if (wrap) {
+ hint = low;
+ wrap = false;
+ continue;
+ }
+ return -1;
+ }
+ if (!test_and_set_bit_lock(nr, word))
+ break;
+ hint = nr + 1;
+ if (hint >= high) {
+ if (!wrap)
+ return -1;
+ hint = low;
+ wrap = false;
+ }
+ }
+
+ return nr;
+}
+
+static int sbitmap_find_bit_in_word_range(struct sbitmap_word *map,
+ unsigned int low, unsigned int high,
+ unsigned int hint)
+{
+ int nr;
+
+ do {
+ nr = __sbitmap_get_word_range(&map->word, low, high, hint);
+ if (nr != -1)
+ break;
+ if (!sbitmap_deferred_clear(map, high, low, false))
+ break;
+ } while (1);
+
+ return nr;
+}
+
+/*
+ * Allocate a bit restricted to the [min, max) range of the bitmap. Used to
+ * carve the shared depth into (possibly overlapping) per-user windows without
+ * a separate accounting counter: a user simply cannot allocate outside its
+ * window.
+ */
+static int __sbitmap_get_range(struct sbitmap *sb, unsigned int min,
+ unsigned int max, unsigned int hint)
+{
+ unsigned int min_index, max_index, index;
+ unsigned int nwords, i;
+ int nr = -1;
+
+ if (max > sb->depth)
+ max = sb->depth;
+ if (min >= max)
+ return -1;
+
+ min_index = SB_NR_TO_INDEX(sb, min);
+ max_index = SB_NR_TO_INDEX(sb, max - 1);
+ nwords = max_index - min_index + 1;
+
+ if (hint < min || hint >= max)
+ hint = min;
+ index = SB_NR_TO_INDEX(sb, hint);
+
+ for (i = 0; i < nwords; i++) {
+ unsigned int word_base = index << sb->shift;
+ unsigned int word_depth = __map_depth(sb, index);
+ unsigned int low, high, h;
+
+ low = (index == min_index) ? SB_NR_TO_BIT(sb, min) : 0;
+ high = (index == max_index) ? SB_NR_TO_BIT(sb, max - 1) + 1 :
+ word_depth;
+ h = (hint >= word_base && hint < word_base + word_depth) ?
+ SB_NR_TO_BIT(sb, hint) : low;
+
+ nr = sbitmap_find_bit_in_word_range(&sb->map[index], low,
+ high, h);
+ if (nr != -1) {
+ nr += word_base;
+ break;
+ }
+
+ if (++index > max_index)
+ index = min_index;
+ }
+
+ return nr;
+}
+
+static int sbitmap_get_range(struct sbitmap *sb, unsigned int min,
+ unsigned int max)
+{
+ int nr;
+ unsigned int hint, depth;
+
+ if (WARN_ON_ONCE(unlikely(!sb->alloc_hint)))
+ return -1;
+
+ depth = READ_ONCE(sb->depth);
+ hint = update_alloc_hint_before_get(sb, depth);
+ nr = __sbitmap_get_range(sb, min, max, hint);
+ update_alloc_hint_after_get(sb, depth, hint, nr);
+
+ return nr;
+}
+
bool sbitmap_any_bit_set(const struct sbitmap *sb)
{
unsigned int i;
@@ -377,6 +501,38 @@ unsigned int sbitmap_weight(const struct sbitmap *sb)
}
EXPORT_SYMBOL_GPL(sbitmap_weight);
+unsigned int sbitmap_weight_range(const struct sbitmap *sb, unsigned int off,
+ unsigned int len)
+{
+ unsigned int weight = 0;
+ unsigned int end = off + len;
+ unsigned int index;
+
+ if (end > sb->depth)
+ end = sb->depth;
+
+ for (index = SB_NR_TO_INDEX(sb, off); index < sb->map_nr; index++) {
+ const struct sbitmap_word *word = &sb->map[index];
+ unsigned int base = index << sb->shift;
+ unsigned int wbits = __map_depth(sb, index);
+ unsigned int lo, hi;
+ unsigned long val;
+
+ if (base >= end)
+ break;
+ val = READ_ONCE(word->word) & ~READ_ONCE(word->cleared);
+ lo = (off > base) ? off - base : 0;
+ hi = (end - base < wbits) ? end - base : wbits;
+ if (lo)
+ val &= ~((1UL << lo) - 1);
+ if (hi < BITS_PER_LONG)
+ val &= (1UL << hi) - 1;
+ weight += hweight_long(val);
+ }
+ return weight;
+}
+EXPORT_SYMBOL_GPL(sbitmap_weight_range);
+
void sbitmap_show(struct sbitmap *sb, struct seq_file *m)
{
seq_printf(m, "depth=%u\n", sb->depth);
@@ -462,6 +618,7 @@ int sbitmap_queue_init_node(struct sbitmap_queue *sbq, unsigned int depth,
atomic_set(&sbq->ws_active, 0);
atomic_set(&sbq->completion_cnt, 0);
atomic_set(&sbq->wakeup_cnt, 0);
+ atomic_set(&sbq->relay_credits, 0);
sbq->ws = kzalloc_node(SBQ_WAIT_QUEUES * sizeof(*sbq->ws), flags, node);
if (!sbq->ws) {
@@ -573,6 +730,13 @@ int sbitmap_queue_get_shallow(struct sbitmap_queue *sbq,
}
EXPORT_SYMBOL_GPL(sbitmap_queue_get_shallow);
+int sbitmap_queue_get_range(struct sbitmap_queue *sbq, unsigned int min,
+ unsigned int max)
+{
+ return sbitmap_get_range(&sbq->sb, min, max);
+}
+EXPORT_SYMBOL_GPL(sbitmap_queue_get_range);
+
void sbitmap_queue_min_shallow_depth(struct sbitmap_queue *sbq,
unsigned int min_shallow_depth)
{
@@ -629,10 +793,46 @@ void sbitmap_queue_wake_up(struct sbitmap_queue *sbq, int nr)
} while (!atomic_try_cmpxchg(&sbq->wakeup_cnt,
&wakeups, wakeups + wake_batch));
+ atomic_set(&sbq->relay_credits, SBQ_RELAY_BUDGET);
__sbitmap_queue_wake_up(sbq, wake_batch);
}
EXPORT_SYMBOL_GPL(sbitmap_queue_wake_up);
+/*
+ * Wake a single waiter on the next active wait queue, bypassing the
+ * wake_batch accounting (no bit was freed). This "relays" a wakeup that the
+ * current waiter consumed but could not use - e.g. the freed bit fell outside
+ * its allowed allocation range - so a waiter that *can* use a free bit gets a
+ * chance to run. Relaying continues, one hop per failed waiter, until a usable
+ * waiter is found or the chain credits drain.
+ */
+void sbitmap_queue_wake_up_relay(struct sbitmap_queue *sbq,
+ struct sbq_wait_state *cur)
+{
+ int wake_index, i;
+
+ if (!atomic_read(&sbq->ws_active))
+ return;
+
+ if (atomic_dec_if_positive(&sbq->relay_credits) < 0)
+ return;
+
+ wake_index = atomic_read(&sbq->wake_index);
+ for (i = 0; i < SBQ_WAIT_QUEUES; i++) {
+ struct sbq_wait_state *ws = &sbq->ws[wake_index];
+
+ wake_index = sbq_index_inc(wake_index);
+ if (ws != cur && waitqueue_active(&ws->wait)) {
+ wake_up_nr(&ws->wait, 1);
+ break;
+ }
+ }
+
+ if (wake_index != atomic_read(&sbq->wake_index))
+ atomic_set(&sbq->wake_index, wake_index);
+}
+EXPORT_SYMBOL_GPL(sbitmap_queue_wake_up_relay);
+
static inline void sbitmap_update_cpu_hint(struct sbitmap *sb, int cpu, int tag)
{
if (likely(!sb->round_robin && tag < sb->depth))
--
2.52.0
^ permalink raw reply related [flat|nested] 9+ messages in thread* [RFC PATCH 2/6] blk-mq: replace shared-tag fairness counter with allocation windows
2026-07-06 17:34 [RFC PATCH 0/6] sbitmap enforced fairness for blk-mq Keith Busch
2026-07-06 17:34 ` [RFC PATCH 1/6] lib/sbitmap: add ranged allocation, bounded wakeup relay, and ranged weight Keith Busch
@ 2026-07-06 17:34 ` Keith Busch
2026-07-06 17:34 ` [RFC PATCH 3/6] blk-mq: factor out a per-hctx tag busy iterator Keith Busch
` (4 subsequent siblings)
6 siblings, 0 replies; 9+ messages in thread
From: Keith Busch @ 2026-07-06 17:34 UTC (permalink / raw)
To: linux-block; +Cc: linux-scsi, axboe, hch, bvanassche, sumit.saxena, Keith Busch
From: Keith Busch <kbusch@kernel.org>
The per-request atomic active counter that hctx_may_queue() checks on
every shared-tag allocation is costly under contention. Instead, confine
each active queue to a slice of the shared bitmap and allocate from
that, so one busy queue cannot starve the others. This removes per-IO
atomic on the hot path. Fairness becomes structural to the sbitmap
allocation range, so the counter and hctx_may_queue are removed.
Link: https://lore.kernel.org/linux-block/20240529213921.3166462-1-bvanassche@acm.org/
Link: https://lore.kernel.org/linux-block/20260609121806.2121755-1-sumit.saxena@broadcom.com/
Signed-off-by: Keith Busch <kbusch@kernel.org>
---
block/blk-core.c | 2 +-
block/blk-mq-debugfs.c | 2 +-
block/blk-mq-tag.c | 113 +++++++++++++++++++++++++++++++++++++++--
block/blk-mq.c | 21 ++------
block/blk-mq.h | 102 +------------------------------------
include/linux/blk-mq.h | 15 ++++--
include/linux/blkdev.h | 6 ++-
7 files changed, 135 insertions(+), 126 deletions(-)
diff --git a/block/blk-core.c b/block/blk-core.c
index 365641266c9e8..7e719b90d8a66 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -464,7 +464,7 @@ struct request_queue *blk_alloc_queue(struct queue_limits *lim, int node_id)
q->node = node_id;
- atomic_set(&q->nr_active_requests_shared_tags, 0);
+ q->tag_win_slot = -1;
timer_setup(&q->timeout, blk_rq_timed_out_timer, 0);
INIT_WORK(&q->timeout_work, blk_timeout_work);
diff --git a/block/blk-mq-debugfs.c b/block/blk-mq-debugfs.c
index 6754d8f9449c1..4b47a3322ff72 100644
--- a/block/blk-mq-debugfs.c
+++ b/block/blk-mq-debugfs.c
@@ -480,7 +480,7 @@ static int hctx_active_show(void *data, struct seq_file *m)
{
struct blk_mq_hw_ctx *hctx = data;
- seq_printf(m, "%d\n", __blk_mq_active_requests(hctx));
+ seq_printf(m, "%u\n", blk_mq_hctx_active(hctx));
return 0;
}
diff --git a/block/blk-mq-tag.c b/block/blk-mq-tag.c
index 35deee5bbc739..0dd497225c74a 100644
--- a/block/blk-mq-tag.c
+++ b/block/blk-mq-tag.c
@@ -33,6 +33,18 @@ static void blk_mq_update_wake_batch(struct blk_mq_tags *tags,
users);
}
+/*
+ * Where this queue's fairness allocation-window slot is stored. HCTX_SHARED
+ * tag sets divide one global bitmap among request_queues (slot per queue);
+ * QUEUE_SHARED tag sets divide each per-hw-queue bitmap among the hctxs
+ * sharing it (slot per hctx).
+ */
+static int *blk_mq_tag_win_slot(struct blk_mq_hw_ctx *hctx)
+{
+ return blk_mq_is_shared_tags(hctx->flags) ?
+ &hctx->queue->tag_win_slot : &hctx->tag_win_slot;
+}
+
/*
* If a previously inactive queue goes active, bump the active user count.
* We need to do this before try to allocate driver tag, then even if fail
@@ -65,6 +77,15 @@ void __blk_mq_tag_busy(struct blk_mq_hw_ctx *hctx)
users = tags->active_queues + 1;
WRITE_ONCE(tags->active_queues, users);
blk_mq_update_wake_batch(tags, users);
+ if (tags->active_slots) {
+ unsigned int nbits = tags->bitmap_tags.sb.depth;
+ unsigned int slot = find_first_zero_bit(tags->active_slots, nbits);
+
+ if (slot < nbits) {
+ set_bit(slot, tags->active_slots);
+ WRITE_ONCE(*blk_mq_tag_win_slot(hctx), slot);
+ }
+ }
spin_unlock_irqrestore(&tags->lock, flags);
}
@@ -102,17 +123,82 @@ void __blk_mq_tag_idle(struct blk_mq_hw_ctx *hctx)
users = tags->active_queues - 1;
WRITE_ONCE(tags->active_queues, users);
blk_mq_update_wake_batch(tags, users);
+ if (tags->active_slots) {
+ int *slotp = blk_mq_tag_win_slot(hctx);
+ int slot = READ_ONCE(*slotp);
+
+ if (slot >= 0) {
+ clear_bit(slot, tags->active_slots);
+ WRITE_ONCE(*slotp, -1);
+ }
+ }
spin_unlock_irq(&tags->lock);
blk_mq_tag_wakeup_all(tags, false);
}
+static inline bool blk_mq_tag_is_windowed(struct blk_mq_alloc_data *data)
+{
+ return !data->q->elevator &&
+ !(data->flags & BLK_MQ_REQ_RESERVED) &&
+ (data->hctx->flags & BLK_MQ_F_TAG_QUEUE_SHARED);
+}
+
+static bool blk_mq_tag_active_window(struct blk_mq_hw_ctx *hctx,
+ unsigned int depth, unsigned int *min,
+ unsigned int *max)
+{
+ unsigned int users = READ_ONCE(hctx->tags->active_queues);
+ int slot = READ_ONCE(*blk_mq_tag_win_slot(hctx));
+ unsigned int p, rem, pos;
+
+ if (users <= 1 || slot < 0 || depth <= 1)
+ return false;
+
+ p = depth / users;
+ if (!p)
+ return false;
+
+ /*
+ * Slots are allocated from a bitmap and freed out of order, so a slot
+ * index can exceed the active-user count once lower slots are freed.
+ * Convert it to a position based on how many active slots precede it.
+ */
+ pos = bitmap_weight(hctx->tags->active_slots, slot);
+ rem = depth - p * users;
+ *min = pos * p;
+ *max = pos * p + p + rem;
+ if (*max > depth)
+ *max = depth;
+
+ return true;
+}
+
+int blk_mq_get_tag_window(struct blk_mq_hw_ctx *hctx, struct sbitmap_queue *bt)
+{
+ unsigned int min, max;
+
+ if (blk_mq_tag_active_window(hctx, bt->sb.depth, &min, &max))
+ return sbitmap_queue_get_range(bt, min, max);
+ return __sbitmap_queue_get(bt);
+
+}
+
+unsigned int blk_mq_hctx_active(struct blk_mq_hw_ctx *hctx)
+{
+ struct sbitmap *sb = &hctx->tags->bitmap_tags.sb;
+ unsigned int min, max;
+
+ if (!blk_mq_tag_active_window(hctx, sb->depth, &min, &max))
+ return sbitmap_weight(sb);
+ return sbitmap_weight_range(sb, min, max - min);
+}
+
static int __blk_mq_get_tag(struct blk_mq_alloc_data *data,
struct sbitmap_queue *bt)
{
- if (!data->q->elevator && !(data->flags & BLK_MQ_REQ_RESERVED) &&
- !hctx_may_queue(data->hctx, bt))
- return BLK_MQ_NO_TAG;
+ if (blk_mq_tag_is_windowed(data))
+ return blk_mq_get_tag_window(data->hctx, bt);
if (data->shallow_depth)
return sbitmap_queue_get_shallow(bt, data->shallow_depth);
@@ -142,6 +228,7 @@ unsigned int blk_mq_get_tag(struct blk_mq_alloc_data *data)
struct sbq_wait_state *ws;
DEFINE_SBQ_WAIT(wait);
unsigned int tag_offset;
+ bool slept = false;
int tag;
if (data->flags & BLK_MQ_REQ_RESERVED) {
@@ -193,8 +280,17 @@ unsigned int blk_mq_get_tag(struct blk_mq_alloc_data *data)
if (tag != BLK_MQ_NO_TAG)
break;
+ /*
+ * We were previously woken but still cannot allocate within our
+ * window: the freed bit belonged to another queue's window.
+ * Relay the wakeup so a waiter that can use it gets to run.
+ */
+ if (slept && blk_mq_tag_is_windowed(data))
+ sbitmap_queue_wake_up_relay(bt, ws);
+
bt_prev = bt;
io_schedule();
+ slept = true;
sbitmap_finish_wait(bt, ws, &wait);
@@ -574,8 +670,14 @@ struct blk_mq_tags *blk_mq_init_tags(unsigned int total_tags,
spin_lock_init(&tags->lock);
INIT_LIST_HEAD(&tags->page_list);
+ if (depth) {
+ tags->active_slots = bitmap_zalloc(depth, GFP_KERNEL);
+ if (!tags->active_slots)
+ goto out_free_tags;
+ }
+
if (bt_alloc(&tags->bitmap_tags, depth, round_robin, node))
- goto out_free_tags;
+ goto out_free_slots;
if (bt_alloc(&tags->breserved_tags, reserved_tags, round_robin, node))
goto out_free_bitmap_tags;
@@ -583,6 +685,8 @@ struct blk_mq_tags *blk_mq_init_tags(unsigned int total_tags,
out_free_bitmap_tags:
sbitmap_queue_free(&tags->bitmap_tags);
+out_free_slots:
+ bitmap_free(tags->active_slots);
out_free_tags:
kfree(tags);
return NULL;
@@ -611,6 +715,7 @@ void blk_mq_free_tags(struct blk_mq_tag_set *set, struct blk_mq_tags *tags)
{
sbitmap_queue_free(&tags->bitmap_tags);
sbitmap_queue_free(&tags->breserved_tags);
+ bitmap_free(tags->active_slots);
/* if tags pages is not allocated yet, free tags directly */
if (list_empty(&tags->page_list)) {
diff --git a/block/blk-mq.c b/block/blk-mq.c
index 2c850330a32bc..56ab6ac5ec696 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -489,8 +489,6 @@ __blk_mq_alloc_requests_batch(struct blk_mq_alloc_data *data)
}
} while (data->nr_tags > nr);
- if (!(data->rq_flags & RQF_SCHED_TAGS))
- blk_mq_add_active_requests(data->hctx, nr);
/* caller already holds a reference, add for remainder */
percpu_ref_get_many(&data->q->q_usage_counter, nr - 1);
data->nr_tags -= nr;
@@ -587,8 +585,6 @@ static struct request *__blk_mq_alloc_requests(struct blk_mq_alloc_data *data)
goto retry;
}
- if (!(data->rq_flags & RQF_SCHED_TAGS))
- blk_mq_inc_active_requests(data->hctx);
rq = blk_mq_rq_ctx_init(data, blk_mq_tags_from_data(data), tag);
blk_mq_rq_time_init(rq, alloc_time_ns);
return rq;
@@ -763,8 +759,6 @@ struct request *blk_mq_alloc_request_hctx(struct request_queue *q,
tag = blk_mq_get_tag(&data);
if (tag == BLK_MQ_NO_TAG)
goto out_queue_exit;
- if (!(data.rq_flags & RQF_SCHED_TAGS))
- blk_mq_inc_active_requests(data.hctx);
rq = blk_mq_rq_ctx_init(&data, blk_mq_tags_from_data(&data), tag);
blk_mq_rq_time_init(rq, alloc_time_ns);
rq->__data_len = 0;
@@ -807,10 +801,8 @@ static void __blk_mq_free_request(struct request *rq)
blk_pm_mark_last_busy(rq);
rq->mq_hctx = NULL;
- if (rq->tag != BLK_MQ_NO_TAG) {
- blk_mq_dec_active_requests(hctx);
+ if (rq->tag != BLK_MQ_NO_TAG)
blk_mq_put_tag(hctx->tags, ctx, rq->tag);
- }
if (sched_tag != BLK_MQ_NO_TAG)
blk_mq_put_tag(hctx->sched_tags, ctx, sched_tag);
blk_mq_sched_restart(hctx);
@@ -1157,8 +1149,6 @@ static inline void blk_mq_flush_tag_batch(struct blk_mq_hw_ctx *hctx,
{
struct request_queue *q = hctx->queue;
- blk_mq_sub_active_requests(hctx, nr_tags);
-
blk_mq_put_tags(hctx->tags, tag_array, nr_tags);
percpu_ref_put_many(&q->q_usage_counter, nr_tags);
}
@@ -1844,17 +1834,16 @@ bool __blk_mq_alloc_driver_tag(struct request *rq)
if (blk_mq_tag_is_reserved(rq->mq_hctx->sched_tags, rq->internal_tag)) {
bt = &rq->mq_hctx->tags->breserved_tags;
tag_offset = 0;
+ tag = __sbitmap_queue_get(bt);
} else {
- if (!hctx_may_queue(rq->mq_hctx, bt))
- return false;
+ /* Fairness on the shared driver tags via the allocation window. */
+ tag = blk_mq_get_tag_window(rq->mq_hctx, bt);
}
- tag = __sbitmap_queue_get(bt);
if (tag == BLK_MQ_NO_TAG)
return false;
rq->tag = tag + tag_offset;
- blk_mq_inc_active_requests(rq->mq_hctx);
return true;
}
@@ -4028,7 +4017,7 @@ blk_mq_alloc_hctx(struct request_queue *q, struct blk_mq_tag_set *set,
if (!zalloc_cpumask_var_node(&hctx->cpumask, gfp, node))
goto free_hctx;
- atomic_set(&hctx->nr_active, 0);
+ hctx->tag_win_slot = -1;
if (node == NUMA_NO_NODE)
node = set->numa_node;
hctx->numa_node = node;
diff --git a/block/blk-mq.h b/block/blk-mq.h
index aa15d31aaae9b..920bef5e7ce6e 100644
--- a/block/blk-mq.h
+++ b/block/blk-mq.h
@@ -205,6 +205,8 @@ static inline struct sbq_wait_state *bt_wait_ptr(struct sbitmap_queue *bt,
void __blk_mq_tag_busy(struct blk_mq_hw_ctx *);
void __blk_mq_tag_idle(struct blk_mq_hw_ctx *);
+unsigned int blk_mq_hctx_active(struct blk_mq_hw_ctx *hctx);
+int blk_mq_get_tag_window(struct blk_mq_hw_ctx *hctx, struct sbitmap_queue *bt);
static inline void blk_mq_tag_busy(struct blk_mq_hw_ctx *hctx)
{
@@ -291,70 +293,9 @@ static inline int blk_mq_get_rq_budget_token(struct request *rq)
return -1;
}
-static inline void __blk_mq_add_active_requests(struct blk_mq_hw_ctx *hctx,
- int val)
-{
- if (blk_mq_is_shared_tags(hctx->flags))
- atomic_add(val, &hctx->queue->nr_active_requests_shared_tags);
- else
- atomic_add(val, &hctx->nr_active);
-}
-
-static inline void __blk_mq_inc_active_requests(struct blk_mq_hw_ctx *hctx)
-{
- __blk_mq_add_active_requests(hctx, 1);
-}
-
-static inline void __blk_mq_sub_active_requests(struct blk_mq_hw_ctx *hctx,
- int val)
-{
- if (blk_mq_is_shared_tags(hctx->flags))
- atomic_sub(val, &hctx->queue->nr_active_requests_shared_tags);
- else
- atomic_sub(val, &hctx->nr_active);
-}
-
-static inline void __blk_mq_dec_active_requests(struct blk_mq_hw_ctx *hctx)
-{
- __blk_mq_sub_active_requests(hctx, 1);
-}
-
-static inline void blk_mq_add_active_requests(struct blk_mq_hw_ctx *hctx,
- int val)
-{
- if (hctx->flags & BLK_MQ_F_TAG_QUEUE_SHARED)
- __blk_mq_add_active_requests(hctx, val);
-}
-
-static inline void blk_mq_inc_active_requests(struct blk_mq_hw_ctx *hctx)
-{
- if (hctx->flags & BLK_MQ_F_TAG_QUEUE_SHARED)
- __blk_mq_inc_active_requests(hctx);
-}
-
-static inline void blk_mq_sub_active_requests(struct blk_mq_hw_ctx *hctx,
- int val)
-{
- if (hctx->flags & BLK_MQ_F_TAG_QUEUE_SHARED)
- __blk_mq_sub_active_requests(hctx, val);
-}
-
-static inline void blk_mq_dec_active_requests(struct blk_mq_hw_ctx *hctx)
-{
- if (hctx->flags & BLK_MQ_F_TAG_QUEUE_SHARED)
- __blk_mq_dec_active_requests(hctx);
-}
-
-static inline int __blk_mq_active_requests(struct blk_mq_hw_ctx *hctx)
-{
- if (blk_mq_is_shared_tags(hctx->flags))
- return atomic_read(&hctx->queue->nr_active_requests_shared_tags);
- return atomic_read(&hctx->nr_active);
-}
static inline void __blk_mq_put_driver_tag(struct blk_mq_hw_ctx *hctx,
struct request *rq)
{
- blk_mq_dec_active_requests(hctx);
blk_mq_put_tag(hctx->tags, rq->mq_ctx, rq->tag);
rq->tag = BLK_MQ_NO_TAG;
}
@@ -396,45 +337,6 @@ static inline void blk_mq_free_requests(struct list_head *list)
}
}
-/*
- * For shared tag users, we track the number of currently active users
- * and attempt to provide a fair share of the tag depth for each of them.
- */
-static inline bool hctx_may_queue(struct blk_mq_hw_ctx *hctx,
- struct sbitmap_queue *bt)
-{
- unsigned int depth, users;
-
- if (!hctx || !(hctx->flags & BLK_MQ_F_TAG_QUEUE_SHARED))
- return true;
-
- /*
- * Don't try dividing an ant
- */
- if (bt->sb.depth == 1)
- return true;
-
- if (blk_mq_is_shared_tags(hctx->flags)) {
- struct request_queue *q = hctx->queue;
-
- if (!test_bit(QUEUE_FLAG_HCTX_ACTIVE, &q->queue_flags))
- return true;
- } else {
- if (!test_bit(BLK_MQ_S_TAG_ACTIVE, &hctx->state))
- return true;
- }
-
- users = READ_ONCE(hctx->tags->active_queues);
- if (!users)
- return true;
-
- /*
- * Allow at least some tags
- */
- depth = max((bt->sb.depth + users - 1) / users, 4U);
- return __blk_mq_active_requests(hctx) < depth;
-}
-
/* run the code block in @dispatch_ops with rcu/srcu read lock held */
#define __blk_mq_run_dispatch_ops(q, check_sleep, dispatch_ops) \
do { \
diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h
index af878597afb8c..22cc09d5ef320 100644
--- a/include/linux/blk-mq.h
+++ b/include/linux/blk-mq.h
@@ -433,10 +433,12 @@ struct blk_mq_hw_ctx {
unsigned int queue_num;
/**
- * @nr_active: Number of active requests. Only used when a tag set is
- * shared across request queues.
+ * @tag_win_slot: Assigned fairness allocation-window slot within this
+ * hctx's tags, or -1 if none. Used for non-HCTX_SHARED shared tag sets
+ * (the per-hctx analogue of request_queue.tag_win_slot). Protected by
+ * the tags->lock.
*/
- atomic_t nr_active;
+ int tag_win_slot;
/** @cpuhp_online: List to store request if CPU is going to die */
struct hlist_node cpuhp_online;
@@ -776,6 +778,13 @@ struct blk_mq_tags {
unsigned int nr_reserved_tags;
unsigned int active_queues;
+ /*
+ * Bitmap of assigned per-queue fairness window slots, @nr_tags -
+ * @nr_reserved_tags bits wide (windowing is moot once active users
+ * exceed that). Protected by @lock.
+ */
+ unsigned long *active_slots;
+
struct sbitmap_queue bitmap_tags;
struct sbitmap_queue breserved_tags;
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index 9213a5716f95a..86b16bd8b9c17 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -573,7 +573,11 @@ struct request_queue {
struct timer_list timeout;
struct work_struct timeout_work;
- atomic_t nr_active_requests_shared_tags;
+ /*
+ * Assigned allocation-window slot for shared (HCTX_SHARED) tag-set
+ * fairness, or -1 if none. Protected by the shared tags->lock.
+ */
+ int tag_win_slot;
struct blk_mq_tags *sched_shared_tags;
--
2.52.0
^ permalink raw reply related [flat|nested] 9+ messages in thread* [RFC PATCH 3/6] blk-mq: factor out a per-hctx tag busy iterator
2026-07-06 17:34 [RFC PATCH 0/6] sbitmap enforced fairness for blk-mq Keith Busch
2026-07-06 17:34 ` [RFC PATCH 1/6] lib/sbitmap: add ranged allocation, bounded wakeup relay, and ranged weight Keith Busch
2026-07-06 17:34 ` [RFC PATCH 2/6] blk-mq: replace shared-tag fairness counter with allocation windows Keith Busch
@ 2026-07-06 17:34 ` Keith Busch
2026-07-06 17:34 ` [RFC PATCH 4/6] blk-mq: add a shared zone to tag fairness Keith Busch
` (3 subsequent siblings)
6 siblings, 0 replies; 9+ messages in thread
From: Keith Busch @ 2026-07-06 17:34 UTC (permalink / raw)
To: linux-block; +Cc: linux-scsi, axboe, hch, bvanassche, sumit.saxena, Keith Busch
From: Keith Busch <kbusch@kernel.org>
Make a helper for walking the busy tags of specific hardware queue based
on the per-hctx walk into __blk_mq_hctx_tag_busy_iter() so it can be
reused to iterate a single hardware queue's tags. No functional change.
Signed-off-by: Keith Busch <kbusch@kernel.org>
---
block/blk-mq-tag.c | 40 ++++++++++++++++++++++++----------------
1 file changed, 24 insertions(+), 16 deletions(-)
diff --git a/block/blk-mq-tag.c b/block/blk-mq-tag.c
index 0dd497225c74a..58cf480df9c69 100644
--- a/block/blk-mq-tag.c
+++ b/block/blk-mq-tag.c
@@ -582,6 +582,28 @@ void blk_mq_tagset_wait_completed_request(struct blk_mq_tag_set *tagset)
}
EXPORT_SYMBOL(blk_mq_tagset_wait_completed_request);
+static void __blk_mq_hctx_tag_busy_iter(struct blk_mq_hw_ctx *hctx,
+ busy_tag_iter_fn *fn, void *priv)
+{
+ struct blk_mq_tags *tags = hctx->tags;
+
+ /*
+ * If no software queues are currently mapped to this hardware queue,
+ * there's nothing to check
+ */
+ if (!blk_mq_hw_queue_mapped(hctx))
+ return;
+
+ if (tags->nr_reserved_tags)
+ bt_for_each(hctx, hctx->queue, &tags->breserved_tags, fn, priv, true);
+ bt_for_each(hctx, hctx->queue, &tags->bitmap_tags, fn, priv, false);
+}
+
/**
* blk_mq_queue_tag_busy_iter - iterate over all requests with a driver tag
* @q: Request queue to examine.
@@ -621,22 +643,8 @@ void blk_mq_queue_tag_busy_iter(struct request_queue *q, busy_tag_iter_fn *fn,
struct blk_mq_hw_ctx *hctx;
unsigned long i;
- queue_for_each_hw_ctx(q, hctx, i) {
- struct blk_mq_tags *tags = hctx->tags;
- struct sbitmap_queue *bresv = &tags->breserved_tags;
- struct sbitmap_queue *btags = &tags->bitmap_tags;
-
- /*
- * If no software queues are currently mapped to this
- * hardware queue, there's nothing to check
- */
- if (!blk_mq_hw_queue_mapped(hctx))
- continue;
-
- if (tags->nr_reserved_tags)
- bt_for_each(hctx, q, bresv, fn, priv, true);
- bt_for_each(hctx, q, btags, fn, priv, false);
- }
+ queue_for_each_hw_ctx(q, hctx, i)
+ __blk_mq_hctx_tag_busy_iter(hctx, fn, priv);
}
srcu_read_unlock(&q->tag_set->tags_srcu, srcu_idx);
blk_queue_exit(q);
--
2.52.0
^ permalink raw reply related [flat|nested] 9+ messages in thread* [RFC PATCH 4/6] blk-mq: add a shared zone to tag fairness
2026-07-06 17:34 [RFC PATCH 0/6] sbitmap enforced fairness for blk-mq Keith Busch
` (2 preceding siblings ...)
2026-07-06 17:34 ` [RFC PATCH 3/6] blk-mq: factor out a per-hctx tag busy iterator Keith Busch
@ 2026-07-06 17:34 ` Keith Busch
2026-07-06 17:34 ` [RFC PATCH 5/6] blk-mq: cache shared-tag fairness windows Keith Busch
` (2 subsequent siblings)
6 siblings, 0 replies; 9+ messages in thread
From: Keith Busch @ 2026-07-06 17:34 UTC (permalink / raw)
To: linux-block; +Cc: linux-scsi, axboe, hch, bvanassche, sumit.saxena, Keith Busch
From: Keith Busch <kbusch@kernel.org>
Private slices of the sbitmap can strand an under utilized queue's tags.
Allow a driver to request some percentage of tags to be reserved into a
common shared pool that anyone may allocate from. The default 0 means
the entire tag set is divided up among its users; 100 means no one has
an exclusive window and can therefore allocate from the entire tag
space.
Shared tags being outside the private zone makes it so we can't do a
sbitmap range weight to find its active commands, so the debugfs count
may no longer accurate. Count the hctx's actual tags via
__blk_mq_hctx_tag_busy_iter() to provide accuracy over performance for
debugfs.
Signed-off-by: Keith Busch <kbusch@kernel.org>
---
block/blk-mq-tag.c | 94 +++++++++++++++++++++++++++++++-----------
block/blk-mq.c | 3 ++
include/linux/blk-mq.h | 5 +++
3 files changed, 79 insertions(+), 23 deletions(-)
diff --git a/block/blk-mq-tag.c b/block/blk-mq-tag.c
index 58cf480df9c69..aa7dbceb60dcd 100644
--- a/block/blk-mq-tag.c
+++ b/block/blk-mq-tag.c
@@ -144,54 +144,79 @@ static inline bool blk_mq_tag_is_windowed(struct blk_mq_alloc_data *data)
(data->hctx->flags & BLK_MQ_F_TAG_QUEUE_SHARED);
}
+struct blk_mq_tag_win {
+ unsigned int priv_min;
+ unsigned int priv_max;
+ unsigned int shared_min;
+ unsigned int shared_max;
+};
+
static bool blk_mq_tag_active_window(struct blk_mq_hw_ctx *hctx,
- unsigned int depth, unsigned int *min,
- unsigned int *max)
+ unsigned int depth,
+ struct blk_mq_tag_win *w)
{
unsigned int users = READ_ONCE(hctx->tags->active_queues);
int slot = READ_ONCE(*blk_mq_tag_win_slot(hctx));
- unsigned int p, rem, pos;
+ unsigned int fair, priv, pct, pos;
+
+ if (users <= 1 || depth <= 1)
+ return false;
- if (users <= 1 || slot < 0 || depth <= 1)
+ fair = depth / users;
+ if (!fair)
return false;
- p = depth / users;
- if (!p)
+ pct = 100 - min(hctx->queue->tag_set->shared_pct, 100u);
+ priv = fair * pct / 100;
+ if (!priv)
return false;
/*
* Slots are allocated from a bitmap and freed out of order, so a slot
* index can exceed the active-user count once lower slots are freed.
* Convert it to a position based on how many active slots precede it.
+ * If a slot could not be assigned, the window is confined to the
+ * shared range.
*/
- pos = bitmap_weight(hctx->tags->active_slots, slot);
- rem = depth - p * users;
- *min = pos * p;
- *max = pos * p + p + rem;
- if (*max > depth)
- *max = depth;
+ pos = slot >= 0 ? bitmap_weight(hctx->tags->active_slots, slot) : users;
+ if (pos < users) {
+ w->priv_min = pos * priv;
+ w->priv_max = w->priv_min + priv;
+ } else {
+ w->priv_min = w->priv_max = 0;
+ }
+ w->shared_min = min(users * priv, depth);
+ w->shared_max = depth;
+
+ if (w->priv_min == w->priv_max && w->shared_min == w->shared_max)
+ return false;
return true;
}
-int blk_mq_get_tag_window(struct blk_mq_hw_ctx *hctx, struct sbitmap_queue *bt)
+static int __blk_mq_get_tag_window(struct sbitmap_queue *bt,
+ struct blk_mq_tag_win *w)
{
- unsigned int min, max;
+ if (w->priv_max > w->priv_min) {
+ int tag = sbitmap_queue_get_range(bt, w->priv_min, w->priv_max);
- if (blk_mq_tag_active_window(hctx, bt->sb.depth, &min, &max))
- return sbitmap_queue_get_range(bt, min, max);
- return __sbitmap_queue_get(bt);
+ if (tag >= 0)
+ return tag;
+ }
+
+ if (w->shared_max > w->shared_min)
+ return sbitmap_queue_get_range(bt, w->shared_min, w->shared_max);
+ return BLK_MQ_NO_TAG;
}
-unsigned int blk_mq_hctx_active(struct blk_mq_hw_ctx *hctx)
+int blk_mq_get_tag_window(struct blk_mq_hw_ctx *hctx, struct sbitmap_queue *bt)
{
- struct sbitmap *sb = &hctx->tags->bitmap_tags.sb;
- unsigned int min, max;
+ struct blk_mq_tag_win w;
- if (!blk_mq_tag_active_window(hctx, sb->depth, &min, &max))
- return sbitmap_weight(sb);
- return sbitmap_weight_range(sb, min, max - min);
+ if (blk_mq_tag_active_window(hctx, bt->sb.depth, &w))
+ return __blk_mq_get_tag_window(bt, &w);
+ return __sbitmap_queue_get(bt);
}
static int __blk_mq_get_tag(struct blk_mq_alloc_data *data,
@@ -604,6 +629,29 @@ static void __blk_mq_hctx_tag_busy_iter(struct blk_mq_hw_ctx *hctx,
bt_for_each(hctx, hctx->queue, &tags->bitmap_tags, fn, priv, false);
}
+static bool blk_mq_count_active(struct request *rq, void *priv)
+{
+ (*(unsigned int *)priv)++;
+ return true;
+}
+
+unsigned int blk_mq_hctx_active(struct blk_mq_hw_ctx *hctx)
+{
+ struct request_queue *q = hctx->queue;
+ unsigned int count = 0;
+ int srcu_idx;
+
+ if (!percpu_ref_tryget(&q->q_usage_counter))
+ return 0;
+
+ srcu_idx = srcu_read_lock(&q->tag_set->tags_srcu);
+ __blk_mq_hctx_tag_busy_iter(hctx, blk_mq_count_active, &count);
+ srcu_read_unlock(&q->tag_set->tags_srcu, srcu_idx);
+
+ blk_queue_exit(q);
+ return count;
+}
+
/**
* blk_mq_queue_tag_busy_iter - iterate over all requests with a driver tag
* @q: Request queue to examine.
diff --git a/block/blk-mq.c b/block/blk-mq.c
index 56ab6ac5ec696..cc56baee74fe8 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -4845,6 +4845,9 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set)
set->queue_depth = BLK_MQ_MAX_DEPTH;
}
+ if (set->shared_pct > 100)
+ set->shared_pct = 100;
+
if (!set->nr_maps)
set->nr_maps = 1;
else if (set->nr_maps > HCTX_MAX_TYPES)
diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h
index 22cc09d5ef320..59847ac7d319c 100644
--- a/include/linux/blk-mq.h
+++ b/include/linux/blk-mq.h
@@ -515,6 +515,10 @@ enum hctx_type {
* @numa_node: NUMA node the storage adapter has been connected to.
* @timeout: Request processing timeout in jiffies.
* @flags: Zero or more BLK_MQ_F_* flags.
+ * @shared_pct: Percent of tags dedicated to non-exclusive use. 0 means
+ * the entire tagset is divided among the users for exclusive
+ * per-user access, and 100 means the entire tag space is
+ * available to all queue contexts that share it.
* @driver_data: Pointer to data owned by the block driver that created this
* tag set.
* @tags: Tag sets. One tag set per hardware queue. Has @nr_hw_queues
@@ -544,6 +548,7 @@ struct blk_mq_tag_set {
int numa_node;
unsigned int timeout;
unsigned int flags;
+ unsigned int shared_pct;
void *driver_data;
struct blk_mq_tags **tags;
--
2.52.0
^ permalink raw reply related [flat|nested] 9+ messages in thread* [RFC PATCH 5/6] blk-mq: cache shared-tag fairness windows
2026-07-06 17:34 [RFC PATCH 0/6] sbitmap enforced fairness for blk-mq Keith Busch
` (3 preceding siblings ...)
2026-07-06 17:34 ` [RFC PATCH 4/6] blk-mq: add a shared zone to tag fairness Keith Busch
@ 2026-07-06 17:34 ` Keith Busch
2026-07-06 17:34 ` [RFC PATCH 6/6] scsi: add shared-tag fairness to host_tagset drivers Keith Busch
2026-07-06 17:56 ` [RFC PATCH 0/6] sbitmap enforced fairness for blk-mq Bart Van Assche
6 siblings, 0 replies; 9+ messages in thread
From: Keith Busch @ 2026-07-06 17:34 UTC (permalink / raw)
To: linux-block; +Cc: linux-scsi, axboe, hch, bvanassche, sumit.saxena, Keith Busch
From: Keith Busch <kbusch@kernel.org>
The windowed allocation path recomputed each queue's window on every tag
allocation, which includes costly division and bitmap weight operations.
Compute the window only when the active set changes and cache it per
queue/hctx, packed into a single u64 so the allocation fast path reads
it with one READ_ONCE and no arithmetic. A lock-free rebalance
recomputes the windows at each busy/idle transition for all contexts
using that tagset, assigning positions in tag_list order. This also
drops the need for the per-queue/hctx slot bitmap.
Signed-off-by: Keith Busch <kbusch@kernel.org>
---
block/blk-core.c | 2 +-
block/blk-mq-tag.c | 180 ++++++++++++++++++-----------------------
block/blk-mq.c | 2 +-
include/linux/blk-mq.h | 16 +---
include/linux/blkdev.h | 19 ++++-
5 files changed, 102 insertions(+), 117 deletions(-)
diff --git a/block/blk-core.c b/block/blk-core.c
index 7e719b90d8a66..85f7ad22f6461 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -464,7 +464,7 @@ struct request_queue *blk_alloc_queue(struct queue_limits *lim, int node_id)
q->node = node_id;
- q->tag_win_slot = -1;
+ q->tag_win = (struct blk_mq_tag_win){ .shared_max = U16_MAX };
timer_setup(&q->timeout, blk_rq_timed_out_timer, 0);
INIT_WORK(&q->timeout_work, blk_timeout_work);
diff --git a/block/blk-mq-tag.c b/block/blk-mq-tag.c
index aa7dbceb60dcd..d92496378a834 100644
--- a/block/blk-mq-tag.c
+++ b/block/blk-mq-tag.c
@@ -34,15 +34,77 @@ static void blk_mq_update_wake_batch(struct blk_mq_tags *tags,
}
/*
- * Where this queue's fairness allocation-window slot is stored. HCTX_SHARED
- * tag sets divide one global bitmap among request_queues (slot per queue);
- * QUEUE_SHARED tag sets divide each per-hw-queue bitmap among the hctxs
- * sharing it (slot per hctx).
+ * Where this queue's cached fairness window lives. HCTX_SHARED tag sets divide
+ * one global bitmap among request_queues (window per queue); QUEUE_SHARED tag
+ * sets divide each per-hw-queue bitmap among the hctxs sharing it (window per
+ * hctx). The window packs into a u64 only because every bound fits in a u16.
*/
-static int *blk_mq_tag_win_slot(struct blk_mq_hw_ctx *hctx)
+static_assert(BLK_MQ_MAX_DEPTH <= U16_MAX);
+
+static struct blk_mq_tag_win *blk_mq_tag_win_ptr(struct blk_mq_hw_ctx *hctx)
{
return blk_mq_is_shared_tags(hctx->flags) ?
- &hctx->queue->tag_win_slot : &hctx->tag_win_slot;
+ &hctx->queue->tag_win : &hctx->tag_win;
+}
+
+static void __blk_mq_rebalance(struct blk_mq_tag_set *set,
+ struct blk_mq_tags *tags, bool shared,
+ unsigned int idx)
+{
+ unsigned int depth = tags->bitmap_tags.sb.depth;
+ unsigned int users, priv, smin, pos;
+ struct request_queue *q;
+
+ users = READ_ONCE(tags->active_queues);
+ priv = smin = pos = 0;
+ if (users > 1 && depth > 1) {
+ unsigned int fair = depth / users;
+ unsigned int pct = 100 - min(set->shared_pct, 100u);
+
+ priv = fair * pct / 100;
+ smin = min(users * priv, depth);
+ }
+
+ rcu_read_lock();
+ list_for_each_entry_rcu(q, &set->tag_list, tag_set_list) {
+ struct blk_mq_tag_win *w, new_w;
+ bool active;
+
+ if (shared) {
+ w = &q->tag_win;
+ active = test_bit(QUEUE_FLAG_HCTX_ACTIVE, &q->queue_flags);
+ } else {
+ struct blk_mq_hw_ctx *hctx = queue_hctx(q, idx);
+
+ if (!hctx)
+ continue;
+ w = &hctx->tag_win;
+ active = test_bit(BLK_MQ_S_TAG_ACTIVE, &hctx->state);
+ }
+
+ if (!priv || !active) {
+ new_w = (struct blk_mq_tag_win) {
+ .shared_max = U16_MAX
+ };
+ } else {
+ new_w = (struct blk_mq_tag_win) {
+ .priv_min = min(pos * priv, depth),
+ .priv_max = min(pos * priv + priv, depth),
+ .shared_min = smin,
+ .shared_max = depth,
+ };
+ pos++;
+ }
+ WRITE_ONCE(w->v, new_w.v);
+ }
+ rcu_read_unlock();
+}
+
+/* Rebalance the pool @hctx draws from after its active set changed. */
+static void blk_mq_rebalance(struct blk_mq_hw_ctx *hctx)
+{
+ __blk_mq_rebalance(hctx->queue->tag_set, hctx->tags,
+ blk_mq_is_shared_tags(hctx->flags), hctx->queue_num);
}
/*
@@ -77,16 +139,9 @@ void __blk_mq_tag_busy(struct blk_mq_hw_ctx *hctx)
users = tags->active_queues + 1;
WRITE_ONCE(tags->active_queues, users);
blk_mq_update_wake_batch(tags, users);
- if (tags->active_slots) {
- unsigned int nbits = tags->bitmap_tags.sb.depth;
- unsigned int slot = find_first_zero_bit(tags->active_slots, nbits);
-
- if (slot < nbits) {
- set_bit(slot, tags->active_slots);
- WRITE_ONCE(*blk_mq_tag_win_slot(hctx), slot);
- }
- }
spin_unlock_irqrestore(&tags->lock, flags);
+
+ blk_mq_rebalance(hctx);
}
/*
@@ -123,17 +178,9 @@ void __blk_mq_tag_idle(struct blk_mq_hw_ctx *hctx)
users = tags->active_queues - 1;
WRITE_ONCE(tags->active_queues, users);
blk_mq_update_wake_batch(tags, users);
- if (tags->active_slots) {
- int *slotp = blk_mq_tag_win_slot(hctx);
- int slot = READ_ONCE(*slotp);
-
- if (slot >= 0) {
- clear_bit(slot, tags->active_slots);
- WRITE_ONCE(*slotp, -1);
- }
- }
spin_unlock_irq(&tags->lock);
+ blk_mq_rebalance(hctx);
blk_mq_tag_wakeup_all(tags, false);
}
@@ -144,81 +191,22 @@ static inline bool blk_mq_tag_is_windowed(struct blk_mq_alloc_data *data)
(data->hctx->flags & BLK_MQ_F_TAG_QUEUE_SHARED);
}
-struct blk_mq_tag_win {
- unsigned int priv_min;
- unsigned int priv_max;
- unsigned int shared_min;
- unsigned int shared_max;
-};
-
-static bool blk_mq_tag_active_window(struct blk_mq_hw_ctx *hctx,
- unsigned int depth,
- struct blk_mq_tag_win *w)
-{
- unsigned int users = READ_ONCE(hctx->tags->active_queues);
- int slot = READ_ONCE(*blk_mq_tag_win_slot(hctx));
- unsigned int fair, priv, pct, pos;
-
- if (users <= 1 || depth <= 1)
- return false;
-
- fair = depth / users;
- if (!fair)
- return false;
-
- pct = 100 - min(hctx->queue->tag_set->shared_pct, 100u);
- priv = fair * pct / 100;
- if (!priv)
- return false;
-
- /*
- * Slots are allocated from a bitmap and freed out of order, so a slot
- * index can exceed the active-user count once lower slots are freed.
- * Convert it to a position based on how many active slots precede it.
- * If a slot could not be assigned, the window is confined to the
- * shared range.
- */
- pos = slot >= 0 ? bitmap_weight(hctx->tags->active_slots, slot) : users;
- if (pos < users) {
- w->priv_min = pos * priv;
- w->priv_max = w->priv_min + priv;
- } else {
- w->priv_min = w->priv_max = 0;
- }
-
- w->shared_min = min(users * priv, depth);
- w->shared_max = depth;
-
- if (w->priv_min == w->priv_max && w->shared_min == w->shared_max)
- return false;
- return true;
-}
-
-static int __blk_mq_get_tag_window(struct sbitmap_queue *bt,
- struct blk_mq_tag_win *w)
+int blk_mq_get_tag_window(struct blk_mq_hw_ctx *hctx, struct sbitmap_queue *bt)
{
- if (w->priv_max > w->priv_min) {
- int tag = sbitmap_queue_get_range(bt, w->priv_min, w->priv_max);
+ struct blk_mq_tag_win w = { .v = READ_ONCE(blk_mq_tag_win_ptr(hctx)->v) };
+ int tag;
+ if (w.priv_max > w.priv_min) {
+ tag = sbitmap_queue_get_range(bt, w.priv_min, w.priv_max);
if (tag >= 0)
return tag;
}
- if (w->shared_max > w->shared_min)
- return sbitmap_queue_get_range(bt, w->shared_min, w->shared_max);
-
+ if (w.shared_max > w.shared_min)
+ return sbitmap_queue_get_range(bt, w.shared_min, w.shared_max);
return BLK_MQ_NO_TAG;
}
-int blk_mq_get_tag_window(struct blk_mq_hw_ctx *hctx, struct sbitmap_queue *bt)
-{
- struct blk_mq_tag_win w;
-
- if (blk_mq_tag_active_window(hctx, bt->sb.depth, &w))
- return __blk_mq_get_tag_window(bt, &w);
- return __sbitmap_queue_get(bt);
-}
-
static int __blk_mq_get_tag(struct blk_mq_alloc_data *data,
struct sbitmap_queue *bt)
{
@@ -726,14 +714,8 @@ struct blk_mq_tags *blk_mq_init_tags(unsigned int total_tags,
spin_lock_init(&tags->lock);
INIT_LIST_HEAD(&tags->page_list);
- if (depth) {
- tags->active_slots = bitmap_zalloc(depth, GFP_KERNEL);
- if (!tags->active_slots)
- goto out_free_tags;
- }
-
if (bt_alloc(&tags->bitmap_tags, depth, round_robin, node))
- goto out_free_slots;
+ goto out_free_tags;
if (bt_alloc(&tags->breserved_tags, reserved_tags, round_robin, node))
goto out_free_bitmap_tags;
@@ -741,8 +723,6 @@ struct blk_mq_tags *blk_mq_init_tags(unsigned int total_tags,
out_free_bitmap_tags:
sbitmap_queue_free(&tags->bitmap_tags);
-out_free_slots:
- bitmap_free(tags->active_slots);
out_free_tags:
kfree(tags);
return NULL;
@@ -771,7 +751,6 @@ void blk_mq_free_tags(struct blk_mq_tag_set *set, struct blk_mq_tags *tags)
{
sbitmap_queue_free(&tags->bitmap_tags);
sbitmap_queue_free(&tags->breserved_tags);
- bitmap_free(tags->active_slots);
/* if tags pages is not allocated yet, free tags directly */
if (list_empty(&tags->page_list)) {
@@ -786,6 +765,7 @@ void blk_mq_tag_resize_shared_tags(struct blk_mq_tag_set *set, unsigned int size
{
struct blk_mq_tags *tags = set->shared_tags;
+ __blk_mq_rebalance(set, tags, true, 0);
sbitmap_queue_resize(&tags->bitmap_tags, size - set->reserved_tags);
}
diff --git a/block/blk-mq.c b/block/blk-mq.c
index cc56baee74fe8..ef8699735aa95 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -4017,7 +4017,7 @@ blk_mq_alloc_hctx(struct request_queue *q, struct blk_mq_tag_set *set,
if (!zalloc_cpumask_var_node(&hctx->cpumask, gfp, node))
goto free_hctx;
- hctx->tag_win_slot = -1;
+ hctx->tag_win = (struct blk_mq_tag_win){ .shared_max = U16_MAX };
if (node == NUMA_NO_NODE)
node = set->numa_node;
hctx->numa_node = node;
diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h
index 59847ac7d319c..0b1e1697e1736 100644
--- a/include/linux/blk-mq.h
+++ b/include/linux/blk-mq.h
@@ -433,12 +433,11 @@ struct blk_mq_hw_ctx {
unsigned int queue_num;
/**
- * @tag_win_slot: Assigned fairness allocation-window slot within this
- * hctx's tags, or -1 if none. Used for non-HCTX_SHARED shared tag sets
- * (the per-hctx analogue of request_queue.tag_win_slot). Protected by
- * the tags->lock.
+ * @tag_win: Cached fairness allocation window for non-HCTX_SHARED shared
+ * tag sets (each per-hw-queue bitmap divided among its hctxs). Written by
+ * the rebalance walk, read locklessly on the allocation path.
*/
- int tag_win_slot;
+ struct blk_mq_tag_win tag_win;
/** @cpuhp_online: List to store request if CPU is going to die */
struct hlist_node cpuhp_online;
@@ -783,13 +782,6 @@ struct blk_mq_tags {
unsigned int nr_reserved_tags;
unsigned int active_queues;
- /*
- * Bitmap of assigned per-queue fairness window slots, @nr_tags -
- * @nr_reserved_tags bits wide (windowing is moot once active users
- * exceed that). Protected by @lock.
- */
- unsigned long *active_slots;
-
struct sbitmap_queue bitmap_tags;
struct sbitmap_queue breserved_tags;
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index 86b16bd8b9c17..7b55ce6a3fd2b 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -487,6 +487,18 @@ struct blk_independent_access_ranges {
struct blk_independent_access_range ia_range[];
};
+struct blk_mq_tag_win {
+ union {
+ struct {
+ u16 priv_min;
+ u16 priv_max;
+ u16 shared_min;
+ u16 shared_max;
+ };
+ u64 v;
+ };
+};
+
struct request_queue {
/*
* The queue owner gets to use this for whatever they like.
@@ -574,10 +586,11 @@ struct request_queue {
struct work_struct timeout_work;
/*
- * Assigned allocation-window slot for shared (HCTX_SHARED) tag-set
- * fairness, or -1 if none. Protected by the shared tags->lock.
+ * Cached fairness allocation window for HCTX_SHARED tag sets (the one
+ * global bitmap divided among request_queues). Written by the rebalance
+ * walk, read locklessly on the allocation path.
*/
- int tag_win_slot;
+ struct blk_mq_tag_win tag_win;
struct blk_mq_tags *sched_shared_tags;
--
2.52.0
^ permalink raw reply related [flat|nested] 9+ messages in thread* [RFC PATCH 6/6] scsi: add shared-tag fairness to host_tagset drivers
2026-07-06 17:34 [RFC PATCH 0/6] sbitmap enforced fairness for blk-mq Keith Busch
` (4 preceding siblings ...)
2026-07-06 17:34 ` [RFC PATCH 5/6] blk-mq: cache shared-tag fairness windows Keith Busch
@ 2026-07-06 17:34 ` Keith Busch
2026-07-06 17:56 ` [RFC PATCH 0/6] sbitmap enforced fairness for blk-mq Bart Van Assche
6 siblings, 0 replies; 9+ messages in thread
From: Keith Busch @ 2026-07-06 17:34 UTC (permalink / raw)
To: linux-block; +Cc: linux-scsi, axboe, hch, bvanassche, sumit.saxena, Keith Busch
From: Keith Busch <kbusch@kernel.org>
Introduce a per-host shared_pct into the scsi block tag set so
host_tagset drivers can choose how much of the shared pool stays carved
into per-LUN exclusivity versus shared by everyone. This may be useful
for UFS where the sharing is harmful to performance.
Signed-off-by: Keith Busch <kbusch@kernel.org>
---
drivers/scsi/scsi_debug.c | 8 +++++++-
drivers/scsi/scsi_lib.c | 4 +++-
include/scsi/scsi_host.h | 8 ++++++++
3 files changed, 18 insertions(+), 2 deletions(-)
diff --git a/drivers/scsi/scsi_debug.c b/drivers/scsi/scsi_debug.c
index 9d1c9c41d0f99..cb2bec76ba3de 100644
--- a/drivers/scsi/scsi_debug.c
+++ b/drivers/scsi/scsi_debug.c
@@ -905,6 +905,7 @@ static int sdebug_every_nth = DEF_EVERY_NTH;
static int sdebug_fake_rw = DEF_FAKE_RW;
static unsigned int sdebug_guard = DEF_GUARD;
static int sdebug_host_max_queue; /* per host */
+static int sdebug_shared_pct; /* host_tagset shared tag pool % */
static int sdebug_lowest_aligned = DEF_LOWEST_ALIGNED;
static int sdebug_max_luns = DEF_MAX_LUNS;
static int sdebug_max_queue = SDEBUG_CANQUEUE; /* per submit queue */
@@ -7343,6 +7344,7 @@ module_param_named(fake_rw, sdebug_fake_rw, int, S_IRUGO | S_IWUSR);
module_param_named(guard, sdebug_guard, uint, S_IRUGO);
module_param_named(host_lock, sdebug_host_lock, bool, S_IRUGO | S_IWUSR);
module_param_named(host_max_queue, sdebug_host_max_queue, int, S_IRUGO);
+module_param_named(shared_pct, sdebug_shared_pct, int, S_IRUGO);
module_param_string(inq_product, sdebug_inq_product_id,
sizeof(sdebug_inq_product_id), S_IRUGO | S_IWUSR);
module_param_string(inq_rev, sdebug_inq_product_rev,
@@ -7427,6 +7429,8 @@ MODULE_PARM_DESC(guard, "protection checksum: 0=crc, 1=ip (def=0)");
MODULE_PARM_DESC(host_lock, "host_lock is ignored (def=0)");
MODULE_PARM_DESC(host_max_queue,
"host max # of queued cmds (0 to max(def) [max_queue fixed equal for !0])");
+MODULE_PARM_DESC(shared_pct,
+ "host_tagset: %% of the shared tag pool shared by all LUNs (0=exclusive, 100=no fairness, def=0)");
MODULE_PARM_DESC(inq_product, "SCSI INQUIRY product string (def=\"scsi_debug\")");
MODULE_PARM_DESC(inq_rev, "SCSI INQUIRY revision string (def=\""
SDEBUG_VERSION "\")");
@@ -9569,8 +9573,10 @@ static int sdebug_driver_probe(struct device *dev)
* following should give the same answer for each host.
*/
hpnt->nr_hw_queues = submit_queues;
- if (sdebug_host_max_queue)
+ if (sdebug_host_max_queue) {
hpnt->host_tagset = 1;
+ hpnt->shared_pct = sdebug_shared_pct;
+ }
/* poll queues are possible for nr_hw_queues > 1 */
if (hpnt->nr_hw_queues == 1 || (poll_queues < 1)) {
diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
index 22e2e3223440d..efcb49af899a2 100644
--- a/drivers/scsi/scsi_lib.c
+++ b/drivers/scsi/scsi_lib.c
@@ -2152,8 +2152,10 @@ int scsi_mq_setup_tags(struct Scsi_Host *shost)
if (shost->queuecommand_may_block)
tag_set->flags |= BLK_MQ_F_BLOCKING;
tag_set->driver_data = shost;
- if (shost->host_tagset)
+ if (shost->host_tagset) {
tag_set->flags |= BLK_MQ_F_TAG_HCTX_SHARED;
+ tag_set->shared_pct = shost->shared_pct;
+ }
return blk_mq_alloc_tag_set(tag_set);
}
diff --git a/include/scsi/scsi_host.h b/include/scsi/scsi_host.h
index 7e2011830ba4b..68c99bc7ad865 100644
--- a/include/scsi/scsi_host.h
+++ b/include/scsi/scsi_host.h
@@ -694,6 +694,14 @@ struct Scsi_Host {
/* The queuecommand callback may block. See also BLK_MQ_F_BLOCKING. */
unsigned queuecommand_may_block:1;
+ /*
+ * For host_tagset hosts: percent of the shared tag pool made available
+ * to all LUNs rather than reserved as per-LUN exclusive floors. See
+ * blk_mq_tag_set.shared_pct. 0 (default) divides the pool exclusively;
+ * 100 lets every LUN allocate from the whole pool with no fairness.
+ */
+ unsigned int shared_pct;
+
/* Host responded with short (<36 bytes) INQUIRY result */
unsigned short_inquiry:1;
--
2.52.0
^ permalink raw reply related [flat|nested] 9+ messages in thread* Re: [RFC PATCH 0/6] sbitmap enforced fairness for blk-mq
2026-07-06 17:34 [RFC PATCH 0/6] sbitmap enforced fairness for blk-mq Keith Busch
` (5 preceding siblings ...)
2026-07-06 17:34 ` [RFC PATCH 6/6] scsi: add shared-tag fairness to host_tagset drivers Keith Busch
@ 2026-07-06 17:56 ` Bart Van Assche
2026-07-06 18:26 ` Keith Busch
6 siblings, 1 reply; 9+ messages in thread
From: Bart Van Assche @ 2026-07-06 17:56 UTC (permalink / raw)
To: Keith Busch, linux-block
Cc: linux-scsi, axboe, hch, sumit.saxena, Keith Busch
On 7/6/26 10:34 AM, Keith Busch wrote:
> There have been a few proposals to remove the blk-mq tag fairness
> algorithm:
>
> https://lore.kernel.org/linux-block/20240529213921.3166462-1-bvanassche@acm.org/
> https://lore.kernel.org/linux-block/20260609121806.2121755-1-sumit.saxena@broadcom.com/
There have been more proposals to remove the fairness code from other
kernel contributors. I proposed to remove the tag fairness code because
prior attempts to solve UFS performance issues were not good enough for
the upstream kernel.
> Both abandon blk-mq's attempt to enforce tag allocation limits on the
> per-queue/per-hctx users that share tag space. This can harm resource
> allocation for lesser devices sharing the space, potentially starving,
> them from fair progress by a highly utilized device.
Has this starving phenomenon ever been observed? I think that the
measurement data that I published shows that the fairness properties of
the current sbitmap implementation are good enough.
> This series proposes an entirely different fairness mechanism that
> doesn't require per-IO atomic accounting:
>
> First, the sbitmap API is augmented with a ranged allocator. This
> allows a client to carve the depth into exclusive ranges for specific
> users.
>
> Second, you can optionally declare a percentage of that pool to be
> fair game for anyone to allocate from. This provides a way to
> guarantee minimum tag space for each client while allowing a user to
> over-allocate its fair budget on demand into the shared zone.
What software layer is expected to set these percentages? My patch
series intentionally removes the fairness mechanism because that's
exactly what the legacy code path in the UFS driver needs. The UFS
WLUN and data logical units share a single tag set. Any activity on
the WLUN increases the number of active queues and hence reduces the
number of tags available for any data logical units.
Additionally, what is the performance impact of this patch series?
Removing the tag fairness code increases IOPS so this patch series
probably has a performance impact.
Thanks,
Bart.
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [RFC PATCH 0/6] sbitmap enforced fairness for blk-mq
2026-07-06 17:56 ` [RFC PATCH 0/6] sbitmap enforced fairness for blk-mq Bart Van Assche
@ 2026-07-06 18:26 ` Keith Busch
0 siblings, 0 replies; 9+ messages in thread
From: Keith Busch @ 2026-07-06 18:26 UTC (permalink / raw)
To: Bart Van Assche
Cc: Keith Busch, linux-block, linux-scsi, axboe, hch, sumit.saxena
On Mon, Jul 06, 2026 at 10:56:05AM -0700, Bart Van Assche wrote:
> On 7/6/26 10:34 AM, Keith Busch wrote:
> > There have been a few proposals to remove the blk-mq tag fairness
> > algorithm:
> >
> > https://lore.kernel.org/linux-block/20240529213921.3166462-1-bvanassche@acm.org/
> > https://lore.kernel.org/linux-block/20260609121806.2121755-1-sumit.saxena@broadcom.com/
>
> There have been more proposals to remove the fairness code from other
> kernel contributors. I proposed to remove the tag fairness code because
> prior attempts to solve UFS performance issues were not good enough for
> the upstream kernel.
>
> > Both abandon blk-mq's attempt to enforce tag allocation limits on the
> > per-queue/per-hctx users that share tag space. This can harm resource
> > allocation for lesser devices sharing the space, potentially starving,
> > them from fair progress by a highly utilized device.
>
> Has this starving phenomenon ever been observed?
If you're asking if a production workload sees the phenomenon, no
because they run the code that has enforced fairness.
> I think that the
> measurement data that I published shows that the fairness properties of
> the current sbitmap implementation are good enough.
I can trivially wire up a test where one queue allocates 100% of the
tags for long running commands, and all the other queues that want to
start a short running command can't proceed.
> > Second, you can optionally declare a percentage of that pool to be
> > fair game for anyone to allocate from. This provides a way to
> > guarantee minimum tag space for each client while allowing a user to
> > over-allocate its fair budget on demand into the shared zone.
>
> What software layer is expected to set these percentages?
It's a tagset parameter that the LLD can set to whatever it needs.
> My patch
> series intentionally removes the fairness mechanism because that's
> exactly what the legacy code path in the UFS driver needs. The UFS
> WLUN and data logical units share a single tag set. Any activity on
> the WLUN increases the number of active queues and hence reduces the
> number of tags available for any data logical units.
>
> Additionally, what is the performance impact of this patch series?
It removes the per-io atomics, so impact is it goes faster if that was
your bottle-neck.
> Removing the tag fairness code increases IOPS so this patch series
> probably has a performance impact.
You can declare the entire tagset is shared if you want. That should get
you what you were going for, but we still have an option to spatially
constrain others that want fairness with this series.
^ permalink raw reply [flat|nested] 9+ messages in thread