[PATCH 0/6] md/raid5: size stripe-cache and worker tuning from the hardware

[Hiroshi Nishida <nishidafmly@gmail.com>]

md/raid5 has several tuning values that are compile-time constants chosen
for the modest systems of years ago:

  - NR_STRIPE_HASH_LOCKS   (8)    stripe-cache hash lock striping
  - the 32768 stripe_cache_size ceiling
  - NR_STRIPES             (256)  initial stripe cache size
  - MAX_STRIPE_BATCH       (8)    stripes handled per device_lock
  - worker_cnt_per_group   (0)    raid5 worker threads (group_thread_cnt)

On a large-memory, many-core host backing a wide array these are all too
small, and the only remedy today is to retune each one by hand after every
assembly, or to recompile.  On a small NAS a fixed larger value would just
waste memory.

This series makes each value derive a default from the hardware -- system
memory for the cache sizes, CPU count for the lock and worker counts --
while keeping every one overridable:

  1/6 (prerequisite) size the worker_groups[] array by nr_node_ids
  2/6 size the stripe-cache hash locks from the CPU count (8..32)
  3/6 scale the stripe_cache_size limit with memory (never below 32768)
  4/6 make the stripe batch size a module parameter
  5/6 scale the default stripe cache size with memory (256..4096)
  6/6 derive the default group_thread_cnt from the CPU count

Patch 1 is a prerequisite correctness fix that patch 6 depends on:
alloc_thread_groups() sizes the worker_groups[] array by
num_possible_nodes() (a node count) but indexes it by cpu_to_node() (a
node id), so a sparse NUMA node map can index out of bounds.  That is only
reachable once worker groups are enabled, which patch 6 does by default --
so the fix goes first.  It stands alone (Fixes: 851c30c9badf) and can be
taken independently.

Each tunable default only rises on hardware that can back it: the lock and
worker counts (2, 6) scale with the CPU count, the cache sizes (3, 5) with
RAM.  A genuinely small system -- a few cores and a few GB -- therefore
keeps today's values and today's memory footprint.  Patches 2-6 each add a
module parameter to override or pin the value (patch 6 also keeps the
existing per-array group_thread_cnt sysfs attribute), so an administrator
can force any of them, including back to the historical behaviour.

Patches 2-5 are sizing/capability changes with no throughput claim -- the
intent is simply that the out-of-the-box configuration tracks the machine
instead of a fixed constant.  Patch 6 does move throughput: enabling md's
worker groups by default measures 2.1-3.2x on a 16-disk NVMe array (a
32-vCPU / 16-core host, steady state), with no regression on small
machines -- a 4-CPU box gets 2 workers and is never slower, a box with two
or fewer CPUs gets 0 and is unchanged.

Tested on a KASAN + lockdep + DEBUG_LIST kernel (RAID5 on loop devices):
create, 180MB write + sha256 verify, disk fail + spare rebuild (data
verified), scrub (mismatch_cnt=0), and bitmap add/remove -- which drives
raid5_quiesce()'s lock_all_device_hash_locks_irq() path -- with the
parameters both left at their hardware-derived defaults and pinned to
explicit values, including nr_stripe_hash_locks=32 (the 33-lock quiesce
path).  No KASAN, lockdep, or list-debug reports.

Hiroshi Nishida (6):
  md/raid5: size the worker group array by nr_node_ids
  md/raid5: size stripe-cache hash locks from the CPU count
  md/raid5: scale the stripe_cache_size limit with system memory
  md/raid5: make the stripe batch size a module parameter
  md/raid5: scale the default stripe cache size with system memory
  md/raid5: derive the default group_thread_cnt from the hardware

 drivers/md/raid5-cache.c |   2 +-
 drivers/md/raid5.c       | 233 +++++++++++++++++++++++++++++++++------
 drivers/md/raid5.h       |  43 ++++++--
 3 files changed, 233 insertions(+), 45 deletions(-)


base-commit: 55b77337bdd088c77461588e5ec094421b89911b
--
2.43.0