From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-dy1-f201.google.com (mail-dy1-f201.google.com [74.125.82.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 36B4A33B6DA for ; Wed, 15 Apr 2026 23:21:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.82.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776295295; cv=none; b=KFoQ1SegpNN3qwwpJeIaXBs+utZ8SA3qpNl12xS4QCpy9wOvS2IP3ABM12LErPDW8kSNsCgnzsa/vkQZNbrdwDFDp+qbv2RDNePuENK++F4o1fkEu+s06CZnEDneEevPvq2rMhvar5UKp8HGBJYpG/4KcOJkYGU5fbIGIlXEf/E= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776295295; c=relaxed/simple; bh=6dAhuECT/i1VihQi3r45JGDd6acMTRJ9plcxtde7Fn0=; h=Date:Mime-Version:Message-ID:Subject:From:To:Cc:Content-Type; b=Y59+1BA7P40By16WzycMc3LRhfppK8woN5hf0+SoQLxYzO1tSNp5CGgEUSSK4mX/0B+weT+OFKHJEXg7eehdPazKD4Qfqogmv1xGCwbPLfkmcKN27AV7coWOK5EpxI5Vzz2gC8+0HukxDafc52i88usVT32JwU9MVOx79DBP05w= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--aniketgattani.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=X4b63/zq; arc=none smtp.client-ip=74.125.82.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--aniketgattani.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="X4b63/zq" Received: by mail-dy1-f201.google.com with SMTP id 5a478bee46e88-2bda35eab74so110168eec.0 for ; Wed, 15 Apr 2026 16:21:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1776295293; x=1776900093; darn=vger.kernel.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=+r+WpZUUQ/YvdDlyZscQdfNrzTqbtr5QHQkAJv55K3U=; b=X4b63/zqCfUiwaG554XMiwn1ZyYAa/xLbA5PczHnIp/PY2F7kWK36iy7vad6m5jtMz qda7Cy1zcLlxsrJNpJkxizbmSiUYsvLMBwfMcP/3q3bQ3AmQz4Mve9rhs4EkV9RPMV7F fmfjEzDE1YLTblWaktV97NK3SPXDAFaIORK9PCPy5ia6q8mnTdscxn9brN/yb3MHCVNh gJrGEuXWiKD2l3qHoHkuksHepnaPelGvuMAjrf41E9EbjlVKd5EyZZxKd2ffZHBrJ1ys ii86s5KS0nZllGQAQYnOy8shvJAAbxQVClXX1uuutgLD+0GKelQJuda9YeZmtdIOGHSk jxFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1776295293; x=1776900093; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=+r+WpZUUQ/YvdDlyZscQdfNrzTqbtr5QHQkAJv55K3U=; b=aTzkbL2Q1w/zlkl5694qCWxXXkJJKCUyzkNR5mNQxfDEms1MAn2XNr7jk9AKgwc0Yx YvCFHBRzsO69pRc3Y45arGTK+T5REVvJ3bxuT9DWkDgMh3MjxAIVVPznDJGRnILPZGId Igbcr9XACMSi26cTR0Zpi1yzNq5qVY8KCNt4LumeA4NMGwRiLBzDWv7bA62KyUmy4Y4K UPK1SgSZ1OrS1wphldCBMbiHtqCp5/q5O0yt+r3dhyS0Z/9L06465mdpHRS00nFpIZta cY/RzT+ntOMQGcW68qznxxRUSdLnTKb2pLRyS0/Ebeo5lBx9td0gIT3AZ1WSbrjs5GId lS1w== X-Forwarded-Encrypted: i=1; AFNElJ+9PV+2ZYhn+7BG75GL+dSRIkM2Chtc7H7EehCiD5gkvAIwFZBIQsAGr4qujKCZOG03oETFGjAeK5LkGwc=@vger.kernel.org X-Gm-Message-State: AOJu0YzI0Dueyh9eNn0ZLl8QCmcqTYQv0+/OHVfc3cxq8Qq5wRbg7jRI vOGTeXsmPGBqPZJWqiWAXM6sSo89VdIDg8lfisBUcqx7J30N7PV9ByFbGQLcEtxNxcgQYHAmn1M pS0KI2+XqerrQZF+OM4IFiJDufhOM06YqXQ== X-Received: from dybdm8.prod.google.com ([2002:a05:7300:bb08:b0:2d8:5557:aec]) (user=aniketgattani job=prod-delivery.src-stubby-dispatcher) by 2002:a05:7300:ad26:b0:2de:e194:5fb1 with SMTP id 5a478bee46e88-2df7dcdd7b4mr803163eec.7.1776295293029; Wed, 15 Apr 2026 16:21:33 -0700 (PDT) Date: Wed, 15 Apr 2026 23:21:04 +0000 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 X-Mailer: git-send-email 2.54.0.rc1.513.gad8abe7a5a-goog Message-ID: <20260415232106.2803644-1-aniketgattani@google.com> Subject: [PATCH v2 0/2] sched/membarrier: Use per-CPU mutexes for targeted commands From: Aniket Gattani To: Mathieu Desnoyers , "Paul E . McKenney" Cc: Peter Zijlstra , Ingo Molnar , Ben Segall , Josh Don , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, Aniket Gattani Content-Type: text/plain; charset="UTF-8" This series mitigates a severe contention issue in the membarrier system call by replacing the global membarrier_ipi_mutex with per-CPU mutexes for targeted expedited commands. Problem: Currently, MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ relies on a single global mutex to serialize IPIs. On large systems with heavy concurrent usage, this creates significant contention. The issue cascades into hard lockups when combined with CFS bandwidth throttling, during which target CPUs may have interrupts disabled for extended periods. If membarrier is waiting on such a CPU, it holds the global mutex, stalling all other membarrier callers system-wide. Solution: Patch 1 introduces membarrier_cpu_mutexes to serialize expedited commands specifically when a target CPU is provided, isolating the lock contention to the targeted CPU. Broadcast commands continue to use the global mutex. Patch 2 introduces a dedicated kselftest reproducer (membarrier_rseq_stress) to permanently test this interaction. It aggressively hammers targeted membarrier commands within a deep, aggressively throttled cgroup hierarchy to prove the lockup scenario and validate the fix. Results: As measured by the stress test introduced in Patch 2, testing on an AMD Turin machine with 384 CPUs (2 NUMA nodes, SMT=2) shows: Throughput: ~200x increase in successful membarrier calls. --- Changes in v2: v1: https://lore.kernel.org/lkml/20260409212223.2072418-1-aniketgattani@google.com - Use different mutex macros for global vs targeted cpu membarrier (Mathieu). - Use (unsigned int) cpu_id >= nr_cpu_id (Peter). - Removed #warning on unsupported architectures that causes build failures with W=1 reported by the kernel test robot. Aniket Gattani (2): sched/membarrier: Use per-CPU mutexes for targeted commands selftests/membarrier: Add rseq stress test for CFS throttle interactions kernel/sched/membarrier.c | 36 +- tools/testing/selftests/membarrier/Makefile | 5 +- .../membarrier/membarrier_rseq_stress.c | 951 ++++++++++++++++++ 3 files changed, 979 insertions(+), 13 deletions(-) create mode 100644 tools/testing/selftests/membarrier/membarrier_rseq_stress.c -- 2.54.0.rc1.513.gad8abe7a5a-goog