From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wr1-f54.google.com (mail-wr1-f54.google.com [209.85.221.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 660153B6C09 for ; Mon, 13 Jul 2026 10:22:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.54 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783938158; cv=none; b=hBQb9ompfIXQcaTxIukPzXLXvBz+QZ5adlswt6Rk+zCebPxT7hCRRpLVb6ZTaNLg3Th4kZhN0euYOwYTF/6xHLOCBlxJySsByKgNMVR2qVvejE8tfD8LxAu4dOqMm0u+y/9BAGWKXlBi8CXWz0RVgvtOHsMN/tx1k26Mlu9pN5E= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783938158; c=relaxed/simple; bh=l8axt9Tk9geBEkN8bylkeCe0FVdnGDnLH1I3RigAbJU=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=hkOZEMpS3wqCclrpNuRkUOgtAxurYHnjjhbl+7sKMcQYF7yPVH1sxYRz0S7+6ZTACmNSXEbKVehIwjyqrsx5SK5qeZbH1CtzQsipog6XDyOeWKgT+xKxSAs44975vFhfOFqarn54sfISn1LHy+RYnfqQwA8devYYAFZww4Y+XaU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=ionos.com; spf=pass smtp.mailfrom=ionos.com; dkim=pass (2048-bit key) header.d=ionos.com header.i=@ionos.com header.b=QeyyZXiI; arc=none smtp.client-ip=209.85.221.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=ionos.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ionos.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ionos.com header.i=@ionos.com header.b="QeyyZXiI" Received: by mail-wr1-f54.google.com with SMTP id ffacd0b85a97d-47f3b39f2a1so869666f8f.2 for ; Mon, 13 Jul 2026 03:22:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; t=1783938155; x=1784542955; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to:content-type; bh=uuK9JeqClTwXWplH5M0lC/009LSJw3bFvJurNVCcSgk=; b=QeyyZXiI2sEENdtWCA1M47I2OgG3d4R+FWr78FcvSnsEthd/VsZvwuY5S3k9V8Uh/Z eVuax1pheEmYQwZy5AhD+9tooQG9YpauilvJ9cg3yE4LVqmC18VqxP3v8/bwHGMkBPTw ZE3C03dACJbERXWvJsHRTP6srzdSmPNgZi4xKlTWf61tE84MijLT6p07QEgqm52lczmh /V1jGkXN5ceP7UHUeQ6XJnIuoP8Irh5DRhfeVgEq6vcb01DdQgsWcOxved1pOgFO+/ZY QVABYjs1GaYVQ9420gId0zHCec6sd/h+/H8jkUtiZHkLGHn3C1ysjfbxpqjLmn4bZxae ejWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1783938155; x=1784542955; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=uuK9JeqClTwXWplH5M0lC/009LSJw3bFvJurNVCcSgk=; b=D1wNiv64zQ6/9B6HMC4/N7XIsdxikkS3yBwweSPgYM3bGgyKqwYRdgc2RmTY+f0YIy YMBSNESJ30WQb/5luzDyX7IKNjxCNdJKSkTjrPQFotjmbo7bud2wLurHH2ainL62hQcC me7dsQPrX2vFPMbUuLmoskdB9HExtbwf3zN6GmUgavjPSkHhfRDr8QYiWhRkXGIHnV3s pTEehjN5hoA4fhGGTJGQYR74gHdIU87kM7SHEl45ek5GtPqjLzNpS+2DZhNSZXajp9lu thwz7zXEnsWdLbmktVh6CC+PxICWRIzmElbAk8sC4qd7cpBIilvs67w/oN01Jl5ZDLYT yHbA== X-Forwarded-Encrypted: i=1; AHgh+Rr9GLRuTqEwGCg9rpOOhfiNtEG4KtYgYtNQyGlYKNDrN8JdbL0NPT2VMWuLPXPqhteA+QdkRrhzLvNCLJc=@vger.kernel.org X-Gm-Message-State: AOJu0Yw/7IFXQ+UB5kLCN8tX1nFsZJGDnuLYAjrSRNPF5kikzWyPzM0C 0SRoOCgv4ffW4+bT8mugTDK1x9G9UpmCFj5SqHrTCpjjUaYn8ZAvQubiA0A4oIlpb6E= X-Gm-Gg: AfdE7ck3lKP9b9XcxQ4X/icjC8Xh43fHCMOaoOVSBgN6nUEZilZGPb1RxG/cCEGRskd ZGrOgsQlXkSf2uBgb+Ie4iaOEjoWanubyYW3LnfQFh96F8DAmOZ5XQTya09gU4byWAAiiDbxjzF mAcAYuxzHrr9fF6cOfcm/R3Us32HYfSMmSyABt0ADP0BGNFQHhlqIE1sMCjdS5U4/X994LhIYhp v/tyJklRvwQYuiVrgRIWYJOjI/b9XArgJr1zm7YjAWfPLkGGfK9mfmfE2Wad83ZesillUGBUVrI qnN2D//IVN8NMWY9fQC1P7aM8CBQ56Z4KqWIA4PL7YbJ7DPOqAYP+5u1YZx8CNzdJ3jrNwuA14z LNST6iD9QC4uZmbnfUtg+eKW2ZwbVTtlhXFn3EwEcucBVtxgfwl1UUjhDS6Fm/28ayI5oBdjdh+ ElVJ4u2BKB4Kl+F/hKMniTi36S63Pfp/JTeAlFgSjMrtNHzQxqbATKSC5zB0UTpIMFf8DJ10BeJ +i1ViFSGZDeEc// X-Received: by 2002:a05:6000:1a86:b0:45e:f439:9def with SMTP id ffacd0b85a97d-47f2dcbffdemr9467962f8f.15.1783938154686; Mon, 13 Jul 2026 03:22:34 -0700 (PDT) Received: from raven.intern.cm-ag (p200300dc6f484700023064fffe740809.dip0.t-ipconnect.de. [2003:dc:6f48:4700:230:64ff:fe74:809]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-47aa09608d4sm84858344f8f.25.2026.07.13.03.22.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Jul 2026 03:22:33 -0700 (PDT) From: Max Kellermann To: tytso@mit.edu, jack@suse.com, linux-ext4@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Max Kellermann Subject: [PATCH 0/2] jbd2: bound j_list_lock hold time in the checkpoint shrinker Date: Mon, 13 Jul 2026 12:22:27 +0200 Message-ID: <20260713102229.1598812-1-max.kellermann@ionos.com> X-Mailer: git-send-email 2.47.3 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit We hit recurring RCU stalls / soft lockups on a busy ext4 filesystem that serves as the CacheFiles backing store for a Ceph (fscache) client. The stall is always the journal commit thread spinning on journal->j_list_lock: rcu: INFO: rcu_sched self-detected stall on CPU rcu: 106-....: (2099 ticks this GP) ... (t=2100 jiffies) CPU: 106 Comm: jbd2/nvme1n1p1- RIP: 0010:queued_spin_lock_slowpath+0x20a/0x240 Call Trace: jbd2_journal_write_metadata_buffer+0x1c0/0x310 jbd2_journal_commit_transaction+0x5e2/0x16e0 kjournald2+0xa1/0x220 kthread+0xe4/0x1d0 kjournald2 has already passed the "wait for outstanding handles" barrier and is in the metadata write-out loop; it is simply unable to acquire j_list_lock for >21s. The lock holder is the jbd2 checkpoint shrinker. Under memory pressure, the shrinker (jbd2_journal_shrink_scan -> jbd2_journal_shrink_checkpoint_list -> journal_shrink_one_cp_list) walks a transaction's checkpoint list under j_list_lock. On this workload, the lists are large and dominated by busy buffers (dirty / under writeback / attached to the running transaction), and journal_shrink_one_cp_list() can therefore hold j_list_lock for a time proportional to the whole list length, for two reasons: 1. The JBD2_SHRINK_BUSY_SKIP path uses "continue", which also skips the need_resched() check, so the scan does not yield even when a reschedule is pending. 2. The scan budget (nr_to_scan) is decremented only for buffers that are actually freed, so busy buffers do not consume it and the loop is not bounded by the shrinker's batch at all. Both were introduced by commit b98dba273a0e ("jbd2: remove journal_clean_one_cp_list()"), which folded journal_clean_one_cp_list() into journal_shrink_one_cp_list() and dropped the per-examined-buffer budget that the shrinker previously had. The result is a self-reinforcing collapse: shrinker instances across many CPUs hold/contend j_list_lock, kjournald2 cannot commit, the journal fills, CacheFiles lookups block in ext4, fscache cookies get stuck in LOOKING_UP, and netfs I/O times out. Max Kellermann (2): jbd2: check need_resched() when skipping busy checkpoint buffers jbd2: bound shrinker scans by examined checkpoint buffers fs/jbd2/checkpoint.c | 28 +++++++++++++++------------- 1 file changed, 15 insertions(+), 13 deletions(-) -- 2.47.3