All of lore.kernel.org
 help / color / mirror / Atom feed
From: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
To: linux-kernel@vger.kernel.org
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	stable@vger.kernel.org, Tejun Heo <tj@kernel.org>,
	Baolin Wang <baolin.wang@linux.alibaba.com>,
	Jens Axboe <axboe@kernel.dk>, Sasha Levin <sashal@kernel.org>
Subject: [PATCH 5.10 42/54] blk-cgroup: Use cond_resched() when destroy blkgs
Date: Thu, 11 Feb 2021 16:02:26 +0100	[thread overview]
Message-ID: <20210211150154.707204813@linuxfoundation.org> (raw)
In-Reply-To: <20210211150152.885701259@linuxfoundation.org>

From: Baolin Wang <baolin.wang@linux.alibaba.com>

[ Upstream commit 6c635caef410aa757befbd8857c1eadde5cc22ed ]

On !PREEMPT kernel, we can get below softlockup when doing stress
testing with creating and destroying block cgroup repeatly. The
reason is it may take a long time to acquire the queue's lock in
the loop of blkcg_destroy_blkgs(), or the system can accumulate a
huge number of blkgs in pathological cases. We can add a need_resched()
check on each loop and release locks and do cond_resched() if true
to avoid this issue, since the blkcg_destroy_blkgs() is not called
from atomic contexts.

[ 4757.010308] watchdog: BUG: soft lockup - CPU#11 stuck for 94s!
[ 4757.010698] Call trace:
[ 4757.010700]  blkcg_destroy_blkgs+0x68/0x150
[ 4757.010701]  cgwb_release_workfn+0x104/0x158
[ 4757.010702]  process_one_work+0x1bc/0x3f0
[ 4757.010704]  worker_thread+0x164/0x468
[ 4757.010705]  kthread+0x108/0x138

Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
 block/blk-cgroup.c | 18 +++++++++++++-----
 1 file changed, 13 insertions(+), 5 deletions(-)

diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
index 54fbe1e80cc41..f13688c4b9317 100644
--- a/block/blk-cgroup.c
+++ b/block/blk-cgroup.c
@@ -1017,6 +1017,8 @@ static void blkcg_css_offline(struct cgroup_subsys_state *css)
  */
 void blkcg_destroy_blkgs(struct blkcg *blkcg)
 {
+	might_sleep();
+
 	spin_lock_irq(&blkcg->lock);
 
 	while (!hlist_empty(&blkcg->blkg_list)) {
@@ -1024,14 +1026,20 @@ void blkcg_destroy_blkgs(struct blkcg *blkcg)
 						struct blkcg_gq, blkcg_node);
 		struct request_queue *q = blkg->q;
 
-		if (spin_trylock(&q->queue_lock)) {
-			blkg_destroy(blkg);
-			spin_unlock(&q->queue_lock);
-		} else {
+		if (need_resched() || !spin_trylock(&q->queue_lock)) {
+			/*
+			 * Given that the system can accumulate a huge number
+			 * of blkgs in pathological cases, check to see if we
+			 * need to rescheduling to avoid softlockup.
+			 */
 			spin_unlock_irq(&blkcg->lock);
-			cpu_relax();
+			cond_resched();
 			spin_lock_irq(&blkcg->lock);
+			continue;
 		}
+
+		blkg_destroy(blkg);
+		spin_unlock(&q->queue_lock);
 	}
 
 	spin_unlock_irq(&blkcg->lock);
-- 
2.27.0




  parent reply	other threads:[~2021-02-11 15:40 UTC|newest]

Thread overview: 68+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-02-11 15:01 [PATCH 5.10 00/54] 5.10.16-rc1 review Greg Kroah-Hartman
2021-02-11 15:01 ` [PATCH 5.10 01/54] io_uring: simplify io_task_match() Greg Kroah-Hartman
2021-02-11 15:01 ` [PATCH 5.10 02/54] io_uring: add a {task,files} pair matching helper Greg Kroah-Hartman
2021-02-11 15:01 ` [PATCH 5.10 03/54] io_uring: dont iterate io_uring_cancel_files() Greg Kroah-Hartman
2021-02-11 15:01 ` [PATCH 5.10 04/54] io_uring: pass files into kill timeouts/poll Greg Kroah-Hartman
2021-02-11 15:01 ` [PATCH 5.10 05/54] io_uring: always batch cancel in *cancel_files() Greg Kroah-Hartman
2021-02-11 15:01 ` [PATCH 5.10 06/54] io_uring: fix files cancellation Greg Kroah-Hartman
2021-02-11 15:01 ` [PATCH 5.10 07/54] io_uring: account io_uring internal files as REQ_F_INFLIGHT Greg Kroah-Hartman
2021-02-11 15:01 ` [PATCH 5.10 08/54] io_uring: if we see flush on exit, cancel related tasks Greg Kroah-Hartman
2021-02-11 15:01 ` [PATCH 5.10 09/54] io_uring: fix __io_uring_files_cancel() with TASK_UNINTERRUPTIBLE Greg Kroah-Hartman
2021-02-11 15:01 ` [PATCH 5.10 10/54] io_uring: replace inflight_wait with tctx->wait Greg Kroah-Hartman
2021-02-11 15:01 ` [PATCH 5.10 11/54] io_uring: fix cancellation taking mutex while TASK_UNINTERRUPTIBLE Greg Kroah-Hartman
2021-02-11 15:01 ` [PATCH 5.10 12/54] io_uring: fix flush cqring overflow list while TASK_INTERRUPTIBLE Greg Kroah-Hartman
2021-02-11 15:01 ` [PATCH 5.10 13/54] io_uring: fix list corruption for splice file_get Greg Kroah-Hartman
2021-02-11 15:01 ` [PATCH 5.10 14/54] io_uring: fix sqo ownership false positive warning Greg Kroah-Hartman
2021-02-11 15:01 ` [PATCH 5.10 15/54] io_uring: reinforce cancel on flush during exit Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.10 16/54] io_uring: drop mm/files between task_work_submit Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.10 17/54] gpiolib: cdev: clear debounce period if line set to output Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.10 18/54] powerpc/64/signal: Fix regression in __kernel_sigtramp_rt64() semantics Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.10 19/54] af_key: relax availability checks for skb size calculation Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.10 20/54] regulator: core: avoid regulator_resolve_supply() race condition Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.10 21/54] ASoC: wm_adsp: Fix control name parsing for multi-fw Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.10 22/54] drm/nouveau/nvif: fix method count when pushing an array Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.10 23/54] mac80211: 160MHz with extended NSS BW in CSA Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.10 24/54] ASoC: Intel: Skylake: Zero snd_ctl_elem_value Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.10 25/54] chtls: Fix potential resource leak Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.10 26/54] pNFS/NFSv4: Try to return invalid layout in pnfs_layout_process() Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.10 27/54] pNFS/NFSv4: Improve rejection of out-of-order layouts Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.10 28/54] ALSA: hda: intel-dsp-config: add PCI id for TGL-H Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.10 29/54] ASoC: ak4458: correct reset polarity Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.10 30/54] ASoC: Intel: sof_sdw: set proper flags for Dell TGL-H SKU 0A5E Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.10 31/54] iwlwifi: mvm: skip power command when unbinding vif during CSA Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.10 32/54] iwlwifi: mvm: take mutex for calling iwl_mvm_get_sync_time() Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.10 33/54] iwlwifi: pcie: add a NULL check in iwl_pcie_txq_unmap Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.10 34/54] iwlwifi: pcie: fix context info memory leak Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.10 35/54] iwlwifi: mvm: invalidate IDs of internal stations at mvm start Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.10 36/54] iwlwifi: pcie: add rules to match Qu with Hr2 Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.10 37/54] iwlwifi: mvm: guard against device removal in reprobe Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.10 38/54] iwlwifi: queue: bail out on invalid freeing Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.10 39/54] SUNRPC: Move simple_get_bytes and simple_get_netobj into private header Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.10 40/54] SUNRPC: Handle 0 length opaque XDR object data properly Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.10 41/54] i2c: mediatek: Move suspend and resume handling to NOIRQ phase Greg Kroah-Hartman
2021-02-11 15:02 ` Greg Kroah-Hartman [this message]
2021-02-11 15:02 ` [PATCH 5.10 43/54] regulator: Fix lockdep warning resolving supplies Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.10 44/54] bpf: Fix verifier jmp32 pruning decision logic Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.10 45/54] bpf: Fix 32 bit src register truncation on div/mod Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.10 46/54] bpf: Fix verifier jsgt branch analysis on max bound Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.10 47/54] drm/i915: Fix ICL MG PHY vswing handling Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.10 48/54] drm/i915: Skip vswing programming for TBT Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.10 49/54] nilfs2: make splice write available again Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.10 50/54] Revert "mm: memcontrol: avoid workload stalls when lowering memory.high" Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.10 51/54] squashfs: avoid out of bounds writes in decompressors Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.10 52/54] squashfs: add more sanity checks in id lookup Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.10 53/54] squashfs: add more sanity checks in inode lookup Greg Kroah-Hartman
2021-02-11 15:02 ` [PATCH 5.10 54/54] squashfs: add more sanity checks in xattr id lookup Greg Kroah-Hartman
2021-02-12  3:16 ` [PATCH 5.10 00/54] 5.10.16-rc1 review Naresh Kamboju
2021-02-13 12:58   ` Greg Kroah-Hartman
2021-02-12 16:17 ` Shuah Khan
2021-02-13 12:58   ` Greg Kroah-Hartman
2021-02-17 22:45     ` Shuah Khan
2021-02-12 18:08 ` Guenter Roeck
2021-02-13 12:58   ` Greg Kroah-Hartman
2021-02-12 19:21 ` Florian Fainelli
2021-02-13 12:58   ` Greg Kroah-Hartman
2021-02-12 19:54 ` Pavel Machek
2021-02-13 12:58   ` Greg Kroah-Hartman
2021-02-13  3:20 ` Ross Schmidt
2021-02-13 12:57   ` Greg Kroah-Hartman

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210211150154.707204813@linuxfoundation.org \
    --to=gregkh@linuxfoundation.org \
    --cc=axboe@kernel.dk \
    --cc=baolin.wang@linux.alibaba.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=sashal@kernel.org \
    --cc=stable@vger.kernel.org \
    --cc=tj@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.