public inbox for linux-kernel@vger.kernel.org
 help / color / mirror / Atom feed
From: Peter Zijlstra <peterz@infradead.org>
To: Andrew Morton <akpm@linux-foundation.org>
Cc: Christoph Hellwig <hch@infradead.org>,
	Julian Sun <sunjunchao@bytedance.com>,
	cgroups@vger.kernel.org, linux-fsdevel@vger.kernel.org,
	linux-kernel@vger.kernel.org, viro@zeniv.linux.org.uk,
	brauner@kernel.org, jack@suse.cz, mingo@redhat.com,
	juri.lelli@redhat.com, vincent.guittot@linaro.org,
	dietmar.eggemann@arm.com, rostedt@goodmis.org,
	bsegall@google.com, mgorman@suse.de, vschneid@redhat.com,
	lance.yang@linux.dev, mhiramat@kernel.org, agruenba@redhat.com,
	hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev,
	shakeel.butt@linux.dev, muchun.song@linux.dev
Subject: Re: [PATCH 0/3] Suppress undesirable hung task warnings.
Date: Tue, 23 Sep 2025 09:16:07 +0200	[thread overview]
Message-ID: <20250923071607.GR3245006@noisy.programming.kicks-ass.net> (raw)
In-Reply-To: <20250922145045.afc6593b4e91c55d8edefabb@linux-foundation.org>

On Mon, Sep 22, 2025 at 02:50:45PM -0700, Andrew Morton wrote:
> On Mon, 22 Sep 2025 11:08:32 -0700 Christoph Hellwig <hch@infradead.org> wrote:
> 
> > On Mon, Sep 22, 2025 at 03:27:18PM +0200, Peter Zijlstra wrote:
> > > > Julian Sun (3):
> > > >   sched: Introduce a new flag PF_DONT_HUNG.
> > > >   writeback: Introduce wb_wait_for_completion_no_hung().
> > > >   memcg: Don't trigger hung task when memcg is releasing.
> > > 
> > > This is all quite terrible. I'm not at all sure why a task that is
> > > genuinely not making progress and isn't killable should not be reported.
> > 
> > The hung device detector is way to aggressive for very slow I/O.
> > See blk_wait_io, which has been around for a long time to work
> > around just that.  Given that this series targets writeback I suspect
> > it is about an overloaded device as well.
> 
> Yup, it's writeback - the bug report is in
> https://lkml.kernel.org/r/20250917212959.355656-1-sunjunchao@bytedance.com
> 
> Memory is big and storage is slow, there's nothing wrong if a task
> which is designed to wait for writeback waits for a long time.
> 
> Of course, there's something wrong if some other task which isn't
> designed to wait for writeback gets stuck waiting for the task which
> *is* designed to wait for writeback, but we'll still warn about that.
> 
> 
> Regarding an implementation, I'm wondering if we can put a flag in
> `struct completion' telling the hung task detector that this one is
> expected to wait for long periods sometimes.  Probably messy and it
> only works for completions (not semaphores, mutexes, etc).  Just
> putting it out there ;)

So the problem is that there *is* progress (albeit rather slowly), the
watchdog just doesn't see that. Perhaps that is the thing we should look
at fixing.

How about something like the below? That will 'spuriously' wake up the
waiters as long as there is some progress being made. Thereby increasing
the context switch counters of the tasks and thus the hung_task watchdog
sees progress.

This approach should be safer than the blk_wait_io() hack, which has a
timer ticking, regardless of actual completions happening or not.

---

diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c
index a07b8cf73ae2..1326193b4d95 100644
--- a/fs/fs-writeback.c
+++ b/fs/fs-writeback.c
@@ -174,9 +174,10 @@ static void finish_writeback_work(struct wb_writeback_work *work)
 		kfree(work);
 	if (done) {
 		wait_queue_head_t *waitq = done->waitq;
+		bool force_wake = (jiffies - done->stamp) > HZ/2;
 
 		/* @done can't be accessed after the following dec */
-		if (atomic_dec_and_test(&done->cnt))
+		if (atomic_dec_and_test(&done->cnt) || force_wake)
 			wake_up_all(waitq);
 	}
 }
@@ -213,7 +214,7 @@ static void wb_queue_work(struct bdi_writeback *wb,
 void wb_wait_for_completion(struct wb_completion *done)
 {
 	atomic_dec(&done->cnt);		/* put down the initial count */
-	wait_event(*done->waitq, !atomic_read(&done->cnt));
+	wait_event(*done->waitq, ({ done->stamp = jiffies; !atomic_read(&done->cnt); }));
 }
 
 #ifdef CONFIG_CGROUP_WRITEBACK
diff --git a/include/linux/backing-dev-defs.h b/include/linux/backing-dev-defs.h
index 2ad261082bba..197593193ce3 100644
--- a/include/linux/backing-dev-defs.h
+++ b/include/linux/backing-dev-defs.h
@@ -63,6 +63,7 @@ enum wb_reason {
 struct wb_completion {
 	atomic_t		cnt;
 	wait_queue_head_t	*waitq;
+	unsigned long		stamp;
 };
 
 #define __WB_COMPLETION_INIT(_waitq)	\

  reply	other threads:[~2025-09-23  7:16 UTC|newest]

Thread overview: 23+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2025-09-22  9:41 [PATCH 0/3] Suppress undesirable hung task warnings Julian Sun
2025-09-22  9:41 ` [PATCH 1/3] sched: Introduce a new flag PF_DONT_HUNG Julian Sun
2025-09-22  9:41 ` [PATCH 2/3] writeback: Introduce wb_wait_for_completion_no_hung() Julian Sun
2025-09-22  9:41 ` [PATCH 3/3] memcg: Don't trigger hung task warnings when memcg is releasing resources Julian Sun
2025-09-22 11:38 ` [PATCH 0/3] Suppress undesirable hung task warnings Lance Yang
2025-09-22 12:40   ` Julian Sun
2025-09-22 13:12     ` Lance Yang
2025-09-22 21:57   ` Andrew Morton
2025-09-23  2:30     ` Lance Yang
2025-09-23  2:45       ` [External] " Julian Sun
2025-09-23  3:18         ` Lance Yang
2025-09-22 13:07 ` Michal Hocko
2025-09-22 14:24   ` [External] " Julian Sun
2025-09-22 13:27 ` Peter Zijlstra
2025-09-22 14:29   ` [External] " Julian Sun
2025-09-22 15:27   ` Jan Kara
2025-09-22 18:08   ` Christoph Hellwig
2025-09-22 21:50     ` Andrew Morton
2025-09-23  7:16       ` Peter Zijlstra [this message]
2025-09-23 12:44         ` Masami Hiramatsu
2025-09-24 10:34         ` Jan Kara
2025-09-25 15:07           ` [External] " Julian Sun
2025-09-25 16:30             ` Jan Kara

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20250923071607.GR3245006@noisy.programming.kicks-ass.net \
    --to=peterz@infradead.org \
    --cc=agruenba@redhat.com \
    --cc=akpm@linux-foundation.org \
    --cc=brauner@kernel.org \
    --cc=bsegall@google.com \
    --cc=cgroups@vger.kernel.org \
    --cc=dietmar.eggemann@arm.com \
    --cc=hannes@cmpxchg.org \
    --cc=hch@infradead.org \
    --cc=jack@suse.cz \
    --cc=juri.lelli@redhat.com \
    --cc=lance.yang@linux.dev \
    --cc=linux-fsdevel@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mgorman@suse.de \
    --cc=mhiramat@kernel.org \
    --cc=mhocko@kernel.org \
    --cc=mingo@redhat.com \
    --cc=muchun.song@linux.dev \
    --cc=roman.gushchin@linux.dev \
    --cc=rostedt@goodmis.org \
    --cc=shakeel.butt@linux.dev \
    --cc=sunjunchao@bytedance.com \
    --cc=vincent.guittot@linaro.org \
    --cc=viro@zeniv.linux.org.uk \
    --cc=vschneid@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox