[RFC PATCH] kernfs: release kernfs_mutex before the inode allocation

[RFC PATCH] kernfs: release kernfs_mutex before the inode allocation

[Minchan Kim]

The kernfs implementation has big lock granularity(kernfs_rwsem) so
every kernfs-based(e.g., sysfs, cgroup, dmabuf) fs are able to compete
the lock. Thus, if one of userspace goes the sleep under holding
the lock for a long time, rest of them should wait it. A example is
the holder goes direct reclaim with the lock since it needs memory
allocation. Let's fix it at common technique that release the lock
and then allocate the memory. Fortunately, kernfs looks like have
an refcount so I hope it's fine.

Signed-off-by: Minchan Kim <minchan@kernel.org>
---
 fs/kernfs/dir.c             | 14 +++++++++++---
 fs/kernfs/inode.c           |  2 +-
 fs/kernfs/kernfs-internal.h |  1 +
 3 files changed, 13 insertions(+), 4 deletions(-)

diff --git a/fs/kernfs/dir.c b/fs/kernfs/dir.c
index 8e0a1378a4b1..ecdb2975060d 100644
--- a/fs/kernfs/dir.c
+++ b/fs/kernfs/dir.c
@@ -1119,9 +1119,17 @@ static struct dentry *kernfs_iop_lookup(struct inode *dir,
 			up_read(&kernfs_rwsem);
 			return NULL;
 		}
-		inode = kernfs_get_inode(dir->i_sb, kn);
-		if (!inode)
-			inode = ERR_PTR(-ENOMEM);
+		kernfs_get(kn);
+		up_read(&kernfs_rwsem);
+		inode = iget_locked(dir->i_sb, kernfs_ino(kn));
+		if (!inode) {
+			kernfs_put(kn);
+			return ERR_PTR(-ENOMEM);
+		}
+		down_read(&kernfs_rwsem);
+		if (inode->i_state & I_NEW)
+			kernfs_init_inode(kn, inode);
+		kernfs_put(kn);
 	}
 	/*
 	 * Needed for negative dentry validation.
diff --git a/fs/kernfs/inode.c b/fs/kernfs/inode.c
index c0eae1725435..6e2004010435 100644
--- a/fs/kernfs/inode.c
+++ b/fs/kernfs/inode.c
@@ -195,7 +195,7 @@ int kernfs_iop_getattr(struct user_namespace *mnt_userns,
 	return 0;
 }
 
-static void kernfs_init_inode(struct kernfs_node *kn, struct inode *inode)
+void kernfs_init_inode(struct kernfs_node *kn, struct inode *inode)
 {
 	kernfs_get(kn);
 	inode->i_private = kn;
diff --git a/fs/kernfs/kernfs-internal.h b/fs/kernfs/kernfs-internal.h
index f9cc912c31e1..eef7656f7cd8 100644
--- a/fs/kernfs/kernfs-internal.h
+++ b/fs/kernfs/kernfs-internal.h
@@ -118,6 +118,7 @@ int kernfs_iop_getattr(struct user_namespace *mnt_userns,
 		       u32 request_mask, unsigned int query_flags);
 ssize_t kernfs_iop_listxattr(struct dentry *dentry, char *buf, size_t size);
 int __kernfs_setattr(struct kernfs_node *kn, const struct iattr *iattr);
+void kernfs_init_inode(struct kernfs_node *kn, struct inode *inode);
 
 /*
  * dir.c
-- 
2.34.0.rc1.387.gb447b232ab-goog

Re: [RFC PATCH] kernfs: release kernfs_mutex before the inode allocation

[Greg Kroah-Hartman]

On Tue, Nov 16, 2021 at 11:43:17AM -0800, Minchan Kim wrote:
> The kernfs implementation has big lock granularity(kernfs_rwsem) so
> every kernfs-based(e.g., sysfs, cgroup, dmabuf) fs are able to compete
> the lock. Thus, if one of userspace goes the sleep under holding
> the lock for a long time, rest of them should wait it. A example is
> the holder goes direct reclaim with the lock since it needs memory
> allocation. Let's fix it at common technique that release the lock
> and then allocate the memory. Fortunately, kernfs looks like have
> an refcount so I hope it's fine.
> 
> Signed-off-by: Minchan Kim <minchan@kernel.org>
> ---
>  fs/kernfs/dir.c             | 14 +++++++++++---
>  fs/kernfs/inode.c           |  2 +-
>  fs/kernfs/kernfs-internal.h |  1 +
>  3 files changed, 13 insertions(+), 4 deletions(-)

What workload hits this lock to cause it to be noticable?

There was a bunch of recent work in this area to make this much more
fine-grained, and the theoritical benchmarks that people created (adding
10s of thousands of scsi disks at boot time) have gotten better.

But in that work, no one could find a real benchmark or use case that
anyone could even notice this type of thing.  What do you have that
shows this?
thanks,

greg k-h

Re: [RFC PATCH] kernfs: release kernfs_mutex before the inode allocation

[Minchan Kim]

On Tue, Nov 16, 2021 at 08:49:46PM +0100, Greg Kroah-Hartman wrote:
> On Tue, Nov 16, 2021 at 11:43:17AM -0800, Minchan Kim wrote:
> > The kernfs implementation has big lock granularity(kernfs_rwsem) so
> > every kernfs-based(e.g., sysfs, cgroup, dmabuf) fs are able to compete
> > the lock. Thus, if one of userspace goes the sleep under holding
> > the lock for a long time, rest of them should wait it. A example is
> > the holder goes direct reclaim with the lock since it needs memory
> > allocation. Let's fix it at common technique that release the lock
> > and then allocate the memory. Fortunately, kernfs looks like have
> > an refcount so I hope it's fine.
> > 
> > Signed-off-by: Minchan Kim <minchan@kernel.org>
> > ---
> >  fs/kernfs/dir.c             | 14 +++++++++++---
> >  fs/kernfs/inode.c           |  2 +-
> >  fs/kernfs/kernfs-internal.h |  1 +
> >  3 files changed, 13 insertions(+), 4 deletions(-)
> 
> What workload hits this lock to cause it to be noticable?

A app launching since it was dropping the frame since the
latency was too long.

> 
> There was a bunch of recent work in this area to make this much more
> fine-grained, and the theoritical benchmarks that people created (adding
> 10s of thousands of scsi disks at boot time) have gotten better.
> 
> But in that work, no one could find a real benchmark or use case that
> anyone could even notice this type of thing.  What do you have that
> shows this?

https://developer.android.com/studio/command-line/perfetto
https://perfetto.dev/docs/data-sources/cpu-scheduling

Android has perfetto tracing system and can show where processes
were stuck. This case was the lock since holder was in direct reclaim
path.

Re: [RFC PATCH] kernfs: release kernfs_mutex before the inode allocation

[Greg Kroah-Hartman]

On Tue, Nov 16, 2021 at 01:36:01PM -0800, Minchan Kim wrote:
> On Tue, Nov 16, 2021 at 08:49:46PM +0100, Greg Kroah-Hartman wrote:
> > On Tue, Nov 16, 2021 at 11:43:17AM -0800, Minchan Kim wrote:
> > > The kernfs implementation has big lock granularity(kernfs_rwsem) so
> > > every kernfs-based(e.g., sysfs, cgroup, dmabuf) fs are able to compete
> > > the lock. Thus, if one of userspace goes the sleep under holding
> > > the lock for a long time, rest of them should wait it. A example is
> > > the holder goes direct reclaim with the lock since it needs memory
> > > allocation. Let's fix it at common technique that release the lock
> > > and then allocate the memory. Fortunately, kernfs looks like have
> > > an refcount so I hope it's fine.
> > > 
> > > Signed-off-by: Minchan Kim <minchan@kernel.org>
> > > ---
> > >  fs/kernfs/dir.c             | 14 +++++++++++---
> > >  fs/kernfs/inode.c           |  2 +-
> > >  fs/kernfs/kernfs-internal.h |  1 +
> > >  3 files changed, 13 insertions(+), 4 deletions(-)
> > 
> > What workload hits this lock to cause it to be noticable?
> 
> A app launching since it was dropping the frame since the
> latency was too long.

How does running a program interact with kernfs filesystems?  Which
one(s)?

> > There was a bunch of recent work in this area to make this much more
> > fine-grained, and the theoritical benchmarks that people created (adding
> > 10s of thousands of scsi disks at boot time) have gotten better.
> > 
> > But in that work, no one could find a real benchmark or use case that
> > anyone could even notice this type of thing.  What do you have that
> > shows this?
> 
> https://developer.android.com/studio/command-line/perfetto
> https://perfetto.dev/docs/data-sources/cpu-scheduling

That is links to a tool, not a test we can run ourselves.

Or how about the output of that tool?

> Android has perfetto tracing system and can show where processes
> were stuck. This case was the lock since holder was in direct reclaim
> path.

Reclaim of what?  What is the interaction here with kernfs?  Normally
this filesystem is not on any "fast paths" that I know of.

More specifics would be nice :)

thanks,

greg k-h

Re: [RFC PATCH] kernfs: release kernfs_mutex before the inode allocation

[Minchan Kim]

On Wed, Nov 17, 2021 at 07:44:44AM +0100, Greg Kroah-Hartman wrote:
> On Tue, Nov 16, 2021 at 01:36:01PM -0800, Minchan Kim wrote:
> > On Tue, Nov 16, 2021 at 08:49:46PM +0100, Greg Kroah-Hartman wrote:
> > > On Tue, Nov 16, 2021 at 11:43:17AM -0800, Minchan Kim wrote:
> > > > The kernfs implementation has big lock granularity(kernfs_rwsem) so
> > > > every kernfs-based(e.g., sysfs, cgroup, dmabuf) fs are able to compete
> > > > the lock. Thus, if one of userspace goes the sleep under holding
> > > > the lock for a long time, rest of them should wait it. A example is
> > > > the holder goes direct reclaim with the lock since it needs memory
> > > > allocation. Let's fix it at common technique that release the lock
> > > > and then allocate the memory. Fortunately, kernfs looks like have
> > > > an refcount so I hope it's fine.
> > > > 
> > > > Signed-off-by: Minchan Kim <minchan@kernel.org>
> > > > ---
> > > >  fs/kernfs/dir.c             | 14 +++++++++++---
> > > >  fs/kernfs/inode.c           |  2 +-
> > > >  fs/kernfs/kernfs-internal.h |  1 +
> > > >  3 files changed, 13 insertions(+), 4 deletions(-)
> > > 
> > > What workload hits this lock to cause it to be noticable?
> > 
> > A app launching since it was dropping the frame since the
> > latency was too long.
> 
> How does running a program interact with kernfs filesystems?  Which
> one(s)?

A app launching involves dma_buf exports which creates kobject
and add it to the kernfs with down_write - kernfs_add_one.

At the same time in other CPU, a random process was accessing
sysfs and the kernfs_iop_lookup was already hoding the kernfs_rwsem
and ran under direct reclaim patch due to alloc_inode in
kerfs_get_inode.

Therefore, the app is stuck on the lock and lose frames so enduser
sees the jank.

> 
> > > There was a bunch of recent work in this area to make this much more
> > > fine-grained, and the theoritical benchmarks that people created (adding
> > > 10s of thousands of scsi disks at boot time) have gotten better.
> > > 
> > > But in that work, no one could find a real benchmark or use case that
> > > anyone could even notice this type of thing.  What do you have that
> > > shows this?
> > 
> > https://developer.android.com/studio/command-line/perfetto
> > https://perfetto.dev/docs/data-sources/cpu-scheduling
> 
> That is links to a tool, not a test we can run ourselves.
> 
> Or how about the output of that tool?
> 
> > Android has perfetto tracing system and can show where processes
> > were stuck. This case was the lock since holder was in direct reclaim
> > path.
> 
> Reclaim of what?  What is the interaction here with kernfs?  Normally
> this filesystem is not on any "fast paths" that I know of.
> 
> More specifics would be nice :)

I hope it's enough above.

Re: [RFC PATCH] kernfs: release kernfs_mutex before the inode allocation

[Greg Kroah-Hartman]

On Tue, Nov 16, 2021 at 11:27:56PM -0800, Minchan Kim wrote:
> On Wed, Nov 17, 2021 at 07:44:44AM +0100, Greg Kroah-Hartman wrote:
> > On Tue, Nov 16, 2021 at 01:36:01PM -0800, Minchan Kim wrote:
> > > On Tue, Nov 16, 2021 at 08:49:46PM +0100, Greg Kroah-Hartman wrote:
> > > > On Tue, Nov 16, 2021 at 11:43:17AM -0800, Minchan Kim wrote:
> > > > > The kernfs implementation has big lock granularity(kernfs_rwsem) so
> > > > > every kernfs-based(e.g., sysfs, cgroup, dmabuf) fs are able to compete
> > > > > the lock. Thus, if one of userspace goes the sleep under holding
> > > > > the lock for a long time, rest of them should wait it. A example is
> > > > > the holder goes direct reclaim with the lock since it needs memory
> > > > > allocation. Let's fix it at common technique that release the lock
> > > > > and then allocate the memory. Fortunately, kernfs looks like have
> > > > > an refcount so I hope it's fine.
> > > > > 
> > > > > Signed-off-by: Minchan Kim <minchan@kernel.org>
> > > > > ---
> > > > >  fs/kernfs/dir.c             | 14 +++++++++++---
> > > > >  fs/kernfs/inode.c           |  2 +-
> > > > >  fs/kernfs/kernfs-internal.h |  1 +
> > > > >  3 files changed, 13 insertions(+), 4 deletions(-)
> > > > 
> > > > What workload hits this lock to cause it to be noticable?
> > > 
> > > A app launching since it was dropping the frame since the
> > > latency was too long.
> > 
> > How does running a program interact with kernfs filesystems?  Which
> > one(s)?
> 
> A app launching involves dma_buf exports which creates kobject
> and add it to the kernfs with down_write - kernfs_add_one.

I thought the "create a dma_buf kobject" kernel change was fixed up to
not do that anymore as that was a known performance issue.

Creating kobjects should NOT be on a fast path, if they are, that needs
to be fixed.

> At the same time in other CPU, a random process was accessing
> sysfs and the kernfs_iop_lookup was already hoding the kernfs_rwsem
> and ran under direct reclaim patch due to alloc_inode in
> kerfs_get_inode.

What program is constantly hitting sysfs?  sysfs is not for
performance-critical things, right?

> Therefore, the app is stuck on the lock and lose frames so enduser
> sees the jank.

But how does this patch work around it?  It seems like you are
special-casing the kobject creation path only.

And is this the case really on 5.15?  I thought the kernfs locks were
broken up again to not cause this problem in 5.14 or so.

thanks,

greg k-h

Re: [RFC PATCH] kernfs: release kernfs_mutex before the inode allocation

[Minchan Kim]

On Wed, Nov 17, 2021 at 08:39:44AM +0100, Greg Kroah-Hartman wrote:
> On Tue, Nov 16, 2021 at 11:27:56PM -0800, Minchan Kim wrote:
> > On Wed, Nov 17, 2021 at 07:44:44AM +0100, Greg Kroah-Hartman wrote:
> > > On Tue, Nov 16, 2021 at 01:36:01PM -0800, Minchan Kim wrote:
> > > > On Tue, Nov 16, 2021 at 08:49:46PM +0100, Greg Kroah-Hartman wrote:
> > > > > On Tue, Nov 16, 2021 at 11:43:17AM -0800, Minchan Kim wrote:
> > > > > > The kernfs implementation has big lock granularity(kernfs_rwsem) so
> > > > > > every kernfs-based(e.g., sysfs, cgroup, dmabuf) fs are able to compete
> > > > > > the lock. Thus, if one of userspace goes the sleep under holding
> > > > > > the lock for a long time, rest of them should wait it. A example is
> > > > > > the holder goes direct reclaim with the lock since it needs memory
> > > > > > allocation. Let's fix it at common technique that release the lock
> > > > > > and then allocate the memory. Fortunately, kernfs looks like have
> > > > > > an refcount so I hope it's fine.
> > > > > > 
> > > > > > Signed-off-by: Minchan Kim <minchan@kernel.org>
> > > > > > ---
> > > > > >  fs/kernfs/dir.c             | 14 +++++++++++---
> > > > > >  fs/kernfs/inode.c           |  2 +-
> > > > > >  fs/kernfs/kernfs-internal.h |  1 +
> > > > > >  3 files changed, 13 insertions(+), 4 deletions(-)
> > > > > 
> > > > > What workload hits this lock to cause it to be noticable?
> > > > 
> > > > A app launching since it was dropping the frame since the
> > > > latency was too long.
> > > 
> > > How does running a program interact with kernfs filesystems?  Which
> > > one(s)?
> > 
> > A app launching involves dma_buf exports which creates kobject
> > and add it to the kernfs with down_write - kernfs_add_one.
> 
> I thought the "create a dma_buf kobject" kernel change was fixed up to
> not do that anymore as that was a known performance issue.
> 
> Creating kobjects should NOT be on a fast path, if they are, that needs
> to be fixed.

Ccing Hridya
I also already mentioned before but it's the as-is.
It would be great to be fixed but kernfs still has the problem
regardless of the dmabuf.

For example, process A hold the lock as read-side and try to
allocate memory and entered the reclaim. process B try to
hold the lock as writes-side(e.g., kernfs_iop_setattr) but
he should wait until process A completes. Next, processs C is
also stuck on the lock as read-side when it tries to access
sysfs since the process B spent a threahold timeout in rwsem.
Here, process C could be critical role to contribute the jank.
What it was doing is just access sysfs but was stuck.

> 
> > At the same time in other CPU, a random process was accessing
> > sysfs and the kernfs_iop_lookup was already hoding the kernfs_rwsem
> > and ran under direct reclaim patch due to alloc_inode in
> > kerfs_get_inode.
> 
> What program is constantly hitting sysfs?  sysfs is not for
> performance-critical things, right?

Constantly hitting sysfs itself is no problem since they need
to read telemetry data from sysfs and it's not latency sensitive.
Here, problem is the reading kernfs could make trouble others who
want to access once the rwsem is involved with exclusive lock mode.
Please look at above scenario.

> 
> > Therefore, the app is stuck on the lock and lose frames so enduser
> > sees the jank.
> 
> But how does this patch work around it?  It seems like you are
> special-casing the kobject creation path only.

It's true. If other path also has the memory allocation with holding
kernfs_rwsem, it could make trouble.

> 
> And is this the case really on 5.15?  I thought the kernfs locks were
> broken up again to not cause this problem in 5.14 or so.

It happened on 5.10 but the path I mentioned were still vulnerable path
with rwsem so it could happen on 5.15, too.

Re: [RFC PATCH] kernfs: release kernfs_mutex before the inode allocation

[Tejun Heo]

Hello,

On Tue, Nov 16, 2021 at 11:27:56PM -0800, Minchan Kim wrote:
> A app launching involves dma_buf exports which creates kobject
> and add it to the kernfs with down_write - kernfs_add_one.
> 
> At the same time in other CPU, a random process was accessing
> sysfs and the kernfs_iop_lookup was already hoding the kernfs_rwsem
> and ran under direct reclaim patch due to alloc_inode in
> kerfs_get_inode.
> 
> Therefore, the app is stuck on the lock and lose frames so enduser
> sees the jank.

So, one really low hanging fruit here would be using a separate rwsem per
superblock. Nothing needs synchronization across different users of kernfs
and the locking is shared just because nobody bothered to separate them out
while generalizing it from sysfs.

Thanks.

-- 
tejun

Re: [RFC PATCH] kernfs: release kernfs_mutex before the inode allocation

[Minchan Kim]

Hi Tejun,

On Wed, Nov 17, 2021 at 11:45:46AM -1000, Tejun Heo wrote:
> Hello,
> 
> On Tue, Nov 16, 2021 at 11:27:56PM -0800, Minchan Kim wrote:
> > A app launching involves dma_buf exports which creates kobject
> > and add it to the kernfs with down_write - kernfs_add_one.
> > 
> > At the same time in other CPU, a random process was accessing
> > sysfs and the kernfs_iop_lookup was already hoding the kernfs_rwsem
> > and ran under direct reclaim patch due to alloc_inode in
> > kerfs_get_inode.
> > 
> > Therefore, the app is stuck on the lock and lose frames so enduser
> > sees the jank.
> 
> So, one really low hanging fruit here would be using a separate rwsem per
> superblock. Nothing needs synchronization across different users of kernfs
> and the locking is shared just because nobody bothered to separate them out
> while generalizing it from sysfs.

That's really what I wanted but had a question whether we can access
superblock from the kernfs_node all the time since there are some
functions to access the kernfs_rwsem without ionde, sb context.

Is it doable to get the superblock from the kernfs_node all the time?

Re: [RFC PATCH] kernfs: release kernfs_mutex before the inode allocation

[Tejun Heo]

Hello,

On Wed, Nov 17, 2021 at 02:13:35PM -0800, Minchan Kim wrote:
> > So, one really low hanging fruit here would be using a separate rwsem per
> > superblock. Nothing needs synchronization across different users of kernfs
> > and the locking is shared just because nobody bothered to separate them out
> > while generalizing it from sysfs.
> 
> That's really what I wanted but had a question whether we can access
> superblock from the kernfs_node all the time since there are some
> functions to access the kernfs_rwsem without ionde, sb context.
> 
> Is it doable to get the superblock from the kernfs_node all the time?

Ah, right, kernfs_node doesn't point back to kernfs_root. I guess it can go
one of three ways:

a. Follow parent until root kernfs_node and make that guy point to
   kernfs_root through its parent field. This isn't great but the hotter
   paths all have sb / inode already, I think, so if we do this only in the
   really cold paths, it likely isn't too bad.

b. Change the interface so that the callers have to provide kernfs_root. I
   don't think this is gonna be a huge problem. There are a few users of
   kernfs and they always know their roots.

c. Add a field to kernfs_node so that we can always find kernfs_root.

I think b is likely the cheapest && cleanest.

Thanks.

-- 
tejun

Re: [RFC PATCH] kernfs: release kernfs_mutex before the inode allocation

[Minchan Kim]

On Wed, Nov 17, 2021 at 12:23:55PM -1000, Tejun Heo wrote:
> Hello,
> 
> On Wed, Nov 17, 2021 at 02:13:35PM -0800, Minchan Kim wrote:
> > > So, one really low hanging fruit here would be using a separate rwsem per
> > > superblock. Nothing needs synchronization across different users of kernfs
> > > and the locking is shared just because nobody bothered to separate them out
> > > while generalizing it from sysfs.
> > 
> > That's really what I wanted but had a question whether we can access
> > superblock from the kernfs_node all the time since there are some
> > functions to access the kernfs_rwsem without ionde, sb context.
> > 
> > Is it doable to get the superblock from the kernfs_node all the time?
> 
> Ah, right, kernfs_node doesn't point back to kernfs_root. I guess it can go
> one of three ways:

Thanks for the suggestion, Tejun.

I found kernfs_root and it seems like to return kernfs_root from kernfs_node.
If it's true all the case, we would put the rwsem in kernfs_root and change
would be straightforward. Do you see any problem?

> 
> a. Follow parent until root kernfs_node and make that guy point to
>    kernfs_root through its parent field. This isn't great but the hotter
>    paths all have sb / inode already, I think, so if we do this only in the
>    really cold paths, it likely isn't too bad.
> 
> b. Change the interface so that the callers have to provide kernfs_root. I
>    don't think this is gonna be a huge problem. There are a few users of
>    kernfs and they always know their roots.
> 
> c. Add a field to kernfs_node so that we can always find kernfs_root.
> 
> I think b is likely the cheapest && cleanest.
> 
> Thanks.
> 
> -- 
> tejun

Re: [RFC PATCH] kernfs: release kernfs_mutex before the inode allocation

[Tejun Heo]

On Wed, Nov 17, 2021 at 05:55:33PM -0800, Minchan Kim wrote:
> > Ah, right, kernfs_node doesn't point back to kernfs_root. I guess it can go
> > one of three ways:
> 
> Thanks for the suggestion, Tejun.
> 
> I found kernfs_root and it seems like to return kernfs_root from kernfs_node.
> If it's true all the case, we would put the rwsem in kernfs_root and change
> would be straightforward. Do you see any problem?

Ah, you're right. I forgot that directory kernfs_node always points to the
root so we need at most one walk up the tree to locate kernfs_root. Yeah,
that should definitely work.

Thanks.

-- 
tejun

Re: [RFC PATCH] kernfs: release kernfs_mutex before the inode allocation

[Wenwu Hou]

On Tue, Nov 16, 2021 at 08:49:46PM +0100, Greg Kroah-Hartman wrote:
> On Tue, Nov 16, 2021 at 11:43:17AM -0800, Minchan Kim wrote:
> > The kernfs implementation has big lock granularity(kernfs_rwsem) so
> > every kernfs-based(e.g., sysfs, cgroup, dmabuf) fs are able to compete
> > the lock. Thus, if one of userspace goes the sleep under holding
> > the lock for a long time, rest of them should wait it. A example is
> > the holder goes direct reclaim with the lock since it needs memory
> > allocation. Let's fix it at common technique that release the lock
> > and then allocate the memory. Fortunately, kernfs looks like have
> > an refcount so I hope it's fine.
> >
> > Signed-off-by: Minchan Kim <minchan@kernel.org>
> > ---
> >  fs/kernfs/dir.c             | 14 +++++++++++---
> >  fs/kernfs/inode.c           |  2 +-
> >  fs/kernfs/kernfs-internal.h |  1 +
> >  3 files changed, 13 insertions(+), 4 deletions(-)
>
> What workload hits this lock to cause it to be noticable?
>
> There was a bunch of recent work in this area to make this much more
> fine-grained, and the theoritical benchmarks that people created (adding
> 10s of thousands of scsi disks at boot time) have gotten better.

Hi,

By 2026, the kernfs_rwsem has been split into per-fs. But the problem still
exists.

In a k8s cluster environment, there are a lot of processes reading /sys, for example:

Container runtime: runc
Prometheus Exporter: node_exporter

If a tenant's container processes read /sys and sleep on the OOM (or direct
reclaim) path, both runc and node_exporter will be blocked, which will cause
the cluster worker node to become unschedulable.

For example, the nodejs runtime reads
/sys/devices/system/cpu/cpu%u/cpufreq/scaling_cur_freq [1]

Jun 22 21:43:17 kernel: task:node            state:D stack:0     pid:2621530 ppid:1062663 flags:0x00004002
Jun 22 21:43:17 kernel: Call Trace:
Jun 22 21:43:17 kernel:  <TASK>
Jun 22 21:43:17 kernel:  __schedule+0x278/0x740
Jun 22 21:43:17 kernel:  schedule+0x5a/0xd0
Jun 22 21:43:17 kernel:  schedule_preempt_disabled+0x11/0x20
Jun 22 21:43:17 kernel:  __mutex_lock.constprop.0+0x376/0x680
Jun 22 21:43:17 kernel:  ? kvm_sched_clock_read+0xd/0x20
Jun 22 21:43:17 kernel:  mem_cgroup_out_of_memory+0x53/0x150
Jun 22 21:43:17 kernel:  try_charge_memcg+0x682/0x7d0
Jun 22 21:43:17 kernel:  ? memcg_list_lru_alloc+0xa7/0x330
Jun 22 21:43:17 kernel:  obj_cgroup_charge+0x70/0x170
Jun 22 21:43:17 kernel:  slab_pre_alloc_hook.constprop.0+0xb6/0x1d0
Jun 22 21:43:17 kernel:  ? alloc_inode+0x59/0xc0
Jun 22 21:43:17 kernel:  kmem_cache_alloc_lru+0x54/0x2c0
Jun 22 21:43:17 kernel:  alloc_inode+0x59/0xc0
Jun 22 21:43:17 kernel:  iget_locked+0xe3/0x220
Jun 22 21:43:17 kernel:  kernfs_get_inode+0x18/0x110
Jun 22 21:43:17 kernel:  kernfs_iop_lookup+0x74/0xd0
Jun 22 21:43:17 kernel:  __lookup_slow+0x82/0x130
Jun 22 21:43:17 kernel:  walk_component+0xdb/0x150
Jun 22 21:43:17 kernel:  link_path_walk.part.0.constprop.0+0x240/0x380
Jun 22 21:43:17 kernel:  ? path_init+0x293/0x3d0
Jun 22 21:43:17 kernel:  path_openat+0x85/0x2a0
Jun 22 21:43:17 kernel:  ? seq_printf+0x8e/0xb0
Jun 22 21:43:17 kernel:  do_filp_open+0xb4/0x160
Jun 22 21:43:17 kernel:  ? __check_object_size.part.0+0x5e/0x130
Jun 22 21:43:17 kernel:  do_sys_openat2+0x91/0xc0
Jun 22 21:43:17 kernel:  __x64_sys_openat+0x53/0xa0
Jun 22 21:43:17 kernel:  do_syscall_64+0x35/0x80
Jun 22 21:43:17 kernel:  entry_SYSCALL_64_after_hwframe+0x78/0xe2

A rwsem writer is involved:

Jun 22 21:46:17 kernel: task:kworker/u232:0  state:D stack:0     pid:2886179 ppid:2      flags:0x00004000
Jun 22 21:46:17 kernel: Workqueue: netns cleanup_net
Jun 22 21:46:17 kernel: Call Trace:
Jun 22 21:46:17 kernel:  <TASK>
Jun 22 21:46:17 kernel:  __schedule+0x278/0x740
Jun 22 21:46:17 kernel:  ? psi_group_change+0x226/0x3d0
Jun 22 21:46:17 kernel:  schedule+0x5a/0xd0
Jun 22 21:46:17 kernel:  schedule_preempt_disabled+0x11/0x20
Jun 22 21:46:17 kernel:  rwsem_down_write_slowpath+0x1e2/0x4f0
Jun 22 21:46:17 kernel:  down_write+0x57/0x60
Jun 22 21:46:17 kernel:  kernfs_remove_by_name_ns+0x38/0xc0
Jun 22 21:46:17 kernel:  remove_files+0x2b/0x70
Jun 22 21:46:17 kernel:  sysfs_remove_group+0x38/0x80
Jun 22 21:46:17 kernel:  sysfs_remove_groups+0x29/0x50
Jun 22 21:46:17 kernel:  ib_free_port_attrs+0x92/0x170 [ib_core]
Jun 22 21:46:17 kernel:  rdma_dev_exit_net+0x117/0x1d0 [ib_core]
Jun 22 21:46:17 kernel:  ops_exit_list+0x30/0x70
Jun 22 21:46:17 kernel:  cleanup_net+0x273/0x430
Jun 22 21:46:17 kernel:  process_one_work+0x18a/0x3a0
Jun 22 21:46:17 kernel:  worker_thread+0x277/0x3a0
Jun 22 21:46:17 kernel:  ? __pfx_worker_thread+0x10/0x10
Jun 22 21:46:17 kernel:  kthread+0xe1/0x110
Jun 22 21:46:17 kernel:  ? __pfx_kthread+0x10/0x10
Jun 22 21:46:17 kernel:  ret_from_fork+0x2d/0x50
Jun 22 21:46:17 kernel:  ? __pfx_kthread+0x10/0x10
Jun 22 21:46:17 kernel:  ret_from_fork_asm+0x1b/0x30
Jun 22 21:46:17 kernel:  </TASK>


In our real-world case, thousands of nodejs processes hit the cgroup memory
limit and sleep on the OOM path, runc gets blocked, and consequently the node
remains in an unschedulable state for many hours.


Link:
- [1] https://github.com/nodejs/node/blob/v22.19.0/deps/uv/src/unix/linux.c#L1889

> But in that work, no one could find a real benchmark or use case that
> anyone could even notice this type of thing.  What do you have that
> shows this?
> thanks,
>
> greg k-h

Re: [RFC PATCH] kernfs: release kernfs_mutex before the inode allocation

[Tejun Heo]

Hello,

On Tue, Jun 30, 2026 at 11:26:31AM +0800, Wenwu Hou wrote:
...
> In our real-world case, thousands of nodejs processes hit the cgroup memory
> limit and sleep on the OOM path, runc gets blocked, and consequently the node
> remains in an unschedulable state for many hours.

I don't think we can work around thousands of processes stalling in OOM path
with kernfs locking changes. There are better ways to manage memory
pressure. Please look into memory.high and PSI driven userspace OOM
handling.

Thanks.

-- 
tejun

Re: [RFC PATCH] kernfs: release kernfs_mutex before the inode allocation

[Ian Kent]

On 30/6/26 11:26, Wenwu Hou wrote:
> On Tue, Nov 16, 2021 at 08:49:46PM +0100, Greg Kroah-Hartman wrote:
>> On Tue, Nov 16, 2021 at 11:43:17AM -0800, Minchan Kim wrote:
>>> The kernfs implementation has big lock granularity(kernfs_rwsem) so
>>> every kernfs-based(e.g., sysfs, cgroup, dmabuf) fs are able to compete
>>> the lock. Thus, if one of userspace goes the sleep under holding
>>> the lock for a long time, rest of them should wait it. A example is
>>> the holder goes direct reclaim with the lock since it needs memory
>>> allocation. Let's fix it at common technique that release the lock
>>> and then allocate the memory. Fortunately, kernfs looks like have
>>> an refcount so I hope it's fine.
>>>
>>> Signed-off-by: Minchan Kim <minchan@kernel.org>
>>> ---
>>>   fs/kernfs/dir.c             | 14 +++++++++++---
>>>   fs/kernfs/inode.c           |  2 +-
>>>   fs/kernfs/kernfs-internal.h |  1 +
>>>   3 files changed, 13 insertions(+), 4 deletions(-)
>> What workload hits this lock to cause it to be noticable?

Hi Greg, ;)


Sadly, perhaps by coincidence, I have a case that looks almost identical

to this so I'm pretty interested in the discussion. This scenario didn't

immediately occur to me looking at the vmcores but, yes, I think this is

a serious (potential) problem. But I don't have enough information to put

together a causal scenario either which is a problem.


The type of systems that show these sort of problem are extremely heavy

kernfs users (obviously) so sleeping, even under a read lock, can be a

problem, again obviously.


I'm not sure about dropping and re-taking the lock, although at first sight

it looks like it would be ok, I just don't like this sort of execution

continuity break but a refactor that avoids memory allocations under a lock

seems like a very sensible thing to do. I don't yet have a feel for how far

to go with it either. But given this code path is a very hot in the systems

for which I get these problem reports the inode lookup op might be enough.


>>
>> There was a bunch of recent work in this area to make this much more
>> fine-grained, and the theoritical benchmarks that people created (adding
>> 10s of thousands of scsi disks at boot time) have gotten better.
> Hi,
>
> By 2026, the kernfs_rwsem has been split into per-fs. But the problem still
> exists.

Yes, but for my case I'm not sue how much of that has made it into kernels

people with these systems are using. I have done a fair bit to add these

improvements but change is slow in larger environments.


>
> In a k8s cluster environment, there are a lot of processes reading /sys, for example:
>
> Container runtime: runc
> Prometheus Exporter: node_exporter

Indeed, these and a few other friends too, thend to show up again and again.


>
> If a tenant's container processes read /sys and sleep on the OOM (or direct
> reclaim) path, both runc and node_exporter will be blocked, which will cause
> the cluster worker node to become unschedulable.

In the case I saw it looked like there was a directory remove going on and

there was about 7M negative dentries, roughly double the positives, so I've

been chasing info. to work out if the number of directory children was large

enough to cause this.


Perhaps, as demand grows over time, we will see other scenarios beside the

allocation delays you have seen.


>
> For example, the nodejs runtime reads
> /sys/devices/system/cpu/cpu%u/cpufreq/scaling_cur_freq [1]
>
> Jun 22 21:43:17 kernel: task:node            state:D stack:0     pid:2621530 ppid:1062663 flags:0x00004002
> Jun 22 21:43:17 kernel: Call Trace:
> Jun 22 21:43:17 kernel:  <TASK>
> Jun 22 21:43:17 kernel:  __schedule+0x278/0x740
> Jun 22 21:43:17 kernel:  schedule+0x5a/0xd0
> Jun 22 21:43:17 kernel:  schedule_preempt_disabled+0x11/0x20
> Jun 22 21:43:17 kernel:  __mutex_lock.constprop.0+0x376/0x680
> Jun 22 21:43:17 kernel:  ? kvm_sched_clock_read+0xd/0x20
> Jun 22 21:43:17 kernel:  mem_cgroup_out_of_memory+0x53/0x150
> Jun 22 21:43:17 kernel:  try_charge_memcg+0x682/0x7d0
> Jun 22 21:43:17 kernel:  ? memcg_list_lru_alloc+0xa7/0x330
> Jun 22 21:43:17 kernel:  obj_cgroup_charge+0x70/0x170
> Jun 22 21:43:17 kernel:  slab_pre_alloc_hook.constprop.0+0xb6/0x1d0
> Jun 22 21:43:17 kernel:  ? alloc_inode+0x59/0xc0
> Jun 22 21:43:17 kernel:  kmem_cache_alloc_lru+0x54/0x2c0
> Jun 22 21:43:17 kernel:  alloc_inode+0x59/0xc0
> Jun 22 21:43:17 kernel:  iget_locked+0xe3/0x220
> Jun 22 21:43:17 kernel:  kernfs_get_inode+0x18/0x110
> Jun 22 21:43:17 kernel:  kernfs_iop_lookup+0x74/0xd0
> Jun 22 21:43:17 kernel:  __lookup_slow+0x82/0x130
> Jun 22 21:43:17 kernel:  walk_component+0xdb/0x150
> Jun 22 21:43:17 kernel:  link_path_walk.part.0.constprop.0+0x240/0x380
> Jun 22 21:43:17 kernel:  ? path_init+0x293/0x3d0
> Jun 22 21:43:17 kernel:  path_openat+0x85/0x2a0
> Jun 22 21:43:17 kernel:  ? seq_printf+0x8e/0xb0
> Jun 22 21:43:17 kernel:  do_filp_open+0xb4/0x160
> Jun 22 21:43:17 kernel:  ? __check_object_size.part.0+0x5e/0x130
> Jun 22 21:43:17 kernel:  do_sys_openat2+0x91/0xc0
> Jun 22 21:43:17 kernel:  __x64_sys_openat+0x53/0xa0
> Jun 22 21:43:17 kernel:  do_syscall_64+0x35/0x80
> Jun 22 21:43:17 kernel:  entry_SYSCALL_64_after_hwframe+0x78/0xe2
>
> A rwsem writer is involved:
>
> Jun 22 21:46:17 kernel: task:kworker/u232:0  state:D stack:0     pid:2886179 ppid:2      flags:0x00004000
> Jun 22 21:46:17 kernel: Workqueue: netns cleanup_net
> Jun 22 21:46:17 kernel: Call Trace:
> Jun 22 21:46:17 kernel:  <TASK>
> Jun 22 21:46:17 kernel:  __schedule+0x278/0x740
> Jun 22 21:46:17 kernel:  ? psi_group_change+0x226/0x3d0
> Jun 22 21:46:17 kernel:  schedule+0x5a/0xd0
> Jun 22 21:46:17 kernel:  schedule_preempt_disabled+0x11/0x20
> Jun 22 21:46:17 kernel:  rwsem_down_write_slowpath+0x1e2/0x4f0
> Jun 22 21:46:17 kernel:  down_write+0x57/0x60
> Jun 22 21:46:17 kernel:  kernfs_remove_by_name_ns+0x38/0xc0
> Jun 22 21:46:17 kernel:  remove_files+0x2b/0x70
> Jun 22 21:46:17 kernel:  sysfs_remove_group+0x38/0x80
> Jun 22 21:46:17 kernel:  sysfs_remove_groups+0x29/0x50
> Jun 22 21:46:17 kernel:  ib_free_port_attrs+0x92/0x170 [ib_core]
> Jun 22 21:46:17 kernel:  rdma_dev_exit_net+0x117/0x1d0 [ib_core]
> Jun 22 21:46:17 kernel:  ops_exit_list+0x30/0x70
> Jun 22 21:46:17 kernel:  cleanup_net+0x273/0x430
> Jun 22 21:46:17 kernel:  process_one_work+0x18a/0x3a0
> Jun 22 21:46:17 kernel:  worker_thread+0x277/0x3a0
> Jun 22 21:46:17 kernel:  ? __pfx_worker_thread+0x10/0x10
> Jun 22 21:46:17 kernel:  kthread+0xe1/0x110
> Jun 22 21:46:17 kernel:  ? __pfx_kthread+0x10/0x10
> Jun 22 21:46:17 kernel:  ret_from_fork+0x2d/0x50
> Jun 22 21:46:17 kernel:  ? __pfx_kthread+0x10/0x10
> Jun 22 21:46:17 kernel:  ret_from_fork_asm+0x1b/0x30
> Jun 22 21:46:17 kernel:  </TASK>

This looks very much like the senario I've seen recently, that remove looked

like a sub-tree removal to me at the time so the question I have is what's

your dentry-state like?


If I could get our customer to do so I'd be asking for a locking status 
report

but I wouldn't want to use a debug kernel build, just a build the lock 
reporting

enabled ... that;s just not going to happen.


>
>
> In our real-world case, thousands of nodejs processes hit the cgroup memory
> limit and sleep on the OOM path, runc gets blocked, and consequently the node
> remains in an unschedulable state for many hours.

Right, that does sound like a different workload to my case so maybe 
your not

alone, ;)


>
>
> Link:
> - [1] https://github.com/nodejs/node/blob/v22.19.0/deps/uv/src/unix/linux.c#L1889
>
>> But in that work, no one could find a real benchmark or use case that
>> anyone could even notice this type of thing.  What do you have that
>> shows this?

I wish I could be more helpful but I too am guilty of not being able to

get enough info. and have no reproducer either.


Ian

Re: [RFC PATCH] kernfs: release kernfs_mutex before the inode allocation

[Ian Kent]

On 30/6/26 21:50, Tejun Heo wrote:
> Hello,
>
> On Tue, Jun 30, 2026 at 11:26:31AM +0800, Wenwu Hou wrote:
> ...
>> In our real-world case, thousands of nodejs processes hit the cgroup memory
>> limit and sleep on the OOM path, runc gets blocked, and consequently the node
>> remains in an unschedulable state for many hours.
> I don't think we can work around thousands of processes stalling in OOM path
> with kernfs locking changes. There are better ways to manage memory
> pressure. Please look into memory.high and PSI driven userspace OOM

Sounds like your assuming misbehaviour or errant processes but if that's not

the case then it would be better to try and avoid the situation.


Ian

Re: [RFC PATCH] kernfs: release kernfs_mutex before the inode allocation

[Ian Kent]

On 30/6/26 11:26, Wenwu Hou wrote:
> On Tue, Nov 16, 2021 at 08:49:46PM +0100, Greg Kroah-Hartman wrote:
>> On Tue, Nov 16, 2021 at 11:43:17AM -0800, Minchan Kim wrote:
>>> The kernfs implementation has big lock granularity(kernfs_rwsem) so
>>> every kernfs-based(e.g., sysfs, cgroup, dmabuf) fs are able to compete
>>> the lock. Thus, if one of userspace goes the sleep under holding
>>> the lock for a long time, rest of them should wait it. A example is
>>> the holder goes direct reclaim with the lock since it needs memory
>>> allocation. Let's fix it at common technique that release the lock
>>> and then allocate the memory. Fortunately, kernfs looks like have
>>> an refcount so I hope it's fine.
>>>
>>> Signed-off-by: Minchan Kim <minchan@kernel.org>
>>> ---
>>>   fs/kernfs/dir.c             | 14 +++++++++++---
>>>   fs/kernfs/inode.c           |  2 +-
>>>   fs/kernfs/kernfs-internal.h |  1 +
>>>   3 files changed, 13 insertions(+), 4 deletions(-)
>> What workload hits this lock to cause it to be noticable?
>>
>> There was a bunch of recent work in this area to make this much more
>> fine-grained, and the theoritical benchmarks that people created (adding
>> 10s of thousands of scsi disks at boot time) have gotten better.
> Hi,
>
> By 2026, the kernfs_rwsem has been split into per-fs. But the problem still
> exists.

In spite of my confusion due to your reference to a really old post I do 
have

a recent similar report (although I'm still struggling to work out the cause

of my report) I do agree that moving the inode allocation out from under the

rwsem is a good idea regardless. And I do still think the old patch you are

referring to isn't quite good enough.


Once I have worked through the back trace I'm looking at I'll have a go 
at doing

it, unless Greg or Tejun has any objections?


Ian

>
> In a k8s cluster environment, there are a lot of processes reading /sys, for example:
>
> Container runtime: runc
> Prometheus Exporter: node_exporter
>
> If a tenant's container processes read /sys and sleep on the OOM (or direct
> reclaim) path, both runc and node_exporter will be blocked, which will cause
> the cluster worker node to become unschedulable.
>
> For example, the nodejs runtime reads
> /sys/devices/system/cpu/cpu%u/cpufreq/scaling_cur_freq [1]
>
> Jun 22 21:43:17 kernel: task:node            state:D stack:0     pid:2621530 ppid:1062663 flags:0x00004002
> Jun 22 21:43:17 kernel: Call Trace:
> Jun 22 21:43:17 kernel:  <TASK>
> Jun 22 21:43:17 kernel:  __schedule+0x278/0x740
> Jun 22 21:43:17 kernel:  schedule+0x5a/0xd0
> Jun 22 21:43:17 kernel:  schedule_preempt_disabled+0x11/0x20
> Jun 22 21:43:17 kernel:  __mutex_lock.constprop.0+0x376/0x680
> Jun 22 21:43:17 kernel:  ? kvm_sched_clock_read+0xd/0x20
> Jun 22 21:43:17 kernel:  mem_cgroup_out_of_memory+0x53/0x150
> Jun 22 21:43:17 kernel:  try_charge_memcg+0x682/0x7d0
> Jun 22 21:43:17 kernel:  ? memcg_list_lru_alloc+0xa7/0x330
> Jun 22 21:43:17 kernel:  obj_cgroup_charge+0x70/0x170
> Jun 22 21:43:17 kernel:  slab_pre_alloc_hook.constprop.0+0xb6/0x1d0
> Jun 22 21:43:17 kernel:  ? alloc_inode+0x59/0xc0
> Jun 22 21:43:17 kernel:  kmem_cache_alloc_lru+0x54/0x2c0
> Jun 22 21:43:17 kernel:  alloc_inode+0x59/0xc0
> Jun 22 21:43:17 kernel:  iget_locked+0xe3/0x220
> Jun 22 21:43:17 kernel:  kernfs_get_inode+0x18/0x110
> Jun 22 21:43:17 kernel:  kernfs_iop_lookup+0x74/0xd0
> Jun 22 21:43:17 kernel:  __lookup_slow+0x82/0x130
> Jun 22 21:43:17 kernel:  walk_component+0xdb/0x150
> Jun 22 21:43:17 kernel:  link_path_walk.part.0.constprop.0+0x240/0x380
> Jun 22 21:43:17 kernel:  ? path_init+0x293/0x3d0
> Jun 22 21:43:17 kernel:  path_openat+0x85/0x2a0
> Jun 22 21:43:17 kernel:  ? seq_printf+0x8e/0xb0
> Jun 22 21:43:17 kernel:  do_filp_open+0xb4/0x160
> Jun 22 21:43:17 kernel:  ? __check_object_size.part.0+0x5e/0x130
> Jun 22 21:43:17 kernel:  do_sys_openat2+0x91/0xc0
> Jun 22 21:43:17 kernel:  __x64_sys_openat+0x53/0xa0
> Jun 22 21:43:17 kernel:  do_syscall_64+0x35/0x80
> Jun 22 21:43:17 kernel:  entry_SYSCALL_64_after_hwframe+0x78/0xe2
>
> A rwsem writer is involved:
>
> Jun 22 21:46:17 kernel: task:kworker/u232:0  state:D stack:0     pid:2886179 ppid:2      flags:0x00004000
> Jun 22 21:46:17 kernel: Workqueue: netns cleanup_net
> Jun 22 21:46:17 kernel: Call Trace:
> Jun 22 21:46:17 kernel:  <TASK>
> Jun 22 21:46:17 kernel:  __schedule+0x278/0x740
> Jun 22 21:46:17 kernel:  ? psi_group_change+0x226/0x3d0
> Jun 22 21:46:17 kernel:  schedule+0x5a/0xd0
> Jun 22 21:46:17 kernel:  schedule_preempt_disabled+0x11/0x20
> Jun 22 21:46:17 kernel:  rwsem_down_write_slowpath+0x1e2/0x4f0
> Jun 22 21:46:17 kernel:  down_write+0x57/0x60
> Jun 22 21:46:17 kernel:  kernfs_remove_by_name_ns+0x38/0xc0
> Jun 22 21:46:17 kernel:  remove_files+0x2b/0x70
> Jun 22 21:46:17 kernel:  sysfs_remove_group+0x38/0x80
> Jun 22 21:46:17 kernel:  sysfs_remove_groups+0x29/0x50
> Jun 22 21:46:17 kernel:  ib_free_port_attrs+0x92/0x170 [ib_core]
> Jun 22 21:46:17 kernel:  rdma_dev_exit_net+0x117/0x1d0 [ib_core]
> Jun 22 21:46:17 kernel:  ops_exit_list+0x30/0x70
> Jun 22 21:46:17 kernel:  cleanup_net+0x273/0x430
> Jun 22 21:46:17 kernel:  process_one_work+0x18a/0x3a0
> Jun 22 21:46:17 kernel:  worker_thread+0x277/0x3a0
> Jun 22 21:46:17 kernel:  ? __pfx_worker_thread+0x10/0x10
> Jun 22 21:46:17 kernel:  kthread+0xe1/0x110
> Jun 22 21:46:17 kernel:  ? __pfx_kthread+0x10/0x10
> Jun 22 21:46:17 kernel:  ret_from_fork+0x2d/0x50
> Jun 22 21:46:17 kernel:  ? __pfx_kthread+0x10/0x10
> Jun 22 21:46:17 kernel:  ret_from_fork_asm+0x1b/0x30
> Jun 22 21:46:17 kernel:  </TASK>
>
>
> In our real-world case, thousands of nodejs processes hit the cgroup memory
> limit and sleep on the OOM path, runc gets blocked, and consequently the node
> remains in an unschedulable state for many hours.
>
>
> Link:
> - [1] https://github.com/nodejs/node/blob/v22.19.0/deps/uv/src/unix/linux.c#L1889
>
>> But in that work, no one could find a real benchmark or use case that
>> anyone could even notice this type of thing.  What do you have that
>> shows this?
>> thanks,
>>
>> greg k-h