[PATCH net-next v4 0/1] af_unix: Fix priority inversion issue

[PATCH net-next v4 0/1] af_unix: Fix priority inversion issue

[Nam Cao]

Hi,

While auditing AF_UNIX sockets, I noticed that the sendmsg() code paths may
block on the garbage collector running as workqueue. This can cause
priority inversion and latency for real-time users.

The implementation does kindly avoid blocking "sane users". However, it is
impossible to tell whether the kernel's definition of "sane users"
accurately describes all users out there.

Digging into history and figuring out the reasons why sendmsg() needs to
wait for garbage collector, it is determined that those reasons no longer
apply.

v4:
  - Downsize this series to just removing the priority
    inversion. Garbage collector scheduling should still be cleaned up,
    but that is non-trivial and I do not want that to stand in the way
    of this simple fix.

v3:
  - Move unix_schedule_gc() to be after exit_task_work()

v2:
  - Add patch [1/3]
  - Rebase the other two patches onto the new patch
  - Change commit message to be more precise

Nam Cao (1):
  af_unix: Do not wait for garbage collector in sendmsg()

 net/unix/garbage.c | 3 ---
 1 file changed, 3 deletions(-)

-- 
2.47.3

[PATCH net-next v4 1/1] af_unix: Do not wait for garbage collector in sendmsg()

[Nam Cao]

AF_UNIX sockets' sendmsg() schedules and blocks on the garbage collector if
user has too many inflight unix sockets and there is cyclic reference in
the system. This causes real-time issues, as cyclic reference can be
created by any task in the system, and high priority tasks who do need to
send lots of AF_UNIX sockets get blocked by the garbage collector which
runs as workqueue, causing a priority inversion scenario.

The reason for blocking on garbage collector goes back to 2008, when
it was reported that "Local/unprivileged users can cause soft lockups
and take out system processes by triggering the OOM killer":
https://bugzilla.redhat.com/show_bug.cgi?id=470201

The soft lockup was because a process can keep queueing AF_UNIX sockets to
another process that is exiting. Back in 2008, the garbage collector was
run synchronously by the exiting process, therefore keep queueing AF_UNIX
sockets blocks that process from exiting.

The solution to that issue was forcing sendmsg() to wait for ongoing
garbage collector.

The OOM killer issue was brought up again in 2010:
https://lore.kernel.org/lkml/AANLkTi=Q967xpX0KLMwX-=_4_1AKO5wjHEuJ1TrNjCj9@mail.gmail.com/

To resolve that report, beside blocking on the garbage collector, sendmsg()
also schedules the garbage collector if the number of inflight AF_UNIX
sockets in the system is too high.

Then in 2015, once again, the OOM killer problem was brought up:
https://lore.kernel.org/lkml/20151228141435.GA13351@1wt.eu/

That time, the issue was resolved by disallowing a user from having more
inflight AF_UNIX sockets than their RLIMIT_NOFILE. That was done by commit
712f4aad406b ("unix: properly account for FDs passed over unix sockets")
and commit 415e3d3e90ce ("unix: correctly track in-flight fds in sending
process user_struct").

Now, sendmsg() does not have to block on the garbage collector anymore,
because:

  - The OOM killer issue has already been addressed by checking
    RLIMIT_NOFILE.

  - The soft lockup issue is no longer relevant, because the garbage
    collector now runs asynchronously since commit d9f21b361333 ("af_unix:
    Try to run GC async.")

Therefore, remove that to prevent priority inversion. Running all the
reproducers from the mentioned bug reports after this patch, no problem is
observed.

Signed-off-by: Nam Cao <namcao@linutronix.de>
---
 net/unix/garbage.c | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/net/unix/garbage.c b/net/unix/garbage.c
index 0783555e2526..e0702171d246 100644
--- a/net/unix/garbage.c
+++ b/net/unix/garbage.c
@@ -653,7 +653,4 @@ void unix_schedule_gc(struct user_struct *user)
 
 	if (!READ_ONCE(gc_in_progress))
 		queue_work(system_dfl_wq, &unix_gc_work);
-
-	if (user && READ_ONCE(unix_graph_cyclic_sccs))
-		flush_work(&unix_gc_work);
 }
-- 
2.47.3

Re: [PATCH net-next v4 1/1] af_unix: Do not wait for garbage collector in sendmsg()

[Jakub Kicinski]

On Tue,  4 Aug 2026 08:46:16 +0200 Nam Cao wrote:
> AF_UNIX sockets' sendmsg() schedules and blocks on the garbage collector if
> user has too many inflight unix sockets and there is cyclic reference in
> the system. This causes real-time issues, as cyclic reference can be
> created by any task in the system, and high priority tasks who do need to
> send lots of AF_UNIX sockets get blocked by the garbage collector which
> runs as workqueue, causing a priority inversion scenario.

Hi Kuniyuki! Any thoughts?

Re: [PATCH net-next v4 1/1] af_unix: Do not wait for garbage collector in sendmsg()

[Kuniyuki Iwashima]

On Fri, Aug 7, 2026 at 6:07 PM Jakub Kicinski <kuba@kernel.org> wrote:
>
> On Tue,  4 Aug 2026 08:46:16 +0200 Nam Cao wrote:
> > AF_UNIX sockets' sendmsg() schedules and blocks on the garbage collector if
> > user has too many inflight unix sockets and there is cyclic reference in
> > the system. This causes real-time issues, as cyclic reference can be
> > created by any task in the system, and high priority tasks who do need to
> > send lots of AF_UNIX sockets get blocked by the garbage collector which
> > runs as workqueue, causing a priority inversion scenario.
>
> Hi Kuniyuki! Any thoughts?

flush_work() makes an insane process slow down by itself and
the sane users (whose peers receive fd in timely manner) can
likely acquire the gc lock before insane users.

The commit message assumes the GC is low priority, and it will
be easier to trigger OOM and soft lockup without the penalty.

If priority inversion on RT kernel is the only problem here, there will
be more flush_work() instances triggered from userspace, and the real
problem is that flush_work() does not implement priority inheritance
like mutex.

Re: [PATCH net-next v4 1/1] af_unix: Do not wait for garbage collector in sendmsg()

[Nam Cao]

Kuniyuki Iwashima <kuniyu@google.com> writes:

> On Fri, Aug 7, 2026 at 6:07 PM Jakub Kicinski <kuba@kernel.org> wrote:
>>
>> On Tue,  4 Aug 2026 08:46:16 +0200 Nam Cao wrote:
>> > AF_UNIX sockets' sendmsg() schedules and blocks on the garbage collector if
>> > user has too many inflight unix sockets and there is cyclic reference in
>> > the system. This causes real-time issues, as cyclic reference can be
>> > created by any task in the system, and high priority tasks who do need to
>> > send lots of AF_UNIX sockets get blocked by the garbage collector which
>> > runs as workqueue, causing a priority inversion scenario.
>>
>> Hi Kuniyuki! Any thoughts?
>
> flush_work() makes an insane process slow down by itself and
> the sane users (whose peers receive fd in timely manner) can
> likely acquire the gc lock before insane users.

As mentioned in the cover letter, our definition of "sane users" is
quite arbitrary. I acknowledge that it likely cover most users out
there, possibly all of them. But we can never tell for sure.

> The commit message assumes the GC is low priority,

GC runs as workqueue and thus is low priority.

> and it will be easier to trigger OOM and soft lockup without the
> penalty.

Sorry, perhaps I am still confused somewhere, but how can OOM or soft
lockup be triggered?

At best (or worst?), a task can make the GC iterates over RLIMIT_NOFILE
inflight sockets, so there wouldn't be a soft lockup.

And the number of inflight sockets is capped at RLIMIT_NOFILE, so there
wouldn't be an OOM issue either.

> If priority inversion on RT kernel is the only problem here, there will
> be more flush_work() instances triggered from userspace,

Yes, you are correct and this happens often. The solution usually is
avoiding workqueue. Some times we replace the workqueue with a dedicated
kthread, other times we drop the workqueue and execute the work
synchronously. Otherwise we accept that such API is not real-time-safe
and should not be used by real-time application (we have on-going work
in run-time verification monitors that tell user if they are using an
unsafe API).

> and the real problem is that flush_work() does not implement priority
> inheritance like mutex.

This idea has been brought up in the past. Unfortunately it is no easy
feat.

Nam