[PATCH] keys: use long workqueue for garbage collection

[PATCH] keys: use long workqueue for garbage collection

[Eduard Zateev]

The key garbage collector can wait for RCU grace periods. This can occupy a
normal events worker for a long time and contribute to workqueue lockups.

Queue key GC on system_long_wq because its work may wait for long periods.

Reported-by: syzbot+c15de43f1e7fffbb7b9b@syzkaller.appspotmail.com
Link: https://syzkaller.appspot.com/bug?extid=c15de43f1e7fffbb7b9b
Signed-off-by: Eduard Zateev <hackerowskiy@gmail.com>
---
 security/keys/gc.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/security/keys/gc.c b/security/keys/gc.c
index eda445f815d..b174b8c511b 100644
--- a/security/keys/gc.c
+++ b/security/keys/gc.c
@@ -19,6 +19,7 @@ unsigned key_gc_delay = 5 * 60;
  * Reaper for unused keys.
  */
 static void key_garbage_collector(struct work_struct *work);
+/* Key GC may wait for RCU grace periods, so queue it on system_long_wq. */
 DECLARE_WORK(key_gc_work, key_garbage_collector);
 
 /*
@@ -57,7 +58,7 @@ void key_schedule_gc(time64_t gc_at)
 
 	if (gc_at <= now || test_bit(KEY_GC_REAP_KEYTYPE, &key_gc_flags)) {
 		kdebug("IMMEDIATE");
-		schedule_work(&key_gc_work);
+		queue_work(system_long_wq, &key_gc_work);
 	} else if (gc_at < key_gc_next_run) {
 		kdebug("DEFERRED");
 		key_gc_next_run = gc_at;
@@ -85,7 +86,7 @@ void key_set_expiry(struct key *key, time64_t expiry)
 void key_schedule_gc_links(void)
 {
 	set_bit(KEY_GC_KEY_EXPIRED, &key_gc_flags);
-	schedule_work(&key_gc_work);
+	queue_work(system_long_wq, &key_gc_work);
 }
 
 /*
@@ -119,7 +120,7 @@ void key_gc_keytype(struct key_type *ktype)
 	set_bit(KEY_GC_REAP_KEYTYPE, &key_gc_flags);
 
 	kdebug("schedule");
-	schedule_work(&key_gc_work);
+	queue_work(system_long_wq, &key_gc_work);
 
 	kdebug("sleep");
 	wait_on_bit(&key_gc_flags, KEY_GC_REAPING_KEYTYPE,
@@ -322,7 +323,7 @@ static void key_garbage_collector(struct work_struct *work)
 	}
 
 	if (gc_state & KEY_GC_REAP_AGAIN)
-		schedule_work(&key_gc_work);
+		queue_work(system_long_wq, &key_gc_work);
 	kleave(" [end %x]", gc_state);
 	return;
 
-- 
2.55.0

Re: [PATCH] keys: use long workqueue for garbage collection

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Critical] keys: concurrent key_garbage_collector() via schedule_work() in key_put()
--

commit 49fec25be24d41f55255824b44e92e9f57128440
Author: Eduard Zateev <hackerowskiy@gmail.com>

keys: use long workqueue for garbage collection

This commit switches the key garbage collector work item (key_gc_work) in
security/keys/gc.c from system_percpu_wq to system_long_wq because key
garbage collection can block waiting for RCU grace periods.

Link: https://syzkaller.appspot.com/bug?extid=c15de43f1e7fffbb7b9b

> diff --git a/security/keys/gc.c b/security/keys/gc.c
> index eda445f815d47..b174b8c511b4b 100644
> --- a/security/keys/gc.c
> +++ b/security/keys/gc.c
> @@ -19,6 +19,7 @@ unsigned key_gc_delay = 5 * 60;
>   * Reaper for unused keys.
>   */
>  static void key_garbage_collector(struct work_struct *work);
> +/* Key GC may wait for RCU grace periods, so queue it on system_long_wq. */
>  DECLARE_WORK(key_gc_work, key_garbage_collector);
>  
>  /*
> @@ -57,7 +58,7 @@ void key_schedule_gc(time64_t gc_at)
>  
>  	if (gc_at <= now || test_bit(KEY_GC_REAP_KEYTYPE, &key_gc_flags)) {
>  		kdebug("IMMEDIATE");
> -		schedule_work(&key_gc_work);
> +		queue_work(system_long_wq, &key_gc_work);

[Severity: Critical]
Does this allow key_garbage_collector() to run concurrently with itself across
two CPUs because key_put() still queues key_gc_work on system_percpu_wq?

While this patch updates the four schedule_work(&key_gc_work) call sites in
security/keys/gc.c to use system_long_wq, key_put() in security/keys/key.c
still calls schedule_work(&key_gc_work):

security/keys/key.c:key_put() {
	if (key) {
		key_check(key);

		if (refcount_dec_and_test(&key->usage)) {
			/* Mark key as safe for GC after key->user done. */
			clear_bit_unlock(KEY_FLAG_USER_ALIVE, &key->flags);
			schedule_work(&key_gc_work);
		}
	}
}

First, since key_put() is the path that clears KEY_FLAG_USER_ALIVE to trigger
reaping of unreferenced keys and the resulting synchronize_rcu() call, normal
system_percpu_wq workers will still execute key_garbage_collector() and block
on RCU grace periods.

Second, in __queue_work(), cross-CPU non-reentrancy for a work_struct that is
already executing relies on matching the target workqueue against the running
worker's workqueue:

kernel/workqueue.c:__queue_work() {
	...
	last_pool = get_work_pool(work);
	if (last_pool && last_pool != pool) {
		worker = find_worker_executing_work(last_pool, work);

		if (worker && worker->current_pwq->wq == wq) {
			pwq = worker->current_pwq;\
			pool = pwq->pool;
			WARN_ON_ONCE(pool != last_pool);
		} else {
			/* meh... not running there, queue here */
			raw_spin_unlock(&last_pool->lock);
			raw_spin_lock(&pool->lock);
		}
	}
	...
}

Once process_one_work() clears WORK_STRUCT_PENDING_BIT and begins executing
key_garbage_collector() on one CPU under system_long_wq, a concurrent
key_put() -> schedule_work(&key_gc_work) on another CPU passes wq ==
system_percpu_wq (or vice versa). Because worker->current_pwq->wq == wq is
false, __queue_work() queues key_gc_work on the second CPU's pool and runs a
second instance of key_garbage_collector() in parallel while the first is
sleeping in synchronize_rcu(), cond_resched(), or down_write(&key->sem).

key_garbage_collector() assumes only a single instance can run at a time:

security/keys/gc.c:key_garbage_collector() {
	static LIST_HEAD(graveyard);
	static u8 gc_state;		/* Internal persistent state */
	...
	/* As only this function is permitted to remove things from the key
	 * serial tree, if cursor is non-NULL then it will always point to a
	 * valid node in the tree - even if lock got dropped.
	 */
	spin_lock(&key_serial_lock);
	cursor = rb_first(&key_serial_tree);
	...
}

Can concurrent execution of key_garbage_collector() corrupt the static
graveyard list and gc_state, free keys in key_gc_unused_keys() before the
other worker's synchronize_rcu() completes, or rb_erase() and free the key
pointed to by the other worker's cursor while key_serial_lock is dropped?

Should key_put() in security/keys/key.c also be updated to call
queue_work(system_long_wq, &key_gc_work)?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261024042944.52493-1-hackerowskiy@gmail.com?part=1

Re: [PATCH] keys: use long workqueue for garbage collection

[Jarkko Sakkinen]

On Sat, Oct 24, 2026 at 06:29:44AM +0200, Eduard Zateev wrote:
> The key garbage collector can wait for RCU grace periods. This can occupy a
> normal events worker for a long time and contribute to workqueue lockups.
> 
> Queue key GC on system_long_wq because its work may wait for long periods.
> 
> Reported-by: syzbot+c15de43f1e7fffbb7b9b@syzkaller.appspotmail.com
> Link: https://syzkaller.appspot.com/bug?extid=c15de43f1e7fffbb7b9b
> Signed-off-by: Eduard Zateev <hackerowskiy@gmail.com>

This commit message is not huge selling point for any possible argument.

> ---
>  security/keys/gc.c | 9 +++++----
>  1 file changed, 5 insertions(+), 4 deletions(-)
> 
> diff --git a/security/keys/gc.c b/security/keys/gc.c
> index eda445f815d..b174b8c511b 100644
> --- a/security/keys/gc.c
> +++ b/security/keys/gc.c
> @@ -19,6 +19,7 @@ unsigned key_gc_delay = 5 * 60;
>   * Reaper for unused keys.
>   */
>  static void key_garbage_collector(struct work_struct *work);
> +/* Key GC may wait for RCU grace periods, so queue it on system_long_wq. */
>  DECLARE_WORK(key_gc_work, key_garbage_collector);
>  
>  /*
> @@ -57,7 +58,7 @@ void key_schedule_gc(time64_t gc_at)
>  
>  	if (gc_at <= now || test_bit(KEY_GC_REAP_KEYTYPE, &key_gc_flags)) {
>  		kdebug("IMMEDIATE");
> -		schedule_work(&key_gc_work);
> +		queue_work(system_long_wq, &key_gc_work);
>  	} else if (gc_at < key_gc_next_run) {
>  		kdebug("DEFERRED");
>  		key_gc_next_run = gc_at;
> @@ -85,7 +86,7 @@ void key_set_expiry(struct key *key, time64_t expiry)
>  void key_schedule_gc_links(void)
>  {
>  	set_bit(KEY_GC_KEY_EXPIRED, &key_gc_flags);
> -	schedule_work(&key_gc_work);
> +	queue_work(system_long_wq, &key_gc_work);
>  }
>  
>  /*
> @@ -119,7 +120,7 @@ void key_gc_keytype(struct key_type *ktype)
>  	set_bit(KEY_GC_REAP_KEYTYPE, &key_gc_flags);
>  
>  	kdebug("schedule");
> -	schedule_work(&key_gc_work);
> +	queue_work(system_long_wq, &key_gc_work);
>  
>  	kdebug("sleep");
>  	wait_on_bit(&key_gc_flags, KEY_GC_REAPING_KEYTYPE,
> @@ -322,7 +323,7 @@ static void key_garbage_collector(struct work_struct *work)
>  	}
>  
>  	if (gc_state & KEY_GC_REAP_AGAIN)
> -		schedule_work(&key_gc_work);
> +		queue_work(system_long_wq, &key_gc_work);
>  	kleave(" [end %x]", gc_state);
>  	return;
>  
> -- 
> 2.55.0
> 

Br, Jarkko

Re: [PATCH] keys: use long workqueue for garbage collection

[Jarkko Sakkinen]

On Sat, Oct 24, 2026 at 08:20:55AM +0300, Jarkko Sakkinen wrote:
> On Sat, Oct 24, 2026 at 06:29:44AM +0200, Eduard Zateev wrote:
> > The key garbage collector can wait for RCU grace periods. This can occupy a
> > normal events worker for a long time and contribute to workqueue lockups.
> > 
> > Queue key GC on system_long_wq because its work may wait for long periods.
> > 
> > Reported-by: syzbot+c15de43f1e7fffbb7b9b@syzkaller.appspotmail.com
> > Link: https://syzkaller.appspot.com/bug?extid=c15de43f1e7fffbb7b9b
> > Signed-off-by: Eduard Zateev <hackerowskiy@gmail.com>
> 
> This commit message is not huge selling point for any possible argument.

You ignored key_gc_timed_out(), which leaves code to a broken state and
switching workqueue is not solving any possible root cause of anything.
The long queue exists mostly for the sake of flush_workqueue().

Please stop sending patches like this in the future.

Br, Jarkko

Re: [PATCH] keys: use long workqueue for garbage collection

[Jarkko Sakkinen]

On Sat, Oct 24, 2026 at 08:37:42AM +0300, Jarkko Sakkinen wrote:
> On Sat, Oct 24, 2026 at 08:20:55AM +0300, Jarkko Sakkinen wrote:
> > On Sat, Oct 24, 2026 at 06:29:44AM +0200, Eduard Zateev wrote:
> > > The key garbage collector can wait for RCU grace periods. This can occupy a
> > > normal events worker for a long time and contribute to workqueue lockups.
> > > 
> > > Queue key GC on system_long_wq because its work may wait for long periods.
> > > 
> > > Reported-by: syzbot+c15de43f1e7fffbb7b9b@syzkaller.appspotmail.com
> > > Link: https://syzkaller.appspot.com/bug?extid=c15de43f1e7fffbb7b9b
> > > Signed-off-by: Eduard Zateev <hackerowskiy@gmail.com>
> > 
> > This commit message is not huge selling point for any possible argument.
> 
> You ignored key_gc_timed_out(), which leaves code to a broken state and
> switching workqueue is not solving any possible root cause of anything.
> The long queue exists mostly for the sake of flush_workqueue().
> 
> Please stop sending patches like this in the future.

Makes me wonder was this even tried a single time TBH given the obvious
glitch.  I would bet for no.

Br, Jarkko