Re: [PATCH bpf-next 2/4] bpf: add bpf_thread_wq kthread-backed workqueue with cgroup placement

["Hui Zhu" <hui.zhu@linux.dev>]

> 
> On Fri Aug 7, 2026 at 9:01 AM CEST, Hui Zhu wrote:
> 
> > 
> > From: Hui Zhu <zhuhui@kylinos.cn>
> > 
> >  Introduce bpf_thread_wq, a new BPF embedded map field similar to
> >  bpf_wq but backed by a dedicated kthread_worker instead of a system
> >  workqueue. The worker kthread can be attached to a specific cgroup at
> >  init time so BPF-deferred callbacks run under the resource limits of
> >  the target cgroup.
> > 
> >  Three kfuncs are exposed:
> >  bpf_thread_wq_init(twq, map, cgroup_id, flags) [KF_SLEEPABLE]
> >  bpf_thread_wq_set_callback(twq, cb, flags, aux)
> >  bpf_thread_wq_start(twq, flags)
> > 
> >  bpf_thread_wq_init() is registered only for BPF_PROG_TYPE_SYSCALL
> >  programs. It creates a kthread worker and may attach it to a cgroup;
> >  those paths can sleep and acquire kthread and cgroup locks. Restricting
> >  init to syscall programs prevents it from running in BPF contexts that
> >  may already hold locks which could deadlock with those paths.
> > 
> >  bpf_thread_wq intentionally avoids the bpf_async infrastructure used by
> >  bpf_timer and bpf_wq. That infrastructure drives cleanup from irq_work
> >  in hardirq context, while bpf_thread_wq cancellation and final teardown
> >  may need to sleep through kthread_cancel_work_sync(),
> >  kthread_destroy_worker() and a final cgroup_put().
> >  bpf_thread_wq_cancel_and_free() therefore cancels work synchronously and
> >  drops the context reference; the last put waits for tasks-trace RCU
> >  readers and then schedules process-context work to run bpf_prog_put(),
> >  cgroup_put(), kthread_destroy_worker() and kfree().
> > 
> >  Add BTF/map support for bpf_thread_wq fields, map teardown hooks,
> >  verifier handling for the callback kfunc, and cgroup_kthread_attach() to
> >  move the worker into the requested cgroup.
> > 
> >  Supported map types are BPF_MAP_TYPE_HASH, BPF_MAP_TYPE_LRU_HASH, and
> >  BPF_MAP_TYPE_ARRAY, consistent with bpf_wq and bpf_task_work.
> > 
> >  Signed-off-by: Hui Zhu <zhuhui@kylinos.cn>
> >  ---
> > 
> Hi Hui,
> 
> Thanks for sharing the patches. I think Sashiko and Mykyta already pointed out a
> couple of issues with the current implementation, but I would like to comment on
> the higher-level approach.
> 
> If I understood the past discussions and current set correctly, the reason for
> your choice to move from bpf_wq to bpf_thread_wq was primarily to enable correct
> CPU accounting of the work done by threads to specific cgroups.
> 
> I think this is a step in the right direction, but looking at the bigger
> picture, I feel we need a more flexible solution.
> 
> In practice, users deciding to do async reclaim through such a BPF interface
> would want to scale and compact the number of threads doing reclaim-related work
> dynamically, based on available idle resources, but also application-specific
> metrics, and I do not think the bpf_thread_wq abstraction provides enough
> flexibility in managing work scheduling related aspects precisely.
> 
> What we probably should expose is the ability for programs to manage their own
> wait queues and subscribe kthreads managed by BPF programs dynamically to them.
> User-defined policies can dictate how many threads remain active, whether they
> busy poll, and when they go to sleep, completely under the program's control.
> 
> Looking beyond this particular example, there are cases where work is stashed in
> queues, and threads draw items from the pool and process them. In such cases
> having flexibility in deciding the mapping between threads and queues is also
> important. We want to thus allow management of the work item queues from the
> program itself, and not hide it behind the API to offer the desired level of
> control. The order in which items are ranked in individual queues, and the
> order in which queues are processed horizontally by threads is also a desirable
> property.
> 
> Correctly associating the BPF-managed kthread to a cgroup should still be
> possible in a manner similar to what you did in this set. But the amount of
> control programs can exert over how work is scheduled and the level of
> concurrency will be much higher if we disaggregate and generalize each part of
> the picture (wait queues, kthreads, and work item queues, which can be
> implemented in BPF itself).
> 
> I am not aware of ways to back charge time spent to remote cgroups, but if
> desired we could also explore that option when a single thread does work on
> behalf of multiple cgroups. It is something to be explored.
> 
> I've been working on related patches, and will post RFC set for bpf_kthread and
> bpf_waitq management in due time. Until then I recommend that you continue
> experimenting with the existing async execution primitives for now.

Hi Kumar,

Agreed, the disaggregated design sounds like the right direction.

I'll keep updating the remaining commits of this series, and wait
for your bpf_kthread/bpf_waitq RFC.

Best,
Hui

> 
> > 
> > [...]
> >
>