Re: [RFC PATCH 0/3] workqueue: Control cpu affinity of !WQ_SYSFS unbound workqueues

[Kevin Hilman <khilman@linaro.org>]

Frederic Weisbecker <fweisbec@gmail.com> writes:

> There are several types of workqueues. Some of them are bound to specific
> CPUs, some others are unbound and can be executed on any CPU.
>
> A tiny subset of the unbound workqueues have a sysfs representation
> in /sys/devices/virtual/workqueue/ and have a cpumask file than can
> be used to tweak their CPU affinity.
>
> But the vast majority of unbound workqueues aren't visible in sysfs.
>
> They are a problem nowaday because people who work on CPU isolation
> (HPC, Real time, etc...) want to be able to migrate all the unbound
> workqueues away from specific CPUs.
>
> There are several possible solutions to solve this:
>
> 1) Affine the !WQ_SYSFS unbound workqueues to the CPUs outside the
> full dynticks mask. Full dynticks is expected to be a component in
> many CPU isolation configurations and its CPU mapping can be a good
> way to retrieve the desired set of isolated CPUs.
>
> On the drawbacks though we can notice the lack of consistency with
> WQ_SYSFS workqueue affinity interface, issues with ordering between
> workqueue and dynticks subsystem initialization, intrusion from
> the workqueue subsystem on dynticks internals.
>
> 2) Implement a sysfs directory for each unbound !WQ_SYSFS. That sounds
> like a very nice solution as it uses existing and known interface.
> But workqueues appearing in the sysfs hierarchy are subject to become
> stable ABIs. And this is definetly not what we want.
>
> This could be worked around with a specific Kconfig to make sure that
> these workqueues won't be considered as a stable ABI. But we all know
> that all distros will enable this Kconfig symbol and that nobody
> reads, nor care about, warnings in Kconfig help text which thereby won't
> protect us against anything.
>
> 3) Implement a single sysfs directory to map properties of all !WQ_SYSFS
> unbound workqueues. It would contain only the cpumask file to control
> the affinity of all these workqueues. But more can be added later.
>
> This complexifies the code a bit although not that much compared to
> solution 2 which requires some plumbling to cope with workqueues created
> before sysfs, as reported by Mike (I played with that a bit as well, as I
> took that direction initially). But it deals with all issues previously
> described.

Since I also tinkered with (1) and (2) and ran into some of the same
issues, I think (3) is a good way.  It also doesn't tie the affinity to
full_nohz, and leaves it up to userspace which addresses a concern of
Mike's in earlier proposals as well.

Kevin