[PATCH-cgroup v2] cgroup: Move rcu_head up near the top of cgroup_root

[PATCH-cgroup v2] cgroup: Move rcu_head up near the top of cgroup_root

[Waiman Long]

Commit d23b5c577715 ("cgroup: Make operations on the cgroup root_list RCU
safe") adds a new rcu_head to the cgroup_root structure and kvfree_rcu()
for freeing the cgroup_root.

The current implementation of kvfree_rcu(), however, has the limitation
that the offset of the rcu_head structure within the larger data
structure must be less than 4096 or the compilation will fail. See the
macro definition of __is_kvfree_rcu_offset() in include/linux/rcupdate.h
for more information.

By putting rcu_head below the large cgroup structure, any change to the
cgroup structure that makes it larger run the risk of causing build
failure under certain configurations. Commit 77070eeb8821 ("cgroup:
Avoid false cacheline sharing of read mostly rstat_cpu") happens to be
the last straw that breaks it. Fix this problem by moving the rcu_head
structure up before the cgroup structure.

Fixes: d23b5c577715 ("cgroup: Make operations on the cgroup root_list RCU safe")
Reported-by: Stephen Rothwell <sfr@canb.auug.org.au>
Closes: https://lore.kernel.org/lkml/20231207143806.114e0a74@canb.auug.org.au/
Signed-off-by: Waiman Long <longman@redhat.com>
Acked-by: Yafang Shao <laoar.shao@gmail.com>
---
 include/linux/cgroup-defs.h | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/include/linux/cgroup-defs.h b/include/linux/cgroup-defs.h
index 5a97ea95b564..ea48c861cd36 100644
--- a/include/linux/cgroup-defs.h
+++ b/include/linux/cgroup-defs.h
@@ -562,6 +562,10 @@ struct cgroup_root {
 	/* Unique id for this hierarchy. */
 	int hierarchy_id;
 
+	/* A list running through the active hierarchies */
+	struct list_head root_list;
+	struct rcu_head rcu;	/* Must be near the top */
+
 	/*
 	 * The root cgroup. The containing cgroup_root will be destroyed on its
 	 * release. cgrp->ancestors[0] will be used overflowing into the
@@ -575,10 +579,6 @@ struct cgroup_root {
 	/* Number of cgroups in the hierarchy, used only for /proc/cgroups */
 	atomic_t nr_cgrps;
 
-	/* A list running through the active hierarchies */
-	struct list_head root_list;
-	struct rcu_head rcu;
-
 	/* Hierarchy-specific flags */
 	unsigned int flags;
 
-- 
2.39.3

Re: [PATCH-cgroup v2] cgroup: Move rcu_head up near the top of cgroup_root

[Michal Koutný]

[-- Attachment #1: Type: text/plain, Size: 1895 bytes --]

On Thu, Dec 07, 2023 at 08:46:14AM -0500, Waiman Long <longman@redhat.com> wrote:
> Commit 77070eeb8821 ("cgroup: Avoid false cacheline sharing of read
> mostly rstat_cpu") happens to be the last straw that breaks it.

FTR, when I build kernel from that commit, I can see

> struct cgroup_root {
> 	struct kernfs_root *       kf_root;              /*     0     8 */
> 	unsigned int               subsys_mask;          /*     8     4 */
> 	int                        hierarchy_id;         /*    12     4 */
> 
> 	/* XXX 48 bytes hole, try to pack */
> 
> 	/* --- cacheline 1 boundary (64 bytes) --- */
> 	struct cgroup              cgrp __attribute__((__aligned__(64))); /*    64  2368 */
> 
> 	/* XXX last struct has 8 bytes of padding */
> 
> 	/* --- cacheline 38 boundary (2432 bytes) --- */
> 	struct cgroup *            cgrp_ancestor_storage; /*  2432     8 */
> 	atomic_t                   nr_cgrps;             /*  2440     4 */
> 
> 	/* XXX 4 bytes hole, try to pack */
> 
> 	struct list_head           root_list;            /*  2448    16 */
> 	struct callback_head       rcu __attribute__((__aligned__(8))); /*  2464    16 */
> 	unsigned int               flags;                /*  2480     4 */
> 	char                       release_agent_path[4096]; /*  2484  4096 */
> 	/* --- cacheline 102 boundary (6528 bytes) was 52 bytes ago --- */
> 	char                       name[64];             /*  6580    64 */
> 
> 	/* size: 6656, cachelines: 104, members: 11 */
> 	/* sum members: 6592, holes: 2, sum holes: 52 */
> 	/* padding: 12 */
> 	/* paddings: 1, sum paddings: 8 */
> 	/* forced alignments: 2, forced holes: 1, sum forced holes: 48 */
> } __attribute__((__aligned__(64)));

2480 has still quite a reserve below 4096. (I can't see an CONFIG_*
affecting this.)

Perhaps, I missed something from the linux-next merging thread?


Michal

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [PATCH-cgroup v2] cgroup: Move rcu_head up near the top of cgroup_root

[Waiman Long]

On 12/7/23 11:46, Michal Koutný wrote:
> On Thu, Dec 07, 2023 at 08:46:14AM -0500, Waiman Long <longman@redhat.com> wrote:
>> Commit 77070eeb8821 ("cgroup: Avoid false cacheline sharing of read
>> mostly rstat_cpu") happens to be the last straw that breaks it.
> FTR, when I build kernel from that commit, I can see
>
>> struct cgroup_root {
>> 	struct kernfs_root *       kf_root;              /*     0     8 */
>> 	unsigned int               subsys_mask;          /*     8     4 */
>> 	int                        hierarchy_id;         /*    12     4 */
>>
>> 	/* XXX 48 bytes hole, try to pack */
>>
>> 	/* --- cacheline 1 boundary (64 bytes) --- */
>> 	struct cgroup              cgrp __attribute__((__aligned__(64))); /*    64  2368 */
>>
>> 	/* XXX last struct has 8 bytes of padding */
>>
>> 	/* --- cacheline 38 boundary (2432 bytes) --- */
>> 	struct cgroup *            cgrp_ancestor_storage; /*  2432     8 */
>> 	atomic_t                   nr_cgrps;             /*  2440     4 */
>>
>> 	/* XXX 4 bytes hole, try to pack */
>>
>> 	struct list_head           root_list;            /*  2448    16 */
>> 	struct callback_head       rcu __attribute__((__aligned__(8))); /*  2464    16 */
>> 	unsigned int               flags;                /*  2480     4 */
>> 	char                       release_agent_path[4096]; /*  2484  4096 */
>> 	/* --- cacheline 102 boundary (6528 bytes) was 52 bytes ago --- */
>> 	char                       name[64];             /*  6580    64 */
>>
>> 	/* size: 6656, cachelines: 104, members: 11 */
>> 	/* sum members: 6592, holes: 2, sum holes: 52 */
>> 	/* padding: 12 */
>> 	/* paddings: 1, sum paddings: 8 */
>> 	/* forced alignments: 2, forced holes: 1, sum forced holes: 48 */
>> } __attribute__((__aligned__(64)));
> 2480 has still quite a reserve below 4096. (I can't see an CONFIG_*
> affecting this.)
>
> Perhaps, I missed something from the linux-next merging thread?

CONFIG_LOCKDEP and some other debug configs are enabled with 
allmodconfig. This can greatly increase the size of some of the 
structures. I am not able to use pahole due to missing BTF info so I 
don't the exact size. However, I can reproduce the build failure and the 
patch is  able to fix it.

Cheers,
Longman

Re: [PATCH-cgroup v2] cgroup: Move rcu_head up near the top of cgroup_root

[Michal Koutný]

[-- Attachment #1: Type: text/plain, Size: 478 bytes --]

On Thu, Dec 07, 2023 at 12:40:33PM -0500, Waiman Long <longman@redhat.com> wrote:
> CONFIG_LOCKDEP and some other debug configs are enabled with allmodconfig.

Interesting, I had CONFIG_LOCKDEP=y.

I shared the numbers to rule out you're after something unrelated

> However, I can reproduce the build failure and the patch is  able to
> fix it.

That is a good proof :-)
The patch makes sense for future robustness,

Reviewed-by: Michal Koutný <mkoutny@suse.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [PATCH-cgroup v2] cgroup: Move rcu_head up near the top of cgroup_root

[Yosry Ahmed]

On Thu, Dec 7, 2023 at 5:46 AM Waiman Long <longman@redhat.com> wrote:
>
> Commit d23b5c577715 ("cgroup: Make operations on the cgroup root_list RCU
> safe") adds a new rcu_head to the cgroup_root structure and kvfree_rcu()
> for freeing the cgroup_root.
>
> The current implementation of kvfree_rcu(), however, has the limitation
> that the offset of the rcu_head structure within the larger data
> structure must be less than 4096 or the compilation will fail. See the
> macro definition of __is_kvfree_rcu_offset() in include/linux/rcupdate.h
> for more information.
>
> By putting rcu_head below the large cgroup structure, any change to the
> cgroup structure that makes it larger run the risk of causing build
> failure under certain configurations. Commit 77070eeb8821 ("cgroup:
> Avoid false cacheline sharing of read mostly rstat_cpu") happens to be
> the last straw that breaks it. Fix this problem by moving the rcu_head
> structure up before the cgroup structure.
>
> Fixes: d23b5c577715 ("cgroup: Make operations on the cgroup root_list RCU safe")
> Reported-by: Stephen Rothwell <sfr@canb.auug.org.au>
> Closes: https://lore.kernel.org/lkml/20231207143806.114e0a74@canb.auug.org.au/
> Signed-off-by: Waiman Long <longman@redhat.com>
> Acked-by: Yafang Shao <laoar.shao@gmail.com>

 Reviewed-by: Yosry Ahmed <yosryahmed@google.com>

Re: [PATCH-cgroup v2] cgroup: Move rcu_head up near the top of cgroup_root

[Tejun Heo]

On Thu, Dec 07, 2023 at 08:46:14AM -0500, Waiman Long wrote:
> Commit d23b5c577715 ("cgroup: Make operations on the cgroup root_list RCU
> safe") adds a new rcu_head to the cgroup_root structure and kvfree_rcu()
> for freeing the cgroup_root.
> 
> The current implementation of kvfree_rcu(), however, has the limitation
> that the offset of the rcu_head structure within the larger data
> structure must be less than 4096 or the compilation will fail. See the
> macro definition of __is_kvfree_rcu_offset() in include/linux/rcupdate.h
> for more information.
> 
> By putting rcu_head below the large cgroup structure, any change to the
> cgroup structure that makes it larger run the risk of causing build
> failure under certain configurations. Commit 77070eeb8821 ("cgroup:
> Avoid false cacheline sharing of read mostly rstat_cpu") happens to be
> the last straw that breaks it. Fix this problem by moving the rcu_head
> structure up before the cgroup structure.
> 
> Fixes: d23b5c577715 ("cgroup: Make operations on the cgroup root_list RCU safe")
> Reported-by: Stephen Rothwell <sfr@canb.auug.org.au>
> Closes: https://lore.kernel.org/lkml/20231207143806.114e0a74@canb.auug.org.au/
> Signed-off-by: Waiman Long <longman@redhat.com>
> Acked-by: Yafang Shao <laoar.shao@gmail.com>

Applied to cgroup/for-6.8.

Thanks.

-- 
tejun