From: "Michal Koutný" <mkoutny@suse.com>
To: Tadeusz Struk <tadeusz.struk@linaro.org>
Cc: cgroups@vger.kernel.org, Tejun Heo <tj@kernel.org>,
Zefan Li <lizefan.x@bytedance.com>,
Johannes Weiner <hannes@cmpxchg.org>,
Christian Brauner <brauner@kernel.org>,
Alexei Starovoitov <ast@kernel.org>,
Daniel Borkmann <daniel@iogearbox.net>,
Andrii Nakryiko <andrii@kernel.org>,
Martin KaFai Lau <kafai@fb.com>, Song Liu <songliubraving@fb.com>,
Yonghong Song <yhs@fb.com>,
John Fastabend <john.fastabend@gmail.com>,
KP Singh <kpsingh@kernel.org>,
netdev@vger.kernel.org, bpf@vger.kernel.org,
stable@vger.kernel.org, linux-kernel@vger.kernel.org,
syzbot+e42ae441c3b10acf9e9d@syzkaller.appspotmail.com
Subject: Re: [PATCH] cgroup: don't queue css_release_work if one already pending
Date: Thu, 14 Apr 2022 18:44:09 +0200 [thread overview]
Message-ID: <20220414164409.GA5404@blackbody.suse.cz> (raw)
In-Reply-To: <20220412192459.227740-1-tadeusz.struk@linaro.org>
Hello Tadeusz.
Thanks for analyzing this syzbot report. Let me provide my understanding
of the test case and explanation why I think your patch fixes it but is
not fully correct.
On Tue, Apr 12, 2022 at 12:24:59PM -0700, Tadeusz Struk <tadeusz.struk@linaro.org> wrote:
> Syzbot found a corrupted list bug scenario that can be triggered from
> cgroup css_create(). The reproduces writes to cgroup.subtree_control
> file, which invokes cgroup_apply_control_enable(), css_create(), and
> css_populate_dir(), which then randomly fails with a fault injected -ENOMEM.
The reproducer code makes it hard for me to understand which function
fails with ENOMEM.
But I can see your patch fixes the reproducer and your additional debug
patch which proves that css->destroy_work is re-queued.
> In such scenario the css_create() error path rcu enqueues css_free_rwork_fn
> work for an css->refcnt initialized with css_release() destructor,
Note that css_free_rwork_fn() utilizes css->destroy_*r*work.
The error path in css_create() open codes relevant parts of
css_release_work_fn() so that css_release() can be skipped and the
refcnt is eventually just percpu_ref_exit()'d.
> and there is a chance that the css_release() function will be invoked
> for a cgroup_subsys_state, for which a destroy_work has already been
> queued via css_create() error path.
But I think the problem is css_populate_dir() failing in
cgroup_apply_control_enable(). (Is this what you actually meant?
css_create() error path is then irrelevant, no?)
The already created csses should then be rolled back via
cgroup_restore_control(cgrp);
cgroup_apply_control_disable(cgrp);
...
kill_css(css)
I suspect the double-queuing is a result of the fact that there exists
only the single reference to the css->refcnt. I.e. it's
percpu_ref_kill_and_confirm()'d and released both at the same time.
(Normally (when not killing the last reference), css->destroy_work reuse
is not a problem because of the sequenced chain
css_killed_work_fn()->css_put()->css_release().)
> This can be avoided by adding a check to css_release() that checks
> if it has already been enqueued.
If that's what's happening, then your patch omits the final
css_release_work_fn() in favor of css_killed_work_fn() but both should
be run during the rollback upon css_populate_dir() failure.
So an alternative approach to tackle this situation would be to split
css->destroy_work into two work work_structs (one for killing, one for
releasing) at the cost of inflating cgroup_subsys_state.
Take my hypothesis with a grain of salt maybe the assumption (last
reference == initial reference) is not different from normal operation.
Regards,
Michal
next prev parent reply other threads:[~2022-04-14 17:04 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-04-12 19:24 [PATCH] cgroup: don't queue css_release_work if one already pending Tadeusz Struk
2022-04-14 16:44 ` Michal Koutný [this message]
2022-04-14 17:51 ` Tadeusz Struk
2022-04-21 23:43 ` Tejun Heo
2022-04-22 0:00 ` Tejun Heo
2022-04-22 11:05 ` Michal Koutný
2022-05-18 16:48 ` Tadeusz Struk
[not found] ` <20220519112319.2455-1-hdanton@sina.com>
2022-05-19 23:26 ` Tadeusz Struk
2022-05-20 8:13 ` Tejun Heo
2022-05-20 16:38 ` Tadeusz Struk
2022-05-20 16:42 ` Michal Koutný
2022-05-20 16:56 ` Tadeusz Struk
2022-05-23 19:00 ` Tadeusz Struk
2022-05-23 19:02 ` Tejun Heo
2022-05-23 19:08 ` Tadeusz Struk
2022-05-23 20:05 ` Tadeusz Struk
2022-05-23 21:27 ` [PATCH v2] cgroups: separate destroy_work into two separate wq Tadeusz Struk
[not found] <20220413095606.1182-1-hdanton@sina.com>
2022-04-13 15:39 ` [PATCH] cgroup: don't queue css_release_work if one already pending Tadeusz Struk
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20220414164409.GA5404@blackbody.suse.cz \
--to=mkoutny@suse.com \
--cc=andrii@kernel.org \
--cc=ast@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=brauner@kernel.org \
--cc=cgroups@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=hannes@cmpxchg.org \
--cc=john.fastabend@gmail.com \
--cc=kafai@fb.com \
--cc=kpsingh@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=lizefan.x@bytedance.com \
--cc=netdev@vger.kernel.org \
--cc=songliubraving@fb.com \
--cc=stable@vger.kernel.org \
--cc=syzbot+e42ae441c3b10acf9e9d@syzkaller.appspotmail.com \
--cc=tadeusz.struk@linaro.org \
--cc=tj@kernel.org \
--cc=yhs@fb.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox