From: Lance Yang <ioworker0@gmail.com>
To: akpm@linux-foundation.org
Cc: 21cnbao@gmail.com, ryan.roberts@arm.com, david@redhat.com,
shy828301@gmail.com, ziy@nvidia.com, libang.li@antgroup.com,
baolin.wang@linux.alibaba.com, linux-kernel@vger.kernel.org,
linux-mm@kvack.org, Lance Yang <ioworker0@gmail.com>
Subject: [BUG] mm/cgroupv2: memory.min may lead to an OOM error
Date: Thu, 1 Aug 2024 12:54:27 +0800 [thread overview]
Message-ID: <20240801045430.48694-1-ioworker0@gmail.com> (raw)
Hi all,
It's possible to encounter an OOM error if both parent and child cgroups are
configured such that memory.min and memory.max are set to the same values, as
is practice in Kubernetes.
Hmm... I'm not sure that whether this behavior is a bug or an expected aspect of
the kernel design.
To reproduce the bug, we can follow these command-based steps:
1. Check Kernel Version and OS release:
```
$ uname -r
6.10.0-rc5+
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="<https://www.ubuntu.com/>"
SUPPORT_URL="<https://help.ubuntu.com/>"
BUG_REPORT_URL="<https://bugs.launchpad.net/ubuntu/>"
PRIVACY_POLICY_URL="<https://www.ubuntu.com/legal/terms-and-policies/privacy-policy>"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo
```
2. Navigate to the cgroup v2 filesystem, create a test cgroup, and set memory settings:
```
$ cd /sys/fs/cgroup/
$ stat -fc %T /sys/fs/cgroup
cgroup2fs
$ mkdir test
$ echo "+memory" > cgroup.subtree_control
$ mkdir test/test-child
$ echo 1073741824 > memory.max
$ echo 1073741824 > memory.min
$ cat memory.max
1073741824
$ cat memory.min
1073741824
$ cat memory.low
0
$ cat memory.high
max
```
3. Set up and check memory settings in the child cgroup:
```
$ cd test-child
$ echo 1073741824 > memory.max
$ echo 1073741824 > memory.min
$ cat memory.max
1073741824
$ cat memory.min
1073741824
$ cat memory.low
0
$ cat memory.high
max
```
4. Add process to the child cgroup and verify:
```
$ echo $$ > cgroup.procs
$ cat cgroup.procs
1131
1320
$ ps -ef|grep 1131
root 1131 1014 0 10:45 pts/0 00:00:00 -bash
root 1321 1131 99 11:06 pts/0 00:00:00 ps -ef
root 1322 1131 0 11:06 pts/0 00:00:00 grep --color=auto 1131
```
5. Attempt to create a large file using dd and observe the process being killed:
```
$ dd if=/dev/zero of=/tmp/2gbfile bs=10M count=200
Killed
```
6. Check kernel messages related to the OOM event:
```
$ dmesg
...
[ 1341.112388] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/test,task_memcg=/test/test-child,task=dd,pid=1324,uid=0
[ 1341.112418] Memory cgroup out of memory: Killed process 1324 (dd) total-vm:15548kB, anon-rss:10240kB, file-rss:1764kB, shmem-rss:0kB, UID:0 pgtables:76kB oom_score_adj:0
```
7. Reduce the `memory.min` setting in the child cgroup and attempt the same large file creation, and then this issue is resolved.
```
# echo 107374182 > memory.min
# dd if=/dev/zero of=/tmp/2gbfile bs=10M count=200
200+0 records in
200+0 records out
2097152000 bytes (2.1 GB, 2.0 GiB) copied, 1.8713 s, 1.1 GB/s
```
Thanks,
Lance
next reply other threads:[~2024-08-01 4:54 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-08-01 4:54 Lance Yang [this message]
2024-08-01 10:35 ` [BUG] mm/cgroupv2: memory.min may lead to an OOM error Vlastimil Babka (SUSE)
2024-08-01 11:40 ` Lance Yang
2024-08-01 22:58 ` Michal Koutný
2024-08-02 1:56 ` Lance Yang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240801045430.48694-1-ioworker0@gmail.com \
--to=ioworker0@gmail.com \
--cc=21cnbao@gmail.com \
--cc=akpm@linux-foundation.org \
--cc=baolin.wang@linux.alibaba.com \
--cc=david@redhat.com \
--cc=libang.li@antgroup.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=ryan.roberts@arm.com \
--cc=shy828301@gmail.com \
--cc=ziy@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).