"make -j" with memory.(memsw.)limit_in_bytes smaller than required -> livelock, even for unlimited processes

Lutz Vieweg
2011-06-21T14:51:32Z

[-- Attachment #1: Type: text/plain, Size: 2527 bytes --]

Dear Memory Ressource Controller maintainers,

by using per-user control groups with a limit on memory (and swap) I am
trying to secure a shared development server against memory exhaustion
by any one single user - as it happened before when somebody imprudently
issued "make -j" (which has the infamous habit to spawn an unlimited
number of processes) on a large software project with many source files.

The memory limitation using control groups works just fine when
only a few processes sum up to a usage that exceeds the limits - the
processes are OOM-killed, then, and the others users are unaffected.

But the original cause, a "make -j" on many source files, leads to
the following ugly symptom:

- make starts numerous (~ 100 < x < 200) gcc processes

- some of those gcc processes get OOM-killed quickly, then
a few more are killed, but with increasing pauses in between

- then after a few seconds, no more gcc processes are killed, but
the "make" process and its childs do not show any progress anymore

- at this time, top indicates 100% "system" CPU usage, mostly by
"[kworker/*]" threads (one per CPU). But processes from other
users, that only require CPU, proceed to run.

- but also at this time, if any other user (who has not exhausted
his memory limits) tries to access any file (at least on /tmp/,
as e.g. gcc does), even a simple "ls /tmp/", this operation
waits forever. (But "iostat" does not indicate any I/O activity.)

- as soon as you press "CTRL-C" to abort the "make -j", everything
goes back to normal, quickly - also the other users' processes proceed.

To reproduce the problem, the attached "Makefile" to a directory
on a filesystem with at least 70MB free space, then

mount -o memory none /cgroup
mkdir /cgroup/test
echo 64M >/cgroup/test/memory.limit_in_bytes
echo 64M >/cgroup/test/memory.memsw.limit_in_bytes

cd /somewhere/with/70mb/free
echo $$ >/cgroup/test/tasks
make sources
make -j compile

Notice that "make sources" will create 200 bogus "*.c" files from
/dev/urandom to make sure that "gcc" will use up some memory.

The "make -j compile" reliably reproduces the above mentioned syndrome,
here.

Please notice that the livelock does happen only with a significant
number of parallel compiler runs - it did e.g. not happen with
only 100 for me, and it also did not happen when I started "make"
with "strace" - so timing seems to be an issue, here.

Thanks for any hints towards a solution of this issue in advance!

Regards,

Lutz Vieweg

[-- Attachment #2: Makefile --]
[-- Type: text/plain, Size: 542 bytes --]

all:
echo "first 'make sources', then 'make -j compile'

N=200

clean:
rm -f file_*.o lib.so

mrproper:
rm -f file_*.c file_*.o lib.so

sources: clean
for (( I=0 ; $$I < $(N) ; I=`expr $$I + 1` )) ; do \
echo $$I; \
echo "char array_$$I [] = " >file_$$I.c ;\
dd if=/dev/urandom bs=256k count=1 | base64 | sed 's/^.*/"\0"/g' >>file_$$I.c ;\
echo ";" >>file_$$I.c ;\
done

OBJ = $(addsuffix .o, $(basename $(notdir $(wildcard file_*.c))))

compile: $(OBJ)
gcc -shared -O3 -o lib.so $(OBJ)

%.o: ./%.c
gcc -O3 -c $< -o $@