"make -j" with memory.(memsw.)limit_in_bytes smaller than required -> livelock, even for unlimited processes

[Lutz Vieweg <lvml@5t9.de>]

[-- Attachment #1: Type: text/plain, Size: 2527 bytes --]

Dear Memory Ressource Controller maintainers,

by using per-user control groups with a limit on memory (and swap) I am
trying to secure a shared development server against memory exhaustion
by any one single user - as it happened before when somebody imprudently
issued "make -j" (which has the infamous habit to spawn an unlimited
number of processes) on a large software project with many source files.

The memory limitation using control groups works just fine when
only a few processes sum up to a usage that exceeds the limits - the
processes are OOM-killed, then, and the others users are unaffected.

But the original cause, a "make -j" on many source files, leads to
the following ugly symptom:

- make starts numerous (~ 100 < x < 200) gcc processes

- some of those gcc processes get OOM-killed quickly, then
   a few more are killed, but with increasing pauses in between

- then after a few seconds, no more gcc processes are killed, but
   the "make" process and its childs do not show any progress anymore

- at this time, top indicates 100% "system" CPU usage, mostly by
   "[kworker/*]" threads (one per CPU). But processes from other
   users, that only require CPU, proceed to run.

- but also at this time, if any other user (who has not exhausted
   his memory limits) tries to access any file (at least on /tmp/,
   as e.g. gcc does), even a simple "ls /tmp/", this operation
   waits forever. (But "iostat" does not indicate any I/O activity.)

- as soon as you press "CTRL-C" to abort the "make -j", everything
   goes back to normal, quickly - also the other users' processes proceed.


To reproduce the problem, the attached "Makefile" to a directory
on a filesystem with at least 70MB free space, then

  mount -o memory none /cgroup
  mkdir /cgroup/test
  echo 64M >/cgroup/test/memory.limit_in_bytes
  echo 64M >/cgroup/test/memory.memsw.limit_in_bytes

  cd /somewhere/with/70mb/free
  echo $$ >/cgroup/test/tasks
  make sources
  make -j compile

Notice that "make sources" will create 200 bogus "*.c" files from
/dev/urandom to make sure that "gcc" will use up some memory.

The "make -j compile" reliably reproduces the above mentioned syndrome,
here.

Please notice that the livelock does happen only with a significant
number of parallel compiler runs - it did e.g. not happen with
only 100 for me, and it also did not happen when I started "make"
with "strace" - so timing seems to be an issue, here.

Thanks for any hints towards a solution of this issue in advance!

Regards,

Lutz Vieweg

[-- Attachment #2: Makefile --]
[-- Type: text/plain, Size: 542 bytes --]


all:
	echo "first 'make sources', then 'make -j compile'


N=200 

clean:
	rm -f file_*.o lib.so


mrproper:
	rm -f file_*.c file_*.o lib.so
	

sources: clean
	for (( I=0 ; $$I < $(N) ; I=`expr $$I + 1` )) ; do \
		echo $$I; \
		echo "char array_$$I [] = " >file_$$I.c ;\
		dd if=/dev/urandom bs=256k count=1 | base64 | sed 's/^.*/"\0"/g' >>file_$$I.c ;\
		echo ";" >>file_$$I.c ;\
	done


OBJ = $(addsuffix .o, $(basename $(notdir $(wildcard file_*.c))))

compile: $(OBJ)
	gcc -shared -O3 -o lib.so $(OBJ)	

%.o: ./%.c
	gcc -O3 -c $< -o $@