From: Jan Stancek <jstancek@redhat.com>
To: ltp@lists.linux.it
Subject: [LTP] [BUG] oom hangs the system, NMI backtrace shows most CPUs in shrink_slab
Date: Tue, 26 Jan 2016 08:48:01 +0100 [thread overview]
Message-ID: <56A724B1.3000407@redhat.com> (raw)
In-Reply-To: <56A24760.5020503@redhat.com>
On 01/22/2016 04:14 PM, Jan Stancek wrote:
> On 01/19/2016 11:29 AM, Tetsuo Handa wrote:
>> although I
>> couldn't find evidence that mlock() and madvice() are related with this hangup,
>
> I simplified reproducer by having only single thread allocating
> memory when OOM triggers:
> http://jan.stancek.eu/tmp/oom_hangs/console.log.3-v4.4-8606-with-memalloc.txt
>
> In this instance it was mmap + mlock, as you can see from oom call trace.
> It made it to do_exit(), but couldn't complete it:
I have extracted test from LTP into standalone reproducer (attached),
if you want to give a try. It usually hangs my system within ~30
minutes. If it takes too long, you can try disabling swap. From my past
experience this usually helped to reproduce it faster on small KVM guests.
# gcc oom_mlock.c -pthread -O2
# echo 1 > /proc/sys/vm/overcommit_memory
(optionally) # swapoff -a
# ./a.out
Also, it's interesting to note, that when I disabled mlock() calls
test ran fine over night. I'll look into confirming this observation
on more systems.
Regards,
Jan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: oom_mlock.c
Type: text/x-csrc
Size: 1974 bytes
Desc: not available
URL: <http://lists.linux.it/pipermail/ltp/attachments/20160126/f54f7c7b/attachment.c>
WARNING: multiple messages have this Message-ID (diff)
From: Jan Stancek <jstancek@redhat.com>
To: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>, linux-mm@kvack.org
Cc: ltp@lists.linux.it
Subject: Re: [LTP] [BUG] oom hangs the system, NMI backtrace shows most CPUs in shrink_slab
Date: Tue, 26 Jan 2016 08:48:01 +0100 [thread overview]
Message-ID: <56A724B1.3000407@redhat.com> (raw)
In-Reply-To: <56A24760.5020503@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 1058 bytes --]
On 01/22/2016 04:14 PM, Jan Stancek wrote:
> On 01/19/2016 11:29 AM, Tetsuo Handa wrote:
>> although I
>> couldn't find evidence that mlock() and madvice() are related with this hangup,
>
> I simplified reproducer by having only single thread allocating
> memory when OOM triggers:
> http://jan.stancek.eu/tmp/oom_hangs/console.log.3-v4.4-8606-with-memalloc.txt
>
> In this instance it was mmap + mlock, as you can see from oom call trace.
> It made it to do_exit(), but couldn't complete it:
I have extracted test from LTP into standalone reproducer (attached),
if you want to give a try. It usually hangs my system within ~30
minutes. If it takes too long, you can try disabling swap. From my past
experience this usually helped to reproduce it faster on small KVM guests.
# gcc oom_mlock.c -pthread -O2
# echo 1 > /proc/sys/vm/overcommit_memory
(optionally) # swapoff -a
# ./a.out
Also, it's interesting to note, that when I disabled mlock() calls
test ran fine over night. I'll look into confirming this observation
on more systems.
Regards,
Jan
[-- Attachment #2: oom_mlock.c --]
[-- Type: text/x-csrc, Size: 1974 bytes --]
#include <errno.h>
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/mman.h>
#include <sys/wait.h>
/*
* oom hang reproducer v1
*
* # gcc oom_mlock.c -pthread -O2
* # echo 1 > /proc/sys/vm/overcommit_memory
* (optionally) # swapoff -a
* # ./a.out
*/
#define _1GB (1024L*1024*1024)
static do_mlock = 1;
static int alloc_mem(long int length)
{
char *s;
long i, pagesz = getpagesize();
int loop = 10;
printf("thread (%lx), allocating %ld bytes, do_mlock: %d\n",
(unsigned long) pthread_self(), length, do_mlock);
s = mmap(NULL, length, PROT_READ | PROT_WRITE,
MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
if (s == MAP_FAILED)
return errno;
if (do_mlock) {
while (mlock(s, length) == -1 && loop > 0) {
if (EAGAIN != errno)
return errno;
usleep(300000);
loop--;
}
}
for (i = 0; i < length; i += pagesz)
s[i] = '\a';
return 0;
}
void *alloc_thread(void *args)
{
int ret;
do {
ret = alloc_mem(3 * _1GB);
} while (ret == 0);
exit(ret);
}
int trigger_oom(void)
{
int i, ret, child, status, threads;
pthread_t *th;
threads = sysconf(_SC_NPROCESSORS_ONLN) - 1;
th = malloc(sizeof(pthread_t) * threads);
if (!th) {
printf("malloc failed\n");
exit(2);
}
do_mlock = !do_mlock;
child = fork();
if (child == 0) {
for (i = 0; i < threads - 1; i++) {
ret = pthread_create(&th[i], NULL, alloc_thread, NULL);
if (ret) {
printf("pthread_create failed with %d\n", ret);
exit(3);
}
}
pause();
}
if (waitpid(-1, &status, 0) == -1) {
perror("waitpid");
exit(1);
}
if (WIFSIGNALED(status)) {
printf("child killed by %d\n", WTERMSIG(status));
if (WTERMSIG(status) != SIGKILL)
exit(1);
}
if (WIFEXITED(status)) {
printf("child exited with %d\n", WEXITSTATUS(status));
if (WEXITSTATUS(status) != ENOMEM)
exit(1);
}
}
int main(void)
{
int i = 1;
while (1) {
printf("starting iteration %d\n", i++);
trigger_oom();
}
return 0;
}
next prev parent reply other threads:[~2016-01-26 7:48 UTC|newest]
Thread overview: 26+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-01-18 15:38 [LTP] [BUG] oom hangs the system, NMI backtrace shows most CPUs in shrink_slab Jan Stancek
2016-01-18 15:38 ` Jan Stancek
2016-01-19 10:29 ` [LTP] " Tetsuo Handa
2016-01-19 10:29 ` Tetsuo Handa
2016-01-19 15:13 ` [LTP] " Jan Stancek
2016-01-19 15:13 ` Jan Stancek
2016-01-20 10:23 ` [LTP] [BUG] oom hangs the system, NMI backtrace shows most CPUs inshrink_slab Tetsuo Handa
2016-01-20 10:23 ` Tetsuo Handa
2016-01-20 13:17 ` [LTP] [BUG] oom hangs the system, NMI backtrace shows most CPUs in shrink_slab Tetsuo Handa
2016-01-20 13:17 ` Tetsuo Handa
2016-01-20 15:10 ` [LTP] " Tejun Heo
2016-01-20 15:10 ` Tejun Heo
2016-01-20 15:54 ` [LTP] " Tetsuo Handa
2016-01-20 15:54 ` Tetsuo Handa
2016-01-22 15:14 ` [LTP] " Jan Stancek
2016-01-22 15:14 ` Jan Stancek
2016-01-23 6:30 ` [LTP] " Tetsuo Handa
2016-01-23 6:30 ` Tetsuo Handa
2016-01-26 7:48 ` Jan Stancek [this message]
2016-01-26 7:48 ` [LTP] " Jan Stancek
2016-01-26 14:46 ` Tetsuo Handa
2016-01-26 14:46 ` Tetsuo Handa
2016-01-27 11:02 ` Tetsuo Handa
2016-01-28 15:48 ` Tetsuo Handa
2016-01-29 7:32 ` Jan Stancek
2016-01-29 12:35 ` Tetsuo Handa
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=56A724B1.3000407@redhat.com \
--to=jstancek@redhat.com \
--cc=ltp@lists.linux.it \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.