From: Dave Jones <davej@codemonkey.org.uk>
To: Michal Hocko <mhocko@kernel.org>
Cc: linux-mm@kvack.org
Subject: Re: oom-reaper choosing wrong processes.
Date: Tue, 19 Jul 2016 11:33:35 -0400 [thread overview]
Message-ID: <20160719153335.GA11863@codemonkey.org.uk> (raw)
In-Reply-To: <20160719090857.GB9490@dhcp22.suse.cz>
On Tue, Jul 19, 2016 at 11:08:58AM +0200, Michal Hocko wrote:
> On Mon 18-07-16 19:18:50, Dave Jones wrote:
> [...]
> > [ 4607.765352] sendmail-mta invoked oom-killer: gfp_mask=0x24201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), order=0, oom_score_adj=0
> > [ 4607.765359] sendmail-mta cpuset=/ mems_allowed=0
> [...]
> > [ 4607.765619] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
> > [ 4607.765637] [ 749] 0 749 13116 782 29 3 385 0 systemd-journal
> > [ 4607.765641] [ 793] 0 793 10640 10 23 3 285 -1000 systemd-udevd
> > [ 4607.765647] [ 1647] 0 1647 11928 16 27 3 111 0 rpcbind
> > [ 4607.765651] [ 1653] 0 1653 5841 0 15 3 54 0 rpc.idmapd
> > [ 4607.765656] [ 1655] 0 1655 11052 24 26 3 114 0 systemd-logind
> > [ 4607.765687] [ 1657] 0 1657 64579 181 28 3 161 0 rsyslogd
> > [ 4607.765691] [ 1660] 0 1660 1058 0 8 3 38 0 acpid
> > [ 4607.765696] [ 1661] 0 1661 7414 22 18 3 52 0 cron
> > [ 4607.765700] [ 1662] 0 1662 6993 0 19 3 54 0 atd
> > [ 4607.765704] [ 1664] 105 1664 10744 40 26 3 79 -900 dbus-daemon
> > [ 4607.765708] [ 1671] 0 1671 6264 29 17 3 157 0 smartd
> > [ 4607.765712] [ 1738] 0 1738 16948 0 37 3 204 -1000 sshd
> > [ 4607.765716] [ 1742] 0 1742 9461 0 22 3 195 0 rpc.mountd
> > [ 4607.765721] [ 1776] 0 1776 3624 0 12 3 39 0 agetty
> > [ 4607.765725] [ 1797] 0 1797 3319 0 10 3 48 0 mcelog
> > [ 4607.765729] [ 1799] 0 1799 4824 21 15 3 39 0 irqbalance
> > [ 4607.765733] [ 1803] 108 1803 25492 42 24 3 124 0 ntpd
> > [ 4607.765737] [ 1842] 0 1842 19793 48 39 3 410 0 sendmail-mta
> > [ 4607.765746] [ 1878] 0 1878 5121 0 13 3 262 0 dhclient
> > [ 4607.765752] [ 2145] 1000 2145 15627 0 33 3 213 0 systemd
> > [ 4607.765756] [ 2148] 1000 2148 19584 4 40 3 438 0 (sd-pam)
> > [ 4607.765760] [ 2643] 1000 2643 7465 433 19 3 152 0 tmux
> > [ 4607.765764] [ 2644] 1000 2644 5864 0 16 3 508 0 bash
> > [ 4607.765768] [ 2678] 1000 2678 3328 89 11 3 19 0 test-multi.sh
> > [ 4607.765774] [ 2693] 1000 2693 5864 1 16 3 507 0 bash
> > [ 4607.765782] [ 6456] 1000 6456 3091 21 11 3 24 0 dmesg
> > [ 4607.765787] [18624] 1000 18624 750863 43368 520 6 0 500 trinity-c10
> > [ 4607.765792] [21525] 1000 21525 797320 20517 493 7 0 500 trinity-c15
> > [ 4607.765796] [22023] 1000 22023 797349 1985 319 7 0 500 trinity-c2
> > [ 4607.765814] [22658] 1000 22658 797382 1 458 7 0 500 trinity-c0
> > [ 4607.765818] [26334] 1000 26334 797217 34960 412 7 0 500 trinity-c4
> > [ 4607.765823] [26388] 1000 26388 797383 9401 118 7 0 500 trinity-c11
> > [ 4607.765826] oom_kill_process: would have killed process 749 (systemd-journal), but continuing instead...
> > [ 4608.147644] oom_reaper: reaped process 26334 (trinity-c4), now anon-rss:0kB, file-rss:0kB, shmem-rss:136724kB
> > [ 4608.148218] oom_reaper: reaped process 18624 (trinity-c10), now anon-rss:0kB, file-rss:0kB, shmem-rss:174356kB
> > [ 4608.149795] oom_reaper: reaped process 21525 (trinity-c15), now anon-rss:0kB, file-rss:0kB, shmem-rss:86288kB
> > [ 4608.150734] oom_reaper: reaped process 18624 (trinity-c10), now anon-rss:0kB, file-rss:0kB, shmem-rss:174348kB
> > [ 4608.152489] oom_reaper: reaped process 21525 (trinity-c15), now anon-rss:0kB, file-rss:0kB, shmem-rss:86288kB
> > [ 4608.156127] oom_reaper: reaped process 18624 (trinity-c10), now anon-rss:0kB, file-rss:0kB, shmem-rss:174336kB
> > [ 4608.158798] oom_reaper: reaped process 26334 (trinity-c4), now anon-rss:0kB, file-rss:0kB, shmem-rss:136652kB
> > [ 4608.161336] oom_reaper: reaped process 26334 (trinity-c4), now anon-rss:0kB, file-rss:0kB, shmem-rss:136652kB
> > [ 4608.163836] oom_reaper: reaped process 26334 (trinity-c4), now anon-rss:0kB, file-rss:0kB, shmem-rss:136652kB
> >
> >
> > Whoa. Why did it pick systemd-journal ?
>
> Who has picked that? select_bad process?
>
> > My 'skip over !trinity processes' code kicks in, and it then kills the
> > right processes, and the box lives on, but if I hadn't have had that
> > diff, the wrong process would have been killed.
>
> Could you post your diff please? oom_reaper will reap not only the oom
> victims but also tasks which hit the oom path and they are exiting.
commit fd1fe9a8ad00ee6a2e81a1152c3150d2289fdf10
Author: Dave Jones <davej@codemonkey.org.uk>
Date: Mon Jun 15 23:41:09 2015 -0400
only oom-kill tasks named trinity
diff --git a/mm/oom_kill.c b/mm/oom_kill.c
index ddf74487f848..2635db037dd5 100644
--- a/mm/oom_kill.c
+++ b/mm/oom_kill.c
@@ -767,6 +767,15 @@ void oom_kill_process(struct oom_control *oc, struct task_struct *p,
if (__ratelimit(&oom_rs))
dump_header(oc, p, memcg);
+ task_lock(p);
+ if (strncmp(p->comm, "trinity-", 8) != 0) {
+ printk(KERN_ERR "%s: would have killed process %d (%s), but continuing instead...\n",
+ __func__, task_pid_nr(p), p->comm);
+ task_unlock(p);
+ return;
+ }
+ task_unlock(p);
+
pr_err("%s: Kill process %d (%s) score %u or sacrifice child\n",
message, task_pid_nr(p), p->comm, points);
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2016-07-19 15:33 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-07-18 23:18 oom-reaper choosing wrong processes Dave Jones
2016-07-19 9:08 ` Michal Hocko
2016-07-19 10:52 ` Tetsuo Handa
2016-07-19 15:36 ` Dave Jones
2016-07-20 10:40 ` Tetsuo Handa
2016-07-19 15:33 ` Dave Jones [this message]
2016-07-20 7:09 ` Michal Hocko
2016-07-20 13:23 ` Dave Jones
2016-07-20 13:33 ` Dave Jones
2016-07-20 13:39 ` Michal Hocko
2016-07-20 13:36 ` Michal Hocko
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20160719153335.GA11863@codemonkey.org.uk \
--to=davej@codemonkey.org.uk \
--cc=linux-mm@kvack.org \
--cc=mhocko@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.