From: Trevor Cordes <trevor@tecnopolis.ca>
To: Mel Gorman <mgorman@techsingularity.net>
Cc: Michal Hocko <mhocko@kernel.org>,
linux-kernel@vger.kernel.org,
Joonsoo Kim <iamjoonsoo.kim@lge.com>,
Minchan Kim <minchan@kernel.org>, Rik van Riel <riel@surriel.com>,
Srikar Dronamraju <srikar@linux.vnet.ibm.com>
Subject: Re: mm, vmscan: commit makes PAE kernel crash nightly (bisected)
Date: Wed, 25 Jan 2017 04:02:46 -0600 [thread overview]
Message-ID: <20170125040246.5b211135@pog.tecnopolis.ca> (raw)
In-Reply-To: <20170123104858.gpjy25y2ogju3gkg@techsingularity.net>
[-- Attachment #1: Type: text/plain, Size: 4154 bytes --]
On 2017-01-23 Mel Gorman wrote:
> On Sun, Jan 22, 2017 at 06:45:59PM -0600, Trevor Cordes wrote:
> > On 2017-01-20 Mel Gorman wrote:
> > > >
> > > > Thanks for the OOM report. I was expecting it to be a particular
> > > > shape and my expectations were not matched so it took time to
> > > > consider it further. Can you try the cumulative patch below? It
> > > > combines three patches that
> > > >
> > > > 1. Allow slab shrinking even if the LRU patches are
> > > > unreclaimable in direct reclaim
> > > > 2. Shrinks slab based once based on the contents of all memcgs
> > > > instead of shrinking one at a time
> > > > 3. Tries to shrink slabs if the lowmem usage is too high
> > > >
> > > > Unfortunately it's only boot tested on x86-64 as I didn't get
> > > > the chance to setup an i386 test bed.
> > > >
> > >
> > > There was one major flaw in that patch. This version fixes it and
> > > addresses other minor issues. It may still be too agressive
> > > shrinking slab but worth trying out. Thanks.
> >
> > I ran with your patch below and it oom'd on the first night. It was
> > weird, it didn't hang the system, and my rebooter script started a
> > reboot but the system never got more than half down before it just
> > sat there in a weird state where a local console user could still
> > login but not much was working. So the patches don't seem to solve
> > the problem.
> >
> > For the above compile I applied your patches to 4.10.0-rc4+, I hope
> > that's ok.
> >
>
> It would be strongly preferred to run them on top of Michal's other
> fixes. The main reason it's preferred is because this OOM differs from
> earlier ones in that it OOM killed from GFP_NOFS|__GFP_NOFAIL context.
> That meant that the slab shrinking could not happen from direct
> reclaim so the balancing from my patches would not occur. As
> Michal's other patches affect how kswapd behaves, it's important.
OK, I patched & compiled mhocko's git tree from the other day 4.9.0+.
(To confirm, weird, but mhocko's git tree I'm using from a couple of
weeks ago shows the newest commit (git log) is
69973b830859bc6529a7a0468ba0d80ee5117826 "Linux 4.9"? Let me know if
I'm doing something wrong, see below.)
Anyhow, it oom'd as usual at ~3am, system froze after 20 ooms hit in 7
secs. So no help there. Attached is the oom log from the first oom
hit.
On 2017-01-24 Michal Hocko wrote:
> On Sun 22-01-17 18:45:59, Trevor Cordes wrote:
> [...]
> > Also, completely separate from your patch I ran mhocko's 4.9 tree
> > with mem=2G to see if lower ram amount would help, but it didn't.
> > Even with 2G the system oom and hung same as usual. So far the
> > only thing that helps at all was the cgroup_disable=memory option,
> > which makes the problem disappear completely for me.
>
> OK, can we reduce the problem space slightly more and could you boot
> with kmem accounting enabled? cgroup.memory=nokmem,nosocket
I will try that right now, I'll use the mhocko git tree without Mel's
emailed patch, and I'll refresh the git tree from origin first (let me
know that's a bad move). As usual, I'll report back within 24-48 hours.
Actually, on my tests with mhocko git tree, I'm a bit confused and want
to make sure I'm compiling the right thing. His tree doesn't seem to
have recent commits? I did "git fetch origin" and "git reset --hard
origin/master" to refresh the tree just now and the latest commit is
still the one shown above "Linux 4.9"? Is Michal making changes but
not comitting? How do I ensure I'm compiling the version you guys want
me to test? ("git log mm/vmscan.c" shows newest commit is Dec 2??) Am
I supposed to be testing a specific branch?
If I've been testing the wrong branch, this *only* affects my mhocko
tree tests (not the vanilla or fedora-patched tests). Thankfully I
think I've only done 1 or 2 mhocko tree tests, and I can easily redo
them. If this turns out to be the case, I'm so sorry for the
confusion, the non-vanilla git tree thing is all new to me.
In any event, I'm still trying the above, and will adjust if necessary
if it's confirmed I'm doing something wrong with the mhocko git tree.
Thanks!
[-- Attachment #2: oom5 --]
[-- Type: application/octet-stream, Size: 22459 bytes --]
Jan 25 03:01:39 firewallfsi kernel: [91632.892378] smbd invoked oom-killer: gfp_mask=0x2420848(GFP_NOFS|__GFP_NOFAIL|__GFP_HARDWALL|__GFP_MOVABLE), nodemask=0, order=0, oom_score_adj=0
Jan 25 03:01:39 firewallfsi kernel: [91632.894810] smbd cpuset=/ mems_allowed=0
Jan 25 03:01:39 firewallfsi kernel: [91632.896010] CPU: 0 PID: 1973 Comm: smbd Not tainted 4.9.0+ #2
Jan 25 03:01:39 firewallfsi kernel: [91632.897220] Hardware name: Intel Corporation S1200BTL/S1200BTL, BIOS S1200BT.86B.02.00.0035.030220120927 03/02/2012
Jan 25 03:01:39 firewallfsi kernel: [91632.898418] ed683b5c c9f604e7 ed683c94 f69fc800 ed683b8c c9de17a6 ed683b6c ca363bad
Jan 25 03:01:39 firewallfsi kernel: [91632.899615] ed683b8c c9f6625f ed683b90 f6bcac00 ecb84000 f69fc800 ca568bce ed683c94
Jan 25 03:01:39 firewallfsi kernel: [91632.900797] ed683bd0 c9d7aff7 c9c76d8a ed683bbc c9d7ac6b 00000007 00000000 0000002d
Jan 25 03:01:39 firewallfsi kernel: [91632.901968] Call Trace:
Jan 25 03:01:39 firewallfsi kernel: [91632.903098] [<c9f604e7>] dump_stack+0x58/0x81
Jan 25 03:01:39 firewallfsi kernel: [91632.904214] [<c9de17a6>] dump_header+0x64/0x1a6
Jan 25 03:01:39 firewallfsi kernel: [91632.905311] [<ca363bad>] ? _raw_spin_unlock_irqrestore+0xd/0x10
Jan 25 03:01:39 firewallfsi kernel: [91632.906398] [<c9f6625f>] ? ___ratelimit+0x9f/0x100
Jan 25 03:01:39 firewallfsi kernel: [91632.907466] [<c9d7aff7>] oom_kill_process+0x207/0x3d0
Jan 25 03:01:39 firewallfsi kernel: [91632.908517] [<c9c76d8a>] ? has_capability_noaudit+0x1a/0x30
Jan 25 03:01:39 firewallfsi kernel: [91632.909554] [<c9d7ac6b>] ? oom_badness.part.13+0xcb/0x140
Jan 25 03:01:39 firewallfsi kernel: [91632.910574] [<c9d7b4d8>] out_of_memory+0xf8/0x2a0
Jan 25 03:01:39 firewallfsi kernel: [91632.911576] [<c9d80016>] __alloc_pages_nodemask+0xc46/0xd10
Jan 25 03:01:39 firewallfsi kernel: [91632.912564] [<c9d768c2>] ? find_get_entry+0x22/0x160
Jan 25 03:01:39 firewallfsi kernel: [91632.913534] [<c9d7727e>] pagecache_get_page+0xbe/0x2d0
Jan 25 03:01:39 firewallfsi kernel: [91632.914488] [<c9e19044>] __getblk_gfp+0x104/0x360
Jan 25 03:01:39 firewallfsi kernel: [91632.915422] [<c9e6b973>] ext4_getblk+0xa3/0x1b0
Jan 25 03:01:39 firewallfsi kernel: [91632.916337] [<c9e6baa3>] ext4_bread+0x23/0xb0
Jan 25 03:01:39 firewallfsi kernel: [91632.917240] [<c9e75617>] __ext4_read_dirblock+0x27/0x3f0
Jan 25 03:01:39 firewallfsi kernel: [91632.918130] [<c9def476>] ? lookup_fast+0x46/0x2c0
Jan 25 03:01:39 firewallfsi kernel: [91632.919003] [<c9e76005>] htree_dirblock_to_tree+0x45/0x1a0
Jan 25 03:01:39 firewallfsi kernel: [91632.919860] [<c9f6e90b>] ? lockref_get_not_dead+0xb/0x30
Jan 25 03:01:39 firewallfsi kernel: [91632.920701] [<c9def389>] ? unlazy_walk+0xf9/0x1a0
Jan 25 03:01:39 firewallfsi kernel: [91632.921523] [<c9e77000>] ext4_htree_fill_tree+0x90/0x2e0
Jan 25 03:01:39 firewallfsi kernel: [91632.922331] [<c9e63ef0>] ? ext4_readdir+0x8c0/0x960
Jan 25 03:01:39 firewallfsi kernel: [91632.923119] [<c9e63ef0>] ? ext4_readdir+0x8c0/0x960
Jan 25 03:01:39 firewallfsi kernel: [91632.923883] [<c9e631b6>] ? free_rb_tree_fname+0x16/0x70
Jan 25 03:01:39 firewallfsi kernel: [91632.924630] [<c9e63cff>] ext4_readdir+0x6cf/0x960
Jan 25 03:01:39 firewallfsi kernel: [91632.925358] [<c9f6e7f6>] ? _copy_to_user+0x26/0x30
Jan 25 03:01:39 firewallfsi kernel: [91632.926071] [<c9ee3b6f>] ? security_file_permission+0x9f/0xc0
Jan 25 03:01:39 firewallfsi kernel: [91632.926769] [<c9df84b9>] iterate_dir+0x179/0x1a0
Jan 25 03:01:39 firewallfsi kernel: [91632.927448] [<c9df8a6b>] SyS_getdents64+0x7b/0x110
Jan 25 03:01:39 firewallfsi kernel: [91632.928109] [<c9df84e0>] ? iterate_dir+0x1a0/0x1a0
Jan 25 03:01:39 firewallfsi kernel: [91632.928751] [<c9c0377a>] do_fast_syscall_32+0x8a/0x150
Jan 25 03:01:39 firewallfsi kernel: [91632.929378] [<ca3640ca>] sysenter_past_esp+0x47/0x75
Jan 25 03:01:39 firewallfsi kernel: [91632.929991] Mem-Info:
Jan 25 03:01:39 firewallfsi kernel: [91632.930764] active_anon:185890 inactive_anon:1821 isolated_anon:0
Jan 25 03:01:39 firewallfsi kernel: [91632.930764] active_file:123389 inactive_file:655369 isolated_file:0
Jan 25 03:01:39 firewallfsi kernel: [91632.930764] unevictable:0 dirty:1332 writeback:0 unstable:0
Jan 25 03:01:39 firewallfsi kernel: [91632.930764] slab_reclaimable:166383 slab_unreclaimable:11556
Jan 25 03:01:39 firewallfsi kernel: [91632.930764] mapped:24783 shmem:1249 pagetables:1579 bounce:0
Jan 25 03:01:39 firewallfsi kernel: [91632.930764] free:61842 free_pcp:286 free_cma:0
Jan 25 03:01:39 firewallfsi kernel: [91632.934074] Node 0 active_anon:743560kB inactive_anon:7284kB active_file:493556kB inactive_file:2621476kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:99132kB dirty:5328kB writeback:0kB shmem:4996kB writeback_tmp:0kB unstable:0kB pages_scanned:0 all_unreclaimable? no
Jan 25 03:01:39 firewallfsi kernel: [91632.935671] DMA free:3164kB min:68kB low:84kB high:100kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15976kB managed:15900kB mlocked:0kB slab_reclaimable:12684kB slab_unreclaimable:52kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jan 25 03:01:39 firewallfsi kernel: lowmem_reserve[]: 0 778 4734 4734
Jan 25 03:01:39 firewallfsi kernel: [91632.937897] Normal free:3452kB min:3532kB low:4412kB high:5292kB active_anon:0kB inactive_anon:0kB active_file:78388kB inactive_file:120kB unevictable:0kB writepending:2164kB present:892920kB managed:817332kB mlocked:0kB slab_reclaimable:652848kB slab_unreclaimable:46172kB kernel_stack:2592kB pagetables:0kB bounce:0kB free_pcp:1020kB local_pcp:228kB free_cma:0kB
Jan 25 03:01:39 firewallfsi kernel: lowmem_reserve[]: 0 0 31652 31652
Jan 25 03:01:39 firewallfsi kernel: [91632.940306] HighMem free:240628kB min:512kB low:5000kB high:9488kB active_anon:743560kB inactive_anon:7284kB active_file:415168kB inactive_file:2621356kB unevictable:0kB writepending:3168kB present:4051548kB managed:4051548kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:6316kB bounce:0kB free_pcp:244kB local_pcp:0kB free_cma:0kB
Jan 25 03:01:39 firewallfsi kernel: lowmem_reserve[]: 0 0 0 0
Jan 25 03:01:39 firewallfsi kernel: [91632.942957] DMA: 41*4kB (E) 23*8kB (UE) 2*16kB (UE) 3*32kB (UE) 6*64kB (UE) 6*128kB (UME) 2*256kB (U) 2*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 3164kB
Jan 25 03:01:39 firewallfsi kernel: Normal: 27*4kB (MH) 152*8kB (UMH) 81*16kB (MH) 14*32kB (H) 4*64kB (H) 1*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3452kB
Jan 25 03:01:39 firewallfsi kernel: HighMem: 621*4kB (UM) 168*8kB (UM) 36*16kB (UM) 12*32kB (UM) 11*64kB (UM) 25*128kB (UM) 6*256kB (UM) 60*512kB (M) 13*1024kB (M) 15*2048kB (M) 38*4096kB (M) = 240628kB
Jan 25 03:01:39 firewallfsi kernel: [91632.947548] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jan 25 03:01:39 firewallfsi kernel: [91632.948339] 780026 total pagecache pages
Jan 25 03:01:39 firewallfsi kernel: [91632.949153] 0 pages in swap cache
Jan 25 03:01:39 firewallfsi kernel: [91632.949929] Swap cache stats: add 0, delete 0, find 0/0
Jan 25 03:01:39 firewallfsi kernel: [91632.950772] Free swap = 33784572kB
Jan 25 03:01:39 firewallfsi kernel: [91632.951590] Total swap = 33784572kB
Jan 25 03:01:39 firewallfsi kernel: [91632.952402] 1240111 pages RAM
Jan 25 03:01:39 firewallfsi kernel: [91632.953205] 1012887 pages HighMem/MovableOnly
Jan 25 03:01:39 firewallfsi kernel: [91632.953994] 18916 pages reserved
Jan 25 03:01:39 firewallfsi kernel: [91632.954816] 0 pages hwpoisoned
Jan 25 03:01:39 firewallfsi kernel: [91632.955624] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
Jan 25 03:01:39 firewallfsi kernel: [91632.956471] [ 596] 0 596 5325 3531 14 3 0 0 systemd-journal
Jan 25 03:01:39 firewallfsi kernel: [91632.957312] [ 631] 0 631 3592 1206 9 3 0 -1000 systemd-udevd
Jan 25 03:01:39 firewallfsi kernel: [91632.958137] [ 735] 0 735 8633 1077 11 3 0 0 rsyslogd
Jan 25 03:01:39 firewallfsi kernel: [91632.958979] [ 736] 0 736 1005 740 5 3 0 0 systemd-logind
Jan 25 03:01:39 firewallfsi kernel: [91632.959824] [ 737] 0 737 1704 653 7 3 0 0 sh
Jan 25 03:01:39 firewallfsi kernel: [91632.960680] [ 738] 0 738 1704 652 8 3 0 0 sh
Jan 25 03:01:39 firewallfsi kernel: [91632.961513] [ 742] 0 742 1472 971 7 3 0 0 smartd
Jan 25 03:01:39 firewallfsi kernel: [91632.962346] [ 743] 81 743 1700 1041 7 3 0 -900 dbus-daemon
Jan 25 03:01:39 firewallfsi kernel: [91632.963173] [ 752] 0 752 2708 1666 9 3 0 0 udp-sgr
Jan 25 03:01:39 firewallfsi kernel: [91632.963984] [ 753] 0 753 3695 1992 11 3 0 0 fetchmail
Jan 25 03:01:39 firewallfsi kernel: [91632.964807] [ 757] 0 757 1704 625 6 3 0 0 sh
Jan 25 03:01:39 firewallfsi kernel: [91632.965622] [ 760] 0 760 1704 704 8 3 0 0 sh
Jan 25 03:01:39 firewallfsi kernel: [91632.966434] [ 761] 0 761 800 496 5 3 0 0 mdadm
Jan 25 03:01:39 firewallfsi kernel: [91632.967234] [ 773] 0 773 1033 646 5 3 0 0 irqbalance
Jan 25 03:01:39 firewallfsi kernel: [91632.968016] [ 775] 0 775 1736 722 6 3 0 0 tickle-pog
Jan 25 03:01:39 firewallfsi kernel: [91632.968813] [ 777] 0 777 3309 2235 10 3 0 0 watch-ip
Jan 25 03:01:39 firewallfsi kernel: [91632.969591] [ 788] 0 788 1704 692 6 3 0 0 sh
Jan 25 03:01:39 firewallfsi kernel: [91632.970349] [ 789] 0 789 3239 2159 10 3 0 0 dynamic-ip-upda
Jan 25 03:01:39 firewallfsi kernel: [91632.971086] [ 790] 0 790 1704 624 7 3 0 0 sh
Jan 25 03:01:39 firewallfsi kernel: [91632.971788] [ 791] 0 791 3165 2109 10 3 0 0 mailwarnings
Jan 25 03:01:39 firewallfsi kernel: [91632.972492] [ 792] 0 792 1704 692 6 3 0 0 sh
Jan 25 03:01:39 firewallfsi kernel: [91632.973171] [ 793] 0 793 3264 2170 10 3 0 0 restarter
Jan 25 03:01:39 firewallfsi kernel: [91632.973833] [ 796] 0 796 1704 649 7 3 0 0 sh
Jan 25 03:01:39 firewallfsi kernel: [91632.974526] [ 797] 0 797 1819 793 7 3 0 0 watch-services
Jan 25 03:01:39 firewallfsi kernel: [91632.975162] [ 799] 288 799 14423 1672 18 3 0 0 milter-greylist
Jan 25 03:01:39 firewallfsi kernel: [91632.975765] [ 802] 0 802 583 403 5 3 0 0 acpid
Jan 25 03:01:39 firewallfsi kernel: [91632.976372] [ 805] 0 805 7287 1286 11 3 0 0 apcupsd
Jan 25 03:01:39 firewallfsi kernel: [91632.976957] [ 822] 0 822 860 539 5 3 0 0 atd
Jan 25 03:01:39 firewallfsi kernel: [91632.977505] [ 928] 0 928 2499 430 8 3 0 0 saslauthd
Jan 25 03:01:39 firewallfsi kernel: [91632.978051] [ 929] 0 929 2499 126 8 3 0 0 saslauthd
Jan 25 03:01:39 firewallfsi kernel: [91632.978562] [ 930] 0 930 2499 126 8 3 0 0 saslauthd
Jan 25 03:01:39 firewallfsi kernel: [91632.979065] [ 931] 0 931 2499 126 8 3 0 0 saslauthd
Jan 25 03:01:39 firewallfsi kernel: [91632.979529] [ 932] 0 932 2499 126 8 3 0 0 saslauthd
Jan 25 03:01:39 firewallfsi kernel: [91632.979985] [ 1034] 0 1034 2769 605 9 3 0 -1000 sshd
Jan 25 03:01:39 firewallfsi kernel: [91632.980408] [ 1063] 27 1063 1707 736 6 3 0 0 mysqld_safe
Jan 25 03:01:39 firewallfsi kernel: [91632.980829] [ 1101] 25 1101 81656 47557 113 3 0 0 named
Jan 25 03:01:39 firewallfsi kernel: [91632.981212] [ 1241] 27 1241 126039 14616 64 3 0 0 mysqld
Jan 25 03:01:39 firewallfsi kernel: [91632.981586] [ 1270] 0 1270 12052 6520 28 3 0 0 /usr/sbin/httpd
Jan 25 03:01:39 firewallfsi kernel: [91632.981968] [ 1362] 0 1362 7049 2424 16 3 0 0 nmbd
Jan 25 03:01:39 firewallfsi kernel: [91632.982334] [ 1363] 0 1363 6711 2157 16 3 0 0 nmbd
Jan 25 03:01:39 firewallfsi kernel: [91632.982710] [ 1386] 0 1386 1116 517 6 3 0 0 agetty
Jan 25 03:01:39 firewallfsi kernel: [91632.983062] [ 1387] 0 1387 1116 546 6 3 0 0 agetty
Jan 25 03:01:39 firewallfsi kernel: [91632.983417] [ 1388] 0 1388 1116 531 5 3 0 0 agetty
Jan 25 03:01:39 firewallfsi kernel: [91632.983784] [ 1389] 0 1389 1116 536 6 3 0 0 agetty
Jan 25 03:01:39 firewallfsi kernel: [91632.984128] [ 1390] 0 1390 1116 512 6 3 0 0 agetty
Jan 25 03:01:39 firewallfsi kernel: [91632.984469] [ 1391] 0 1391 1116 530 6 3 0 0 agetty
Jan 25 03:01:39 firewallfsi kernel: [91632.984824] [ 1599] 48 1599 49254 4236 46 3 0 0 /usr/sbin/httpd
Jan 25 03:01:39 firewallfsi kernel: [91632.985154] [ 1601] 48 1601 16220 4548 29 3 0 0 /usr/sbin/httpd
Jan 25 03:01:39 firewallfsi kernel: [91632.985474] [ 1613] 48 1613 16220 4234 29 3 0 0 /usr/sbin/httpd
Jan 25 03:01:39 firewallfsi kernel: [91632.985805] [ 1622] 48 1622 16220 4381 29 3 0 0 /usr/sbin/httpd
Jan 25 03:01:39 firewallfsi kernel: [91632.986110] [ 1625] 48 1625 16216 4375 29 3 0 0 /usr/sbin/httpd
Jan 25 03:01:39 firewallfsi kernel: [91632.986416] [ 1773] 0 1773 1704 645 7 3 0 0 sh
Jan 25 03:01:39 firewallfsi kernel: [91632.986736] [ 1774] 0 1774 2744 1706 9 3 0 0 udp-sgs
Jan 25 03:01:39 firewallfsi kernel: [91632.987041] [ 1779] 0 1779 9561 3569 22 3 0 0 smbd
Jan 25 03:01:39 firewallfsi kernel: [91632.987349] [ 1780] 0 1780 9197 1105 22 3 0 0 smbd
Jan 25 03:01:39 firewallfsi kernel: [91632.987669] [ 1781] 0 1781 9446 1215 22 3 0 0 smbd
Jan 25 03:01:39 firewallfsi kernel: [91632.987965] [ 1854] 0 1854 5116 2463 13 3 0 0 dhclient
Jan 25 03:01:39 firewallfsi kernel: [91632.988262] [ 1947] 0 1947 594 434 5 3 0 0 pptpd
Jan 25 03:01:39 firewallfsi kernel: [91632.988563] [ 1949] 0 1949 954 608 5 3 0 0 dovecot
Jan 25 03:01:39 firewallfsi kernel: [91632.988881] [ 1950] 97 1950 904 574 6 3 0 0 anvil
Jan 25 03:01:39 firewallfsi kernel: [91632.989180] [ 1951] 0 1951 937 597 5 3 0 0 log
Jan 25 03:01:39 firewallfsi kernel: [91632.989481] [ 1953] 0 1953 1136 787 5 3 0 0 config
Jan 25 03:01:39 firewallfsi kernel: [91632.989797] [ 1960] 0 1960 1899 726 8 3 0 0 crond
Jan 25 03:01:39 firewallfsi kernel: [91632.990092] [ 1961] 177 1961 6832 4736 17 3 0 0 dhcpd
Jan 25 03:01:39 firewallfsi kernel: [91632.990390] [ 1962] 38 1962 1536 1092 6 3 0 0 ntpd
Jan 25 03:01:39 firewallfsi kernel: [91632.990702] [ 1965] 48 1965 16217 4397 29 3 0 0 /usr/sbin/httpd
Jan 25 03:01:39 firewallfsi kernel: [91632.990997] [ 1973] 300 1973 11618 4438 27 3 0 0 smbd
Jan 25 03:01:39 firewallfsi kernel: [91632.991291] [ 2562] 273 2562 2208 1393 8 3 0 0 imap-login
Jan 25 03:01:39 firewallfsi kernel: [91632.991591] [ 2564] 301 2564 1676 1089 7 3 0 0 imap
Jan 25 03:01:39 firewallfsi kernel: [91632.991905] [ 2764] 302 2764 1437 1077 7 3 0 0 imap
Jan 25 03:01:39 firewallfsi kernel: [91632.992197] [ 2962] 304 2962 1399 925 6 3 0 0 imap
Jan 25 03:01:39 firewallfsi kernel: [91632.992491] [ 2965] 303 2965 1444 1080 6 3 0 0 imap
Jan 25 03:01:39 firewallfsi kernel: [91632.992796] [ 2967] 301 2967 1647 1141 7 3 0 0 imap
Jan 25 03:01:39 firewallfsi kernel: [91632.993078] [10890] 48 10890 16386 4665 30 3 0 0 /usr/sbin/httpd
Jan 25 03:01:39 firewallfsi kernel: [91632.993369] [14351] 0 14351 1584 1132 6 3 0 0 systemd
Jan 25 03:01:39 firewallfsi kernel: [91632.993678] [14352] 0 14352 2455 384 8 3 0 0 (sd-pam)
Jan 25 03:01:39 firewallfsi kernel: [91632.993967] [14403] 0 14403 1307 906 6 3 0 0 reboot-when-oom
Jan 25 03:01:39 firewallfsi kernel: [91632.994263] [18123] 276 18123 118311 105011 225 3 0 0 clamd
Jan 25 03:01:39 firewallfsi kernel: [91632.994569] [18137] 290 18137 10838 1193 11 3 0 0 clamav-milter
Jan 25 03:01:39 firewallfsi kernel: [91632.994898] [18160] 0 18160 3829 1728 12 3 0 0 sendmail
Jan 25 03:01:39 firewallfsi kernel: [91632.995213] [18175] 51 18175 3501 767 11 3 0 0 sendmail
Jan 25 03:01:39 firewallfsi kernel: [91632.995530] [18306] 23 18306 5440 725 15 3 0 0 squid
Jan 25 03:01:39 firewallfsi kernel: [91632.995859] [18308] 23 18308 9593 6545 21 3 0 0 squid
Jan 25 03:01:39 firewallfsi kernel: [91632.996164] [18309] 23 18309 1179 404 6 3 0 0 unlinkd
Jan 25 03:01:39 firewallfsi kernel: [91632.996472] [22303] 0 22303 11505 5578 27 3 0 0 smbd
Jan 25 03:01:39 firewallfsi kernel: [91632.996794] [25074] 273 25074 2209 1416 8 3 0 0 imap-login
Jan 25 03:01:39 firewallfsi kernel: [91632.997098] [25075] 301 25075 1401 905 6 3 0 0 imap
Jan 25 03:01:39 firewallfsi kernel: [91632.997406] [15747] 48 15747 16220 4394 29 3 0 0 /usr/sbin/httpd
Jan 25 03:01:39 firewallfsi kernel: [91632.997737] [15750] 48 15750 16216 4253 29 3 0 0 /usr/sbin/httpd
Jan 25 03:01:39 firewallfsi kernel: [91632.998049] [15752] 48 15752 16220 4238 29 3 0 0 /usr/sbin/httpd
Jan 25 03:01:39 firewallfsi kernel: [91632.998366] [27260] 273 27260 2209 1372 9 3 0 0 imap-login
Jan 25 03:01:39 firewallfsi kernel: [91632.998704] [27264] 302 27264 1421 1109 6 3 0 0 imap
Jan 25 03:01:39 firewallfsi kernel: [91632.999026] [ 4826] 273 4826 2208 1449 8 3 0 0 imap-login
Jan 25 03:01:39 firewallfsi kernel: [91632.999357] [ 4830] 301 4830 1396 980 7 3 0 0 imap
Jan 25 03:01:39 firewallfsi kernel: [91632.999702] [25329] 0 25329 4181 1979 11 3 0 0 sshd
Jan 25 03:01:39 firewallfsi kernel: [91633.000029] [25331] 0 25331 4214 1018 11 3 0 0 sshd
Jan 25 03:01:39 firewallfsi kernel: [91633.000362] [25343] 0 25343 1236 1008 6 3 0 0 tcsh
Jan 25 03:01:39 firewallfsi kernel: [91633.000713] [ 3159] 0 3159 1396 169 7 3 0 0 sleep
Jan 25 03:01:39 firewallfsi kernel: [91633.001049] [ 4685] 0 4685 589 165 5 3 0 0 tail
Jan 25 03:01:39 firewallfsi kernel: [91633.001384] [ 5660] 0 5660 1900 510 8 3 0 0 crond
Jan 25 03:01:39 firewallfsi kernel: [91633.001736] [ 5661] 0 5661 3185 2084 10 3 0 0 freshclam-tec-w
Jan 25 03:01:39 firewallfsi kernel: [91633.002074] [ 5770] 0 5770 1396 141 6 3 0 0 sleep
Jan 25 03:01:39 firewallfsi kernel: [91633.002416] [ 5771] 0 5771 9575 2310 22 3 0 0 smbd
Jan 25 03:01:39 firewallfsi kernel: [91633.002776] [ 5782] 0 5782 1654 402 7 3 0 0 anacron
Jan 25 03:01:39 firewallfsi kernel: [91633.003117] Out of memory: Kill process 18123 (clamd) score 10 or sacrifice child
Jan 25 03:01:39 firewallfsi kernel: [91633.003473] Killed process 18123 (clamd) total-vm:473244kB, anon-rss:403624kB, file-rss:16420kB, shmem-rss:0kB
Jan 25 03:01:39 firewallfsi kernel: [91633.100395] audit: type=1131 audit(1485334899.957:2534): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=clamd@scan comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
next prev parent reply other threads:[~2017-01-25 10:23 UTC|newest]
Thread overview: 40+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-01-11 10:32 mm, vmscan: commit makes PAE kernel crash nightly (bisected) Trevor Cordes
2017-01-11 12:11 ` Mel Gorman
2017-01-11 12:14 ` Mel Gorman
2017-01-11 22:52 ` Trevor Cordes
2017-01-12 9:36 ` Michal Hocko
2017-01-15 6:27 ` Trevor Cordes
2017-01-16 11:09 ` Mel Gorman
2017-01-17 13:52 ` Michal Hocko
2017-01-17 14:21 ` Mel Gorman
2017-01-17 14:54 ` Michal Hocko
2017-01-18 7:25 ` Trevor Cordes
2017-01-18 17:48 ` Mel Gorman
2017-01-18 18:07 ` Mel Gorman
2017-01-19 9:48 ` Trevor Cordes
2017-01-19 11:37 ` Michal Hocko
2017-01-20 6:35 ` Trevor Cordes
2017-01-20 11:02 ` Mel Gorman
2017-01-20 15:55 ` Mel Gorman
2017-01-23 0:45 ` Trevor Cordes
2017-01-23 10:48 ` Mel Gorman
2017-01-23 11:04 ` Mel Gorman
2017-01-25 9:46 ` Michal Hocko
2017-01-24 12:59 ` Michal Hocko
2017-01-25 10:02 ` Trevor Cordes [this message]
2017-01-25 12:04 ` Michal Hocko
2017-01-29 22:50 ` Trevor Cordes
2017-01-30 7:51 ` Michal Hocko
2017-02-01 9:29 ` Trevor Cordes
2017-02-01 10:14 ` Michal Hocko
2017-02-04 0:36 ` Trevor Cordes
2017-02-04 20:05 ` Rik van Riel
2017-02-05 10:03 ` Michal Hocko
2017-02-05 22:53 ` Trevor Cordes
2017-01-30 9:10 ` Mel Gorman
2017-01-24 12:54 ` Michal Hocko
2017-01-26 23:18 ` Trevor Cordes
2017-01-27 7:36 ` Michal Hocko
2017-01-24 12:51 ` Michal Hocko
2017-01-18 6:52 ` Trevor Cordes
2017-01-17 13:45 ` Michal Hocko
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20170125040246.5b211135@pog.tecnopolis.ca \
--to=trevor@tecnopolis.ca \
--cc=iamjoonsoo.kim@lge.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mgorman@techsingularity.net \
--cc=mhocko@kernel.org \
--cc=minchan@kernel.org \
--cc=riel@surriel.com \
--cc=srikar@linux.vnet.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).