From: Trevor Cordes <trevor@tecnopolis.ca>
To: Michal Hocko <mhocko@kernel.org>
Cc: Mel Gorman <mgorman@techsingularity.net>,
linux-kernel@vger.kernel.org,
Joonsoo Kim <iamjoonsoo.kim@lge.com>,
Minchan Kim <minchan@kernel.org>, Rik van Riel <riel@surriel.com>,
Srikar Dronamraju <srikar@linux.vnet.ibm.com>
Subject: Re: mm, vmscan: commit makes PAE kernel crash nightly (bisected)
Date: Fri, 20 Jan 2017 00:35:44 -0600 [thread overview]
Message-ID: <20170120003544.7e6e34d1@pog.tecnopolis.ca> (raw)
In-Reply-To: <20170119113757.GP30786@dhcp22.suse.cz>
[-- Attachment #1: Type: text/plain, Size: 5869 bytes --]
On 2017-01-19 Michal Hocko wrote:
> On Thu 19-01-17 03:48:50, Trevor Cordes wrote:
> > On 2017-01-17 Michal Hocko wrote:
> > > On Tue 17-01-17 14:21:14, Mel Gorman wrote:
> > > > On Tue, Jan 17, 2017 at 02:52:28PM +0100, Michal Hocko
> > > > wrote:
> > > > > On Mon 16-01-17 11:09:34, Mel Gorman wrote:
> > > > > [...]
> > > > > > diff --git a/mm/vmscan.c b/mm/vmscan.c
> > > > > > index 532a2a750952..46aac487b89a 100644
> > > > > > --- a/mm/vmscan.c
> > > > > > +++ b/mm/vmscan.c
> > > > > > @@ -2684,6 +2684,7 @@ static void shrink_zones(struct
> > > > > > zonelist *zonelist, struct scan_control *sc) continue;
> > > > > >
> > > > > > if (sc->priority != DEF_PRIORITY &&
> > > > > > + !buffer_heads_over_limit &&
> > > > > > !pgdat_reclaimable(zone->zone_pgdat))
> > > > > > continue; /* Let
> > > > > > kswapd poll it */
> > > > >
> > > > > I think we should rather remove pgdat_reclaimable here. This
> > > > > sounds like a wrong layer to decide whether we want to reclaim
> > > > > and how much.
> > > >
> > > > I had considered that but it'd also be important to add the
> > > > other 32-bit patches you have posted to see the impact. Because
> > > > of the ratio of LRU pages to slab pages, it may not have an
> > > > impact but it'd need to be eliminated.
> > >
> > > OK, Trevor you can pull from
> > > git://git.kernel.org/pub/scm/linux/kernel/git/mhocko/mm.git tree
> > > fixes/highmem-node-fixes branch. This contains the current mmotm
> > > tree
> > > + the latest highmem fixes. I also do not expect this would help
> > > much in your case but as Mel've said we should rule that out at
> > > least.
> >
> > Hi! The git tree above version oom'd after < 24 hours (3:02am) so
> > it doesn't solve the bug. If you need a oom messages dump let me
> > know.
>
> Yes please.
The first oom from that night attached. Note, the oom wasn't as dire
with your mhocko/4.9.0+ as it usually is with stock 4.8.x: my oom
detector and reboot script was able to do its thing cleanly before the
system became unusable.
I'll await further instructions and test right away. Maybe I'll try a
few tuning ideas until then. Thanks!
> > Let me know what to try next, guys, and I'll test it out.
> >
> > > > Before prototyping such a thing, I'd like to hear the outcome of
> > > > this heavy hack and then add your 32-bit patches onto the list.
> > > > If the problem is still there then I'd next look at taking slab
> > > > pages into account in pgdat_reclaimable() instead of an
> > > > outright removal that has a much wider impact. If that doesn't
> > > > work then I'll prototype a heavy-handed forced slab reclaim
> > > > when lower zones are almost all slab pages.
> >
> > I don't think I've tried the "heavy hack" patch yet? It's not in
> > the mhocko tree I just tried? Should I try the heavy hack on top
> > of mhocko git or on vanilla or what?
> >
> > I also want to mention that these PAE boxes suffer from another
> > problem/bug that I've worked around for almost a year now. For some
> > reason it keeps gnawing at me that it might be related. The disk
> > I/O goes to pot on this/these PAE boxes after a certain amount of
> > disk writes (like some unknown number of GB, around 10-ish maybe).
> > Like writes go from 500MB/s to 10MB/s!! Reboot and it's magically
> > 500MB/s again. I detail this here:
> > https://muug.ca/pipermail/roundtable/2016-June/004669.html
> > My fix was to mem=XG where X is <8 (like 4 or 6) to force the PAE
> > kernel to be more sane about highmem choices. I never filed a bug
> > because I read a ton of stuff saying Linus hates PAE, don't use over
> > 4G, blah blah. But the other fix is to:
> > set /proc/sys/vm/highmem_is_dirtyable to 1
>
> Yes this sounds like a dirty memory throttling and there were some
> changes in that area. I do not remember when exactly.
I think my PAE-slow-IO bug started way back in Fedora 22 (4.0?), hard
to know exactly when as I didn't discover the bug for maybe a year as I
didn't realize IO was the problem right away. Too late to bisect that
one, I guess. I guess it's not related so we can ignore my tangent!
> > I'm not bringing this up to get attention to a new bug, I bring
> > this up because it smells like it might be related. If something
> > slowly eats away at the box's vm to the point that I/O gets
> > horribly slow, perhaps it's related to the slab and high/lomem
> > issue we have here? And if related, it may help to solve the oom
> > bug. If I'm way off base here, just ignore my tangent!
>
> >From your OOM reports so far it doesn't really seem related because
> >you
> never had large number of pages under the writeback when OOM.
>
> The situation with the PAE kernel is unfortunate but it is really hard
> to do anything about that considering that the kernel and most its
> allocations have to live in a small and scarce lowmem memory. Moreover
> the more memory you have to more you have to allocated from that
> memory.
You're for sure right that the IO-slow bug was definitely worse the more
ram was in a system! (The mem=4G really helps alleviate this bug and is
good enough for me.)
> This is why not only Linus hates 32b systems on a large memory
> systems.
Completely off-topic: it would be great if rather than pretending PAE
should work with large RAM (which seems more broken every day), the
kernel guys put out an officially stated policy of a maximum RAM you
can use, and try to have the kernel behave for <= that size, and then
people could use more RAM but clearly "at your own risk, don't bug us
about problems!". Other than a few posts about Linus hating it,
there's nothing official I can find about it in documentation, etc. It
gives the (mis)impression that it's perfectly fine to run PAE on a
zillion GB modern system. Then we later learn the hard way :-)
[-- Attachment #2: oom3 --]
[-- Type: application/octet-stream, Size: 22921 bytes --]
Jan 19 03:02:19 firewallfsi kernel: [85602.797346] smbd invoked oom-killer: gfp_mask=0x26000d0(GFP_TEMPORARY|__GFP_NOTRACK), nodemask=0, order=0, oom_score_adj=0
Jan 19 03:02:19 firewallfsi kernel: [85602.798595] smbd cpuset=/ mems_allowed=0
Jan 19 03:02:19 firewallfsi kernel: [85602.799868] CPU: 0 PID: 5892 Comm: smbd Not tainted 4.9.0+ #1
Jan 19 03:02:19 firewallfsi kernel: [85602.801084] Hardware name: Intel Corporation S1200BTL/S1200BTL, BIOS S1200BT.86B.02.00.0035.030220120927 03/02/2012
Jan 19 03:02:19 firewallfsi kernel: [85602.802306] df18da9c d67603e7 df18dbd4 ee3e2400 df18dacc d65e16a6 df18daac d6b63aad
Jan 19 03:02:19 firewallfsi kernel: [85602.803529] df18dacc d676615f df18dad0 f6a6d600 e1172400 ee3e2400 d6d68bde df18dbd4
Jan 19 03:02:19 firewallfsi kernel: [85602.804736] df18db10 d657aff7 d6476d8a df18dafc d657ac6b 00000006 00000000 0000000b
Jan 19 03:02:19 firewallfsi kernel: [85602.805933] Call Trace:
Jan 19 03:02:19 firewallfsi kernel: [85602.807087] [<d67603e7>] dump_stack+0x58/0x81
Jan 19 03:02:19 firewallfsi kernel: [85602.808227] [<d65e16a6>] dump_header+0x64/0x1a6
Jan 19 03:02:19 firewallfsi kernel: [85602.809347] [<d6b63aad>] ? _raw_spin_unlock_irqrestore+0xd/0x10
Jan 19 03:02:19 firewallfsi kernel: [85602.810454] [<d676615f>] ? ___ratelimit+0x9f/0x100
Jan 19 03:02:19 firewallfsi kernel: [85602.811544] [<d657aff7>] oom_kill_process+0x207/0x3d0
Jan 19 03:02:19 firewallfsi kernel: [85602.812617] [<d6476d8a>] ? has_capability_noaudit+0x1a/0x30
Jan 19 03:02:19 firewallfsi kernel: [85602.813674] [<d657ac6b>] ? oom_badness.part.13+0xcb/0x140
Jan 19 03:02:19 firewallfsi kernel: [85602.814715] [<d657b4d8>] out_of_memory+0xf8/0x2a0
Jan 19 03:02:19 firewallfsi kernel: [85602.815737] [<d65800ca>] __alloc_pages_nodemask+0xcfa/0xd10
Jan 19 03:02:19 firewallfsi kernel: [85602.816746] [<d65cc928>] new_slab+0x3c8/0x4b0
Jan 19 03:02:19 firewallfsi kernel: [85602.817736] [<d65cddca>] ___slab_alloc.constprop.75+0x42a/0x670
Jan 19 03:02:19 firewallfsi kernel: [85602.818715] [<d65fd583>] ? __d_alloc+0x23/0x190
Jan 19 03:02:19 firewallfsi kernel: [85602.819675] [<d6668ce4>] ? __ext4_get_inode_loc+0x104/0x440
Jan 19 03:02:19 firewallfsi kernel: [85602.820620] [<d65ce03f>] __slab_alloc.constprop.74+0x2f/0x50
Jan 19 03:02:19 firewallfsi kernel: [85602.821550] [<d65cf0ca>] kmem_cache_alloc+0x17a/0x1c0
Jan 19 03:02:19 firewallfsi kernel: [85602.822462] [<d65fd583>] ? __d_alloc+0x23/0x190
Jan 19 03:02:19 firewallfsi kernel: [85602.823355] [<d65fd583>] __d_alloc+0x23/0x190
Jan 19 03:02:19 firewallfsi kernel: [85602.824228] [<d65fd704>] d_alloc+0x14/0x50
Jan 19 03:02:19 firewallfsi kernel: [85602.825082] [<d65fdbd7>] d_alloc_parallel+0x47/0x450
Jan 19 03:02:19 firewallfsi kernel: [85602.825919] [<d65fcecd>] ? d_splice_alias+0x1fd/0x370
Jan 19 03:02:19 firewallfsi kernel: [85602.826739] [<d6677971>] ? ext4_lookup+0x161/0x240
Jan 19 03:02:19 firewallfsi kernel: [85602.827540] [<d676e80b>] ? lockref_get_not_dead+0xb/0x30
Jan 19 03:02:19 firewallfsi kernel: [85602.828325] [<d65ef289>] ? unlazy_walk+0xf9/0x1a0
Jan 19 03:02:19 firewallfsi kernel: [85602.829093] [<d65f00ae>] lookup_slow+0x5e/0x130
Jan 19 03:02:19 firewallfsi kernel: [85602.829842] [<d65f0ca4>] walk_component+0x1e4/0x300
Jan 19 03:02:19 firewallfsi kernel: [85602.830574] [<d65efc8d>] ? path_init+0x14d/0x330
Jan 19 03:02:19 firewallfsi kernel: [85602.831288] [<d65f1e83>] path_lookupat+0x53/0xe0
Jan 19 03:02:19 firewallfsi kernel: [85602.831984] [<d65f4027>] filename_lookup+0x97/0x190
Jan 19 03:02:19 firewallfsi kernel: [85602.832661] [<d65cf047>] ? kmem_cache_alloc+0xf7/0x1c0
Jan 19 03:02:19 firewallfsi kernel: [85602.833322] [<d65f3c6a>] ? getname_flags+0x3a/0x1a0
Jan 19 03:02:19 firewallfsi kernel: [85602.833966] [<d65f3c81>] ? getname_flags+0x51/0x1a0
Jan 19 03:02:19 firewallfsi kernel: [85602.834587] [<d65f41f6>] user_path_at_empty+0x36/0x40
Jan 19 03:02:19 firewallfsi kernel: [85602.835190] [<d65e9910>] vfs_fstatat+0x60/0xb0
Jan 19 03:02:19 firewallfsi kernel: [85602.835774] [<d65ea3aa>] SyS_fstatat64+0x2a/0x50
Jan 19 03:02:19 firewallfsi kernel: [85602.836341] [<d642b168>] ? sched_clock+0x8/0x10
Jan 19 03:02:19 firewallfsi kernel: [85602.836888] [<d649a075>] ? sched_clock_cpu+0x125/0x140
Jan 19 03:02:19 firewallfsi kernel: [85602.837415] [<d64de305>] ? hrtimer_interrupt+0xa5/0x180
Jan 19 03:02:19 firewallfsi kernel: [85602.837926] [<d640377a>] do_fast_syscall_32+0x8a/0x150
Jan 19 03:02:19 firewallfsi kernel: [85602.838420] [<d6b63fca>] sysenter_past_esp+0x47/0x75
Jan 19 03:02:19 firewallfsi kernel: [85602.838902] Mem-Info:
Jan 19 03:02:19 firewallfsi kernel: [85602.840191] active_anon:155033 inactive_anon:32960 isolated_anon:0
Jan 19 03:02:19 firewallfsi kernel: [85602.840191] active_file:171215 inactive_file:640942 isolated_file:0
Jan 19 03:02:19 firewallfsi kernel: [85602.840191] unevictable:0 dirty:1277 writeback:0 unstable:0
Jan 19 03:02:19 firewallfsi kernel: [85602.840191] slab_reclaimable:132504 slab_unreclaimable:11682
Jan 19 03:02:19 firewallfsi kernel: [85602.840191] mapped:22669 shmem:1221 pagetables:1537 bounce:0
Jan 19 03:02:19 firewallfsi kernel: [85602.840191] free:57986 free_pcp:1120 free_cma:0
Jan 19 03:02:19 firewallfsi kernel: [85602.848275] Node 0 active_anon:620132kB inactive_anon:131840kB active_file:684860kB inactive_file:2563768kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:90712kB dirty:5128kB writeback:0kB shmem:4884kB writeback_tmp:0kB unstable:0kB pages_scanned:0 all_unreclaimable? no
Jan 19 03:02:19 firewallfsi kernel: [85602.852375] DMA free:3176kB min:68kB low:84kB high:100kB active_anon:0kB inactive_anon:0kB active_file:4996kB inactive_file:0kB unevictable:0kB writepending:8kB present:15976kB managed:15900kB mlocked:0kB slab_reclaimable:7724kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jan 19 03:02:19 firewallfsi kernel: lowmem_reserve[]: 0 777 4734 4734
Jan 19 03:02:19 firewallfsi kernel: [85602.858232] Normal free:3436kB min:3532kB low:4412kB high:5292kB active_anon:4kB inactive_anon:8kB active_file:193340kB inactive_file:120kB unevictable:0kB writepending:2516kB present:892920kB managed:816932kB mlocked:0kB slab_reclaimable:522292kB slab_unreclaimable:46724kB kernel_stack:2560kB pagetables:0kB bounce:0kB free_pcp:3468kB local_pcp:176kB free_cma:0kB
Jan 19 03:02:19 firewallfsi kernel: lowmem_reserve[]: 0 0 31652 31652
Jan 19 03:02:19 firewallfsi kernel: [85602.864610] HighMem free:225332kB min:512kB low:5004kB high:9496kB active_anon:620128kB inactive_anon:131832kB active_file:486476kB inactive_file:2563648kB unevictable:0kB writepending:2616kB present:4051548kB managed:4051548kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:6148kB bounce:0kB free_pcp:1012kB local_pcp:180kB free_cma:0kB
Jan 19 03:02:19 firewallfsi kernel: lowmem_reserve[]: 0 0 0 0
Jan 19 03:02:19 firewallfsi kernel: [85602.871108] DMA: 14*4kB (UME) 4*8kB (UE) 1*16kB (M) 2*32kB (UE) 1*64kB (U) 1*128kB (U) 1*256kB (U) 3*512kB (UME) 1*1024kB (U) 0*2048kB 0*4096kB = 3176kB
Jan 19 03:02:19 firewallfsi kernel: Normal: 165*4kB (UME) 61*8kB (UME) 19*16kB (UM) 18*32kB (UME) 2*64kB (U) 4*128kB (UE) 3*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3436kB
Jan 19 03:02:19 firewallfsi kernel: HighMem: 243*4kB (M) 167*8kB (UM) 55*16kB (UM) 28*32kB (UM) 15*64kB (UM) 13*128kB (UM) 6*256kB (UM) 0*512kB 2*1024kB (M) 39*2048kB (M) 33*4096kB (M) = 225332kB
Jan 19 03:02:19 firewallfsi kernel: [85602.877129] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jan 19 03:02:19 firewallfsi kernel: [85602.878150] 813455 total pagecache pages
Jan 19 03:02:19 firewallfsi kernel: [85602.879168] 34 pages in swap cache
Jan 19 03:02:19 firewallfsi kernel: [85602.880181] Swap cache stats: add 71, delete 37, find 191/200
Jan 19 03:02:19 firewallfsi kernel: [85602.881205] Free swap = 33784424kB
Jan 19 03:02:19 firewallfsi kernel: [85602.882242] Total swap = 33784572kB
Jan 19 03:02:19 firewallfsi kernel: [85602.883275] 1240111 pages RAM
Jan 19 03:02:19 firewallfsi kernel: [85602.884302] 1012887 pages HighMem/MovableOnly
Jan 19 03:02:19 firewallfsi kernel: [85602.885317] 19016 pages reserved
Jan 19 03:02:19 firewallfsi kernel: [85602.886310] 0 pages hwpoisoned
Jan 19 03:02:19 firewallfsi kernel: [85602.887299] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
Jan 19 03:02:19 firewallfsi kernel: [85602.888342] [ 596] 0 596 3278 1039 9 3 0 0 systemd-journal
Jan 19 03:02:19 firewallfsi kernel: [85602.889382] [ 632] 0 632 3592 1182 9 3 0 -1000 systemd-udevd
Jan 19 03:02:19 firewallfsi kernel: [85602.890420] [ 742] 81 742 1700 1069 7 3 0 -900 dbus-daemon
Jan 19 03:02:19 firewallfsi kernel: [85602.891461] [ 743] 0 743 1704 663 6 3 0 0 sh
Jan 19 03:02:19 firewallfsi kernel: [85602.892502] [ 745] 0 745 1704 646 6 3 0 0 sh
Jan 19 03:02:19 firewallfsi kernel: [85602.893540] [ 747] 0 747 1033 641 6 3 0 0 irqbalance
Jan 19 03:02:19 firewallfsi kernel: [85602.894574] [ 751] 0 751 808 487 5 3 0 0 mdadm
Jan 19 03:02:19 firewallfsi kernel: [85602.895606] [ 752] 288 752 19290 1785 20 3 0 0 milter-greylist
Jan 19 03:02:19 firewallfsi kernel: [85602.896595] [ 753] 0 753 988 716 5 3 0 0 systemd-logind
Jan 19 03:02:19 firewallfsi kernel: [85602.897577] [ 755] 0 755 1472 953 6 3 0 0 smartd
Jan 19 03:02:19 firewallfsi kernel: [85602.898550] [ 757] 0 757 1704 676 7 3 0 0 sh
Jan 19 03:02:19 firewallfsi kernel: [85602.899514] [ 764] 0 764 8633 1110 12 3 0 0 rsyslogd
Jan 19 03:02:19 firewallfsi kernel: [85602.900474] [ 765] 0 765 1704 673 7 3 0 0 sh
Jan 19 03:02:19 firewallfsi kernel: [85602.901429] [ 768] 0 768 1704 629 6 3 0 0 sh
Jan 19 03:02:19 firewallfsi kernel: [85602.902375] [ 770] 0 770 1704 681 7 3 0 0 sh
Jan 19 03:02:19 firewallfsi kernel: [85602.903312] [ 771] 0 771 1704 644 6 3 0 0 sh
Jan 19 03:02:19 firewallfsi kernel: [85602.904225] [ 774] 0 774 1704 680 6 3 0 0 sh
Jan 19 03:02:19 firewallfsi kernel: [85602.905108] [ 799] 0 799 583 436 5 3 0 0 acpid
Jan 19 03:02:19 firewallfsi kernel: [85602.905967] [ 803] 0 803 7287 1234 11 3 0 0 apcupsd
Jan 19 03:02:19 firewallfsi kernel: [85602.906804] [ 810] 0 810 860 487 5 3 0 0 atd
Jan 19 03:02:19 firewallfsi kernel: [85602.907616] [ 868] 0 868 1819 814 8 3 0 0 watch-services
Jan 19 03:02:19 firewallfsi kernel: [85602.908388] [ 869] 0 869 3165 2070 9 3 0 0 mailwarnings
Jan 19 03:02:19 firewallfsi kernel: [85602.909143] [ 870] 0 870 3238 2164 9 3 0 0 dynamic-ip-upda
Jan 19 03:02:19 firewallfsi kernel: [85602.909879] [ 871] 0 871 1736 719 6 3 0 0 tickle-pog
Jan 19 03:02:19 firewallfsi kernel: [85602.910589] [ 872] 0 872 3264 2201 10 3 0 0 restarter
Jan 19 03:02:19 firewallfsi kernel: [85602.911282] [ 875] 0 875 3695 1986 11 3 0 0 fetchmail
Jan 19 03:02:19 firewallfsi kernel: [85602.911955] [ 876] 0 876 2707 1622 9 3 0 0 udp-sgr
Jan 19 03:02:19 firewallfsi kernel: [85602.912607] [ 877] 0 877 2499 414 9 3 0 0 saslauthd
Jan 19 03:02:19 firewallfsi kernel: [85602.913175] [ 878] 0 878 2518 900 9 3 0 0 saslauthd
Jan 19 03:02:19 firewallfsi kernel: [85602.913709] [ 879] 0 879 2499 126 9 3 0 0 saslauthd
Jan 19 03:02:19 firewallfsi kernel: [85602.914216] [ 880] 0 880 2499 126 9 3 0 0 saslauthd
Jan 19 03:02:19 firewallfsi kernel: [85602.914706] [ 881] 0 881 2499 126 9 3 0 0 saslauthd
Jan 19 03:02:19 firewallfsi kernel: [85602.915166] [ 882] 0 882 3310 2223 10 3 0 0 watch-ip
Jan 19 03:02:19 firewallfsi kernel: [85602.915609] [ 938] 0 938 2769 621 9 3 0 -1000 sshd
Jan 19 03:02:19 firewallfsi kernel: [85602.916028] [ 1032] 27 1032 1707 733 7 3 0 0 mysqld_safe
Jan 19 03:02:19 firewallfsi kernel: [85602.916432] [ 1130] 25 1130 82172 47875 114 3 0 0 named
Jan 19 03:02:19 firewallfsi kernel: [85602.916822] [ 1208] 27 1208 126039 14693 65 3 0 0 mysqld
Jan 19 03:02:19 firewallfsi kernel: [85602.917197] [ 1245] 0 1245 7049 2401 16 3 0 0 nmbd
Jan 19 03:02:19 firewallfsi kernel: [85602.917559] [ 1246] 0 1246 6840 2267 16 3 0 0 nmbd
Jan 19 03:02:19 firewallfsi kernel: [85602.917923] [ 1308] 0 1308 12052 6449 26 3 0 0 /usr/sbin/httpd
Jan 19 03:02:19 firewallfsi kernel: [85602.918282] [ 1335] 0 1335 1116 497 6 3 0 0 agetty
Jan 19 03:02:19 firewallfsi kernel: [85602.918640] [ 1336] 0 1336 1116 479 6 3 0 0 agetty
Jan 19 03:02:19 firewallfsi kernel: [85602.918992] [ 1337] 0 1337 1116 521 6 3 0 0 agetty
Jan 19 03:02:19 firewallfsi kernel: [85602.919340] [ 1338] 0 1338 1116 505 6 3 0 0 agetty
Jan 19 03:02:19 firewallfsi kernel: [85602.919688] [ 1339] 0 1339 1116 462 6 3 0 0 agetty
Jan 19 03:02:19 firewallfsi kernel: [85602.920027] [ 1340] 0 1340 1116 473 6 3 0 0 agetty
Jan 19 03:02:19 firewallfsi kernel: [85602.920363] [ 1596] 48 1596 49256 4353 44 3 0 0 /usr/sbin/httpd
Jan 19 03:02:19 firewallfsi kernel: [85602.920697] [ 1597] 48 1597 16386 4635 29 3 0 0 /usr/sbin/httpd
Jan 19 03:02:19 firewallfsi kernel: [85602.921014] [ 1600] 48 1600 16295 5404 29 3 0 0 /usr/sbin/httpd
Jan 19 03:02:19 firewallfsi kernel: [85602.921328] [ 1619] 48 1619 16298 5545 29 3 0 0 /usr/sbin/httpd
Jan 19 03:02:19 firewallfsi kernel: [85602.921640] [ 1622] 48 1622 16220 4205 27 3 0 0 /usr/sbin/httpd
Jan 19 03:02:19 firewallfsi kernel: [85602.921944] [ 1778] 0 1778 1704 682 6 3 0 0 sh
Jan 19 03:02:19 firewallfsi kernel: [85602.922253] [ 1779] 0 1779 2743 1705 9 3 0 0 udp-sgs
Jan 19 03:02:19 firewallfsi kernel: [85602.922562] [ 1781] 0 1781 9561 3567 23 3 0 0 smbd
Jan 19 03:02:19 firewallfsi kernel: [85602.922875] [ 1782] 0 1782 9197 1092 21 3 0 0 smbd
Jan 19 03:02:19 firewallfsi kernel: [85602.923198] [ 1783] 0 1783 9446 1156 21 3 0 0 smbd
Jan 19 03:02:19 firewallfsi kernel: [85602.923498] [ 1870] 0 1870 5116 2471 14 3 0 0 dhclient
Jan 19 03:02:19 firewallfsi kernel: [85602.923793] [ 1948] 0 1948 594 400 5 3 0 0 pptpd
Jan 19 03:02:19 firewallfsi kernel: [85602.924104] [ 1950] 0 1950 954 622 6 3 0 0 dovecot
Jan 19 03:02:19 firewallfsi kernel: [85602.924395] [ 1951] 97 1951 904 582 5 3 0 0 anvil
Jan 19 03:02:19 firewallfsi kernel: [85602.924689] [ 1952] 0 1952 937 613 6 3 0 0 log
Jan 19 03:02:19 firewallfsi kernel: [85602.925000] [ 1954] 0 1954 1133 784 6 3 0 0 config
Jan 19 03:02:19 firewallfsi kernel: [85602.925294] [ 1956] 0 1956 4181 1973 12 3 0 0 sshd
Jan 19 03:02:19 firewallfsi kernel: [85602.925590] [ 1965] 0 1965 1584 1147 6 3 0 0 systemd
Jan 19 03:02:19 firewallfsi kernel: [85602.925904] [ 1971] 0 1971 2455 381 8 3 0 0 (sd-pam)
Jan 19 03:02:19 firewallfsi kernel: [85602.926216] [ 2001] 0 2001 4214 1046 12 3 0 0 sshd
Jan 19 03:02:19 firewallfsi kernel: [85602.926517] [ 2011] 0 2011 1187 957 5 3 0 0 tcsh
Jan 19 03:02:19 firewallfsi kernel: [85602.926821] [ 2035] 48 2035 16220 4208 28 3 0 0 /usr/sbin/httpd
Jan 19 03:02:19 firewallfsi kernel: [85602.927144] [ 2040] 48 2040 16298 5544 29 3 0 0 /usr/sbin/httpd
Jan 19 03:02:19 firewallfsi kernel: [85602.927445] [ 2042] 48 2042 16216 4586 28 3 0 0 /usr/sbin/httpd
Jan 19 03:02:19 firewallfsi kernel: [85602.927746] [ 2055] 177 2055 6833 4786 16 3 0 0 dhcpd
Jan 19 03:02:19 firewallfsi kernel: [85602.928062] [ 2062] 0 2062 1899 698 7 3 29 0 crond
Jan 19 03:02:19 firewallfsi kernel: [85602.928358] [ 2070] 38 2070 1536 1082 6 3 0 0 ntpd
Jan 19 03:02:19 firewallfsi kernel: [85602.928659] [ 2297] 273 2297 2208 1390 8 3 0 0 imap-login
Jan 19 03:02:19 firewallfsi kernel: [85602.928977] [ 2299] 301 2299 1762 1248 7 3 0 0 imap
Jan 19 03:02:19 firewallfsi kernel: [85602.929276] [ 5376] 0 5376 1307 927 6 3 0 0 reboot-when-oom
Jan 19 03:02:19 firewallfsi kernel: [85602.929583] [10505] 276 10505 115120 103978 222 3 0 0 clamd
Jan 19 03:02:19 firewallfsi kernel: [85602.929909] [10519] 290 10519 15704 1191 14 3 0 0 clamav-milter
Jan 19 03:02:19 firewallfsi kernel: [85602.930239] [10539] 0 10539 3829 1778 10 3 0 0 sendmail
Jan 19 03:02:19 firewallfsi kernel: [85602.930565] [10552] 51 10552 3500 765 11 3 0 0 sendmail
Jan 19 03:02:19 firewallfsi kernel: [85602.930902] [10658] 23 10658 5440 724 14 3 0 0 squid
Jan 19 03:02:19 firewallfsi kernel: [85602.931234] [10660] 23 10660 9797 6723 22 3 0 0 squid
Jan 19 03:02:19 firewallfsi kernel: [85602.931551] [10661] 23 10661 1179 419 6 3 0 0 unlinkd
Jan 19 03:02:19 firewallfsi kernel: [85602.931879] [13921] 273 13921 2208 1470 8 3 0 0 imap-login
Jan 19 03:02:19 firewallfsi kernel: [85602.932206] [13924] 301 13924 1400 973 6 3 0 0 imap
Jan 19 03:02:19 firewallfsi kernel: [85602.932519] [13925] 273 13925 2209 1364 9 3 0 0 imap-login
Jan 19 03:02:19 firewallfsi kernel: [85602.932840] [13926] 301 13926 2559 1273 7 3 0 0 imap
Jan 19 03:02:19 firewallfsi kernel: [85602.933173] [15323] 48 15323 16220 4208 27 3 0 0 /usr/sbin/httpd
Jan 19 03:02:19 firewallfsi kernel: [85602.933491] [14826] 273 14826 2208 1448 7 3 0 0 imap-login
Jan 19 03:02:19 firewallfsi kernel: [85602.933836] [14828] 301 14828 3995 1513 8 3 0 0 imap
Jan 19 03:02:19 firewallfsi kernel: [85602.934181] [18753] 48 18753 16220 4338 28 3 0 0 /usr/sbin/httpd
Jan 19 03:02:19 firewallfsi kernel: [85602.934516] [ 5254] 0 5254 9685 3801 23 3 0 0 smbd
Jan 19 03:02:19 firewallfsi kernel: [85602.934879] [ 5892] 300 5892 11287 5069 26 3 0 0 smbd
Jan 19 03:02:19 firewallfsi kernel: [85602.935241] [31892] 273 31892 2209 1455 8 3 0 0 imap-login
Jan 19 03:02:19 firewallfsi kernel: [85602.935583] [31896] 302 31896 1415 1067 7 3 0 0 imap
Jan 19 03:02:19 firewallfsi kernel: [85602.935941] [28129] 0 28129 1396 138 6 3 0 0 sleep
Jan 19 03:02:19 firewallfsi kernel: [85602.936298] [30312] 273 30312 2209 1436 8 3 0 0 imap-login
Jan 19 03:02:19 firewallfsi kernel: [85602.936656] [30315] 301 30315 1432 989 6 3 0 0 imap
Jan 19 03:02:19 firewallfsi kernel: [85602.937030] [32695] 0 32695 1900 494 7 3 17 0 crond
Jan 19 03:02:19 firewallfsi kernel: [85602.937383] [32696] 0 32696 3184 2117 10 3 0 0 freshclam-tec-w
Jan 19 03:02:19 firewallfsi kernel: [85602.937754] [32726] 0 32726 1654 435 7 3 0 0 anacron
Jan 19 03:02:19 firewallfsi kernel: [85602.938122] [ 548] 0 548 1396 139 6 3 0 0 sleep
Jan 19 03:02:19 firewallfsi kernel: [85602.938481] Out of memory: Kill process 10505 (clamd) score 10 or sacrifice child
Jan 19 03:02:19 firewallfsi kernel: [85602.938867] Killed process 10505 (clamd) total-vm:460480kB, anon-rss:399264kB, file-rss:16648kB, shmem-rss:0kB
Jan 19 03:02:19 firewallfsi kernel: [85602.970282] audit: type=1131 audit(1484816539.419:2108): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=clamd@scan comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
next prev parent reply other threads:[~2017-01-20 6:37 UTC|newest]
Thread overview: 40+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-01-11 10:32 mm, vmscan: commit makes PAE kernel crash nightly (bisected) Trevor Cordes
2017-01-11 12:11 ` Mel Gorman
2017-01-11 12:14 ` Mel Gorman
2017-01-11 22:52 ` Trevor Cordes
2017-01-12 9:36 ` Michal Hocko
2017-01-15 6:27 ` Trevor Cordes
2017-01-16 11:09 ` Mel Gorman
2017-01-17 13:52 ` Michal Hocko
2017-01-17 14:21 ` Mel Gorman
2017-01-17 14:54 ` Michal Hocko
2017-01-18 7:25 ` Trevor Cordes
2017-01-18 17:48 ` Mel Gorman
2017-01-18 18:07 ` Mel Gorman
2017-01-19 9:48 ` Trevor Cordes
2017-01-19 11:37 ` Michal Hocko
2017-01-20 6:35 ` Trevor Cordes [this message]
2017-01-20 11:02 ` Mel Gorman
2017-01-20 15:55 ` Mel Gorman
2017-01-23 0:45 ` Trevor Cordes
2017-01-23 10:48 ` Mel Gorman
2017-01-23 11:04 ` Mel Gorman
2017-01-25 9:46 ` Michal Hocko
2017-01-24 12:59 ` Michal Hocko
2017-01-25 10:02 ` Trevor Cordes
2017-01-25 12:04 ` Michal Hocko
2017-01-29 22:50 ` Trevor Cordes
2017-01-30 7:51 ` Michal Hocko
2017-02-01 9:29 ` Trevor Cordes
2017-02-01 10:14 ` Michal Hocko
2017-02-04 0:36 ` Trevor Cordes
2017-02-04 20:05 ` Rik van Riel
2017-02-05 10:03 ` Michal Hocko
2017-02-05 22:53 ` Trevor Cordes
2017-01-30 9:10 ` Mel Gorman
2017-01-24 12:54 ` Michal Hocko
2017-01-26 23:18 ` Trevor Cordes
2017-01-27 7:36 ` Michal Hocko
2017-01-24 12:51 ` Michal Hocko
2017-01-18 6:52 ` Trevor Cordes
2017-01-17 13:45 ` Michal Hocko
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20170120003544.7e6e34d1@pog.tecnopolis.ca \
--to=trevor@tecnopolis.ca \
--cc=iamjoonsoo.kim@lge.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mgorman@techsingularity.net \
--cc=mhocko@kernel.org \
--cc=minchan@kernel.org \
--cc=riel@surriel.com \
--cc=srikar@linux.vnet.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.