From: Trevor Cordes <trevor@tecnopolis.ca>
To: Michal Hocko <mhocko@kernel.org>
Cc: Mel Gorman <mgorman@techsingularity.net>,
linux-kernel@vger.kernel.org,
Joonsoo Kim <iamjoonsoo.kim@lge.com>,
Minchan Kim <minchan@kernel.org>, Rik van Riel <riel@surriel.com>,
Srikar Dronamraju <srikar@linux.vnet.ibm.com>
Subject: Re: mm, vmscan: commit makes PAE kernel crash nightly (bisected)
Date: Fri, 20 Jan 2017 00:35:44 -0600 [thread overview]
Message-ID: <20170120003544.7e6e34d1@pog.tecnopolis.ca> (raw)
In-Reply-To: <20170119113757.GP30786@dhcp22.suse.cz>
[-- Attachment #1: Type: text/plain, Size: 5869 bytes --]
On 2017-01-19 Michal Hocko wrote:
> On Thu 19-01-17 03:48:50, Trevor Cordes wrote:
> > On 2017-01-17 Michal Hocko wrote:
> > > On Tue 17-01-17 14:21:14, Mel Gorman wrote:
> > > > On Tue, Jan 17, 2017 at 02:52:28PM +0100, Michal Hocko
> > > > wrote:
> > > > > On Mon 16-01-17 11:09:34, Mel Gorman wrote:
> > > > > [...]
> > > > > > diff --git a/mm/vmscan.c b/mm/vmscan.c
> > > > > > index 532a2a750952..46aac487b89a 100644
> > > > > > --- a/mm/vmscan.c
> > > > > > +++ b/mm/vmscan.c
> > > > > > @@ -2684,6 +2684,7 @@ static void shrink_zones(struct
> > > > > > zonelist *zonelist, struct scan_control *sc) continue;
> > > > > >
> > > > > > if (sc->priority != DEF_PRIORITY &&
> > > > > > + !buffer_heads_over_limit &&
> > > > > > !pgdat_reclaimable(zone->zone_pgdat))
> > > > > > continue; /* Let
> > > > > > kswapd poll it */
> > > > >
> > > > > I think we should rather remove pgdat_reclaimable here. This
> > > > > sounds like a wrong layer to decide whether we want to reclaim
> > > > > and how much.
> > > >
> > > > I had considered that but it'd also be important to add the
> > > > other 32-bit patches you have posted to see the impact. Because
> > > > of the ratio of LRU pages to slab pages, it may not have an
> > > > impact but it'd need to be eliminated.
> > >
> > > OK, Trevor you can pull from
> > > git://git.kernel.org/pub/scm/linux/kernel/git/mhocko/mm.git tree
> > > fixes/highmem-node-fixes branch. This contains the current mmotm
> > > tree
> > > + the latest highmem fixes. I also do not expect this would help
> > > much in your case but as Mel've said we should rule that out at
> > > least.
> >
> > Hi! The git tree above version oom'd after < 24 hours (3:02am) so
> > it doesn't solve the bug. If you need a oom messages dump let me
> > know.
>
> Yes please.
The first oom from that night attached. Note, the oom wasn't as dire
with your mhocko/4.9.0+ as it usually is with stock 4.8.x: my oom
detector and reboot script was able to do its thing cleanly before the
system became unusable.
I'll await further instructions and test right away. Maybe I'll try a
few tuning ideas until then. Thanks!
> > Let me know what to try next, guys, and I'll test it out.
> >
> > > > Before prototyping such a thing, I'd like to hear the outcome of
> > > > this heavy hack and then add your 32-bit patches onto the list.
> > > > If the problem is still there then I'd next look at taking slab
> > > > pages into account in pgdat_reclaimable() instead of an
> > > > outright removal that has a much wider impact. If that doesn't
> > > > work then I'll prototype a heavy-handed forced slab reclaim
> > > > when lower zones are almost all slab pages.
> >
> > I don't think I've tried the "heavy hack" patch yet? It's not in
> > the mhocko tree I just tried? Should I try the heavy hack on top
> > of mhocko git or on vanilla or what?
> >
> > I also want to mention that these PAE boxes suffer from another
> > problem/bug that I've worked around for almost a year now. For some
> > reason it keeps gnawing at me that it might be related. The disk
> > I/O goes to pot on this/these PAE boxes after a certain amount of
> > disk writes (like some unknown number of GB, around 10-ish maybe).
> > Like writes go from 500MB/s to 10MB/s!! Reboot and it's magically
> > 500MB/s again. I detail this here:
> > https://muug.ca/pipermail/roundtable/2016-June/004669.html
> > My fix was to mem=XG where X is <8 (like 4 or 6) to force the PAE
> > kernel to be more sane about highmem choices. I never filed a bug
> > because I read a ton of stuff saying Linus hates PAE, don't use over
> > 4G, blah blah. But the other fix is to:
> > set /proc/sys/vm/highmem_is_dirtyable to 1
>
> Yes this sounds like a dirty memory throttling and there were some
> changes in that area. I do not remember when exactly.
I think my PAE-slow-IO bug started way back in Fedora 22 (4.0?), hard
to know exactly when as I didn't discover the bug for maybe a year as I
didn't realize IO was the problem right away. Too late to bisect that
one, I guess. I guess it's not related so we can ignore my tangent!
> > I'm not bringing this up to get attention to a new bug, I bring
> > this up because it smells like it might be related. If something
> > slowly eats away at the box's vm to the point that I/O gets
> > horribly slow, perhaps it's related to the slab and high/lomem
> > issue we have here? And if related, it may help to solve the oom
> > bug. If I'm way off base here, just ignore my tangent!
>
> >From your OOM reports so far it doesn't really seem related because
> >you
> never had large number of pages under the writeback when OOM.
>
> The situation with the PAE kernel is unfortunate but it is really hard
> to do anything about that considering that the kernel and most its
> allocations have to live in a small and scarce lowmem memory. Moreover
> the more memory you have to more you have to allocated from that
> memory.
You're for sure right that the IO-slow bug was definitely worse the more
ram was in a system! (The mem=4G really helps alleviate this bug and is
good enough for me.)
> This is why not only Linus hates 32b systems on a large memory
> systems.
Completely off-topic: it would be great if rather than pretending PAE
should work with large RAM (which seems more broken every day), the
kernel guys put out an officially stated policy of a maximum RAM you
can use, and try to have the kernel behave for <= that size, and then
people could use more RAM but clearly "at your own risk, don't bug us
about problems!". Other than a few posts about Linus hating it,
there's nothing official I can find about it in documentation, etc. It
gives the (mis)impression that it's perfectly fine to run PAE on a
zillion GB modern system. Then we later learn the hard way :-)
[-- Attachment #2: oom3 --]
[-- Type: application/octet-stream, Size: 22921 bytes --]
Jan 19 03:02:19 firewallfsi kernel: [85602.797346] smbd invoked oom-killer: gfp_mask=0x26000d0(GFP_TEMPORARY|__GFP_NOTRACK), nodemask=0, order=0, oom_score_adj=0
Jan 19 03:02:19 firewallfsi kernel: [85602.798595] smbd cpuset=/ mems_allowed=0
Jan 19 03:02:19 firewallfsi kernel: [85602.799868] CPU: 0 PID: 5892 Comm: smbd Not tainted 4.9.0+ #1
Jan 19 03:02:19 firewallfsi kernel: [85602.801084] Hardware name: Intel Corporation S1200BTL/S1200BTL, BIOS S1200BT.86B.02.00.0035.030220120927 03/02/2012
Jan 19 03:02:19 firewallfsi kernel: [85602.802306] df18da9c d67603e7 df18dbd4 ee3e2400 df18dacc d65e16a6 df18daac d6b63aad
Jan 19 03:02:19 firewallfsi kernel: [85602.803529] df18dacc d676615f df18dad0 f6a6d600 e1172400 ee3e2400 d6d68bde df18dbd4
Jan 19 03:02:19 firewallfsi kernel: [85602.804736] df18db10 d657aff7 d6476d8a df18dafc d657ac6b 00000006 00000000 0000000b
Jan 19 03:02:19 firewallfsi kernel: [85602.805933] Call Trace:
Jan 19 03:02:19 firewallfsi kernel: [85602.807087] [<d67603e7>] dump_stack+0x58/0x81
Jan 19 03:02:19 firewallfsi kernel: [85602.808227] [<d65e16a6>] dump_header+0x64/0x1a6
Jan 19 03:02:19 firewallfsi kernel: [85602.809347] [<d6b63aad>] ? _raw_spin_unlock_irqrestore+0xd/0x10
Jan 19 03:02:19 firewallfsi kernel: [85602.810454] [<d676615f>] ? ___ratelimit+0x9f/0x100
Jan 19 03:02:19 firewallfsi kernel: [85602.811544] [<d657aff7>] oom_kill_process+0x207/0x3d0
Jan 19 03:02:19 firewallfsi kernel: [85602.812617] [<d6476d8a>] ? has_capability_noaudit+0x1a/0x30
Jan 19 03:02:19 firewallfsi kernel: [85602.813674] [<d657ac6b>] ? oom_badness.part.13+0xcb/0x140
Jan 19 03:02:19 firewallfsi kernel: [85602.814715] [<d657b4d8>] out_of_memory+0xf8/0x2a0
Jan 19 03:02:19 firewallfsi kernel: [85602.815737] [<d65800ca>] __alloc_pages_nodemask+0xcfa/0xd10
Jan 19 03:02:19 firewallfsi kernel: [85602.816746] [<d65cc928>] new_slab+0x3c8/0x4b0
Jan 19 03:02:19 firewallfsi kernel: [85602.817736] [<d65cddca>] ___slab_alloc.constprop.75+0x42a/0x670
Jan 19 03:02:19 firewallfsi kernel: [85602.818715] [<d65fd583>] ? __d_alloc+0x23/0x190
Jan 19 03:02:19 firewallfsi kernel: [85602.819675] [<d6668ce4>] ? __ext4_get_inode_loc+0x104/0x440
Jan 19 03:02:19 firewallfsi kernel: [85602.820620] [<d65ce03f>] __slab_alloc.constprop.74+0x2f/0x50
Jan 19 03:02:19 firewallfsi kernel: [85602.821550] [<d65cf0ca>] kmem_cache_alloc+0x17a/0x1c0
Jan 19 03:02:19 firewallfsi kernel: [85602.822462] [<d65fd583>] ? __d_alloc+0x23/0x190
Jan 19 03:02:19 firewallfsi kernel: [85602.823355] [<d65fd583>] __d_alloc+0x23/0x190
Jan 19 03:02:19 firewallfsi kernel: [85602.824228] [<d65fd704>] d_alloc+0x14/0x50
Jan 19 03:02:19 firewallfsi kernel: [85602.825082] [<d65fdbd7>] d_alloc_parallel+0x47/0x450
Jan 19 03:02:19 firewallfsi kernel: [85602.825919] [<d65fcecd>] ? d_splice_alias+0x1fd/0x370
Jan 19 03:02:19 firewallfsi kernel: [85602.826739] [<d6677971>] ? ext4_lookup+0x161/0x240
Jan 19 03:02:19 firewallfsi kernel: [85602.827540] [<d676e80b>] ? lockref_get_not_dead+0xb/0x30
Jan 19 03:02:19 firewallfsi kernel: [85602.828325] [<d65ef289>] ? unlazy_walk+0xf9/0x1a0
Jan 19 03:02:19 firewallfsi kernel: [85602.829093] [<d65f00ae>] lookup_slow+0x5e/0x130
Jan 19 03:02:19 firewallfsi kernel: [85602.829842] [<d65f0ca4>] walk_component+0x1e4/0x300
Jan 19 03:02:19 firewallfsi kernel: [85602.830574] [<d65efc8d>] ? path_init+0x14d/0x330
Jan 19 03:02:19 firewallfsi kernel: [85602.831288] [<d65f1e83>] path_lookupat+0x53/0xe0
Jan 19 03:02:19 firewallfsi kernel: [85602.831984] [<d65f4027>] filename_lookup+0x97/0x190
Jan 19 03:02:19 firewallfsi kernel: [85602.832661] [<d65cf047>] ? kmem_cache_alloc+0xf7/0x1c0
Jan 19 03:02:19 firewallfsi kernel: [85602.833322] [<d65f3c6a>] ? getname_flags+0x3a/0x1a0
Jan 19 03:02:19 firewallfsi kernel: [85602.833966] [<d65f3c81>] ? getname_flags+0x51/0x1a0
Jan 19 03:02:19 firewallfsi kernel: [85602.834587] [<d65f41f6>] user_path_at_empty+0x36/0x40
Jan 19 03:02:19 firewallfsi kernel: [85602.835190] [<d65e9910>] vfs_fstatat+0x60/0xb0
Jan 19 03:02:19 firewallfsi kernel: [85602.835774] [<d65ea3aa>] SyS_fstatat64+0x2a/0x50
Jan 19 03:02:19 firewallfsi kernel: [85602.836341] [<d642b168>] ? sched_clock+0x8/0x10
Jan 19 03:02:19 firewallfsi kernel: [85602.836888] [<d649a075>] ? sched_clock_cpu+0x125/0x140
Jan 19 03:02:19 firewallfsi kernel: [85602.837415] [<d64de305>] ? hrtimer_interrupt+0xa5/0x180
Jan 19 03:02:19 firewallfsi kernel: [85602.837926] [<d640377a>] do_fast_syscall_32+0x8a/0x150
Jan 19 03:02:19 firewallfsi kernel: [85602.838420] [<d6b63fca>] sysenter_past_esp+0x47/0x75
Jan 19 03:02:19 firewallfsi kernel: [85602.838902] Mem-Info:
Jan 19 03:02:19 firewallfsi kernel: [85602.840191] active_anon:155033 inactive_anon:32960 isolated_anon:0
Jan 19 03:02:19 firewallfsi kernel: [85602.840191] active_file:171215 inactive_file:640942 isolated_file:0
Jan 19 03:02:19 firewallfsi kernel: [85602.840191] unevictable:0 dirty:1277 writeback:0 unstable:0
Jan 19 03:02:19 firewallfsi kernel: [85602.840191] slab_reclaimable:132504 slab_unreclaimable:11682
Jan 19 03:02:19 firewallfsi kernel: [85602.840191] mapped:22669 shmem:1221 pagetables:1537 bounce:0
Jan 19 03:02:19 firewallfsi kernel: [85602.840191] free:57986 free_pcp:1120 free_cma:0
Jan 19 03:02:19 firewallfsi kernel: [85602.848275] Node 0 active_anon:620132kB inactive_anon:131840kB active_file:684860kB inactive_file:2563768kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:90712kB dirty:5128kB writeback:0kB shmem:4884kB writeback_tmp:0kB unstable:0kB pages_scanned:0 all_unreclaimable? no
Jan 19 03:02:19 firewallfsi kernel: [85602.852375] DMA free:3176kB min:68kB low:84kB high:100kB active_anon:0kB inactive_anon:0kB active_file:4996kB inactive_file:0kB unevictable:0kB writepending:8kB present:15976kB managed:15900kB mlocked:0kB slab_reclaimable:7724kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jan 19 03:02:19 firewallfsi kernel: lowmem_reserve[]: 0 777 4734 4734
Jan 19 03:02:19 firewallfsi kernel: [85602.858232] Normal free:3436kB min:3532kB low:4412kB high:5292kB active_anon:4kB inactive_anon:8kB active_file:193340kB inactive_file:120kB unevictable:0kB writepending:2516kB present:892920kB managed:816932kB mlocked:0kB slab_reclaimable:522292kB slab_unreclaimable:46724kB kernel_stack:2560kB pagetables:0kB bounce:0kB free_pcp:3468kB local_pcp:176kB free_cma:0kB
Jan 19 03:02:19 firewallfsi kernel: lowmem_reserve[]: 0 0 31652 31652
Jan 19 03:02:19 firewallfsi kernel: [85602.864610] HighMem free:225332kB min:512kB low:5004kB high:9496kB active_anon:620128kB inactive_anon:131832kB active_file:486476kB inactive_file:2563648kB unevictable:0kB writepending:2616kB present:4051548kB managed:4051548kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:6148kB bounce:0kB free_pcp:1012kB local_pcp:180kB free_cma:0kB
Jan 19 03:02:19 firewallfsi kernel: lowmem_reserve[]: 0 0 0 0
Jan 19 03:02:19 firewallfsi kernel: [85602.871108] DMA: 14*4kB (UME) 4*8kB (UE) 1*16kB (M) 2*32kB (UE) 1*64kB (U) 1*128kB (U) 1*256kB (U) 3*512kB (UME) 1*1024kB (U) 0*2048kB 0*4096kB = 3176kB
Jan 19 03:02:19 firewallfsi kernel: Normal: 165*4kB (UME) 61*8kB (UME) 19*16kB (UM) 18*32kB (UME) 2*64kB (U) 4*128kB (UE) 3*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3436kB
Jan 19 03:02:19 firewallfsi kernel: HighMem: 243*4kB (M) 167*8kB (UM) 55*16kB (UM) 28*32kB (UM) 15*64kB (UM) 13*128kB (UM) 6*256kB (UM) 0*512kB 2*1024kB (M) 39*2048kB (M) 33*4096kB (M) = 225332kB
Jan 19 03:02:19 firewallfsi kernel: [85602.877129] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jan 19 03:02:19 firewallfsi kernel: [85602.878150] 813455 total pagecache pages
Jan 19 03:02:19 firewallfsi kernel: [85602.879168] 34 pages in swap cache
Jan 19 03:02:19 firewallfsi kernel: [85602.880181] Swap cache stats: add 71, delete 37, find 191/200
Jan 19 03:02:19 firewallfsi kernel: [85602.881205] Free swap = 33784424kB
Jan 19 03:02:19 firewallfsi kernel: [85602.882242] Total swap = 33784572kB
Jan 19 03:02:19 firewallfsi kernel: [85602.883275] 1240111 pages RAM
Jan 19 03:02:19 firewallfsi kernel: [85602.884302] 1012887 pages HighMem/MovableOnly
Jan 19 03:02:19 firewallfsi kernel: [85602.885317] 19016 pages reserved
Jan 19 03:02:19 firewallfsi kernel: [85602.886310] 0 pages hwpoisoned
Jan 19 03:02:19 firewallfsi kernel: [85602.887299] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
Jan 19 03:02:19 firewallfsi kernel: [85602.888342] [ 596] 0 596 3278 1039 9 3 0 0 systemd-journal
Jan 19 03:02:19 firewallfsi kernel: [85602.889382] [ 632] 0 632 3592 1182 9 3 0 -1000 systemd-udevd
Jan 19 03:02:19 firewallfsi kernel: [85602.890420] [ 742] 81 742 1700 1069 7 3 0 -900 dbus-daemon
Jan 19 03:02:19 firewallfsi kernel: [85602.891461] [ 743] 0 743 1704 663 6 3 0 0 sh
Jan 19 03:02:19 firewallfsi kernel: [85602.892502] [ 745] 0 745 1704 646 6 3 0 0 sh
Jan 19 03:02:19 firewallfsi kernel: [85602.893540] [ 747] 0 747 1033 641 6 3 0 0 irqbalance
Jan 19 03:02:19 firewallfsi kernel: [85602.894574] [ 751] 0 751 808 487 5 3 0 0 mdadm
Jan 19 03:02:19 firewallfsi kernel: [85602.895606] [ 752] 288 752 19290 1785 20 3 0 0 milter-greylist
Jan 19 03:02:19 firewallfsi kernel: [85602.896595] [ 753] 0 753 988 716 5 3 0 0 systemd-logind
Jan 19 03:02:19 firewallfsi kernel: [85602.897577] [ 755] 0 755 1472 953 6 3 0 0 smartd
Jan 19 03:02:19 firewallfsi kernel: [85602.898550] [ 757] 0 757 1704 676 7 3 0 0 sh
Jan 19 03:02:19 firewallfsi kernel: [85602.899514] [ 764] 0 764 8633 1110 12 3 0 0 rsyslogd
Jan 19 03:02:19 firewallfsi kernel: [85602.900474] [ 765] 0 765 1704 673 7 3 0 0 sh
Jan 19 03:02:19 firewallfsi kernel: [85602.901429] [ 768] 0 768 1704 629 6 3 0 0 sh
Jan 19 03:02:19 firewallfsi kernel: [85602.902375] [ 770] 0 770 1704 681 7 3 0 0 sh
Jan 19 03:02:19 firewallfsi kernel: [85602.903312] [ 771] 0 771 1704 644 6 3 0 0 sh
Jan 19 03:02:19 firewallfsi kernel: [85602.904225] [ 774] 0 774 1704 680 6 3 0 0 sh
Jan 19 03:02:19 firewallfsi kernel: [85602.905108] [ 799] 0 799 583 436 5 3 0 0 acpid
Jan 19 03:02:19 firewallfsi kernel: [85602.905967] [ 803] 0 803 7287 1234 11 3 0 0 apcupsd
Jan 19 03:02:19 firewallfsi kernel: [85602.906804] [ 810] 0 810 860 487 5 3 0 0 atd
Jan 19 03:02:19 firewallfsi kernel: [85602.907616] [ 868] 0 868 1819 814 8 3 0 0 watch-services
Jan 19 03:02:19 firewallfsi kernel: [85602.908388] [ 869] 0 869 3165 2070 9 3 0 0 mailwarnings
Jan 19 03:02:19 firewallfsi kernel: [85602.909143] [ 870] 0 870 3238 2164 9 3 0 0 dynamic-ip-upda
Jan 19 03:02:19 firewallfsi kernel: [85602.909879] [ 871] 0 871 1736 719 6 3 0 0 tickle-pog
Jan 19 03:02:19 firewallfsi kernel: [85602.910589] [ 872] 0 872 3264 2201 10 3 0 0 restarter
Jan 19 03:02:19 firewallfsi kernel: [85602.911282] [ 875] 0 875 3695 1986 11 3 0 0 fetchmail
Jan 19 03:02:19 firewallfsi kernel: [85602.911955] [ 876] 0 876 2707 1622 9 3 0 0 udp-sgr
Jan 19 03:02:19 firewallfsi kernel: [85602.912607] [ 877] 0 877 2499 414 9 3 0 0 saslauthd
Jan 19 03:02:19 firewallfsi kernel: [85602.913175] [ 878] 0 878 2518 900 9 3 0 0 saslauthd
Jan 19 03:02:19 firewallfsi kernel: [85602.913709] [ 879] 0 879 2499 126 9 3 0 0 saslauthd
Jan 19 03:02:19 firewallfsi kernel: [85602.914216] [ 880] 0 880 2499 126 9 3 0 0 saslauthd
Jan 19 03:02:19 firewallfsi kernel: [85602.914706] [ 881] 0 881 2499 126 9 3 0 0 saslauthd
Jan 19 03:02:19 firewallfsi kernel: [85602.915166] [ 882] 0 882 3310 2223 10 3 0 0 watch-ip
Jan 19 03:02:19 firewallfsi kernel: [85602.915609] [ 938] 0 938 2769 621 9 3 0 -1000 sshd
Jan 19 03:02:19 firewallfsi kernel: [85602.916028] [ 1032] 27 1032 1707 733 7 3 0 0 mysqld_safe
Jan 19 03:02:19 firewallfsi kernel: [85602.916432] [ 1130] 25 1130 82172 47875 114 3 0 0 named
Jan 19 03:02:19 firewallfsi kernel: [85602.916822] [ 1208] 27 1208 126039 14693 65 3 0 0 mysqld
Jan 19 03:02:19 firewallfsi kernel: [85602.917197] [ 1245] 0 1245 7049 2401 16 3 0 0 nmbd
Jan 19 03:02:19 firewallfsi kernel: [85602.917559] [ 1246] 0 1246 6840 2267 16 3 0 0 nmbd
Jan 19 03:02:19 firewallfsi kernel: [85602.917923] [ 1308] 0 1308 12052 6449 26 3 0 0 /usr/sbin/httpd
Jan 19 03:02:19 firewallfsi kernel: [85602.918282] [ 1335] 0 1335 1116 497 6 3 0 0 agetty
Jan 19 03:02:19 firewallfsi kernel: [85602.918640] [ 1336] 0 1336 1116 479 6 3 0 0 agetty
Jan 19 03:02:19 firewallfsi kernel: [85602.918992] [ 1337] 0 1337 1116 521 6 3 0 0 agetty
Jan 19 03:02:19 firewallfsi kernel: [85602.919340] [ 1338] 0 1338 1116 505 6 3 0 0 agetty
Jan 19 03:02:19 firewallfsi kernel: [85602.919688] [ 1339] 0 1339 1116 462 6 3 0 0 agetty
Jan 19 03:02:19 firewallfsi kernel: [85602.920027] [ 1340] 0 1340 1116 473 6 3 0 0 agetty
Jan 19 03:02:19 firewallfsi kernel: [85602.920363] [ 1596] 48 1596 49256 4353 44 3 0 0 /usr/sbin/httpd
Jan 19 03:02:19 firewallfsi kernel: [85602.920697] [ 1597] 48 1597 16386 4635 29 3 0 0 /usr/sbin/httpd
Jan 19 03:02:19 firewallfsi kernel: [85602.921014] [ 1600] 48 1600 16295 5404 29 3 0 0 /usr/sbin/httpd
Jan 19 03:02:19 firewallfsi kernel: [85602.921328] [ 1619] 48 1619 16298 5545 29 3 0 0 /usr/sbin/httpd
Jan 19 03:02:19 firewallfsi kernel: [85602.921640] [ 1622] 48 1622 16220 4205 27 3 0 0 /usr/sbin/httpd
Jan 19 03:02:19 firewallfsi kernel: [85602.921944] [ 1778] 0 1778 1704 682 6 3 0 0 sh
Jan 19 03:02:19 firewallfsi kernel: [85602.922253] [ 1779] 0 1779 2743 1705 9 3 0 0 udp-sgs
Jan 19 03:02:19 firewallfsi kernel: [85602.922562] [ 1781] 0 1781 9561 3567 23 3 0 0 smbd
Jan 19 03:02:19 firewallfsi kernel: [85602.922875] [ 1782] 0 1782 9197 1092 21 3 0 0 smbd
Jan 19 03:02:19 firewallfsi kernel: [85602.923198] [ 1783] 0 1783 9446 1156 21 3 0 0 smbd
Jan 19 03:02:19 firewallfsi kernel: [85602.923498] [ 1870] 0 1870 5116 2471 14 3 0 0 dhclient
Jan 19 03:02:19 firewallfsi kernel: [85602.923793] [ 1948] 0 1948 594 400 5 3 0 0 pptpd
Jan 19 03:02:19 firewallfsi kernel: [85602.924104] [ 1950] 0 1950 954 622 6 3 0 0 dovecot
Jan 19 03:02:19 firewallfsi kernel: [85602.924395] [ 1951] 97 1951 904 582 5 3 0 0 anvil
Jan 19 03:02:19 firewallfsi kernel: [85602.924689] [ 1952] 0 1952 937 613 6 3 0 0 log
Jan 19 03:02:19 firewallfsi kernel: [85602.925000] [ 1954] 0 1954 1133 784 6 3 0 0 config
Jan 19 03:02:19 firewallfsi kernel: [85602.925294] [ 1956] 0 1956 4181 1973 12 3 0 0 sshd
Jan 19 03:02:19 firewallfsi kernel: [85602.925590] [ 1965] 0 1965 1584 1147 6 3 0 0 systemd
Jan 19 03:02:19 firewallfsi kernel: [85602.925904] [ 1971] 0 1971 2455 381 8 3 0 0 (sd-pam)
Jan 19 03:02:19 firewallfsi kernel: [85602.926216] [ 2001] 0 2001 4214 1046 12 3 0 0 sshd
Jan 19 03:02:19 firewallfsi kernel: [85602.926517] [ 2011] 0 2011 1187 957 5 3 0 0 tcsh
Jan 19 03:02:19 firewallfsi kernel: [85602.926821] [ 2035] 48 2035 16220 4208 28 3 0 0 /usr/sbin/httpd
Jan 19 03:02:19 firewallfsi kernel: [85602.927144] [ 2040] 48 2040 16298 5544 29 3 0 0 /usr/sbin/httpd
Jan 19 03:02:19 firewallfsi kernel: [85602.927445] [ 2042] 48 2042 16216 4586 28 3 0 0 /usr/sbin/httpd
Jan 19 03:02:19 firewallfsi kernel: [85602.927746] [ 2055] 177 2055 6833 4786 16 3 0 0 dhcpd
Jan 19 03:02:19 firewallfsi kernel: [85602.928062] [ 2062] 0 2062 1899 698 7 3 29 0 crond
Jan 19 03:02:19 firewallfsi kernel: [85602.928358] [ 2070] 38 2070 1536 1082 6 3 0 0 ntpd
Jan 19 03:02:19 firewallfsi kernel: [85602.928659] [ 2297] 273 2297 2208 1390 8 3 0 0 imap-login
Jan 19 03:02:19 firewallfsi kernel: [85602.928977] [ 2299] 301 2299 1762 1248 7 3 0 0 imap
Jan 19 03:02:19 firewallfsi kernel: [85602.929276] [ 5376] 0 5376 1307 927 6 3 0 0 reboot-when-oom
Jan 19 03:02:19 firewallfsi kernel: [85602.929583] [10505] 276 10505 115120 103978 222 3 0 0 clamd
Jan 19 03:02:19 firewallfsi kernel: [85602.929909] [10519] 290 10519 15704 1191 14 3 0 0 clamav-milter
Jan 19 03:02:19 firewallfsi kernel: [85602.930239] [10539] 0 10539 3829 1778 10 3 0 0 sendmail
Jan 19 03:02:19 firewallfsi kernel: [85602.930565] [10552] 51 10552 3500 765 11 3 0 0 sendmail
Jan 19 03:02:19 firewallfsi kernel: [85602.930902] [10658] 23 10658 5440 724 14 3 0 0 squid
Jan 19 03:02:19 firewallfsi kernel: [85602.931234] [10660] 23 10660 9797 6723 22 3 0 0 squid
Jan 19 03:02:19 firewallfsi kernel: [85602.931551] [10661] 23 10661 1179 419 6 3 0 0 unlinkd
Jan 19 03:02:19 firewallfsi kernel: [85602.931879] [13921] 273 13921 2208 1470 8 3 0 0 imap-login
Jan 19 03:02:19 firewallfsi kernel: [85602.932206] [13924] 301 13924 1400 973 6 3 0 0 imap
Jan 19 03:02:19 firewallfsi kernel: [85602.932519] [13925] 273 13925 2209 1364 9 3 0 0 imap-login
Jan 19 03:02:19 firewallfsi kernel: [85602.932840] [13926] 301 13926 2559 1273 7 3 0 0 imap
Jan 19 03:02:19 firewallfsi kernel: [85602.933173] [15323] 48 15323 16220 4208 27 3 0 0 /usr/sbin/httpd
Jan 19 03:02:19 firewallfsi kernel: [85602.933491] [14826] 273 14826 2208 1448 7 3 0 0 imap-login
Jan 19 03:02:19 firewallfsi kernel: [85602.933836] [14828] 301 14828 3995 1513 8 3 0 0 imap
Jan 19 03:02:19 firewallfsi kernel: [85602.934181] [18753] 48 18753 16220 4338 28 3 0 0 /usr/sbin/httpd
Jan 19 03:02:19 firewallfsi kernel: [85602.934516] [ 5254] 0 5254 9685 3801 23 3 0 0 smbd
Jan 19 03:02:19 firewallfsi kernel: [85602.934879] [ 5892] 300 5892 11287 5069 26 3 0 0 smbd
Jan 19 03:02:19 firewallfsi kernel: [85602.935241] [31892] 273 31892 2209 1455 8 3 0 0 imap-login
Jan 19 03:02:19 firewallfsi kernel: [85602.935583] [31896] 302 31896 1415 1067 7 3 0 0 imap
Jan 19 03:02:19 firewallfsi kernel: [85602.935941] [28129] 0 28129 1396 138 6 3 0 0 sleep
Jan 19 03:02:19 firewallfsi kernel: [85602.936298] [30312] 273 30312 2209 1436 8 3 0 0 imap-login
Jan 19 03:02:19 firewallfsi kernel: [85602.936656] [30315] 301 30315 1432 989 6 3 0 0 imap
Jan 19 03:02:19 firewallfsi kernel: [85602.937030] [32695] 0 32695 1900 494 7 3 17 0 crond
Jan 19 03:02:19 firewallfsi kernel: [85602.937383] [32696] 0 32696 3184 2117 10 3 0 0 freshclam-tec-w
Jan 19 03:02:19 firewallfsi kernel: [85602.937754] [32726] 0 32726 1654 435 7 3 0 0 anacron
Jan 19 03:02:19 firewallfsi kernel: [85602.938122] [ 548] 0 548 1396 139 6 3 0 0 sleep
Jan 19 03:02:19 firewallfsi kernel: [85602.938481] Out of memory: Kill process 10505 (clamd) score 10 or sacrifice child
Jan 19 03:02:19 firewallfsi kernel: [85602.938867] Killed process 10505 (clamd) total-vm:460480kB, anon-rss:399264kB, file-rss:16648kB, shmem-rss:0kB
Jan 19 03:02:19 firewallfsi kernel: [85602.970282] audit: type=1131 audit(1484816539.419:2108): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=clamd@scan comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
next prev parent reply other threads:[~2017-01-20 6:37 UTC|newest]
Thread overview: 40+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-01-11 10:32 mm, vmscan: commit makes PAE kernel crash nightly (bisected) Trevor Cordes
2017-01-11 12:11 ` Mel Gorman
2017-01-11 12:14 ` Mel Gorman
2017-01-11 22:52 ` Trevor Cordes
2017-01-12 9:36 ` Michal Hocko
2017-01-15 6:27 ` Trevor Cordes
2017-01-16 11:09 ` Mel Gorman
2017-01-17 13:52 ` Michal Hocko
2017-01-17 14:21 ` Mel Gorman
2017-01-17 14:54 ` Michal Hocko
2017-01-18 7:25 ` Trevor Cordes
2017-01-18 17:48 ` Mel Gorman
2017-01-18 18:07 ` Mel Gorman
2017-01-19 9:48 ` Trevor Cordes
2017-01-19 11:37 ` Michal Hocko
2017-01-20 6:35 ` Trevor Cordes [this message]
2017-01-20 11:02 ` Mel Gorman
2017-01-20 15:55 ` Mel Gorman
2017-01-23 0:45 ` Trevor Cordes
2017-01-23 10:48 ` Mel Gorman
2017-01-23 11:04 ` Mel Gorman
2017-01-25 9:46 ` Michal Hocko
2017-01-24 12:59 ` Michal Hocko
2017-01-25 10:02 ` Trevor Cordes
2017-01-25 12:04 ` Michal Hocko
2017-01-29 22:50 ` Trevor Cordes
2017-01-30 7:51 ` Michal Hocko
2017-02-01 9:29 ` Trevor Cordes
2017-02-01 10:14 ` Michal Hocko
2017-02-04 0:36 ` Trevor Cordes
2017-02-04 20:05 ` Rik van Riel
2017-02-05 10:03 ` Michal Hocko
2017-02-05 22:53 ` Trevor Cordes
2017-01-30 9:10 ` Mel Gorman
2017-01-24 12:54 ` Michal Hocko
2017-01-26 23:18 ` Trevor Cordes
2017-01-27 7:36 ` Michal Hocko
2017-01-24 12:51 ` Michal Hocko
2017-01-18 6:52 ` Trevor Cordes
2017-01-17 13:45 ` Michal Hocko
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20170120003544.7e6e34d1@pog.tecnopolis.ca \
--to=trevor@tecnopolis.ca \
--cc=iamjoonsoo.kim@lge.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mgorman@techsingularity.net \
--cc=mhocko@kernel.org \
--cc=minchan@kernel.org \
--cc=riel@surriel.com \
--cc=srikar@linux.vnet.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).