* Linux 3.19-rc3
[not found] <CA+55aFwsxoyLb9OWMSCL3doe_cz_EQtKsEFCyPUYn_T87pbz0A@mail.gmail.com>
@ 2015-01-08 12:51 ` Mark Langsdorf
2015-01-08 13:45 ` Catalin Marinas
` (2 more replies)
0 siblings, 3 replies; 50+ messages in thread
From: Mark Langsdorf @ 2015-01-08 12:51 UTC (permalink / raw)
To: linux-arm-kernel
On 01/05/2015 07:46 PM, Linus Torvalds wrote:
> It's a day delayed - not because of any particular development issues,
> but simply because I was tiling a bathroom yesterday. But rc3 is out
> there now, and things have stayed reasonably calm. I really hope that
> implies that 3.19 is looking good, but it's equally likely that it's
> just that people are still recovering from the holiday season.
I'm consistently getting an out of memory killer triggered when
compiling the kernel (make -j 16 -s) on a 16 core ARM64 system
with 16 GB of memory. This doesn't happen when running a 3.18
kernel.
I'm going to start bisecting the failure now, but here's the crash
log in case someone can see something obvious in it.
--Mark Langsdorf
[ 137.440443] random: nonblocking pool is initialized
[ 1053.720094] cc1 invoked oom-killer: gfp_mask=0x200da, order=0,
oom_score_adj=0
[ 1053.727292] cc1 cpuset=/ mems_allowed=0
[ 1053.731169] CPU: 5 PID: 32180 Comm: cc1 Not tainted 3.19.0-rc3+ #2
[ 1053.737321] Hardware name: APM X-Gene Mustang board (DT)
[ 1053.742627] Call trace:
[ 1053.745073] [<fffffe0000096954>] dump_backtrace+0x0/0x16c
[ 1053.750465] [<fffffe0000096ad0>] show_stack+0x10/0x1c
[ 1053.755498] [<fffffe000062911c>] dump_stack+0x74/0x98
[ 1053.760543] [<fffffe0000626550>] dump_header.isra.11+0x98/0x1d0
[ 1053.766439] [<fffffe000017c4e0>] oom_kill_process+0x3ac/0x408
[ 1053.772182] [<fffffe000017c9d4>] out_of_memory+0x2a4/0x2d4
[ 1053.777647] [<fffffe0000180b78>] __alloc_pages_nodemask+0x808/0x8d8
[ 1053.783898] [<fffffe00001a443c>] handle_mm_fault+0x938/0xd1c
[ 1053.789544] [<fffffe00000a01c8>] do_page_fault+0x214/0x338
[ 1053.795004] [<fffffe000009022c>] do_mem_abort+0x38/0x9c
[ 1053.800211] Exception stack(0xfffffe0010b1fe30 to 0xfffffe0010b1ff50)
[ 1053.806622] fe20: 00000200
00000000 00f886a8 00000000
[ 1053.814771] fe40: ffffffff ffffffff aa924620 000003ff 00b11000
fffffe00 000f7c84 fffffe00
[ 1053.822915] fe60: 10b1fea0 fffffe00 000903b4 fffffe00 10b1fed0
fffffe00 0002000c fffffc00
[ 1053.831063] fe80: 00b686b8 fffffe00 00020010 fffffc00 10b1feb0
fffffe00 000964e8 fffffe00
[ 1053.839211] fea0: 00000208 00000000 00092f90 fffffe00 cac18530
000003ff 0009308c fffffe00
[ 1053.847348] fec0: 00000200 00000000 00093094 fffffe00 a28e0000
000003ff 00000000 00000000
[ 1053.855489] fee0: 00000248 00000000 a28e0000 000003ff 00000000
00000000 a28e0000 000003ff
[ 1053.863630] ff00: 00000009 00000000 00000009 00000000 00fe8000
00000000 00000480 00000000
[ 1053.871779] ff20: 008213c8 00000000 000000a2 00000000 00000006
00000000 00000018 00000000
[ 1053.879921] ff40: 00000008 00000000 00000000 00000000
[ 1053.884946] Mem-Info:
[ 1053.887206] DMA per-cpu:
[ 1053.889731] CPU 0: hi: 6, btch: 1 usd: 0
[ 1053.894496] CPU 1: hi: 6, btch: 1 usd: 0
[ 1053.899268] CPU 2: hi: 6, btch: 1 usd: 0
[ 1053.904034] CPU 3: hi: 6, btch: 1 usd: 0
[ 1053.908809] CPU 4: hi: 6, btch: 1 usd: 0
[ 1053.913576] CPU 5: hi: 6, btch: 1 usd: 0
[ 1053.918350] CPU 6: hi: 6, btch: 1 usd: 0
[ 1053.923115] CPU 7: hi: 6, btch: 1 usd: 0
[ 1053.927880] Normal per-cpu:
[ 1053.930663] CPU 0: hi: 6, btch: 1 usd: 0
[ 1053.935428] CPU 1: hi: 6, btch: 1 usd: 0
[ 1053.940198] CPU 2: hi: 6, btch: 1 usd: 0
[ 1053.944963] CPU 3: hi: 6, btch: 1 usd: 0
[ 1053.949732] CPU 4: hi: 6, btch: 1 usd: 0
[ 1053.954497] CPU 5: hi: 6, btch: 1 usd: 0
[ 1053.959271] CPU 6: hi: 6, btch: 1 usd: 0
[ 1053.964038] CPU 7: hi: 6, btch: 1 usd: 0
[ 1053.968815] active_anon:207417 inactive_anon:25722 isolated_anon:0
[ 1053.968815] active_file:1300 inactive_file:21234 isolated_file:0
[ 1053.968815] unevictable:0 dirty:0 writeback:0 unstable:0
[ 1053.968815] free:1014 slab_reclaimable:1047 slab_unreclaimable:1758
[ 1053.968815] mapped:733 shmem:58 pagetables:267 bounce:0
[ 1053.968815] free_cma:1
[ 1054.000398] DMA free:52928kB min:4032kB low:4992kB high:6016kB
active_anon:3025728kB inactive_anon:612608kB active_file:20992kB
inactive_file:323072kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:4192256kB managed:4177024kB mlocked:0kB
dirty:0kB writeback:0kB mapped:15168kB shmem:1792kB
slab_reclaimable:26368kB slab_unreclaimable:36160kB kernel_stack:464kB
pagetables:6976kB unstable:0kB bounce:0kB free_cma:64kB
writeback_tmp:0kB pages_scanned:26151872 all_unreclaimable? yes
[ 1054.043628] lowmem_reserve[]: 0 765 765
[ 1054.047498] Normal free:12032kB min:12224kB low:15232kB high:18304kB
active_anon:10248960kB inactive_anon:1033728kB active_file:62208kB
inactive_file:1036032kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:12582912kB managed:12538176kB mlocked:0kB
dirty:0kB writeback:0kB mapped:31744kB shmem:1920kB
slab_reclaimable:40640kB slab_unreclaimable:76352kB kernel_stack:2992kB
pagetables:10112kB unstable:0kB bounce:0kB free_cma:0kB
writeback_tmp:0kB pages_scanned:82150272 all_unreclaimable? yes
[ 1054.091760] lowmem_reserve[]: 0 0 0
[ 1054.095277] DMA: 109*64kB (UR) 53*128kB (R) 8*256kB (R) 0*512kB
0*1024kB 0*2048kB 1*4096kB (R) 0*8192kB 0*16384kB 1*32768kB (R)
0*65536kB = 52672kB
[ 1054.108621] Normal: 191*64kB (MR) 0*128kB 0*256kB 0*512kB 0*1024kB
0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 12224kB
[ 1054.120708] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=524288kB
[ 1054.129280] 43732 total pagecache pages
[ 1054.133096] 25736 pages in swap cache
[ 1054.136739] Swap cache stats: add 27971, delete 2235, find 0/1
[ 1054.142545] Free swap = 6598400kB
[ 1054.145928] Total swap = 8388544kB
[ 1054.149317] 262112 pages RAM
[ 1054.152180] 0 pages HighMem/MovableOnly
[ 1054.155995] 18446744073709544361 pages reserved
[ 1054.160505] 8192 pages cma reserved
[ 1054.163974] [ pid ] uid tgid total_vm rss nr_ptes swapents
oom_score_adj name
[ 1054.171784] [ 481] 0 481 233 89 3 39
0 systemd-journal
[ 1054.180534] [ 487] 0 487 1420 32 3 80
0 lvmetad
[ 1054.188591] [ 512] 0 512 216 40 2 46
-1000 systemd-udevd
[ 1054.197162] [ 598] 0 598 258 45 3 38
-1000 auditd
[ 1054.205133] [ 620] 0 620 6825 213 3 114
0 NetworkManager
[ 1054.213794] [ 623] 0 623 2028 83 3 100
0 abrtd
[ 1054.221678] [ 624] 0 624 2019 78 3 92
0 abrt-watch-log
[ 1054.230338] [ 628] 70 628 87 38 4 34
0 avahi-daemon
[ 1054.238826] [ 629] 998 629 40 13 2 0
0 lsmd
[ 1054.246617] [ 631] 70 631 87 0 4 32
0 avahi-daemon
[ 1054.255106] [ 641] 0 641 2687 48 3 77
0 rsyslogd
[ 1054.263248] [ 642] 0 642 86 42 3 9
0 irqbalance
[ 1054.271563] [ 643] 0 643 6821 196 5 241
0 tuned
[ 1054.279447] [ 647] 0 647 109 44 2 31
0 smartd
[ 1054.287411] [ 651] 997 651 83 45 2 18
0 chronyd
[ 1054.295466] [ 652] 0 652 99 38 3 36
0 systemd-logind
[ 1054.304129] [ 653] 81 653 222 35 2 37
-900 dbus-daemon
[ 1054.312531] [ 667] 0 667 79 27 3 29
0 atd
[ 1054.320242] [ 675] 0 675 1846 68 4 86
0 login
[ 1054.328125] [ 676] 0 676 1717 23 5 10
0 agetty
[ 1054.336089] [ 710] 0 710 1761 47 3 36
0 bash
[ 1054.343887] [ 727] 0 727 240 84 2 74
-1000 sshd
[ 1054.351684] [ 733] 0 733 1739 34 3 9
0 rhsmcertd
[ 1054.359913] [ 741] 999 741 7276 118 4 122
0 polkitd
[ 1054.367964] [ 747] 0 747 502 102 3 281
0 dhclient
[ 1054.376106] [ 1323] 0 1323 334 69 3 69
0 master
[ 1054.384075] [ 1325] 89 1325 337 82 2 78
0 qmgr
[ 1054.391872] [ 1328] 0 1328 358 130 3 111
0 sshd
[ 1054.399669] [ 1331] 0 1331 1762 48 5 39
0 bash
[ 1054.407460] [ 1373] 89 1373 336 89 2 69
0 pickup
[ 1054.415430] [ 2944] 0 2944 1734 50 4 3
0 make
[ 1054.423228] [31425] 0 31425 1760 79 3 0
0 make
[ 1054.431028] [32092] 0 32092 1736 51 4 0
0 sh
[ 1054.438652] [32103] 0 32103 1736 51 4 0
0 sh
[ 1054.446271] [32108] 0 32108 1737 42 4 0
0 gcc
[ 1054.453981] [32110] 0 32110 1736 52 4 0
0 sh
[ 1054.461605] [32111] 0 32111 2479 373 4 0
0 cc1
[ 1054.469315] [32118] 0 32118 1736 52 4 0
0 sh
[ 1054.476934] [32120] 0 32120 1737 42 4 0
0 gcc
[ 1054.484644] [32123] 0 32123 1736 51 4 0
0 sh
[ 1054.492269] [32125] 0 32125 2479 373 4 0
0 cc1
[ 1054.499980] [32127] 0 32127 1737 43 4 0
0 gcc
[ 1054.507684] [32132] 0 32132 1736 52 4 0
0 sh
[ 1054.515308] [32134] 0 32134 1737 42 5 0
0 gcc
[ 1054.523019] [32135] 0 32135 2472 304 4 0
0 cc1
[ 1054.530730] [32136] 0 32136 2474 314 4 0
0 cc1
[ 1054.538440] [32140] 0 32140 1736 52 3 0
0 sh
[ 1054.546058] [32141] 0 32141 1737 42 4 0
0 gcc
[ 1054.553768] [32142] 0 32142 1736 51 3 0
0 sh
[ 1054.561394] [32146] 0 32146 1737 42 4 0
0 gcc
[ 1054.569105] [32150] 0 32150 1736 52 4 0
0 sh
[ 1054.576723] [32151] 0 32151 2474 314 3 0
0 cc1
[ 1054.584433] [32154] 0 32154 1737 42 4 0
0 gcc
[ 1054.592143] [32157] 0 32157 1736 51 3 0
0 sh
[ 1054.599767] [32158] 0 32158 2479 373 4 0
0 cc1
[ 1054.607472] [32164] 0 32164 1737 43 5 0
0 gcc
[ 1054.615183] [32165] 0 32165 1736 52 3 0
0 sh
[ 1054.622808] [32166] 0 32166 1737 42 4 0
0 gcc
[ 1054.630524] [32170] 0 32170 2466 232 3 0
0 cc1
[ 1054.638236] [32173] 0 32173 2470 232 4 0
0 cc1
[ 1054.645941] [32176] 0 32176 1736 52 4 0
0 sh
[ 1054.653566] [32178] 0 32178 1737 42 4 0
0 gcc
[ 1054.661276] [32180] 0 32180 2474 313 3 0
0 cc1
[ 1054.668986] [32182] 0 32182 1736 51 3 0
0 sh
[ 1054.676604] [32184] 0 32184 1737 42 3 0
0 gcc
[ 1054.684314] [32185] 0 32185 1736 52 5 0
0 sh
[ 1054.691940] [32186] 0 32186 1737 42 4 0
0 gcc
[ 1054.699650] [32188] 0 32188 2464 199 4 0
0 cc1
[ 1054.707354] [32190] 0 32190 1736 51 4 0
0 sh
[ 1054.714978] [32194] 0 32194 2466 232 4 0
0 cc1
[ 1054.722688] [32195] 0 32195 1737 43 4 0
0 gcc
[ 1054.730397] [32197] 0 32197 1736 13 4 0
0 sh
[ 1054.738015] [32199] 0 32199 2464 199 4 0
0 cc1
[ 1054.745725] [32200] 0 32200 70 34 3 0
0 mv
[ 1054.753349] [32201] 0 32201 1760 79 3 0
0 make
[ 1054.761146] [32202] 0 32202 1909 157 4 0
0 as
[ 1054.768770] Out of memory: Kill process 643 (tuned) score 1 or
sacrifice child
[ 1054.775957] Killed process 643 (tuned) total-vm:436544kB,
anon-rss:2496kB, file-rss:10048kB
[ 1054.828700] cc1 invoked oom-killer: gfp_mask=0x200da, order=0,
oom_score_adj=0
[ 1054.835892] cc1 cpuset=/ mems_allowed=0
[ 1054.839764] CPU: 0 PID: 32125 Comm: cc1 Not tainted 3.19.0-rc3+ #2
[ 1054.845913] Hardware name: APM X-Gene Mustang board (DT)
[ 1054.851209] Call trace:
[ 1054.853651] [<fffffe0000096954>] dump_backtrace+0x0/0x16c
[ 1054.859033] [<fffffe0000096ad0>] show_stack+0x10/0x1c
[ 1054.864060] [<fffffe000062911c>] dump_stack+0x74/0x98
[ 1054.869094] [<fffffe0000626550>] dump_header.isra.11+0x98/0x1d0
[ 1054.874986] [<fffffe000017c4e0>] oom_kill_process+0x3ac/0x408
[ 1054.880710] [<fffffe000017c9d4>] out_of_memory+0x2a4/0x2d4
[ 1054.886169] [<fffffe0000180b78>] __alloc_pages_nodemask+0x808/0x8d8
[ 1054.892413] [<fffffe00001a443c>] handle_mm_fault+0x938/0xd1c
[ 1054.898045] [<fffffe00000a01c8>] do_page_fault+0x214/0x338
[ 1054.903509] [<fffffe000009022c>] do_mem_abort+0x38/0x9c
[ 1054.908713] Exception stack(0xfffffe011be67e30 to 0xfffffe011be67f50)
[ 1054.915122] 7e20: 00000200
00000000 9d060000 000003ff
[ 1054.923266] 7e40: ffffffff ffffffff a4e64f54 000003ff 00b11000
fffffe00 000f7c84 fffffe00
[ 1054.931408] 7e60: 1be67ea0 fffffe01 000903b4 fffffe00 1be67ed0
fffffe01 0002000c fffffc00
[ 1054.939553] 7e80: 00b686b8 fffffe00 00096520 fffffe00 1be67eb0
fffffe01 000964e8 fffffe00
[ 1054.947689] 7ea0: 00000208 00000000 00092f90 fffffe00 cc09e9e0
000003ff 0009308c fffffe00
[ 1054.955832] 7ec0: 00000200 00000000 007a7d08 00000000 9d05ffe0
000003ff 9cf68c58 000003ff
[ 1054.963977] 7ee0: 00000018 00000000 00000003 00000000 00000004
00000004 a4e64fac 000003ff
[ 1054.972119] 7f00: a4e64fac 000003ff 00000000 00000000 00fe8000
00000000 000005a0 00000000
[ 1054.980262] 7f20: 00a46f60 00000000 0000009d 00000000 00000006
00000000 00000018 00000000
[ 1054.988404] 7f40: 00000008 00000000 00000000 00000000
[ 1054.993428] Mem-Info:
[ 1054.995687] DMA per-cpu:
[ 1054.998213] CPU 0: hi: 6, btch: 1 usd: 0
[ 1055.002978] CPU 1: hi: 6, btch: 1 usd: 0
[ 1055.007743] CPU 2: hi: 6, btch: 1 usd: 0
[ 1055.012514] CPU 3: hi: 6, btch: 1 usd: 0
[ 1055.017279] CPU 4: hi: 6, btch: 1 usd: 0
[ 1055.022050] CPU 5: hi: 6, btch: 1 usd: 0
[ 1055.026815] CPU 6: hi: 6, btch: 1 usd: 0
[ 1055.031586] CPU 7: hi: 6, btch: 1 usd: 0
[ 1055.036350] Normal per-cpu:
[ 1055.039133] CPU 0: hi: 6, btch: 1 usd: 0
[ 1055.043898] CPU 1: hi: 6, btch: 1 usd: 0
[ 1055.048668] CPU 2: hi: 6, btch: 1 usd: 0
[ 1055.053432] CPU 3: hi: 6, btch: 1 usd: 0
[ 1055.058203] CPU 4: hi: 6, btch: 1 usd: 0
[ 1055.062969] CPU 5: hi: 6, btch: 1 usd: 0
[ 1055.067733] CPU 6: hi: 6, btch: 1 usd: 0
[ 1055.072504] CPU 7: hi: 6, btch: 1 usd: 0
[ 1055.077274] active_anon:207422 inactive_anon:25737 isolated_anon:0
[ 1055.077274] active_file:1301 inactive_file:21246 isolated_file:0
[ 1055.077274] unevictable:0 dirty:0 writeback:0 unstable:0
[ 1055.077274] free:1013 slab_reclaimable:1047 slab_unreclaimable:1758
[ 1055.077274] mapped:734 shmem:58 pagetables:267 bounce:0
[ 1055.077274] free_cma:0
[ 1055.108845] DMA free:52672kB min:4032kB low:4992kB high:6016kB
active_anon:3026048kB inactive_anon:612608kB active_file:20992kB
inactive_file:323136kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:4192256kB managed:4177024kB mlocked:0kB
dirty:0kB writeback:0kB mapped:15040kB shmem:1792kB
slab_reclaimable:26368kB slab_unreclaimable:36160kB kernel_stack:464kB
pagetables:6976kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
pages_scanned:26608000 all_unreclaimable? yes
[ 1055.151981] lowmem_reserve[]: 0 765 765
[ 1055.155847] Normal free:11904kB min:12224kB low:15232kB high:18304kB
active_anon:10248960kB inactive_anon:1034816kB active_file:62272kB
inactive_file:1036608kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:12582912kB managed:12538176kB mlocked:0kB
dirty:0kB writeback:0kB mapped:31936kB shmem:1920kB
slab_reclaimable:40640kB slab_unreclaimable:76352kB kernel_stack:2992kB
pagetables:10112kB unstable:0kB bounce:0kB free_cma:0kB
writeback_tmp:0kB pages_scanned:82908864 all_unreclaimable? yes
[ 1055.200104] lowmem_reserve[]: 0 0 0
[ 1055.203616] DMA: 109*64kB (UR) 53*128kB (R) 8*256kB (R) 0*512kB
0*1024kB 0*2048kB 1*4096kB (R) 0*8192kB 0*16384kB 1*32768kB (R)
0*65536kB = 52672kB
[ 1055.216957] Normal: 182*64kB (MR) 0*128kB 0*256kB 0*512kB 0*1024kB
0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 11648kB
[ 1055.229048] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=524288kB
[ 1055.237614] 43737 total pagecache pages
[ 1055.241438] 25743 pages in swap cache
[ 1055.245081] Swap cache stats: add 27980, delete 2237, find 2297/2303
[ 1055.251407] Free swap = 6598528kB
[ 1055.254789] Total swap = 8388544kB
[ 1055.258176] 262112 pages RAM
[ 1055.261040] 0 pages HighMem/MovableOnly
[ 1055.264855] 18446744073709544361 pages reserved
[ 1055.269366] 8192 pages cma reserved
[ 1055.272835] [ pid ] uid tgid total_vm rss nr_ptes swapents
oom_score_adj name
[ 1055.280640] [ 481] 0 481 233 89 3 39
0 systemd-journal
[ 1055.289389] [ 487] 0 487 1420 32 3 80
0 lvmetad
[ 1055.297440] [ 512] 0 512 216 40 2 46
-1000 systemd-udevd
[ 1055.306017] [ 598] 0 598 258 45 3 38
-1000 auditd
[ 1055.313989] [ 620] 0 620 6825 213 3 114
0 NetworkManager
[ 1055.322650] [ 623] 0 623 2028 83 3 100
0 abrtd
[ 1055.330534] [ 624] 0 624 2019 78 3 92
0 abrt-watch-log
[ 1055.339195] [ 628] 70 628 87 38 4 34
0 avahi-daemon
[ 1055.347677] [ 629] 998 629 40 13 2 0
0 lsmd
[ 1055.355474] [ 631] 70 631 87 0 4 32
0 avahi-daemon
[ 1055.363963] [ 641] 0 641 2687 48 3 77
0 rsyslogd
[ 1055.372106] [ 642] 0 642 86 42 3 9
0 irqbalance
[ 1055.380423] [ 681] 0 643 6821 210 5 236
0 tuned
[ 1055.388306] [ 647] 0 647 109 44 2 31
0 smartd
[ 1055.396270] [ 651] 997 651 83 45 2 18
0 chronyd
[ 1055.404326] [ 652] 0 652 99 38 3 36
0 systemd-logind
[ 1055.412987] [ 653] 81 653 222 35 2 37
-900 dbus-daemon
[ 1055.421389] [ 667] 0 667 79 27 3 29
0 atd
[ 1055.429100] [ 675] 0 675 1846 68 4 86
0 login
[ 1055.436978] [ 676] 0 676 1717 23 5 10
0 agetty
[ 1055.444948] [ 710] 0 710 1761 47 3 36
0 bash
[ 1055.452752] [ 727] 0 727 240 84 2 74
-1000 sshd
[ 1055.460553] [ 733] 0 733 1739 34 3 9
0 rhsmcertd
[ 1055.468784] [ 741] 999 741 7276 118 4 122
0 polkitd
[ 1055.476834] [ 747] 0 747 502 102 3 281
0 dhclient
[ 1055.484977] [ 1323] 0 1323 334 69 3 69
0 master
[ 1055.492947] [ 1325] 89 1325 337 82 2 78
0 qmgr
[ 1055.500746] [ 1328] 0 1328 358 130 3 111
0 sshd
[ 1055.508544] [ 1331] 0 1331 1762 48 5 39
0 bash
[ 1055.516335] [ 1373] 89 1373 336 89 2 69
0 pickup
[ 1055.524305] [ 2944] 0 2944 1734 50 4 3
0 make
[ 1055.532102] [31425] 0 31425 1760 79 3 0
0 make
[ 1055.539900] [32092] 0 32092 1736 51 4 0
0 sh
[ 1055.547518] [32103] 0 32103 1736 51 4 0
0 sh
[ 1055.555142] [32108] 0 32108 1737 42 4 0
0 gcc
[ 1055.562852] [32110] 0 32110 1736 52 4 0
0 sh
[ 1055.570477] [32111] 0 32111 2479 373 4 0
0 cc1
[ 1055.578187] [32118] 0 32118 1736 52 4 0
0 sh
[ 1055.585805] [32120] 0 32120 1737 42 4 0
0 gcc
[ 1055.593515] [32123] 0 32123 1736 51 4 0
0 sh
[ 1055.601137] [32125] 0 32125 2479 373 4 0
0 cc1
[ 1055.608848] [32127] 0 32127 1737 43 4 0
0 gcc
[ 1055.616552] [32132] 0 32132 1736 52 4 0
0 sh
[ 1055.624177] [32134] 0 32134 1737 42 5 0
0 gcc
[ 1055.631890] [32135] 0 32135 2472 304 4 0
0 cc1
[ 1055.639601] [32136] 0 32136 2474 314 4 0
0 cc1
[ 1055.647305] [32140] 0 32140 1736 52 3 0
0 sh
[ 1055.654930] [32141] 0 32141 1737 42 4 0
0 gcc
[ 1055.662640] [32142] 0 32142 1736 51 3 0
0 sh
[ 1055.670264] [32146] 0 32146 1737 42 4 0
0 gcc
[ 1055.677968] [32150] 0 32150 1736 52 4 0
0 sh
[ 1055.685591] [32151] 0 32151 2474 314 3 0
0 cc1
[ 1055.693302] [32154] 0 32154 1737 42 4 0
0 gcc
[ 1055.701012] [32157] 0 32157 1736 51 3 0
0 sh
[ 1055.708636] [32158] 0 32158 2479 373 4 0
0 cc1
[ 1055.716340] [32164] 0 32164 1737 43 5 0
0 gcc
[ 1055.724051] [32165] 0 32165 1736 52 3 0
0 sh
[ 1055.731673] [32166] 0 32166 1737 42 4 0
0 gcc
[ 1055.739383] [32170] 0 32170 2466 232 3 0
0 cc1
[ 1055.747087] [32173] 0 32173 2470 232 4 0
0 cc1
[ 1055.754799] [32176] 0 32176 1736 52 4 0
0 sh
[ 1055.762424] [32178] 0 32178 1737 42 4 0
0 gcc
[ 1055.770134] [32180] 0 32180 2474 313 3 0
0 cc1
[ 1055.777839] [32182] 0 32182 1736 51 3 0
0 sh
[ 1055.785462] [32184] 0 32184 1737 42 3 0
0 gcc
[ 1055.793172] [32185] 0 32185 1736 52 5 0
0 sh
[ 1055.800795] [32186] 0 32186 1737 42 4 0
0 gcc
[ 1055.808505] [32188] 0 32188 2464 199 4 0
0 cc1
[ 1055.816209] [32190] 0 32190 1736 51 4 0
0 sh
[ 1055.823833] [32194] 0 32194 2466 232 4 0
0 cc1
[ 1055.831544] [32195] 0 32195 1737 43 4 0
0 gcc
[ 1055.839254] [32197] 0 32197 1736 13 4 0
0 sh
[ 1055.846872] [32199] 0 32199 2464 199 4 0
0 cc1
[ 1055.854582] [32200] 0 32200 70 34 3 0
0 mv
[ 1055.862207] [32201] 0 32201 1760 79 3 0
0 make
[ 1055.870002] [32202] 0 32202 1909 157 4 0
0 as
[ 1055.877620] Out of memory: Kill process 681 (tuned) score 1 or
sacrifice child
[ 1055.884812] Killed process 681 (tuned) total-vm:436544kB,
anon-rss:3200kB, file-rss:10240kB
\aMessage from syslogd at localhost at Jan 8 07:46:37 ...
kernel:Call trace:
\aMessage from syslogd at localhost at Jan 8 07:46:37 ...
kernel:Call trace:
[ 1057.551522] cc1 invoked oom-killer: gfp_mask=0x200da, order=0,
oom_score_adj=0
[ 1057.558741] cc1 cpuset=/ mems_allowed=0
[ 1057.562589] CPU: 3 PID: 32236 Comm: cc1 Not tainted 3.19.0-rc3+ #2
[ 1057.568761] Hardware name: APM X-Gene Mustang board (DT)
[ 1057.574049] Call trace:
[ 1057.576494] [<fffffe0000096954>] dump_backtrace+0x0/0x16c
[ 1057.581888] [<fffffe0000096ad0>] show_stack+0x10/0x1c
[ 1057.586927] [<fffffe000062911c>] dump_stack+0x74/0x98
[ 1057.591973] [<fffffe0000626550>] dump_header.isra.11+0x98/0x1d0
[ 1057.597869] [<fffffe000017c4e0>] oom_kill_process+0x3ac/0x408
[ 1057.603597] [<fffffe000017c9d4>] out_of_memory+0x2a4/0x2d4
[ 1057.609070] [<fffffe0000180b78>] __alloc_pages_nodemask+0x808/0x8d8
[ 1057.615310] [<fffffe00001a1fc8>] do_wp_page+0x444/0x904
[ 1057.620519] [<fffffe00001a3fc8>] handle_mm_fault+0x4c4/0xd1c
[ 1057.626152] [<fffffe00000a01c8>] do_page_fault+0x214/0x338
[ 1057.631625] [<fffffe000009022c>] do_mem_abort+0x38/0x9c
[ 1057.636825] Exception stack(0xfffffe00419e3e30 to 0xfffffe00419e3f50)
[ 1057.643241] 3e20: 00000200
00000000 1ffe5c60 00000000
[ 1057.651385] 3e40: ffffffff ffffffff 006f0560 00000000 00b11000
fffffe00 000f7c84 fffffe00
[ 1057.659528] 3e60: 419e3ea0 fffffe00 000903b4 fffffe00 419e3ed0
fffffe00 0002000c fffffc00
[ 1057.667664] 3e80: 00b686b8 fffffe00 af976004 000003ff 419e3eb0
fffffe00 000964e8 fffffe00
[ 1057.675805] 3ea0: 00000208 00000000 00092f90 fffffe00 e1ae3770
000003ff 0009308c fffffe00
[ 1057.683947] 3ec0: 00000200 00000000 00093094 fffffe00 20006cf0
00000000 20006ce0 00000000
[ 1057.692092] 3ee0: a727db78 000003ff a727db77 000003ff fffffffe
00000000 00000000 00000000
[ 1057.700234] 3f00: 00007fff 00000000 00004000 00000000 4c9abfcd
00000000 00003fcd 00000000
[ 1057.708374] 3f20: 00003fcd 00000000 0000bf97 00000000 00007ffe
00000000 00000000 00000000
[ 1057.716510] 3f40: 1fffffff 00000000 0000ddd0 00000000
[ 1057.721539] Mem-Info:
[ 1057.723800] DMA per-cpu:
[ 1057.726319] CPU 0: hi: 6, btch: 1 usd: 0
[ 1057.731089] CPU 1: hi: 6, btch: 1 usd: 0
[ 1057.735855] CPU 2: hi: 6, btch: 1 usd: 0
[ 1057.740625] CPU 3: hi: 6, btch: 1 usd: 0
[ 1057.745389] CPU 4: hi: 6, btch: 1 usd: 0
[ 1057.750159] CPU 5: hi: 6, btch: 1 usd: 0
[ 1057.754924] CPU 6: hi: 6, btch: 1 usd: 0
[ 1057.759693] CPU 7: hi: 6, btch: 1 usd: 0
[ 1057.764458] Normal per-cpu:
[ 1057.767235] CPU 0: hi: 6, btch: 1 usd: 0
[ 1057.772004] CPU 1: hi: 6, btch: 1 usd: 0
[ 1057.776769] CPU 2: hi: 6, btch: 1 usd: 0
[ 1057.781538] CPU 3: hi: 6, btch: 1 usd: 0
[ 1057.786302] CPU 4: hi: 6, btch: 1 usd: 0
[ 1057.791072] CPU 5: hi: 6, btch: 1 usd: 0
[ 1057.795837] CPU 6: hi: 6, btch: 1 usd: 0
[ 1057.800606] CPU 7: hi: 6, btch: 1 usd: 0
[ 1057.805375] active_anon:207677 inactive_anon:25723 isolated_anon:0
[ 1057.805375] active_file:1246 inactive_file:21239 isolated_file:0
[ 1057.805375] unevictable:0 dirty:0 writeback:0 unstable:0
[ 1057.805375] free:850 slab_reclaimable:1045 slab_unreclaimable:1748
[ 1057.805375] mapped:667 shmem:63 pagetables:232 bounce:0
[ 1057.805375] free_cma:6
[ 1057.836860] DMA free:48576kB min:4032kB low:4992kB high:6016kB
active_anon:3035200kB inactive_anon:612608kB active_file:17856kB
inactive_file:323136kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:4192256kB managed:4177024kB mlocked:0kB
dirty:0kB writeback:0kB mapped:11520kB shmem:2048kB
slab_reclaimable:26304kB slab_unreclaimable:36096kB kernel_stack:352kB
pagetables:5120kB unstable:0kB bounce:0kB free_cma:384kB
writeback_tmp:0kB pages_scanned:25701184 all_unreclaimable? yes
[ 1057.880173] lowmem_reserve[]: 0 765 765
[ 1057.884043] Normal free:5824kB min:12224kB low:15232kB high:18304kB
active_anon:10256128kB inactive_anon:1033664kB active_file:61888kB
inactive_file:1036160kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:12582912kB managed:12538176kB mlocked:0kB
dirty:0kB writeback:0kB mapped:31168kB shmem:1984kB
slab_reclaimable:40576kB slab_unreclaimable:75776kB kernel_stack:3072kB
pagetables:9728kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
pages_scanned:80137088 all_unreclaimable? yes
[ 1057.928132] lowmem_reserve[]: 0 0 0
[ 1057.931662] DMA: 55*64kB (UMRC) 49*128kB (R) 8*256kB (R) 0*512kB
0*1024kB 0*2048kB 1*4096kB (R) 0*8192kB 0*16384kB 1*32768kB (R)
0*65536kB = 48704kB
[ 1057.945093] Normal: 62*64kB (MR) 9*128kB (R) 4*256kB (R) 0*512kB
0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 6144kB
[ 1057.957726] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=524288kB
[ 1057.966305] 43758 total pagecache pages
[ 1057.970133] 25806 pages in swap cache
[ 1057.973778] Swap cache stats: add 28140, delete 2334, find 2447/2608
[ 1057.980108] Free swap = 6619264kB
[ 1057.983491] Total swap = 8388544kB
[ 1057.986874] 262112 pages RAM
[ 1057.989750] 0 pages HighMem/MovableOnly
[ 1057.993567] 18446744073709544361 pages reserved
[ 1057.998072] 8192 pages cma reserved
[ 1058.001550] [ pid ] uid tgid total_vm rss nr_ptes swapents
oom_score_adj name
[ 1058.009370] [ 481] 0 481 233 106 3 26
0 systemd-journal
[ 1058.018115] [ 487] 0 487 1420 32 3 80
0 lvmetad
[ 1058.026179] [ 512] 0 512 216 40 2 46
-1000 systemd-udevd
[ 1058.034756] [ 598] 0 598 258 45 3 38
-1000 auditd
[ 1058.042728] [ 620] 0 620 6825 224 3 106
0 NetworkManager
[ 1058.051400] [ 623] 0 623 2028 83 3 100
0 abrtd
[ 1058.059289] [ 624] 0 624 2019 78 3 92
0 abrt-watch-log
[ 1058.067944] [ 628] 70 628 87 38 4 34
0 avahi-daemon
[ 1058.076440] [ 629] 998 629 40 13 2 0
0 lsmd
[ 1058.084238] [ 631] 70 631 87 0 4 32
0 avahi-daemon
[ 1058.092736] [ 641] 0 641 2687 82 3 56
0 rsyslogd
[ 1058.100881] [ 642] 0 642 86 42 3 9
0 irqbalance
[ 1058.109202] [ 647] 0 647 109 44 2 31
0 smartd
[ 1058.117169] [ 651] 997 651 83 45 2 18
0 chronyd
[ 1058.125230] [ 652] 0 652 99 38 3 36
0 systemd-logind
[ 1058.133900] [ 653] 81 653 222 53 2 21
-900 dbus-daemon
[ 1058.142303] [ 667] 0 667 79 27 3 29
0 atd
[ 1058.150020] [ 675] 0 675 1846 68 4 86
0 login
[ 1058.157900] [ 676] 0 676 1717 23 5 10
0 agetty
[ 1058.165874] [ 710] 0 710 1761 47 3 36
0 bash
[ 1058.173679] [ 727] 0 727 240 84 2 74
-1000 sshd
[ 1058.181476] [ 733] 0 733 1739 34 3 9
0 rhsmcertd
[ 1058.189712] [ 741] 999 741 7276 132 4 105
0 polkitd
[ 1058.197764] [ 747] 0 747 502 102 3 281
0 dhclient
[ 1058.205912] [ 1323] 0 1323 334 69 3 69
0 master
[ 1058.213891] [ 1325] 89 1325 337 82 2 78
0 qmgr
[ 1058.221691] [ 1328] 0 1328 358 130 3 111
0 sshd
[ 1058.229495] [ 1331] 0 1331 1762 48 5 39
0 bash
[ 1058.237289] [ 1373] 89 1373 336 89 2 69
0 pickup
[ 1058.245264] [ 2944] 0 2944 1734 50 4 3
0 make
[ 1058.253069] [31425] 0 31425 1760 79 3 0
0 make
[ 1058.260869] [32190] 0 32190 1736 51 4 0
0 sh
[ 1058.268493] [32195] 0 32195 1737 43 4 0
0 gcc
[ 1058.276199] [32199] 0 32199 2479 382 4 0
0 cc1
[ 1058.283915] [32224] 0 32224 1736 51 4 0
0 sh
[ 1058.291548] [32230] 0 32230 1737 42 4 0
0 gcc
[ 1058.299264] [32232] 0 32232 1736 51 3 0
0 sh
[ 1058.306883] [32236] 0 32236 2479 314 4 0
0 cc1
[ 1058.314594] [32237] 0 32237 1736 52 5 0
0 sh
[ 1058.322217] [32239] 0 32239 1737 42 5 0
0 gcc
[ 1058.329934] [32241] 0 32241 1737 43 5 0
0 gcc
[ 1058.337642] [32245] 0 32245 1736 51 4 0
0 sh
[ 1058.345271] [32246] 0 32246 2474 313 4 0
0 cc1
[ 1058.352992] [32248] 0 32248 1737 42 4 0
0 gcc
[ 1058.360704] [32249] 0 32249 2479 373 3 0
0 cc1
[ 1058.368414] [32250] 0 32250 1736 52 3 0
0 sh
[ 1058.376032] [32252] 0 32252 1737 43 4 0
0 gcc
[ 1058.383750] [32253] 0 32253 1736 51 5 0
0 sh
[ 1058.391376] [32254] 0 32254 2472 303 4 0
0 cc1
[ 1058.399087] [32257] 0 32257 2473 296 3 0
0 cc1
[ 1058.406791] [32258] 0 32258 1737 42 4 0
0 gcc
[ 1058.414501] [32260] 0 32260 1736 51 3 0
0 sh
[ 1058.422132] [32261] 0 32261 1737 42 3 0
0 gcc
[ 1058.429844] [32264] 0 32264 2470 232 3 0
0 cc1
[ 1058.437549] [32268] 0 32268 2479 373 3 0
0 cc1
[ 1058.445259] [32273] 0 32273 1736 51 5 0
0 sh
[ 1058.452890] [32276] 0 32276 1737 43 4 0
0 gcc
[ 1058.460602] [32278] 0 32278 1736 51 4 0
0 sh
[ 1058.468220] [32280] 0 32280 1737 43 4 0
0 gcc
[ 1058.475930] [32282] 0 32282 2470 296 4 0
0 cc1
[ 1058.483652] [32286] 0 32286 1736 52 4 0
0 sh
[ 1058.491278] [32287] 0 32287 2474 313 5 0
0 cc1
[ 1058.498992] [32291] 0 32291 1737 42 3 0
0 gcc
[ 1058.506691] [32294] 0 32294 2470 233 4 0
0 cc1
[ 1058.514405] [32299] 0 32299 1736 51 3 0
0 sh
[ 1058.522036] [32300] 0 32300 1737 42 3 0
0 gcc
[ 1058.529747] [32301] 0 32301 1737 42 3 0
0 gcc
[ 1058.537452] [32302] 0 32302 1760 79 3 0
0 make
[ 1058.545255] Out of memory: Kill process 747 (dhclient) score 0 or
sacrifice child
[ 1058.552708] Killed process 747 (dhclient) total-vm:32128kB,
anon-rss:192kB, file-rss:6336kB
[ 1059.379317] NetworkManager invoked oom-killer: gfp_mask=0xd0,
order=0, oom_score_adj=0
[ 1059.387206] NetworkManager cpuset=/ mems_allowed=0
[ 1059.392022] CPU: 6 PID: 620 Comm: NetworkManager Not tainted
3.19.0-rc3+ #2
[ 1059.398964] Hardware name: APM X-Gene Mustang board (DT)
[ 1059.404251] Call trace:
[ 1059.406694] [<fffffe0000096954>] dump_backtrace+0x0/0x16c
[ 1059.412088] [<fffffe0000096ad0>] show_stack+0x10/0x1c
[ 1059.417121] [<fffffe000062911c>] dump_stack+0x74/0x98
[ 1059.422162] [<fffffe0000626550>] dump_header.isra.11+0x98/0x1d0
[ 1059.428057] [<fffffe000017c4e0>] oom_kill_process+0x3ac/0x408
[ 1059.433788] [<fffffe000017c9d4>] out_of_memory+0x2a4/0x2d4
[ 1059.439264] [<fffffe0000180b78>] __alloc_pages_nodemask+0x808/0x8d8
[ 1059.445504] [<fffffe0000180c5c>] __get_free_pages+0x14/0x48
[ 1059.451063] [<fffffe00001e8454>] __pollwait+0x4c/0xf4
[ 1059.456091] [<fffffe00004ff568>] datagram_poll+0x24/0x11c
[ 1059.461479] [<fffffe0000535a70>] netlink_poll+0xac/0x180
[ 1059.466767] [<fffffe00004ef3dc>] sock_poll+0xf8/0x110
[ 1059.471800] [<fffffe00001e97b0>] do_sys_poll+0x220/0x484
[ 1059.477085] [<fffffe00001e9d24>] SyS_ppoll+0x1c4/0x1dc
[ 1059.482213] Mem-Info:
[ 1059.484476] DMA per-cpu:
[ 1059.486996] CPU 0: hi: 6, btch: 1 usd: 0
[ 1059.491767] CPU 1: hi: 6, btch: 1 usd: 0
[ 1059.496532] CPU 2: hi: 6, btch: 1 usd: 0
[ 1059.501301] CPU 3: hi: 6, btch: 1 usd: 0
[ 1059.506066] CPU 4: hi: 6, btch: 1 usd: 0
[ 1059.510842] CPU 5: hi: 6, btch: 1 usd: 0
[ 1059.515609] CPU 6: hi: 6, btch: 1 usd: 0
[ 1059.520383] CPU 7: hi: 6, btch: 1 usd: 0
[ 1059.525149] Normal per-cpu:
[ 1059.527927] CPU 0: hi: 6, btch: 1 usd: 0
[ 1059.532706] CPU 1: hi: 6, btch: 1 usd: 0
[ 1059.537471] CPU 2: hi: 6, btch: 1 usd: 0
[ 1059.542242] CPU 3: hi: 6, btch: 1 usd: 0
[ 1059.547006] CPU 4: hi: 6, btch: 1 usd: 0
[ 1059.551783] CPU 5: hi: 6, btch: 1 usd: 0
[ 1059.556549] CPU 6: hi: 6, btch: 1 usd: 0
[ 1059.561321] CPU 7: hi: 6, btch: 1 usd: 1
[ 1059.566092] active_anon:207642 inactive_anon:25756 isolated_anon:0
[ 1059.566092] active_file:1244 inactive_file:21227 isolated_file:0
[ 1059.566092] unevictable:0 dirty:0 writeback:0 unstable:0
[ 1059.566092] free:864 slab_reclaimable:1045 slab_unreclaimable:1748
[ 1059.566092] mapped:642 shmem:63 pagetables:229 bounce:0
[ 1059.566092] free_cma:6
[ 1059.597581] DMA free:48768kB min:4032kB low:4992kB high:6016kB
active_anon:3035200kB inactive_anon:612608kB active_file:17792kB
inactive_file:322944kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:4192256kB managed:4177024kB mlocked:0kB
dirty:0kB writeback:0kB mapped:11520kB shmem:2048kB
slab_reclaimable:26304kB slab_unreclaimable:36096kB kernel_stack:336kB
pagetables:5056kB unstable:0kB bounce:0kB free_cma:384kB
writeback_tmp:0kB pages_scanned:24814080 all_unreclaimable? yes
[ 1059.640895] lowmem_reserve[]: 0 765 765
[ 1059.644766] Normal free:6528kB min:12224kB low:15232kB high:18304kB
active_anon:10253888kB inactive_anon:1035776kB active_file:61824kB
inactive_file:1035584kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:12582912kB managed:12538176kB mlocked:0kB
dirty:0kB writeback:0kB mapped:29568kB shmem:1984kB
slab_reclaimable:40576kB slab_unreclaimable:75776kB kernel_stack:2992kB
pagetables:9600kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
pages_scanned:79671936 all_unreclaimable? yes
[ 1059.688856] lowmem_reserve[]: 0 0 0
[ 1059.692375] DMA: 57*64kB (UMRC) 49*128kB (R) 8*256kB (R) 0*512kB
0*1024kB 0*2048kB 1*4096kB (R) 0*8192kB 0*16384kB 1*32768kB (R)
0*65536kB = 48832kB
[ 1059.705808] Normal: 68*64kB (MR) 9*128kB (R) 4*256kB (R) 0*512kB
0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 6528kB
[ 1059.718443] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=524288kB
[ 1059.727021] 43784 total pagecache pages
[ 1059.730850] 25838 pages in swap cache
[ 1059.734493] Swap cache stats: add 28173, delete 2335, find 2447/2609
[ 1059.740820] Free swap = 6635136kB
[ 1059.744204] Total swap = 8388544kB
[ 1059.747587] 262112 pages RAM
[ 1059.750463] 0 pages HighMem/MovableOnly
[ 1059.754280] 18446744073709544361 pages reserved
[ 1059.758794] 8192 pages cma reserved
[ 1059.762265] [ pid ] uid tgid total_vm rss nr_ptes swapents
oom_score_adj name
[ 1059.770084] [ 481] 0 481 233 106 3 26
0 systemd-journal
[ 1059.778838] [ 487] 0 487 1420 32 3 80
0 lvmetad
[ 1059.786890] [ 512] 0 512 216 40 2 46
-1000 systemd-udevd
[ 1059.795474] [ 598] 0 598 258 45 3 38
-1000 auditd
[ 1059.803446] [ 620] 0 620 6825 224 3 106
0 NetworkManager
[ 1059.812116] [ 623] 0 623 2028 83 3 100
0 abrtd
[ 1059.820004] [ 624] 0 624 2019 78 3 92
0 abrt-watch-log
[ 1059.828667] [ 628] 70 628 87 38 4 34
0 avahi-daemon
[ 1059.837151] [ 629] 998 629 40 13 2 0
0 lsmd
[ 1059.844954] [ 631] 70 631 87 0 4 32
0 avahi-daemon
[ 1059.853450] [ 641] 0 641 2687 82 3 56
0 rsyslogd
[ 1059.861594] [ 642] 0 642 86 42 3 9
0 irqbalance
[ 1059.869918] [ 647] 0 647 109 44 2 31
0 smartd
[ 1059.877885] [ 651] 997 651 83 45 2 18
0 chronyd
[ 1059.885945] [ 652] 0 652 99 38 3 36
0 systemd-logind
[ 1059.894614] [ 653] 81 653 222 53 2 21
-900 dbus-daemon
[ 1059.903021] [ 667] 0 667 79 27 3 29
0 atd
[ 1059.910740] [ 675] 0 675 1846 68 4 86
0 login
[ 1059.918629] [ 676] 0 676 1717 23 5 10
0 agetty
[ 1059.926594] [ 710] 0 710 1761 47 3 36
0 bash
[ 1059.934392] [ 727] 0 727 240 84 2 74
-1000 sshd
[ 1059.942197] [ 733] 0 733 1739 34 3 9
0 rhsmcertd
[ 1059.950427] [ 741] 999 741 7276 132 4 105
0 polkitd
[ 1059.958478] [ 1323] 0 1323 334 69 3 69
0 master
[ 1059.966457] [ 1325] 89 1325 337 82 2 78
0 qmgr
[ 1059.974256] [ 1328] 0 1328 358 130 3 111
0 sshd
[ 1059.982061] [ 1331] 0 1331 1762 48 5 39
0 bash
[ 1059.989859] [ 1373] 89 1373 336 89 2 69
0 pickup
[ 1059.997824] [ 2944] 0 2944 1734 50 4 3
0 make
[ 1060.005622] [31425] 0 31425 1760 79 3 0
0 make
[ 1060.013426] [32190] 0 32190 1736 51 4 0
0 sh
[ 1060.021052] [32195] 0 32195 1737 43 4 0
0 gcc
[ 1060.028768] [32199] 0 32199 2479 382 4 0
0 cc1
[ 1060.036475] [32224] 0 32224 1736 51 4 0
0 sh
[ 1060.044104] [32230] 0 32230 1737 42 4 0
0 gcc
[ 1060.051824] [32232] 0 32232 1736 51 3 0
0 sh
[ 1060.059450] [32236] 0 32236 2479 314 4 0
0 cc1
[ 1060.067155] [32237] 0 32237 1736 52 5 0
0 sh
[ 1060.074786] [32239] 0 32239 1737 42 5 0
0 gcc
[ 1060.082496] [32241] 0 32241 1737 43 5 0
0 gcc
[ 1060.090215] [32245] 0 32245 1736 51 4 0
0 sh
[ 1060.097836] [32246] 0 32246 2474 313 4 0
0 cc1
[ 1060.105550] [32248] 0 32248 1737 42 4 0
0 gcc
[ 1060.113274] [32249] 0 32249 2479 373 3 0
0 cc1
[ 1060.120986] [32250] 0 32250 1736 52 3 0
0 sh
[ 1060.128610] [32252] 0 32252 1737 43 4 0
0 gcc
[ 1060.136316] [32253] 0 32253 1736 51 5 0
0 sh
[ 1060.143945] [32254] 0 32254 2472 303 4 0
0 cc1
[ 1060.151664] [32257] 0 32257 2473 296 3 0
0 cc1
[ 1060.159383] [32258] 0 32258 1737 42 4 0
0 gcc
[ 1060.167088] [32260] 0 32260 1736 51 3 0
0 sh
[ 1060.174720] [32261] 0 32261 1737 42 3 0
0 gcc
[ 1060.182432] [32264] 0 32264 2470 232 3 0
0 cc1
[ 1060.190149] [32268] 0 32268 2479 373 3 0
0 cc1
[ 1060.197856] [32273] 0 32273 1736 51 5 0
0 sh
[ 1060.205484] [32276] 0 32276 1737 43 4 0
0 gcc
[ 1060.213202] [32278] 0 32278 1736 51 4 0
0 sh
[ 1060.220827] [32280] 0 32280 1737 43 4 0
0 gcc
[ 1060.228532] [32282] 0 32282 2470 296 4 0
0 cc1
[ 1060.236248] [32286] 0 32286 1736 52 4 0
0 sh
[ 1060.243874] [32287] 0 32287 2474 313 5 0
0 cc1
[ 1060.251591] [32291] 0 32291 1737 42 3 0
0 gcc
[ 1060.259303] [32294] 0 32294 2470 233 4 0
0 cc1
[ 1060.267008] [32299] 0 32299 1736 51 3 0
0 sh
[ 1060.274638] [32300] 0 32300 1737 42 3 0
0 gcc
[ 1060.282349] [32301] 0 32301 1737 42 3 0
0 gcc
[ 1060.290072] [32302] 0 32302 1760 79 3 0
0 make
[ 1060.297866] Out of memory: Kill process 32199 (cc1) score 0 or
sacrifice child
[ 1060.305063] Killed process 32199 (cc1) total-vm:158656kB,
anon-rss:14912kB, file-rss:9536kB
\aMessage from syslogd at localhost at Jan 8 07:46:41 ...
kernel:Call trace:
\aMessage from syslogd at localhost at Jan 8 07:46:41 ...
kernel:Call trace:
[ 1061.074807] cc1 invoked oom-killer: gfp_mask=0x200da, order=0,
oom_score_adj=0
[ 1061.082033] cc1 cpuset=/ mems_allowed=0
[ 1061.085881] CPU: 3 PID: 32257 Comm: cc1 Not tainted 3.19.0-rc3+ #2
[ 1061.092051] Hardware name: APM X-Gene Mustang board (DT)
[ 1061.097339] Call trace:
[ 1061.099794] [<fffffe0000096954>] dump_backtrace+0x0/0x16c
[ 1061.105169] [<fffffe0000096ad0>] show_stack+0x10/0x1c
[ 1061.110205] [<fffffe000062911c>] dump_stack+0x74/0x98
[ 1061.115233] [<fffffe0000626550>] dump_header.isra.11+0x98/0x1d0
[ 1061.121131] [<fffffe000017c4e0>] oom_kill_process+0x3ac/0x408
[ 1061.126848] [<fffffe000017c9d4>] out_of_memory+0x2a4/0x2d4
[ 1061.132314] [<fffffe0000180b78>] __alloc_pages_nodemask+0x808/0x8d8
[ 1061.138550] [<fffffe00001a443c>] handle_mm_fault+0x938/0xd1c
[ 1061.144188] [<fffffe00000a01c8>] do_page_fault+0x214/0x338
[ 1061.149652] [<fffffe000009022c>] do_mem_abort+0x38/0x9c
[ 1061.154849] Exception stack(0xfffffe011be2fe30 to 0xfffffe011be2ff50)
[ 1061.161264] fe20: 00000200
00000000 00f886a8 00000000
[ 1061.169407] fe40: ffffffff ffffffff 969b4620 000003ff 0000011a
00000000 00000015 00000000
[ 1061.177543] fe60: 1be2fea0 fffffe01 00000000 00000000 1be2fe80
fffffe01 000ce950 fffffe00
[ 1061.185685] fe80: 1be2feb0 fffffe01 00096520 fffffe00 00000204
00000000 00000d7e 00000000
[ 1061.193826] fea0: ffffffff ffffffff 969fc628 000003ff ed276070
000003ff 0009308c fffffe00
[ 1061.201968] fec0: 00000200 00000000 000931cc fffffe00 8e7c0000
000003ff 00000000 00000000
[ 1061.210112] fee0: 00000248 00000000 8e7c0000 000003ff 00000000
00000000 8e7c0000 000003ff
[ 1061.218248] ff00: 00000009 00000000 00000009 00000000 00fe8000
00000000 000003a8 00000000
[ 1061.226390] ff20: 006528e0 00000000 0000008e 00000000 00000006
00000000 00000018 00000000
[ 1061.234531] ff40: 00000008 00000000 00000000 00000000
[ 1061.239560] Mem-Info:
[ 1061.241820] DMA per-cpu:
[ 1061.244339] CPU 0: hi: 6, btch: 1 usd: 0
[ 1061.249110] CPU 1: hi: 6, btch: 1 usd: 0
[ 1061.253875] CPU 2: hi: 6, btch: 1 usd: 0
[ 1061.258644] CPU 3: hi: 6, btch: 1 usd: 0
[ 1061.263409] CPU 4: hi: 6, btch: 1 usd: 0
[ 1061.268174] CPU 5: hi: 6, btch: 1 usd: 0
[ 1061.272944] CPU 6: hi: 6, btch: 1 usd: 0
[ 1061.277709] CPU 7: hi: 6, btch: 1 usd: 0
[ 1061.282479] Normal per-cpu:
[ 1061.285257] CPU 0: hi: 6, btch: 1 usd: 0
[ 1061.290027] CPU 1: hi: 6, btch: 1 usd: 0
[ 1061.294792] CPU 2: hi: 6, btch: 1 usd: 0
[ 1061.299562] CPU 3: hi: 6, btch: 1 usd: 0
[ 1061.304327] CPU 4: hi: 6, btch: 1 usd: 0
[ 1061.309096] CPU 5: hi: 6, btch: 1 usd: 0
[ 1061.313861] CPU 6: hi: 6, btch: 1 usd: 0
[ 1061.318625] CPU 7: hi: 6, btch: 1 usd: 0
[ 1061.323400] active_anon:207507 inactive_anon:25760 isolated_anon:0
[ 1061.323400] active_file:1223 inactive_file:21249 isolated_file:0
[ 1061.323400] unevictable:0 dirty:0 writeback:0 unstable:0
[ 1061.323400] free:1016 slab_reclaimable:1044 slab_unreclaimable:1748
[ 1061.323400] mapped:652 shmem:65 pagetables:229 bounce:0
[ 1061.323400] free_cma:1
[ 1061.354973] DMA free:52992kB min:4032kB low:4992kB high:6016kB
active_anon:3030912kB inactive_anon:612608kB active_file:17920kB
inactive_file:323328kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:4192256kB managed:4177024kB mlocked:0kB
dirty:0kB writeback:0kB mapped:11648kB shmem:2112kB
slab_reclaimable:26240kB slab_unreclaimable:36096kB kernel_stack:336kB
pagetables:5056kB unstable:0kB bounce:0kB free_cma:64kB
writeback_tmp:0kB pages_scanned:24781760 all_unreclaimable? yes
[ 1061.398197] lowmem_reserve[]: 0 765 765
[ 1061.402072] Normal free:12032kB min:12224kB low:15232kB high:18304kB
active_anon:10249536kB inactive_anon:1036032kB active_file:60352kB
inactive_file:1036608kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:12582912kB managed:12538176kB mlocked:0kB
dirty:0kB writeback:0kB mapped:30080kB shmem:2048kB
slab_reclaimable:40576kB slab_unreclaimable:75776kB kernel_stack:2992kB
pagetables:9600kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
pages_scanned:76284480 all_unreclaimable? yes
[ 1061.446244] lowmem_reserve[]: 0 0 0
[ 1061.449762] DMA: 122*64kB (UEMR) 49*128kB (R) 8*256kB (R) 0*512kB
0*1024kB 0*2048kB 1*4096kB (R) 0*8192kB 0*16384kB 1*32768kB (R)
0*65536kB = 52992kB
[ 1061.463271] Normal: 134*64kB (MR) 14*128kB (MR) 5*256kB (R) 1*512kB
(M) 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB =
12160kB
[ 1061.476597] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=524288kB
[ 1061.485172] 43768 total pagecache pages
[ 1061.488993] 25841 pages in swap cache
[ 1061.492636] Swap cache stats: add 28188, delete 2347, find 2460/2635
[ 1061.498962] Free swap = 6635968kB
[ 1061.502345] Total swap = 8388544kB
[ 1061.505729] 262112 pages RAM
[ 1061.508592] 0 pages HighMem/MovableOnly
[ 1061.512412] 18446744073709544361 pages reserved
[ 1061.516917] 8192 pages cma reserved
[ 1061.520390] [ pid ] uid tgid total_vm rss nr_ptes swapents
oom_score_adj name
[ 1061.528192] [ 481] 0 481 234 111 3 26
0 systemd-journal
[ 1061.536940] [ 487] 0 487 1420 32 3 80
0 lvmetad
[ 1061.544998] [ 512] 0 512 216 40 2 46
-1000 systemd-udevd
[ 1061.553575] [ 598] 0 598 258 45 3 38
-1000 auditd
[ 1061.561545] [ 620] 0 620 6825 224 3 106
0 NetworkManager
[ 1061.570206] [ 623] 0 623 2028 83 3 100
0 abrtd
[ 1061.578083] [ 624] 0 624 2019 87 3 84
0 abrt-watch-log
[ 1061.586744] [ 628] 70 628 87 38 4 34
0 avahi-daemon
[ 1061.595232] [ 629] 998 629 40 13 2 0
0 lsmd
[ 1061.603030] [ 631] 70 631 87 0 4 32
0 avahi-daemon
[ 1061.611520] [ 641] 0 641 2687 82 3 56
0 rsyslogd
[ 1061.619662] [ 642] 0 642 86 42 3 9
0 irqbalance
[ 1061.627972] [ 647] 0 647 109 44 2 31
0 smartd
[ 1061.635942] [ 651] 997 651 83 45 2 18
0 chronyd
[ 1061.643997] [ 652] 0 652 99 38 3 36
0 systemd-logind
[ 1061.652658] [ 653] 81 653 222 53 2 21
-900 dbus-daemon
[ 1061.661059] [ 667] 0 667 79 27 3 29
0 atd
[ 1061.668769] [ 675] 0 675 1846 68 4 86
0 login
[ 1061.676647] [ 676] 0 676 1717 23 5 10
0 agetty
[ 1061.684616] [ 710] 0 710 1761 47 3 36
0 bash
[ 1061.692413] [ 727] 0 727 240 84 2 74
-1000 sshd
[ 1061.700209] [ 733] 0 733 1739 34 3 9
0 rhsmcertd
[ 1061.708432] [ 741] 999 741 7276 132 4 105
0 polkitd
[ 1061.716488] [ 1323] 0 1323 334 69 3 69
0 master
[ 1061.724457] [ 1325] 89 1325 337 82 2 78
0 qmgr
[ 1061.732254] [ 1328] 0 1328 358 130 3 111
0 sshd
[ 1061.740052] [ 1331] 0 1331 1762 48 5 39
0 bash
[ 1061.747843] [ 1373] 89 1373 336 89 2 69
0 pickup
[ 1061.755813] [ 2944] 0 2944 1734 50 4 3
0 make
[ 1061.763611] [31425] 0 31425 1760 79 3 0
0 make
[ 1061.771408] [32190] 0 32190 1736 51 4 0
0 sh
[ 1061.779032] [32195] 0 32195 1751 52 4 0
0 gcc
[ 1061.786738] [32224] 0 32224 1736 51 4 0
0 sh
[ 1061.794362] [32230] 0 32230 1737 42 4 0
0 gcc
[ 1061.802073] [32232] 0 32232 1736 51 3 0
0 sh
[ 1061.809696] [32236] 0 32236 2479 373 4 0
0 cc1
[ 1061.817401] [32237] 0 32237 1736 52 5 0
0 sh
[ 1061.825024] [32239] 0 32239 1737 42 5 0
0 gcc
[ 1061.832734] [32241] 0 32241 1737 43 5 0
0 gcc
[ 1061.840445] [32245] 0 32245 1736 51 4 0
0 sh
[ 1061.848063] [32246] 0 32246 2474 313 4 0
0 cc1
[ 1061.855773] [32248] 0 32248 1737 42 4 0
0 gcc
[ 1061.863483] [32249] 0 32249 2479 383 3 0
0 cc1
[ 1061.871195] [32250] 0 32250 1736 52 3 0
0 sh
[ 1061.878820] [32252] 0 32252 1737 43 4 0
0 gcc
[ 1061.886525] [32253] 0 32253 1736 51 5 0
0 sh
[ 1061.894148] [32254] 0 32254 2472 303 4 0
0 cc1
[ 1061.901858] [32257] 0 32257 2472 304 3 0
0 cc1
[ 1061.909570] [32258] 0 32258 1737 42 4 0
0 gcc
[ 1061.917274] [32260] 0 32260 1736 51 3 0
0 sh
[ 1061.924898] [32261] 0 32261 1737 42 3 0
0 gcc
[ 1061.932607] [32264] 0 32264 2470 232 3 0
0 cc1
[ 1061.940317] [32268] 0 32268 2479 373 3 0
0 cc1
[ 1061.948022] [32273] 0 32273 1736 51 5 0
0 sh
[ 1061.955646] [32276] 0 32276 1737 43 4 0
0 gcc
[ 1061.963355] [32278] 0 32278 1736 51 4 0
0 sh
[ 1061.970979] [32280] 0 32280 1737 43 4 0
0 gcc
[ 1061.978683] [32282] 0 32282 2473 296 4 0
0 cc1
[ 1061.986394] [32286] 0 32286 1736 52 4 0
0 sh
[ 1061.994017] [32287] 0 32287 2474 313 5 0
0 cc1
[ 1062.001729] [32291] 0 32291 1737 42 3 0
0 gcc
[ 1062.009440] [32294] 0 32294 2470 233 4 0
0 cc1
[ 1062.017144] [32299] 0 32299 1736 51 3 0
0 sh
[ 1062.024768] [32300] 0 32300 1737 42 3 0
0 gcc
[ 1062.032479] [32301] 0 32301 208 11 3 0
0 cc1
[ 1062.040189] [32302] 0 32302 32 11 3 0
0 sh
[ 1062.047807] [32303] 0 32303 1760 79 3 0
0 make
[ 1062.055607] Out of memory: Kill process 32249 (cc1) score 0 or
sacrifice child
[ 1062.062801] Killed process 32249 (cc1) total-vm:158656kB,
anon-rss:14976kB, file-rss:9536kB
\aMessage from syslogd at localhost at Jan 8 07:46:43 ...
kernel:Call trace:
[ 1063.003783] systemd invoked oom-killer: gfp_mask=0x200da, order=0,
oom_score_adj=0
[ 1063.011343] systemd cpuset=/ mems_allowed=0
[ 1063.015536] CPU: 2 PID: 1 Comm: systemd Not tainted 3.19.0-rc3+ #2
[ 1063.021707] Hardware name: APM X-Gene Mustang board (DT)
[ 1063.026995] Call trace:
[ 1063.029450] [<fffffe0000096954>] dump_backtrace+0x0/0x16c
[ 1063.034829] [<fffffe0000096ad0>] show_stack+0x10/0x1c
[ 1063.039872] [<fffffe000062911c>] dump_stack+0x74/0x98
[ 1063.044901] [<fffffe0000626550>] dump_header.isra.11+0x98/0x1d0
[ 1063.050808] [<fffffe000017c4e0>] oom_kill_process+0x3ac/0x408
[ 1063.056531] [<fffffe000017c9d4>] out_of_memory+0x2a4/0x2d4
[ 1063.062007] [<fffffe0000180b78>] __alloc_pages_nodemask+0x808/0x8d8
[ 1063.068246] [<fffffe00001a1c54>] do_wp_page+0xd0/0x904
[ 1063.073374] [<fffffe00001a3fc8>] handle_mm_fault+0x4c4/0xd1c
[ 1063.079020] [<fffffe00000a01c8>] do_page_fault+0x214/0x338
[ 1063.084480] [<fffffe000009022c>] do_mem_abort+0x38/0x9c
[ 1063.089691] Exception stack(0xfffffe03dbe83e30 to 0xfffffe03dbe83f50)
[ 1063.096102] 3e20: 00000000
00000000 00007e36 00000000
[ 1063.104253] 3e40: ffffffff ffffffff b5728d14 000003ff dbe83e60
fffffe03 001a8b80 fffffe00
[ 1063.112403] 3e60: dbe83ea0 fffffe03 001a9968 fffffe00 b4f60000
000003ff 00000000 00000000
[ 1063.120554] 3e80: ffffffff ffffffff 001055ac fffffe00 dbe83ea0
fffffe03 00010000 00000000
[ 1063.128693] 3ea0: e6cf3e80 000003ff 00093170 fffffe00 00000000
00000000 0009308c fffffe00
[ 1063.136843] 3ec0: 00000000 00000000 00010000 00000000 d6cb9ee0
000003ff d6d2fd70 000003ff
[ 1063.144994] 3ee0: d6d307f0 000003ff d6e03e80 000003ff 00000000
00000000 d6de9ea0 000003ff
[ 1063.153148] 3f00: 00000006 00000000 00000006 00000000 0002edc2
00000000 00000030 00000000
[ 1063.161299] 3f20: 00000426 00000000 1009a555 00000000 00000018
00000000 ab5187f2 ffffffff
[ 1063.169446] 3f40: bc000000 003103cf 00000000 003b9aca
[ 1063.174472] Mem-Info:
[ 1063.176732] DMA per-cpu:
[ 1063.179262] CPU 0: hi: 6, btch: 1 usd: 0
[ 1063.184030] CPU 1: hi: 6, btch: 1 usd: 0
[ 1063.188797] CPU 2: hi: 6, btch: 1 usd: 0
[ 1063.193573] CPU 3: hi: 6, btch: 1 usd: 0
[ 1063.198341] CPU 4: hi: 6, btch: 1 usd: 0
[ 1063.203116] CPU 5: hi: 6, btch: 1 usd: 0
[ 1063.207883] CPU 6: hi: 6, btch: 1 usd: 0
[ 1063.212659] CPU 7: hi: 6, btch: 1 usd: 0
[ 1063.217426] Normal per-cpu:
[ 1063.220213] CPU 0: hi: 6, btch: 1 usd: 0
[ 1063.224981] CPU 1: hi: 6, btch: 1 usd: 1
[ 1063.229755] CPU 2: hi: 6, btch: 1 usd: 0
[ 1063.234521] CPU 3: hi: 6, btch: 1 usd: 0
[ 1063.239295] CPU 4: hi: 6, btch: 1 usd: 0
[ 1063.244061] CPU 5: hi: 6, btch: 1 usd: 0
[ 1063.248834] CPU 6: hi: 6, btch: 1 usd: 0
[ 1063.253602] CPU 7: hi: 6, btch: 1 usd: 0
[ 1063.258374] active_anon:207481 inactive_anon:25761 isolated_anon:0
[ 1063.258374] active_file:1241 inactive_file:21246 isolated_file:3
[ 1063.258374] unevictable:0 dirty:0 writeback:0 unstable:0
[ 1063.258374] free:1015 slab_reclaimable:1044 slab_unreclaimable:1748
[ 1063.258374] mapped:660 shmem:66 pagetables:236 bounce:0
[ 1063.258374] free_cma:0
[ 1063.289954] DMA free:52864kB min:4032kB low:4992kB high:6016kB
active_anon:3030784kB inactive_anon:612608kB active_file:18304kB
inactive_file:323456kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:4192256kB managed:4177024kB mlocked:0kB
dirty:0kB writeback:0kB mapped:11968kB shmem:2112kB
slab_reclaimable:26240kB slab_unreclaimable:36096kB kernel_stack:352kB
pagetables:4928kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
pages_scanned:26407104 all_unreclaimable? yes
[ 1063.333094] lowmem_reserve[]: 0 765 765
[ 1063.336965] Normal free:12096kB min:12224kB low:15232kB high:18304kB
active_anon:10248000kB inactive_anon:1036096kB active_file:61120kB
inactive_file:1036288kB unevictable:0kB isolated(anon):0kB
isolated(file):192kB present:12582912kB managed:12538176kB mlocked:0kB
dirty:0kB writeback:0kB mapped:30272kB shmem:2112kB
slab_reclaimable:40576kB slab_unreclaimable:75776kB kernel_stack:2944kB
pagetables:10176kB unstable:0kB bounce:0kB free_cma:0kB
writeback_tmp:0kB pages_scanned:81586304 all_unreclaimable? yes
[ 1063.381407] lowmem_reserve[]: 0 0 0
[ 1063.384924] DMA: 120*64kB (UR) 49*128kB (R) 8*256kB (R) 0*512kB
0*1024kB 0*2048kB 1*4096kB (R) 0*8192kB 0*16384kB 1*32768kB (R)
0*65536kB = 52864kB
[ 1063.398274] Normal: 144*64kB (EMR) 9*128kB (R) 6*256kB (R) 0*512kB
0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 11904kB
[ 1063.411177] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=524288kB
[ 1063.419760] 43809 total pagecache pages
[ 1063.423578] 25864 pages in swap cache
[ 1063.427222] Swap cache stats: add 28224, delete 2360, find 2495/2705
[ 1063.433558] Free swap = 6636800kB
[ 1063.436942] Total swap = 8388544kB
[ 1063.440333] 262112 pages RAM
[ 1063.443199] 0 pages HighMem/MovableOnly
[ 1063.447014] 18446744073709544361 pages reserved
[ 1063.451530] 8192 pages cma reserved
[ 1063.455000] [ pid ] uid tgid total_vm rss nr_ptes swapents
oom_score_adj name
[ 1063.462815] [ 481] 0 481 234 112 3 26
0 systemd-journal
[ 1063.471568] [ 487] 0 487 1420 32 3 80
0 lvmetad
[ 1063.479630] [ 512] 0 512 216 40 2 46
-1000 systemd-udevd
[ 1063.488204] [ 598] 0 598 258 45 3 38
-1000 auditd
[ 1063.496181] [ 620] 0 620 6830 252 3 82
0 NetworkManager
[ 1063.504847] [ 623] 0 623 2028 83 3 100
0 abrtd
[ 1063.512737] [ 624] 0 624 2019 87 3 84
0 abrt-watch-log
[ 1063.521403] [ 628] 70 628 87 55 4 18
0 avahi-daemon
[ 1063.529896] [ 629] 998 629 40 13 2 0
0 lsmd
[ 1063.537689] [ 631] 70 631 87 0 4 32
0 avahi-daemon
[ 1063.546184] [ 641] 0 641 2687 82 3 56
0 rsyslogd
[ 1063.554337] [ 642] 0 642 86 42 3 9
0 irqbalance
[ 1063.562660] [ 647] 0 647 109 44 2 31
0 smartd
[ 1063.570635] [ 651] 997 651 83 45 2 18
0 chronyd
[ 1063.578687] [ 652] 0 652 99 38 3 36
0 systemd-logind
[ 1063.587354] [ 653] 81 653 222 53 2 21
-900 dbus-daemon
[ 1063.595761] [ 667] 0 667 79 27 3 29
0 atd
[ 1063.603476] [ 675] 0 675 1846 68 4 86
0 login
[ 1063.611363] [ 676] 0 676 1717 23 5 10
0 agetty
[ 1063.619338] [ 710] 0 710 1761 47 3 36
0 bash
[ 1063.627132] [ 727] 0 727 240 84 2 74
-1000 sshd
[ 1063.634935] [ 733] 0 733 1739 34 3 9
0 rhsmcertd
[ 1063.643169] [ 741] 999 741 7276 132 4 105
0 polkitd
[ 1063.651230] [ 1323] 0 1323 334 69 3 69
0 master
[ 1063.659205] [ 1325] 89 1325 337 82 2 78
0 qmgr
[ 1063.666998] [ 1328] 0 1328 358 130 3 111
0 sshd
[ 1063.674802] [ 1331] 0 1331 1762 48 5 39
0 bash
[ 1063.682608] [ 1373] 89 1373 336 89 2 69
0 pickup
[ 1063.690584] [ 2944] 0 2944 1734 50 4 3
0 make
[ 1063.698380] [31425] 0 31425 1760 82 3 0
0 make
[ 1063.706184] [32224] 0 32224 1736 51 4 0
0 sh
[ 1063.713814] [32230] 0 32230 1737 42 4 0
0 gcc
[ 1063.721529] [32232] 0 32232 1736 51 3 0
0 sh
[ 1063.729157] [32239] 0 32239 1737 42 5 0
0 gcc
[ 1063.736863] [32245] 0 32245 1736 51 4 0
0 sh
[ 1063.744494] [32246] 0 32246 2474 313 4 0
0 cc1
[ 1063.752208] [32248] 0 32248 1737 42 4 0
0 gcc
[ 1063.759923] [32250] 0 32250 1736 52 3 0
0 sh
[ 1063.767544] [32252] 0 32252 1737 43 4 0
0 gcc
[ 1063.775260] [32253] 0 32253 1736 51 5 0
0 sh
[ 1063.782889] [32254] 0 32254 2474 313 4 0
0 cc1
[ 1063.790605] [32257] 0 32257 2474 314 3 0
0 cc1
[ 1063.798311] [32258] 0 32258 1737 42 4 0
0 gcc
[ 1063.806028] [32260] 0 32260 1736 51 3 0
0 sh
[ 1063.813663] [32261] 0 32261 1737 42 3 0
0 gcc
[ 1063.821379] [32264] 0 32264 2470 295 3 0
0 cc1
[ 1063.829094] [32273] 0 32273 1736 51 5 0
0 sh
[ 1063.836714] [32276] 0 32276 1737 43 4 0
0 gcc
[ 1063.844429] [32278] 0 32278 1736 51 4 0
0 sh
[ 1063.852058] [32280] 0 32280 1737 43 4 0
0 gcc
[ 1063.859774] [32282] 0 32282 2474 314 4 0
0 cc1
[ 1063.867480] [32286] 0 32286 1736 52 4 0
0 sh
[ 1063.875110] [32287] 0 32287 2474 313 5 0
0 cc1
[ 1063.882826] [32291] 0 32291 1737 42 3 0
0 gcc
[ 1063.890541] [32294] 0 32294 2473 296 4 0
0 cc1
[ 1063.898248] [32299] 0 32299 1736 51 3 0
0 sh
[ 1063.905870] [32300] 0 32300 1737 42 3 0
0 gcc
[ 1063.913587] [32301] 0 32301 2464 199 3 0
0 cc1
[ 1063.921303] [32302] 0 32302 1736 51 4 0
0 sh
[ 1063.928931] [32303] 0 32303 1736 51 3 0
0 sh
[ 1063.936552] [32304] 0 32304 1737 42 4 0
0 gcc
[ 1063.944272] [32305] 0 32305 1737 43 4 0
0 gcc
[ 1063.951993] [32306] 0 32306 2464 199 4 0
0 cc1
[ 1063.959708] [32307] 0 32307 45 12 3 0
0 as
[ 1063.967329] [32308] 0 32308 2464 200 4 0
0 cc1
[ 1063.975044] [32309] 0 32309 1925 222 4 0
0 as
[ 1063.982674] [32310] 0 32310 211 61 3 63
0 systemd
[ 1063.990735] Out of memory: Kill process 620 (NetworkManager) score 0
or sacrifice child
[ 1063.998701] Killed process 620 (NetworkManager) total-vm:437120kB,
anon-rss:3712kB, file-rss:12416kB
[ 1064.007830] NetworkManager: page allocation failure: order:0,
mode:0x2015a
[ 1064.014689] CPU: 1 PID: 620 Comm: NetworkManager Not tainted
3.19.0-rc3+ #2
[ 1064.021622] Hardware name: APM X-Gene Mustang board (DT)
[ 1064.026907] Call trace:
[ 1064.029355] [<fffffe0000096954>] dump_backtrace+0x0/0x16c
[ 1064.034728] [<fffffe0000096ad0>] show_stack+0x10/0x1c
[ 1064.039761] [<fffffe000062911c>] dump_stack+0x74/0x98
[ 1064.044789] [<fffffe000017dc10>] warn_alloc_failed+0xe0/0x138
[ 1064.050513] [<fffffe00001809c8>] __alloc_pages_nodemask+0x658/0x8d8
[ 1064.056748] [<fffffe000017a690>] generic_file_read_iter+0x3c4/0x534
[ 1064.063030] [<fffffdfffc196754>] xfs_file_read_iter+0xe8/0x2a0 [xfs]
[ 1064.069367] [<fffffe00001d6500>] new_sync_read+0x84/0xc8
[ 1064.074654] [<fffffe00001d6ff4>] __vfs_read+0x14/0x50
[ 1064.079694] [<fffffe00001d70ac>] vfs_read+0x7c/0x154
[ 1064.084638] [<fffffe00001d71c4>] SyS_read+0x40/0xa0
[ 1064.089503] Mem-Info:
[ 1064.091765] DMA per-cpu:
[ 1064.094285] CPU 0: hi: 6, btch: 1 usd: 0
[ 1064.099056] CPU 1: hi: 6, btch: 1 usd: 0
[ 1064.103822] CPU 2: hi: 6, btch: 1 usd: 0
[ 1064.108586] CPU 3: hi: 6, btch: 1 usd: 0
[ 1064.113368] CPU 4: hi: 6, btch: 1 usd: 0
[ 1064.118134] CPU 5: hi: 6, btch: 1 usd: 0
[ 1064.122905] CPU 6: hi: 6, btch: 1 usd: 0
[ 1064.127671] CPU 7: hi: 6, btch: 1 usd: 0
[ 1064.132442] Normal per-cpu:
[ 1064.135220] CPU 0: hi: 6, btch: 1 usd: 0
[ 1064.139991] CPU 1: hi: 6, btch: 1 usd: 1
[ 1064.144755] CPU 2: hi: 6, btch: 1 usd: 0
[ 1064.149526] CPU 3: hi: 6, btch: 1 usd: 0
[ 1064.154290] CPU 4: hi: 6, btch: 1 usd: 0
[ 1064.159061] CPU 5: hi: 6, btch: 1 usd: 0
[ 1064.163826] CPU 6: hi: 6, btch: 1 usd: 0
[ 1064.168591] CPU 7: hi: 6, btch: 1 usd: 0
[ 1064.173365] active_anon:207484 inactive_anon:25769 isolated_anon:0
[ 1064.173365] active_file:1247 inactive_file:21260 isolated_file:3
[ 1064.173365] unevictable:0 dirty:0 writeback:0 unstable:0
[ 1064.173365] free:1012 slab_reclaimable:1044 slab_unreclaimable:1748
[ 1064.173365] mapped:661 shmem:66 pagetables:233 bounce:0
[ 1064.173365] free_cma:0
[ 1064.204935] DMA free:52864kB min:4032kB low:4992kB high:6016kB
active_anon:3030784kB inactive_anon:612608kB active_file:18304kB
inactive_file:323520kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:4192256kB managed:4177024kB mlocked:0kB
dirty:0kB writeback:0kB mapped:11968kB shmem:2112kB
slab_reclaimable:26240kB slab_unreclaimable:36096kB kernel_stack:336kB
pagetables:4928kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
pages_scanned:32080832 all_unreclaimable? yes
[ 1064.248071] lowmem_reserve[]: 0 765 765
[ 1064.251936] Normal free:11904kB min:12224kB low:15232kB high:18304kB
active_anon:10248192kB inactive_anon:1036608kB active_file:61504kB
inactive_file:1037120kB unevictable:0kB isolated(anon):0kB
isolated(file):192kB present:12582912kB managed:12538176kB mlocked:0kB
dirty:0kB writeback:0kB mapped:30336kB shmem:2112kB
slab_reclaimable:40576kB slab_unreclaimable:75776kB kernel_stack:2976kB
pagetables:9984kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
pages_scanned:98616256 all_unreclaimable? yes
[ 1064.296280] lowmem_reserve[]: 0 0 0
[ 1064.299798] DMA: 120*64kB (UR) 49*128kB (R) 8*256kB (R) 0*512kB
0*1024kB 0*2048kB 1*4096kB (R) 0*8192kB 0*16384kB 1*32768kB (R)
0*65536kB = 52864kB
[ 1064.313134] Normal: 142*64kB (EMR) 9*128kB (R) 6*256kB (R) 0*512kB
0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 11776kB
[ 1064.326021] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=524288kB
[ 1064.334600] 43816 total pagecache pages
[ 1064.338418] 25866 pages in swap cache
[ 1064.342068] Swap cache stats: add 28226, delete 2360, find 2497/2709
[ 1064.348389] Free swap = 6636800kB
[ 1064.351777] Total swap = 8388544kB
[ 1064.355161] 262112 pages RAM
[ 1064.358025] 0 pages HighMem/MovableOnly
[ 1064.361844] 18446744073709544361 pages reserved
[ 1064.366349] 8192 pages cma reserved
\aMessage from syslogd at localhost at Jan 8 07:46:45 ...
kernel:Call trace:
\aMessage from syslogd at localhost at Jan 8 07:46:45 ...
kernel:Call trace:
[ 1065.640588] systemd-journal invoked oom-killer: gfp_mask=0x200da,
order=0, oom_score_adj=0
[ 1065.648823] systemd-journal cpuset=/ mems_allowed=0
[ 1065.653724] CPU: 7 PID: 481 Comm: systemd-journal Not tainted
3.19.0-rc3+ #2
[ 1065.660753] Hardware name: APM X-Gene Mustang board (DT)
[ 1065.666041] Call trace:
[ 1065.668484] [<fffffe0000096954>] dump_backtrace+0x0/0x16c
[ 1065.673878] [<fffffe0000096ad0>] show_stack+0x10/0x1c
[ 1065.678910] [<fffffe000062911c>] dump_stack+0x74/0x98
[ 1065.683957] [<fffffe0000626550>] dump_header.isra.11+0x98/0x1d0
[ 1065.689867] [<fffffe000017c4e0>] oom_kill_process+0x3ac/0x408
[ 1065.695590] [<fffffe000017c9d4>] out_of_memory+0x2a4/0x2d4
[ 1065.701067] [<fffffe0000180b78>] __alloc_pages_nodemask+0x808/0x8d8
[ 1065.707307] [<fffffe00001a443c>] handle_mm_fault+0x938/0xd1c
[ 1065.712945] [<fffffe00000a01c8>] do_page_fault+0x214/0x338
[ 1065.718403] [<fffffe000009022c>] do_mem_abort+0x38/0x9c
[ 1065.723607] Exception stack(0xfffffe03d6323bf0 to 0xfffffe03d6323d10)
[ 1065.730022] 3be0: d6109780
fffffe03 00000000 00000000
[ 1065.738159] 3c00: d6323db0 fffffe03 0031c184 fffffe00 00000005
00000000 d6323cf0 fffffe03
[ 1065.746301] 3c20: d6323d00 fffffe03 ff0a0004 ffffffff d6323c70
fffffe03 001f8b84 fffffe00
[ 1065.754443] 3c40: d6109780 fffffe03 00100073 00000000 d6323cf0
fffffe03 d6323cf0 fffffe03
[ 1065.762584] 3c60: d6323cc0 fffffe03 ffffffd0 00000000 d6323cb0
fffffe03 0023d2b0 fffffe00
[ 1065.770725] 3c80: 00d7d1d8 fffffe00 00000140 00000000 94560000
000003ff ebed0008 fffffe03
[ 1065.778861] 3ca0: 0000006d 00000000 65703a39 655f6672 9456007d
000003ff 002b5d34 00000000
[ 1065.787002] 3cc0: d9ff1000 fffffe03 00000000 00000000 00000000
00000000 fefefefe ff2efefe
[ 1065.795143] 3ce0: ffffffff 7f7fffff 01010101 01010101 00000038
00000000 00000038 00000000
[ 1065.803289] 3d00: ffffffff 0fffffff ffffffed ffffffff
[ 1065.808319] [<fffffe0000092b64>] el1_da+0x14/0x70
[ 1065.813007] [<fffffe00001d6ff4>] __vfs_read+0x14/0x50
[ 1065.818032] [<fffffe00001d70ac>] vfs_read+0x7c/0x154
[ 1065.822978] [<fffffe00001d71c4>] SyS_read+0x40/0xa0
[ 1065.827829] Mem-Info:
[ 1065.830097] DMA per-cpu:
[ 1065.832617] CPU 0: hi: 6, btch: 1 usd: 0
[ 1065.837381] CPU 1: hi: 6, btch: 1 usd: 0
[ 1065.842151] CPU 2: hi: 6, btch: 1 usd: 0
[ 1065.846916] CPU 3: hi: 6, btch: 1 usd: 0
[ 1065.851684] CPU 4: hi: 6, btch: 1 usd: 0
[ 1065.856449] CPU 5: hi: 6, btch: 1 usd: 0
[ 1065.861219] CPU 6: hi: 6, btch: 1 usd: 0
[ 1065.865984] CPU 7: hi: 6, btch: 1 usd: 0
[ 1065.870752] Normal per-cpu:
[ 1065.873529] CPU 0: hi: 6, btch: 1 usd: 0
[ 1065.878294] CPU 1: hi: 6, btch: 1 usd: 0
[ 1065.883063] CPU 2: hi: 6, btch: 1 usd: 0
[ 1065.887828] CPU 3: hi: 6, btch: 1 usd: 0
[ 1065.892598] CPU 4: hi: 6, btch: 1 usd: 0
[ 1065.897362] CPU 5: hi: 6, btch: 1 usd: 0
[ 1065.902133] CPU 6: hi: 6, btch: 1 usd: 0
[ 1065.906897] CPU 7: hi: 6, btch: 1 usd: 0
[ 1065.911671] active_anon:207543 inactive_anon:25751 isolated_anon:0
[ 1065.911671] active_file:1192 inactive_file:21238 isolated_file:0
[ 1065.911671] unevictable:0 dirty:1 writeback:0 unstable:0
[ 1065.911671] free:1016 slab_reclaimable:1044 slab_unreclaimable:1748
[ 1065.911671] mapped:572 shmem:68 pagetables:224 bounce:0
[ 1065.911671] free_cma:1
[ 1065.943242] DMA free:52928kB min:4032kB low:4992kB high:6016kB
active_anon:3030720kB inactive_anon:612544kB active_file:18496kB
inactive_file:323008kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:4192256kB managed:4177024kB mlocked:0kB
dirty:64kB writeback:0kB mapped:11904kB shmem:2112kB
slab_reclaimable:26240kB slab_unreclaimable:36096kB kernel_stack:336kB
pagetables:4736kB unstable:0kB bounce:0kB free_cma:64kB
writeback_tmp:0kB pages_scanned:25027136 all_unreclaimable? yes
[ 1065.986550] lowmem_reserve[]: 0 765 765
[ 1065.990433] Normal free:12480kB min:12224kB low:15232kB high:18304kB
active_anon:10252032kB inactive_anon:1035776kB active_file:57792kB
inactive_file:1036672kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:12582912kB managed:12538176kB mlocked:0kB
dirty:0kB writeback:0kB mapped:24704kB shmem:2240kB
slab_reclaimable:40576kB slab_unreclaimable:75776kB kernel_stack:2928kB
pagetables:9600kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
pages_scanned:81503232 all_unreclaimable? yes
[ 1066.034608] lowmem_reserve[]: 0 0 0
[ 1066.038124] DMA: 122*64kB (UR) 49*128kB (R) 8*256kB (R) 0*512kB
0*1024kB 0*2048kB 1*4096kB (R) 0*8192kB 0*16384kB 1*32768kB (R)
0*65536kB = 52992kB
[ 1066.051465] Normal: 116*64kB (EMR) 18*128kB (MR) 6*256kB (R) 0*512kB
0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 11264kB
[ 1066.064522] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=524288kB
[ 1066.073093] 43745 total pagecache pages
[ 1066.076909] 25845 pages in swap cache
[ 1066.080557] Swap cache stats: add 28232, delete 2387, find 2500/2717
[ 1066.086877] Free swap = 6643328kB
[ 1066.090264] Total swap = 8388544kB
[ 1066.093648] 262112 pages RAM
[ 1066.096511] 0 pages HighMem/MovableOnly
[ 1066.100330] 18446744073709544361 pages reserved
[ 1066.104835] 8192 pages cma reserved
[ 1066.108305] [ pid ] uid tgid total_vm rss nr_ptes swapents
oom_score_adj name
[ 1066.116115] [ 481] 0 481 234 112 3 26
0 systemd-journal
[ 1066.124864] [ 487] 0 487 1420 32 3 80
0 lvmetad
[ 1066.132920] [ 512] 0 512 216 40 2 46
-1000 systemd-udevd
[ 1066.141498] [ 598] 0 598 258 45 3 38
-1000 auditd
[ 1066.149467] [ 623] 0 623 2028 83 3 100
0 abrtd
[ 1066.157345] [ 624] 0 624 2019 87 3 84
0 abrt-watch-log
[ 1066.166006] [ 628] 70 628 87 55 4 18
0 avahi-daemon
[ 1066.174494] [ 629] 998 629 40 13 2 0
0 lsmd
[ 1066.182291] [ 631] 70 631 87 0 4 32
0 avahi-daemon
[ 1066.190779] [ 641] 0 641 2687 82 3 56
0 rsyslogd
[ 1066.198915] [ 642] 0 642 86 42 3 9
0 irqbalance
[ 1066.207230] [ 647] 0 647 109 44 2 31
0 smartd
[ 1066.215199] [ 651] 997 651 83 45 2 18
0 chronyd
[ 1066.223255] [ 652] 0 652 99 38 3 36
0 systemd-logind
[ 1066.231916] [ 653] 81 653 222 53 2 21
-900 dbus-daemon
[ 1066.240319] [ 667] 0 667 79 27 3 29
0 atd
[ 1066.248024] [ 675] 0 675 1846 68 4 86
0 login
[ 1066.255907] [ 676] 0 676 1717 23 5 10
0 agetty
[ 1066.263878] [ 710] 0 710 1761 47 3 36
0 bash
[ 1066.271674] [ 727] 0 727 240 84 2 74
-1000 sshd
[ 1066.279472] [ 733] 0 733 1739 34 3 9
0 rhsmcertd
[ 1066.287695] [ 741] 999 741 7276 132 4 105
0 polkitd
[ 1066.295750] [ 1323] 0 1323 334 69 3 69
0 master
[ 1066.303720] [ 1325] 89 1325 337 82 2 78
0 qmgr
[ 1066.311516] [ 1328] 0 1328 358 130 3 111
0 sshd
[ 1066.319313] [ 1331] 0 1331 1762 48 5 39
0 bash
[ 1066.327105] [ 1373] 89 1373 336 89 2 69
0 pickup
[ 1066.335073] [ 2944] 0 2944 1734 50 4 3
0 make
[ 1066.342871] [31425] 0 31425 1760 82 3 0
0 make
[ 1066.350668] [32224] 0 32224 1736 51 4 0
0 sh
[ 1066.358287] [32232] 0 32232 1736 51 3 0
0 sh
[ 1066.365912] [32239] 0 32239 1737 42 5 0
0 gcc
[ 1066.373624] [32245] 0 32245 1736 51 4 0
0 sh
[ 1066.381249] [32246] 0 32246 2474 313 4 0
0 cc1
[ 1066.388953] [32248] 0 32248 1737 42 4 0
0 gcc
[ 1066.396667] [32250] 0 32250 1736 52 3 0
0 sh
[ 1066.404293] [32252] 0 32252 1737 43 4 0
0 gcc
[ 1066.412005] [32253] 0 32253 1736 51 5 0
0 sh
[ 1066.419631] [32254] 0 32254 2474 313 4 0
0 cc1
[ 1066.427336] [32257] 0 32257 2474 314 3 0
0 cc1
[ 1066.435046] [32258] 0 32258 1737 42 4 0
0 gcc
[ 1066.442757] [32260] 0 32260 1736 51 3 0
0 sh
[ 1066.450387] [32261] 0 32261 1737 42 3 0
0 gcc
[ 1066.458092] [32264] 0 32264 2473 295 3 0
0 cc1
[ 1066.465802] [32273] 0 32273 1736 51 5 0
0 sh
[ 1066.473426] [32276] 0 32276 1737 43 4 0
0 gcc
[ 1066.481136] [32278] 0 32278 1736 51 4 0
0 sh
[ 1066.488755] [32280] 0 32280 1737 43 4 0
0 gcc
[ 1066.496464] [32282] 0 32282 2474 314 4 0
0 cc1
[ 1066.504175] [32286] 0 32286 1736 52 4 0
0 sh
[ 1066.511799] [32287] 0 32287 2474 313 5 0
0 cc1
[ 1066.519509] [32291] 0 32291 1737 42 3 0
0 gcc
[ 1066.527213] [32294] 0 32294 2473 296 4 0
0 cc1
[ 1066.534923] [32299] 0 32299 1736 51 3 0
0 sh
[ 1066.542547] [32300] 0 32300 1737 42 3 0
0 gcc
[ 1066.550257] [32301] 0 32301 2464 199 3 0
0 cc1
[ 1066.557961] [32302] 0 32302 1736 51 4 0
0 sh
[ 1066.565584] [32303] 0 32303 1736 51 3 0
0 sh
[ 1066.573208] [32304] 0 32304 1737 42 4 0
0 gcc
[ 1066.580917] [32305] 0 32305 1737 43 4 0
0 gcc
[ 1066.588622] [32306] 0 32306 2466 232 4 0
0 cc1
[ 1066.596332] [32307] 0 32307 1918 222 5 0
0 as
[ 1066.603955] [32308] 0 32308 2464 200 4 0
0 cc1
[ 1066.611666] [32310] 0 32310 42 13 2 0
0 nm-dispatcher
[ 1066.620239] [32311] 0 32311 7 1 1 0
0 systemd-cgroups
[ 1066.628980] Out of memory: Kill process 32282 (cc1) score 0 or
sacrifice child
[ 1066.636172] Killed process 32282 (cc1) total-vm:158336kB,
anon-rss:10816kB, file-rss:9280kB
\aMessage from syslogd at localhost at Jan 8 07:46:47 ...
kernel:Call trace:
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-08 12:51 ` Linux 3.19-rc3 Mark Langsdorf
@ 2015-01-08 13:45 ` Catalin Marinas
2015-01-08 17:29 ` Mark Langsdorf
2015-01-08 15:08 ` Michal Hocko
2015-01-09 12:13 ` Mark Rutland
2 siblings, 1 reply; 50+ messages in thread
From: Catalin Marinas @ 2015-01-08 13:45 UTC (permalink / raw)
To: linux-arm-kernel
On Thu, Jan 08, 2015 at 12:51:31PM +0000, Mark Langsdorf wrote:
> On 01/05/2015 07:46 PM, Linus Torvalds wrote:
> > It's a day delayed - not because of any particular development issues,
> > but simply because I was tiling a bathroom yesterday. But rc3 is out
> > there now, and things have stayed reasonably calm. I really hope that
> > implies that 3.19 is looking good, but it's equally likely that it's
> > just that people are still recovering from the holiday season.
>
> I'm consistently getting an out of memory killer triggered when
> compiling the kernel (make -j 16 -s) on a 16 core ARM64 system
> with 16 GB of memory. This doesn't happen when running a 3.18
> kernel.
>
> I'm going to start bisecting the failure now, but here's the crash
> log in case someone can see something obvious in it.
[...]
> [ 1053.968815] active_anon:207417 inactive_anon:25722 isolated_anon:0
> [ 1053.968815] active_file:1300 inactive_file:21234 isolated_file:0
> [ 1053.968815] unevictable:0 dirty:0 writeback:0 unstable:0
> [ 1053.968815] free:1014 slab_reclaimable:1047 slab_unreclaimable:1758
> [ 1053.968815] mapped:733 shmem:58 pagetables:267 bounce:0
> [ 1053.968815] free_cma:1
> [ 1054.000398] DMA free:52928kB min:4032kB low:4992kB high:6016kB
> active_anon:3025728kB inactive_anon:612608kB active_file:20992kB
> inactive_file:323072kB unevictable:0kB isolated(anon):0kB
> isolated(file):0kB present:4192256kB managed:4177024kB mlocked:0kB
> dirty:0kB writeback:0kB mapped:15168kB shmem:1792kB
> slab_reclaimable:26368kB slab_unreclaimable:36160kB kernel_stack:464kB
> pagetables:6976kB unstable:0kB bounce:0kB free_cma:64kB
> writeback_tmp:0kB pages_scanned:26151872 all_unreclaimable? yes
> [ 1054.043628] lowmem_reserve[]: 0 765 765
> [ 1054.047498] Normal free:12032kB min:12224kB low:15232kB high:18304kB
> active_anon:10248960kB inactive_anon:1033728kB active_file:62208kB
> inactive_file:1036032kB unevictable:0kB isolated(anon):0kB
> isolated(file):0kB present:12582912kB managed:12538176kB mlocked:0kB
> dirty:0kB writeback:0kB mapped:31744kB shmem:1920kB
> slab_reclaimable:40640kB slab_unreclaimable:76352kB kernel_stack:2992kB
> pagetables:10112kB unstable:0kB bounce:0kB free_cma:0kB
> writeback_tmp:0kB pages_scanned:82150272 all_unreclaimable? yes
> [ 1054.091760] lowmem_reserve[]: 0 0 0
> [ 1054.095277] DMA: 109*64kB (UR) 53*128kB (R) 8*256kB (R) 0*512kB
> 0*1024kB 0*2048kB 1*4096kB (R) 0*8192kB 0*16384kB 1*32768kB (R)
> 0*65536kB = 52672kB
> [ 1054.108621] Normal: 191*64kB (MR) 0*128kB 0*256kB 0*512kB 0*1024kB
> 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 12224kB
> [ 1054.120708] Node 0 hugepages_total=0 hugepages_free=0
> hugepages_surp=0 hugepages_size=524288kB
> [ 1054.129280] 43732 total pagecache pages
> [ 1054.133096] 25736 pages in swap cache
> [ 1054.136739] Swap cache stats: add 27971, delete 2235, find 0/1
> [ 1054.142545] Free swap = 6598400kB
> [ 1054.145928] Total swap = 8388544kB
> [ 1054.149317] 262112 pages RAM
> [ 1054.152180] 0 pages HighMem/MovableOnly
> [ 1054.155995] 18446744073709544361 pages reserved
This looks weird (pages reserved = -7255).
Can you disable (transparent) huge pages? I don't have any better at the
moment suggestion apart from bisecting.
--
Catalin
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-08 12:51 ` Linux 3.19-rc3 Mark Langsdorf
2015-01-08 13:45 ` Catalin Marinas
@ 2015-01-08 15:08 ` Michal Hocko
2015-01-08 16:37 ` Mark Langsdorf
2015-01-09 12:13 ` Mark Rutland
2 siblings, 1 reply; 50+ messages in thread
From: Michal Hocko @ 2015-01-08 15:08 UTC (permalink / raw)
To: linux-arm-kernel
[CCing linux-mm and CMA people]
[Full message here:
http://article.gmane.org/gmane.linux.ports.arm.kernel/383669]
On Thu 08-01-15 06:51:31, Mark Langsdorf wrote:
[...]
> [ 1053.968815] active_anon:207417 inactive_anon:25722 isolated_anon:0
> [ 1053.968815] active_file:1300 inactive_file:21234 isolated_file:0
> [ 1053.968815] unevictable:0 dirty:0 writeback:0 unstable:0
> [ 1053.968815] free:1014 slab_reclaimable:1047 slab_unreclaimable:1758
> [ 1053.968815] mapped:733 shmem:58 pagetables:267 bounce:0
> [ 1053.968815] free_cma:1
Still a lot of pages (~80M) on the file LRU list which should be reclaimable
because they are not dirty apparently.
Anon pages can be reclaimed as well because the swap is basically
unused.
[...]
> [ 1054.095277] DMA: 109*64kB (UR) 53*128kB (R) 8*256kB (R) 0*512kB 0*1024kB
> 0*2048kB 1*4096kB (R) 0*8192kB 0*16384kB 1*32768kB (R) 0*65536kB = 52672kB
> [ 1054.108621] Normal: 191*64kB (MR) 0*128kB 0*256kB 0*512kB 0*1024kB
> 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 12224kB
[...]
> [ 1054.142545] Free swap = 6598400kB
> [ 1054.145928] Total swap = 8388544kB
> [ 1054.149317] 262112 pages RAM
> [ 1054.152180] 0 pages HighMem/MovableOnly
> [ 1054.155995] 18446744073709544361 pages reserved
> [ 1054.160505] 8192 pages cma reserved
Besides underflow in the reserved pages accounting mentioned in other
email the free lists look strange as well. All free blocks with some memory
are marked as reserved. I would suspect something CMA related.
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-08 15:08 ` Michal Hocko
@ 2015-01-08 16:37 ` Mark Langsdorf
2015-01-09 15:56 ` Michal Hocko
0 siblings, 1 reply; 50+ messages in thread
From: Mark Langsdorf @ 2015-01-08 16:37 UTC (permalink / raw)
To: linux-arm-kernel
On 01/08/2015 09:08 AM, Michal Hocko wrote:
> [CCing linux-mm and CMA people]
> [Full message here:
> http://article.gmane.org/gmane.linux.ports.arm.kernel/383669]
>> [ 1054.095277] DMA: 109*64kB (UR) 53*128kB (R) 8*256kB (R) 0*512kB 0*1024kB
>> 0*2048kB 1*4096kB (R) 0*8192kB 0*16384kB 1*32768kB (R) 0*65536kB = 52672kB
>> [ 1054.108621] Normal: 191*64kB (MR) 0*128kB 0*256kB 0*512kB 0*1024kB
>> 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 12224kB
> [...]
>> [ 1054.142545] Free swap = 6598400kB
>> [ 1054.145928] Total swap = 8388544kB
>> [ 1054.149317] 262112 pages RAM
>> [ 1054.152180] 0 pages HighMem/MovableOnly
>> [ 1054.155995] 18446744073709544361 pages reserved
>> [ 1054.160505] 8192 pages cma reserved
>
> Besides underflow in the reserved pages accounting mentioned in other
> email the free lists look strange as well. All free blocks with some memory
> are marked as reserved. I would suspect something CMA related.
I get the same failure with CMA turned off entirely. I assume that means
CMA is not the culprit.
--Mark Langsdorf
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-08 13:45 ` Catalin Marinas
@ 2015-01-08 17:29 ` Mark Langsdorf
2015-01-08 17:34 ` Catalin Marinas
0 siblings, 1 reply; 50+ messages in thread
From: Mark Langsdorf @ 2015-01-08 17:29 UTC (permalink / raw)
To: linux-arm-kernel
On 01/08/2015 07:45 AM, Catalin Marinas wrote:
> On Thu, Jan 08, 2015 at 12:51:31PM +0000, Mark Langsdorf wrote:
>> On 01/05/2015 07:46 PM, Linus Torvalds wrote:
>>> It's a day delayed - not because of any particular development issues,
>>> but simply because I was tiling a bathroom yesterday. But rc3 is out
>>> there now, and things have stayed reasonably calm. I really hope that
>>> implies that 3.19 is looking good, but it's equally likely that it's
>>> just that people are still recovering from the holiday season.
>>
>> I'm consistently getting an out of memory killer triggered when
>> compiling the kernel (make -j 16 -s) on a 16 core ARM64 system
>> with 16 GB of memory. This doesn't happen when running a 3.18
>> kernel.
>>
>> I'm going to start bisecting the failure now, but here's the crash
>> log in case someone can see something obvious in it.
> Can you disable (transparent) huge pages? I don't have any better at the
> moment suggestion apart from bisecting.
I didn't have transparent huge pages on. Turning off hugetblfs didn't
change anything. Turning off 64K pages isn't an option because of
firmware constraints.
--Mark Langsdorf
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-08 17:29 ` Mark Langsdorf
@ 2015-01-08 17:34 ` Catalin Marinas
2015-01-08 18:48 ` Mark Langsdorf
0 siblings, 1 reply; 50+ messages in thread
From: Catalin Marinas @ 2015-01-08 17:34 UTC (permalink / raw)
To: linux-arm-kernel
On Thu, Jan 08, 2015 at 05:29:40PM +0000, Mark Langsdorf wrote:
> On 01/08/2015 07:45 AM, Catalin Marinas wrote:
> > On Thu, Jan 08, 2015 at 12:51:31PM +0000, Mark Langsdorf wrote:
> >> On 01/05/2015 07:46 PM, Linus Torvalds wrote:
> >>> It's a day delayed - not because of any particular development issues,
> >>> but simply because I was tiling a bathroom yesterday. But rc3 is out
> >>> there now, and things have stayed reasonably calm. I really hope that
> >>> implies that 3.19 is looking good, but it's equally likely that it's
> >>> just that people are still recovering from the holiday season.
> >>
> >> I'm consistently getting an out of memory killer triggered when
> >> compiling the kernel (make -j 16 -s) on a 16 core ARM64 system
> >> with 16 GB of memory. This doesn't happen when running a 3.18
> >> kernel.
> >>
> >> I'm going to start bisecting the failure now, but here's the crash
> >> log in case someone can see something obvious in it.
>
> > Can you disable (transparent) huge pages? I don't have any better at the
> > moment suggestion apart from bisecting.
>
> I didn't have transparent huge pages on. Turning off hugetblfs didn't
> change anything. Turning off 64K pages isn't an option because of
> firmware constraints.
What constraints are these? I thought they could only happen the other
way around (4K to 64K).
--
Catalin
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-08 17:34 ` Catalin Marinas
@ 2015-01-08 18:48 ` Mark Langsdorf
2015-01-08 19:21 ` Linus Torvalds
0 siblings, 1 reply; 50+ messages in thread
From: Mark Langsdorf @ 2015-01-08 18:48 UTC (permalink / raw)
To: linux-arm-kernel
On 01/08/2015 11:34 AM, Catalin Marinas wrote:
> On Thu, Jan 08, 2015 at 05:29:40PM +0000, Mark Langsdorf wrote:
>> On 01/08/2015 07:45 AM, Catalin Marinas wrote:
>>> On Thu, Jan 08, 2015 at 12:51:31PM +0000, Mark Langsdorf wrote:
>>>> On 01/05/2015 07:46 PM, Linus Torvalds wrote:
>>>>> It's a day delayed - not because of any particular development issues,
>>>>> but simply because I was tiling a bathroom yesterday. But rc3 is out
>>>>> there now, and things have stayed reasonably calm. I really hope that
>>>>> implies that 3.19 is looking good, but it's equally likely that it's
>>>>> just that people are still recovering from the holiday season.
>>>>
>>>> I'm consistently getting an out of memory killer triggered when
>>>> compiling the kernel (make -j 16 -s) on a 16 core ARM64 system
>>>> with 16 GB of memory. This doesn't happen when running a 3.18
>>>> kernel.
>>>>
>>>> I'm going to start bisecting the failure now, but here's the crash
>>>> log in case someone can see something obvious in it.
>>
>>> Can you disable (transparent) huge pages? I don't have any better at the
>>> moment suggestion apart from bisecting.
>>
>> I didn't have transparent huge pages on. Turning off hugetblfs didn't
>> change anything. Turning off 64K pages isn't an option because of
>> firmware constraints.
>
> What constraints are these? I thought they could only happen the other
> way around (4K to 64K).
I was confused. I can turn off 64K pages with only minor loss of
functionality (network MAC address gets corrupted; I can work around
for testing).
With 4K pages, the oom killer doesn't trigger during `make -j 16 -s`
on a fresh kernel. Thanks for the suggestion. I'm not sure what to do
about that, though.
--Mark Langsdorf
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-08 18:48 ` Mark Langsdorf
@ 2015-01-08 19:21 ` Linus Torvalds
2015-01-09 23:27 ` Catalin Marinas
0 siblings, 1 reply; 50+ messages in thread
From: Linus Torvalds @ 2015-01-08 19:21 UTC (permalink / raw)
To: linux-arm-kernel
On Thu, Jan 8, 2015 at 10:48 AM, Mark Langsdorf <mlangsdo@redhat.com> wrote:
>
> With 4K pages, the oom killer doesn't trigger during `make -j 16 -s`
> on a fresh kernel. Thanks for the suggestion. I'm not sure what to do
> about that, though.
I suspect that bisecting remains the best option.
A 64kB allocation will make various memory pressure issues *much*
worse, and quite frankly, I consider 64kB pages completely
unacceptable for any real life situation for that and other reasons,
but if it used to work, we do want to figure out what actually broke.
The only excuse for 64kB pages is "my hardware TLB is complete crap,
and I have very specialized server-only loads".
People who think 64kB pages are a good idea are wrong. It's that
simple. You seem to be trying to compile stuff, which is not a good
load for 64kB pages (notably the page cache becomes 90% unused due to
wasted space at the end of pages).
But while 64kB pages are a completely braindead idea, the actual *bug*
is likely not the fact that 64kB page thing in itself, but some other
issue that gets triggered by it. Maybe we have some piece of code that
"knows" that a page is 4kB and mis-accounts things due to that, or
similar.
Linus
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-08 12:51 ` Linux 3.19-rc3 Mark Langsdorf
2015-01-08 13:45 ` Catalin Marinas
2015-01-08 15:08 ` Michal Hocko
@ 2015-01-09 12:13 ` Mark Rutland
2015-01-09 14:19 ` Steve Capper
2 siblings, 1 reply; 50+ messages in thread
From: Mark Rutland @ 2015-01-09 12:13 UTC (permalink / raw)
To: linux-arm-kernel
On Thu, Jan 08, 2015 at 12:51:31PM +0000, Mark Langsdorf wrote:
> On 01/05/2015 07:46 PM, Linus Torvalds wrote:
> > It's a day delayed - not because of any particular development issues,
> > but simply because I was tiling a bathroom yesterday. But rc3 is out
> > there now, and things have stayed reasonably calm. I really hope that
> > implies that 3.19 is looking good, but it's equally likely that it's
> > just that people are still recovering from the holiday season.
>
> I'm consistently getting an out of memory killer triggered when
> compiling the kernel (make -j 16 -s) on a 16 core ARM64 system
> with 16 GB of memory. This doesn't happen when running a 3.18
> kernel.
>
> I'm going to start bisecting the failure now, but here's the crash
> log in case someone can see something obvious in it.
FWIW I've just reproduced this with v3.19-rc3 defconfig +
CONFIG_ARM64_64K_PAGES=y by attempting a git clone of mainline. My
system has 16GB of RAM and 6 CPUs.
I have a similarly dodgy looking number of pages reserved
(18446744073709544451 A.K.A. -7165). Log below.
Thanks,
Mark.
git invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
CPU: 2 PID: 9777 Comm: git Not tainted 3.19.0-rc3+ #37
Call trace:
[<fffffe0000096b4c>] dump_backtrace+0x0/0x124
[<fffffe0000096c80>] show_stack+0x10/0x1c
[<fffffe0000552f40>] dump_stack+0x80/0xc4
[<fffffe000013c148>] dump_header.isra.9+0x80/0x1c4
[<fffffe000013c868>] oom_kill_process+0x390/0x3f0
[<fffffe000013cdb0>] out_of_memory+0x2f0/0x324
[<fffffe0000141178>] __alloc_pages_nodemask+0x860/0x874
[<fffffe0000161ae4>] handle_mm_fault+0x7c0/0xe7c
[<fffffe000009f8d0>] do_page_fault+0x188/0x2f8
[<fffffe0000090230>] do_mem_abort+0x38/0x9c
Exception stack(0xfffffe03bf877e30 to 0xfffffe03bf877f50)
7e20: 00000000 00000000 84000078 000003ff
7e40: ffffffff ffffffff abc27210 000003ff 00000006 00000000 001971a4 fffffe00
7e60: bf877ec0 fffffe03 0019725c fffffe00 00000000 00000000 00000028 00000000
7e80: ffffffff ffffffff abc6c718 000003ff 00000000 00000000 00000015 00000000
7ea0: 0000011a 00000000 00024800 00000000 00000024 00000100 00000003 fffffe03
7ec0: 93f7d6c0 000003ff 000939b0 fffffe00 85879770 000003ff 858817b0 000003ff
7ee0: 0000e851 00000000 00008045 00000000 00008041 00000000 04b10000 00000000
7f00: 00008060 00000000 85890000 000003ff 00000038 00000000 6f6c72ff 62606f5e
7f20: 00000040 00000000 01010101 01010101 00000076 00000000 00040000 00000000
7f40: 09fecda5 00000000 ec5a90c4 00000000
Mem-Info:
DMA per-cpu:
CPU 0: hi: 6, btch: 1 usd: 4
CPU 1: hi: 6, btch: 1 usd: 5
CPU 2: hi: 6, btch: 1 usd: 4
CPU 3: hi: 6, btch: 1 usd: 5
CPU 4: hi: 6, btch: 1 usd: 5
CPU 5: hi: 6, btch: 1 usd: 2
Normal per-cpu:
CPU 0: hi: 6, btch: 1 usd: 1
CPU 1: hi: 6, btch: 1 usd: 5
CPU 2: hi: 6, btch: 1 usd: 0
CPU 3: hi: 6, btch: 1 usd: 5
CPU 4: hi: 6, btch: 1 usd: 4
CPU 5: hi: 6, btch: 1 usd: 5
active_anon:241994 inactive_anon:226 isolated_anon:0
active_file:1063 inactive_file:1075 isolated_file:0
unevictable:0 dirty:0 writeback:0 unstable:0
free:13439 slab_reclaimable:363 slab_unreclaimable:1137
mapped:188 shmem:146 pagetables:94 bounce:0
free_cma:2401
DMA free:243840kB min:208000kB low:259968kB high:312000kB active_anon:3800448kB inactive_anon:2624kB active_file:8576kB inactive_file:9344kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:4177920kB managed:4166464kB mlocked:0kB dirty:0kB writeback:0kB mapped:1152kB shmem:2304kB slab_reclaimable:4480kB slab_unreclaimable:16576kB kernel_stack:128kB pagetables:2688kB unstable:0kB bounce:0kB free_cma:153664kB writeback_tmp:0kB pages_scanned:201984 all_unreclaimable? yes
lowmem_reserve[]: 0 764 764
Normal free:616256kB min:625472kB low:781824kB high:938176kB active_anon:11687168kB inactive_anon:11840kB active_file:59456kB inactive_file:59456kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:12582912kB managed:12528640kB mlocked:0kB dirty:0kB writeback:0kB mapped:10880kB shmem:7040kB slab_reclaimable:18752kB slab_unreclaimable:56192kB kernel_stack:2032kB pagetables:3328kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2989056 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0
DMA: 1563*64kB (URC) 527*128kB (UC) 0*256kB 1*512kB (R) 0*1024kB 1*2048kB (R) 0*4096kB 1*8192kB (R) 0*16384kB 0*32768kB 1*65536kB (R) 0*131072kB 0*262144kB 0*524288kB = 243776kB
Normal: 5*64kB (MR) 4*128kB (MR) 4*256kB (R) 2*512kB (MR) 3*1024kB (MR) 2*2048kB (MR) 4*4096kB (MR) 2*8192kB (MR) 3*16384kB (MR) 4*32768kB (MR) 0*65536kB 1*131072kB (R) 1*262144kB (R) 0*524288kB = 616256kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=524288kB
1266 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 0kB
Total swap = 0kB
261888 pages RAM
0 pages HighMem/MovableOnly
18446744073709544451 pages reserved
8192 pages cma reserved
[ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[ 930] 0 930 57 13 2 0 0 upstart-udev-br
[ 935] 0 935 195 60 4 0 -1000 systemd-udevd
[ 1252] 0 1252 51 11 2 0 0 upstart-file-br
[ 1254] 101 1254 3538 56 3 0 0 rsyslogd
[ 1278] 104 1278 88 49 4 0 0 dbus-daemon
[ 1338] 0 1338 96 52 4 0 0 systemd-logind
[ 1376] 0 1376 50 11 2 0 0 upstart-socket-
[ 1416] 0 1416 3682 144 3 0 0 ModemManager
[ 1527] 0 1527 74 34 4 0 0 getty
[ 1528] 0 1528 3872 145 5 0 0 NetworkManager
[ 1529] 0 1529 74 35 4 0 0 getty
[ 1534] 0 1534 74 34 4 0 0 getty
[ 1535] 0 1535 74 34 3 0 0 getty
[ 1537] 0 1537 74 34 3 0 0 getty
[ 1552] 0 1552 151 92 3 0 -1000 sshd
[ 1561] 0 1561 63 38 3 0 0 cron
[ 1565] 0 1565 3602 89 4 0 0 polkitd
[ 1604] 0 1604 126 72 4 0 0 login
[ 1606] 0 1606 74 34 3 0 0 getty
[ 1621] 0 1621 99 68 3 0 0 dhclient
[ 1626] 65534 1626 67 48 4 0 0 dnsmasq
[ 1716] 1000 1716 72 45 4 0 0 bash
[ 1730] 0 1730 238 120 2 0 0 sshd
[ 1743] 1000 1743 238 94 2 0 0 sshd
[ 1744] 1000 1744 96 50 4 0 0 bash
[ 9767] 1000 9767 240 58 3 0 0 git
[ 9773] 1000 9773 11894 9252 6 0 0 git
Out of memory: Kill process 9773 (git) score 35 or sacrifice child
Killed process 9773 (git) total-vm:761216kB, anon-rss:589376kB, file-rss:2752kB
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-09 12:13 ` Mark Rutland
@ 2015-01-09 14:19 ` Steve Capper
2015-01-09 14:27 ` Mark Langsdorf
0 siblings, 1 reply; 50+ messages in thread
From: Steve Capper @ 2015-01-09 14:19 UTC (permalink / raw)
To: linux-arm-kernel
On 9 January 2015 at 12:13, Mark Rutland <mark.rutland@arm.com> wrote:
> On Thu, Jan 08, 2015 at 12:51:31PM +0000, Mark Langsdorf wrote:
>> On 01/05/2015 07:46 PM, Linus Torvalds wrote:
>> > It's a day delayed - not because of any particular development issues,
>> > but simply because I was tiling a bathroom yesterday. But rc3 is out
>> > there now, and things have stayed reasonably calm. I really hope that
>> > implies that 3.19 is looking good, but it's equally likely that it's
>> > just that people are still recovering from the holiday season.
>>
>> I'm consistently getting an out of memory killer triggered when
>> compiling the kernel (make -j 16 -s) on a 16 core ARM64 system
>> with 16 GB of memory. This doesn't happen when running a 3.18
>> kernel.
>>
>> I'm going to start bisecting the failure now, but here's the crash
>> log in case someone can see something obvious in it.
>
> FWIW I've just reproduced this with v3.19-rc3 defconfig +
> CONFIG_ARM64_64K_PAGES=y by attempting a git clone of mainline. My
> system has 16GB of RAM and 6 CPUs.
>
> I have a similarly dodgy looking number of pages reserved
> (18446744073709544451 A.K.A. -7165). Log below.
>
I think the negative page reserved count is a consequence of another bug.
We have the following reporting code in lib/show_mem.c:
#ifdef CONFIG_CMA
printk("%lu pages reserved\n", (reserved - totalcma_pages));
printk("%lu pages cma reserved\n", totalcma_pages);
#else
With totalcma_pages being reported as 8192, that would account for the
-7000ish values reported.
That change appears to have come from:
49abd8c lib/show_mem.c: add cma reserved information
Is the quickest way to exacerbate this OOM a kernel compile?
Cheers,
--
Steve
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-09 14:19 ` Steve Capper
@ 2015-01-09 14:27 ` Mark Langsdorf
2015-01-09 17:57 ` Mark Rutland
0 siblings, 1 reply; 50+ messages in thread
From: Mark Langsdorf @ 2015-01-09 14:27 UTC (permalink / raw)
To: linux-arm-kernel
On 01/09/2015 08:19 AM, Steve Capper wrote:
> On 9 January 2015 at 12:13, Mark Rutland <mark.rutland@arm.com> wrote:
>> On Thu, Jan 08, 2015 at 12:51:31PM +0000, Mark Langsdorf wrote:
>>> I'm consistently getting an out of memory killer triggered when
>>> compiling the kernel (make -j 16 -s) on a 16 core ARM64 system
>>> with 16 GB of memory. This doesn't happen when running a 3.18
>>> kernel.
>>>
>>> I'm going to start bisecting the failure now, but here's the crash
>>> log in case someone can see something obvious in it.
>>
>> FWIW I've just reproduced this with v3.19-rc3 defconfig +
>> CONFIG_ARM64_64K_PAGES=y by attempting a git clone of mainline. My
>> system has 16GB of RAM and 6 CPUs.
>>
>> I have a similarly dodgy looking number of pages reserved
>> (18446744073709544451 A.K.A. -7165). Log below.
>>
>
> I think the negative page reserved count is a consequence of another bug.
>
> We have the following reporting code in lib/show_mem.c:
> #ifdef CONFIG_CMA
> printk("%lu pages reserved\n", (reserved - totalcma_pages));
> printk("%lu pages cma reserved\n", totalcma_pages);
> #else
>
> With totalcma_pages being reported as 8192, that would account for the
> -7000ish values reported.
>
> That change appears to have come from:
> 49abd8c lib/show_mem.c: add cma reserved information
>
> Is the quickest way to exacerbate this OOM a kernel compile?
I haven't really tried to characterize this. Compiling a kernel
on a 64K page machine causes a failure reasonably quickly and
doesn't require a lot of thought. I think that time spent finding
a faster reproducer wouldn't pay off.
Also, contrary to last night's report and in line with Linus'
assumption, the failure still occurs with 4K pages. It just
takes substantially longer to occur.
--Mark Langsdorf
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-08 16:37 ` Mark Langsdorf
@ 2015-01-09 15:56 ` Michal Hocko
0 siblings, 0 replies; 50+ messages in thread
From: Michal Hocko @ 2015-01-09 15:56 UTC (permalink / raw)
To: linux-arm-kernel
On Thu 08-01-15 10:37:50, Mark Langsdorf wrote:
> On 01/08/2015 09:08 AM, Michal Hocko wrote:
> >[CCing linux-mm and CMA people]
> >[Full message here:
> >http://article.gmane.org/gmane.linux.ports.arm.kernel/383669]
>
> >>[ 1054.095277] DMA: 109*64kB (UR) 53*128kB (R) 8*256kB (R) 0*512kB 0*1024kB
> >>0*2048kB 1*4096kB (R) 0*8192kB 0*16384kB 1*32768kB (R) 0*65536kB = 52672kB
> >>[ 1054.108621] Normal: 191*64kB (MR) 0*128kB 0*256kB 0*512kB 0*1024kB
> >>0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 12224kB
> >[...]
> >>[ 1054.142545] Free swap = 6598400kB
> >>[ 1054.145928] Total swap = 8388544kB
> >>[ 1054.149317] 262112 pages RAM
> >>[ 1054.152180] 0 pages HighMem/MovableOnly
> >>[ 1054.155995] 18446744073709544361 pages reserved
> >>[ 1054.160505] 8192 pages cma reserved
> >
> >Besides underflow in the reserved pages accounting mentioned in other
> >email the free lists look strange as well. All free blocks with some memory
> >are marked as reserved. I would suspect something CMA related.
>
> I get the same failure with CMA turned off entirely. I assume that means
> CMA is not the culprit.
OK. Do you see all the free page blocks completely reserved without CMA
as well?
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-09 14:27 ` Mark Langsdorf
@ 2015-01-09 17:57 ` Mark Rutland
2015-01-09 18:37 ` Marc Zyngier
0 siblings, 1 reply; 50+ messages in thread
From: Mark Rutland @ 2015-01-09 17:57 UTC (permalink / raw)
To: linux-arm-kernel
On Fri, Jan 09, 2015 at 02:27:06PM +0000, Mark Langsdorf wrote:
> On 01/09/2015 08:19 AM, Steve Capper wrote:
> > On 9 January 2015 at 12:13, Mark Rutland <mark.rutland@arm.com> wrote:
> >> On Thu, Jan 08, 2015 at 12:51:31PM +0000, Mark Langsdorf wrote:
> >>> I'm consistently getting an out of memory killer triggered when
> >>> compiling the kernel (make -j 16 -s) on a 16 core ARM64 system
> >>> with 16 GB of memory. This doesn't happen when running a 3.18
> >>> kernel.
> >>>
> >>> I'm going to start bisecting the failure now, but here's the crash
> >>> log in case someone can see something obvious in it.
> >>
> >> FWIW I've just reproduced this with v3.19-rc3 defconfig +
> >> CONFIG_ARM64_64K_PAGES=y by attempting a git clone of mainline. My
> >> system has 16GB of RAM and 6 CPUs.
> >>
> >> I have a similarly dodgy looking number of pages reserved
> >> (18446744073709544451 A.K.A. -7165). Log below.
> >>
> >
> > I think the negative page reserved count is a consequence of another bug.
> >
> > We have the following reporting code in lib/show_mem.c:
> > #ifdef CONFIG_CMA
> > printk("%lu pages reserved\n", (reserved - totalcma_pages));
> > printk("%lu pages cma reserved\n", totalcma_pages);
> > #else
> >
> > With totalcma_pages being reported as 8192, that would account for the
> > -7000ish values reported.
> >
> > That change appears to have come from:
> > 49abd8c lib/show_mem.c: add cma reserved information
> >
> > Is the quickest way to exacerbate this OOM a kernel compile?
>
> I haven't really tried to characterize this. Compiling a kernel
> on a 64K page machine causes a failure reasonably quickly and
> doesn't require a lot of thought. I think that time spent finding
> a faster reproducer wouldn't pay off.
I wasn't able to trigger the issue again with git, and the only way I've
managed to trigger the issue is repeatedly building the kernel in a
loop:
while true; do
git clean -fdx > /dev/null 2>&1;
make defconfig > /dev/null 2>&1;
make > /dev/null > 2>&1;
done
Which after a while died:
-bash: fork: Cannot allocate memory
I didn't see anything interesting in dmesg, but I was able to get at
/proc/meminfo:
MemTotal: 16695168 kB
MemFree: 998336 kB
MemAvailable: 325568 kB
Buffers: 51200 kB
Cached: 236224 kB
SwapCached: 0 kB
Active: 14970880 kB
Inactive: 580288 kB
Active(anon): 14834496 kB
Inactive(anon): 5760 kB
Active(file): 136384 kB
Inactive(file): 574528 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 448 kB
Writeback: 0 kB
AnonPages: 22400 kB
Mapped: 10240 kB
Shmem: 8768 kB
Slab: 63744 kB
SReclaimable: 27072 kB
SUnreclaim: 36672 kB
KernelStack: 1824 kB
PageTables: 3776 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 8347584 kB
Committed_AS: 50368 kB
VmallocTotal: 2142764992 kB
VmallocUsed: 283264 kB
VmallocChunk: 2142387200 kB
AnonHugePages: 0 kB
CmaTotal: 524288 kB
CmaFree: 128 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 524288 kB
And also magic-sysrq m:
SysRq : Show Memory
Mem-Info:
DMA per-cpu:
CPU 0: hi: 6, btch: 1 usd: 1
CPU 1: hi: 6, btch: 1 usd: 1
CPU 2: hi: 6, btch: 1 usd: 1
CPU 3: hi: 6, btch: 1 usd: 3
CPU 4: hi: 6, btch: 1 usd: 5
CPU 5: hi: 6, btch: 1 usd: 5
Normal per-cpu:
CPU 0: hi: 6, btch: 1 usd: 0
CPU 1: hi: 6, btch: 1 usd: 5
CPU 2: hi: 6, btch: 1 usd: 1
CPU 3: hi: 6, btch: 1 usd: 5
CPU 4: hi: 6, btch: 1 usd: 5
CPU 5: hi: 6, btch: 1 usd: 5
active_anon:231780 inactive_anon:90 isolated_anon:0
active_file:2131 inactive_file:8977 isolated_file:0
unevictable:0 dirty:8 writeback:0 unstable:0
free:15601 slab_reclaimable:423 slab_unreclaimable:573
mapped:160 shmem:137 pagetables:59 bounce:0
free_cma:2
DMA free:302336kB min:208000kB low:259968kB high:312000kB active_anon:3618432kB inactive_anon:768kB active_file:34432kB inactive_file:131584kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:4177920kB managed:4166528kB mlocked:0kB dirty:192kB writeback:0kB mapped:4736kB shmem:1024kB slab_reclaimable:5184kB slab_unreclaimable:3328kB kernel_stack:0kB pagetables:1600kB unstable:0kB bounce:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:1208448 all_unreclaimable? yes
lowmem_reserve[]: 0 764 764
Normal free:696128kB min:625472kB low:781824kB high:938176kB active_anon:11215488kB inactive_anon:4992kB active_file:101952kB inactive_file:442944kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:12582912kB managed:12528640kB mlocked:0kB dirty:320kB writeback:0kB mapped:5504kB shmem:7744kB slab_reclaimable:21888kB slab_unreclaimable:33344kB kernel_stack:1840kB pagetables:2176kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3331648 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0
DMA: 42*64kB (MRC) 37*128kB (R) 6*256kB (R) 5*512kB (R) 2*1024kB (R) 3*2048kB (R) 1*4096kB (R) 0*8192kB 1*16384kB (R) 0*32768kB 0*65536kB 0*131072kB 1*262144kB (R) 0*524288kB = 302336kB
Normal: 280*64kB (MR) 40*128kB (R) 5*256kB (R) 4*512kB (R) 6*1024kB (R) 4*2048kB (R) 1*4096kB (R) 1*8192kB (R) 1*16384kB (R) 1*32768kB (R) 1*65536kB (R) 0*131072kB 0*262144kB 1*524288kB (R) = 691968kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=524288kB
4492 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 0kB
Total swap = 0kB
261888 pages RAM
0 pages HighMem/MovableOnly
18446744073709544450 pages reserved
8192 pages cma reserved
I also ran ps aux, but I didn't see any stale tasks lying around, nor
did any remaining tasks seem to account for all that active anonymous
memory.
I'll see if I can reproduce on x86.
Thanks,
Mark.
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-09 17:57 ` Mark Rutland
@ 2015-01-09 18:37 ` Marc Zyngier
2015-01-09 19:43 ` Will Deacon
0 siblings, 1 reply; 50+ messages in thread
From: Marc Zyngier @ 2015-01-09 18:37 UTC (permalink / raw)
To: linux-arm-kernel
On 09/01/15 17:57, Mark Rutland wrote:
> On Fri, Jan 09, 2015 at 02:27:06PM +0000, Mark Langsdorf wrote:
>> On 01/09/2015 08:19 AM, Steve Capper wrote:
>>> On 9 January 2015 at 12:13, Mark Rutland <mark.rutland@arm.com> wrote:
>>>> On Thu, Jan 08, 2015 at 12:51:31PM +0000, Mark Langsdorf wrote:
>>>>> I'm consistently getting an out of memory killer triggered when
>>>>> compiling the kernel (make -j 16 -s) on a 16 core ARM64 system
>>>>> with 16 GB of memory. This doesn't happen when running a 3.18
>>>>> kernel.
>>>>>
>>>>> I'm going to start bisecting the failure now, but here's the crash
>>>>> log in case someone can see something obvious in it.
>>>>
>>>> FWIW I've just reproduced this with v3.19-rc3 defconfig +
>>>> CONFIG_ARM64_64K_PAGES=y by attempting a git clone of mainline. My
>>>> system has 16GB of RAM and 6 CPUs.
>>>>
>>>> I have a similarly dodgy looking number of pages reserved
>>>> (18446744073709544451 A.K.A. -7165). Log below.
>>>>
>>>
>>> I think the negative page reserved count is a consequence of another bug.
>>>
>>> We have the following reporting code in lib/show_mem.c:
>>> #ifdef CONFIG_CMA
>>> printk("%lu pages reserved\n", (reserved - totalcma_pages));
>>> printk("%lu pages cma reserved\n", totalcma_pages);
>>> #else
>>>
>>> With totalcma_pages being reported as 8192, that would account for the
>>> -7000ish values reported.
>>>
>>> That change appears to have come from:
>>> 49abd8c lib/show_mem.c: add cma reserved information
>>>
>>> Is the quickest way to exacerbate this OOM a kernel compile?
>>
>> I haven't really tried to characterize this. Compiling a kernel
>> on a 64K page machine causes a failure reasonably quickly and
>> doesn't require a lot of thought. I think that time spent finding
>> a faster reproducer wouldn't pay off.
>
> I wasn't able to trigger the issue again with git, and the only way I've
> managed to trigger the issue is repeatedly building the kernel in a
> loop:
>
> while true; do
> git clean -fdx > /dev/null 2>&1;
> make defconfig > /dev/null 2>&1;
> make > /dev/null > 2>&1;
> done
>
> Which after a while died:
>
> -bash: fork: Cannot allocate memory
>
> I didn't see anything interesting in dmesg, but I was able to get at
> /proc/meminfo:
>
> MemTotal: 16695168 kB
> MemFree: 998336 kB
> MemAvailable: 325568 kB
> Buffers: 51200 kB
> Cached: 236224 kB
> SwapCached: 0 kB
> Active: 14970880 kB
> Inactive: 580288 kB
> Active(anon): 14834496 kB
> Inactive(anon): 5760 kB
> Active(file): 136384 kB
> Inactive(file): 574528 kB
> Unevictable: 0 kB
> Mlocked: 0 kB
> SwapTotal: 0 kB
> SwapFree: 0 kB
> Dirty: 448 kB
> Writeback: 0 kB
> AnonPages: 22400 kB
> Mapped: 10240 kB
> Shmem: 8768 kB
> Slab: 63744 kB
> SReclaimable: 27072 kB
> SUnreclaim: 36672 kB
> KernelStack: 1824 kB
> PageTables: 3776 kB
> NFS_Unstable: 0 kB
> Bounce: 0 kB
> WritebackTmp: 0 kB
> CommitLimit: 8347584 kB
> Committed_AS: 50368 kB
> VmallocTotal: 2142764992 kB
> VmallocUsed: 283264 kB
> VmallocChunk: 2142387200 kB
> AnonHugePages: 0 kB
> CmaTotal: 524288 kB
> CmaFree: 128 kB
> HugePages_Total: 0
> HugePages_Free: 0
> HugePages_Rsvd: 0
> HugePages_Surp: 0
> Hugepagesize: 524288 kB
>
> And also magic-sysrq m:
>
> SysRq : Show Memory
> Mem-Info:
> DMA per-cpu:
> CPU 0: hi: 6, btch: 1 usd: 1
> CPU 1: hi: 6, btch: 1 usd: 1
> CPU 2: hi: 6, btch: 1 usd: 1
> CPU 3: hi: 6, btch: 1 usd: 3
> CPU 4: hi: 6, btch: 1 usd: 5
> CPU 5: hi: 6, btch: 1 usd: 5
> Normal per-cpu:
> CPU 0: hi: 6, btch: 1 usd: 0
> CPU 1: hi: 6, btch: 1 usd: 5
> CPU 2: hi: 6, btch: 1 usd: 1
> CPU 3: hi: 6, btch: 1 usd: 5
> CPU 4: hi: 6, btch: 1 usd: 5
> CPU 5: hi: 6, btch: 1 usd: 5
> active_anon:231780 inactive_anon:90 isolated_anon:0
> active_file:2131 inactive_file:8977 isolated_file:0
> unevictable:0 dirty:8 writeback:0 unstable:0
> free:15601 slab_reclaimable:423 slab_unreclaimable:573
> mapped:160 shmem:137 pagetables:59 bounce:0
> free_cma:2
> DMA free:302336kB min:208000kB low:259968kB high:312000kB active_anon:3618432kB inactive_anon:768kB active_file:34432kB inactive_file:131584kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:4177920kB managed:4166528kB mlocked:0kB dirty:192kB writeback:0kB mapped:4736kB shmem:1024kB slab_reclaimable:5184kB slab_unreclaimable:3328kB kernel_stack:0kB pagetables:1600kB unstable:0kB bounce:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:1208448 all_unreclaimable? yes
> lowmem_reserve[]: 0 764 764
> Normal free:696128kB min:625472kB low:781824kB high:938176kB active_anon:11215488kB inactive_anon:4992kB active_file:101952kB inactive_file:442944kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:12582912kB managed:12528640kB mlocked:0kB dirty:320kB writeback:0kB mapped:5504kB shmem:7744kB slab_reclaimable:21888kB slab_unreclaimable:33344kB kernel_stack:1840kB pagetables:2176kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3331648 all_unreclaimable? yes
> lowmem_reserve[]: 0 0 0
> DMA: 42*64kB (MRC) 37*128kB (R) 6*256kB (R) 5*512kB (R) 2*1024kB (R) 3*2048kB (R) 1*4096kB (R) 0*8192kB 1*16384kB (R) 0*32768kB 0*65536kB 0*131072kB 1*262144kB (R) 0*524288kB = 302336kB
> Normal: 280*64kB (MR) 40*128kB (R) 5*256kB (R) 4*512kB (R) 6*1024kB (R) 4*2048kB (R) 1*4096kB (R) 1*8192kB (R) 1*16384kB (R) 1*32768kB (R) 1*65536kB (R) 0*131072kB 0*262144kB 1*524288kB (R) = 691968kB
> Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=524288kB
> 4492 total pagecache pages
> 0 pages in swap cache
> Swap cache stats: add 0, delete 0, find 0/0
> Free swap = 0kB
> Total swap = 0kB
> 261888 pages RAM
> 0 pages HighMem/MovableOnly
> 18446744073709544450 pages reserved
> 8192 pages cma reserved
>
> I also ran ps aux, but I didn't see any stale tasks lying around, nor
> did any remaining tasks seem to account for all that active anonymous
> memory.
>
> I'll see if I can reproduce on x86.
Just as another data point: I'm reproducing the exact same thing (it
only took a couple of kernel builds to kill the box), with almost all
16GB of RAM stuck in Active(anon). I do *not* have CMA enabled though.
I've kicked another run with 4k pages.
M.
--
Jazz is not dead. It just smells funny...
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-09 18:37 ` Marc Zyngier
@ 2015-01-09 19:43 ` Will Deacon
2015-01-10 3:29 ` Laszlo Ersek
0 siblings, 1 reply; 50+ messages in thread
From: Will Deacon @ 2015-01-09 19:43 UTC (permalink / raw)
To: linux-arm-kernel
On Fri, Jan 09, 2015 at 06:37:36PM +0000, Marc Zyngier wrote:
> On 09/01/15 17:57, Mark Rutland wrote:
> > On Fri, Jan 09, 2015 at 02:27:06PM +0000, Mark Langsdorf wrote:
> >> On 01/09/2015 08:19 AM, Steve Capper wrote:
> >>> On 9 January 2015 at 12:13, Mark Rutland <mark.rutland@arm.com> wrote:
> >>>> On Thu, Jan 08, 2015 at 12:51:31PM +0000, Mark Langsdorf wrote:
> >>>>> I'm consistently getting an out of memory killer triggered when
> >>>>> compiling the kernel (make -j 16 -s) on a 16 core ARM64 system
> >>>>> with 16 GB of memory. This doesn't happen when running a 3.18
> >>>>> kernel.
> >>>>>
> >>>>> I'm going to start bisecting the failure now, but here's the crash
> >>>>> log in case someone can see something obvious in it.
> >>>>
> >>>> FWIW I've just reproduced this with v3.19-rc3 defconfig +
> >>>> CONFIG_ARM64_64K_PAGES=y by attempting a git clone of mainline. My
> >>>> system has 16GB of RAM and 6 CPUs.
[...]
> > I wasn't able to trigger the issue again with git, and the only way I've
> > managed to trigger the issue is repeatedly building the kernel in a
> > loop:
> >
> > while true; do
> > git clean -fdx > /dev/null 2>&1;
> > make defconfig > /dev/null 2>&1;
> > make > /dev/null > 2>&1;
> > done
> >
> > Which after a while died:
> >
> > -bash: fork: Cannot allocate memory
[...]
> Just as another data point: I'm reproducing the exact same thing (it
> only took a couple of kernel builds to kill the box), with almost all
> 16GB of RAM stuck in Active(anon). I do *not* have CMA enabled though.
>
> I've kicked another run with 4k pages.
The `mallocstress' tool from LTP seems to be a quick way to reproduce
the memory leak behind this (leaks 5/8GB on my Juno). It spawns a bunch
of threads, that each call malloc until it returns NULL. I thought maybe
we're leaking page tables, but 5GB is pretty excessive.
However, I'm unable to reproduce the problem under a 32-bit kernel on my
TC2 board or on 3.18 + the 3.19 merge window pull for arm64.
I guess we should try to bisect using the above.
Will
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-08 19:21 ` Linus Torvalds
@ 2015-01-09 23:27 ` Catalin Marinas
2015-01-10 0:35 ` Kirill A. Shutemov
0 siblings, 1 reply; 50+ messages in thread
From: Catalin Marinas @ 2015-01-09 23:27 UTC (permalink / raw)
To: linux-arm-kernel
On Thu, Jan 08, 2015 at 07:21:02PM +0000, Linus Torvalds wrote:
> The only excuse for 64kB pages is "my hardware TLB is complete crap,
> and I have very specialized server-only loads".
I would make a slight correction: s/and/or/.
I agree that for a general purpose system (and even systems like web
hosting servers), 64KB is overkill; 16KB may be a better compromise.
There are however some specialised loads that benefit from this. The
main example here is virtualisation where if both guest and host use 4
levels of page tables each (that's what you may get with 4KB pages on
arm64), a full TLB miss in both stages of translation (the ARM
terminology for nested page tables) needs up to _24_ memory accesses
(though cached). Of course, once the TLB warms up, there will be much
less but for new mmaps you always get some misses.
With 64KB pages (in the host usually), you can reduce the page table
levels to three or two (the latter for 42-bit VA) or you could even
couple this with some insanely huge pages (512MB, the next up from 64KB)
to decrease the number of levels further.
I see three main advantages: the usual reduced TLB pressure (which
arguably can be solved with bigger TLBs), less TLB misses and, pretty
important with virtualisation, the cost of the TLB miss due to a reduced
number of levels. But that's for the user to balance the advantages and
disadvantages you already mentioned based on the planned workload (e.g.
host configured with 64KB pages while guests use 4KB).
Another aspect on ARM is the TLB flushing on (large) MP systems. With a
larger page size, we reduce the number of TLB operation (in-hardware)
broadcasting between CPUs (we could use non-broadcasting ops and IPIs,
not sure they are any faster though).
--
Catalin
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-09 23:27 ` Catalin Marinas
@ 2015-01-10 0:35 ` Kirill A. Shutemov
2015-01-10 2:27 ` Linus Torvalds
0 siblings, 1 reply; 50+ messages in thread
From: Kirill A. Shutemov @ 2015-01-10 0:35 UTC (permalink / raw)
To: linux-arm-kernel
On Fri, Jan 09, 2015 at 11:27:07PM +0000, Catalin Marinas wrote:
> On Thu, Jan 08, 2015 at 07:21:02PM +0000, Linus Torvalds wrote:
> > The only excuse for 64kB pages is "my hardware TLB is complete crap,
> > and I have very specialized server-only loads".
>
> I would make a slight correction: s/and/or/.
>
> I agree that for a general purpose system (and even systems like web
> hosting servers), 64KB is overkill; 16KB may be a better compromise.
>
> There are however some specialised loads that benefit from this. The
> main example here is virtualisation where if both guest and host use 4
> levels of page tables each (that's what you may get with 4KB pages on
> arm64), a full TLB miss in both stages of translation (the ARM
> terminology for nested page tables) needs up to _24_ memory accesses
> (though cached). Of course, once the TLB warms up, there will be much
> less but for new mmaps you always get some misses.
>
> With 64KB pages (in the host usually), you can reduce the page table
> levels to three or two (the latter for 42-bit VA) or you could even
> couple this with some insanely huge pages (512MB, the next up from 64KB)
> to decrease the number of levels further.
>
> I see three main advantages: the usual reduced TLB pressure (which
> arguably can be solved with bigger TLBs), less TLB misses and, pretty
> important with virtualisation, the cost of the TLB miss due to a reduced
> number of levels. But that's for the user to balance the advantages and
> disadvantages you already mentioned based on the planned workload (e.g.
> host configured with 64KB pages while guests use 4KB).
>
> Another aspect on ARM is the TLB flushing on (large) MP systems. With a
> larger page size, we reduce the number of TLB operation (in-hardware)
> broadcasting between CPUs (we could use non-broadcasting ops and IPIs,
> not sure they are any faster though).
With bigger page size there's also reduction in number of entities to
handle by kernel: less memory occupied by struct pages, fewer pages on
lru, etc.
Managing a lot of memory (TiB scale) with 4k chunks is just insane.
We will need to find a way to cluster memory together to manage it
reasonably. Whether it bigger base page size or some other mechanism.
Maybe THP? ;)
--
Kirill A. Shutemov
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-10 0:35 ` Kirill A. Shutemov
@ 2015-01-10 2:27 ` Linus Torvalds
2015-01-10 2:51 ` David Lang
` (2 more replies)
0 siblings, 3 replies; 50+ messages in thread
From: Linus Torvalds @ 2015-01-10 2:27 UTC (permalink / raw)
To: linux-arm-kernel
On Fri, Jan 9, 2015 at 4:35 PM, Kirill A. Shutemov <kirill@shutemov.name> wrote:
>
> With bigger page size there's also reduction in number of entities to
> handle by kernel: less memory occupied by struct pages, fewer pages on
> lru, etc.
Bah. Humbug. You've been listening to HW people too much, but they see
the small details, not the big picture.
Page management was indeed a huge PITA for PAE. But that was because
PAE is garbage.
Really, do the math. 64kB pages waste so much memory due to internal
fragmentation that any other memory use is completely immaterial.
Pretty much the only situation where 64kB pages are fine is when you
have one single meaningful load running on your machine, and that one
single load is likely a database.
Any other time, your filesystem caches will just suck.
Just as an example, look at the kernel tree. Last I did the math
(which is admittedly a long time ago), the median size (not average -
median) of a file was pretty much 4kB.
With a 64kB page, that means that for caching the kernel tree (what,
closer to 50k files by now), you are basically wasting 60kB for most
source files. Say, 60kB * 30k files, or 1.8GB.
Read that again. 1.8 GIGABYTES. Wasted. Completely unused. Thrown
away, with no upside. And that's just from the page cache for the
kernel sources, which would be populated by a single "git grep".
Maybe things have changed, and maybe I did my math wrong, and people
can give a more exact number. But it's an example of why 64kB
granularity is completely unacceptable in any kind of general-purpose
load.
Anybody who brings up TLB costs or costs of maintaining pages is just
jying, or hasn't actually looked at the real issues. Anything you win
in TLB you lose is *lots* and lots of extra IO, because you aren't
using your memory efficiently for caching, and are basically easily
throwing away half your RAM.
Big pages are a bad bad bad idea. They work fine for databases, and
that's pretty much just about it. I'm sure there are some other loads,
but they are few and far between.
4kB works well. 8kB is perfectly acceptable. 16kB is already wasting a
lot of memory. 32kB and up is complete garbage for general-purpose
computing.
And no, source code isn't *that* special. There are lots of other
cases where you have a multitude of small files. And small files are
the things you want to cache - don't tell me about big video files etc
that make a 64kB page size look small, because those files don't
actually tend to be all that relevant for caching.
Linus
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-10 2:27 ` Linus Torvalds
@ 2015-01-10 2:51 ` David Lang
2015-01-10 3:06 ` Linus Torvalds
2015-01-13 3:33 ` Rik van Riel
2015-01-10 3:17 ` Tony Luck
2015-01-10 20:16 ` Arnd Bergmann
2 siblings, 2 replies; 50+ messages in thread
From: David Lang @ 2015-01-10 2:51 UTC (permalink / raw)
To: linux-arm-kernel
On Fri, 9 Jan 2015, Linus Torvalds wrote:
> Big pages are a bad bad bad idea. They work fine for databases, and
> that's pretty much just about it. I'm sure there are some other loads,
> but they are few and far between.
what about a dedicated virtualization host (where your workload is a handful of
virtual machines), would the file cache issue still be overwelming, even though
it's the virtual machines accessing things?
David Lang
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-10 2:51 ` David Lang
@ 2015-01-10 3:06 ` Linus Torvalds
2015-01-10 10:46 ` Andreas Mohr
2015-01-13 3:33 ` Rik van Riel
1 sibling, 1 reply; 50+ messages in thread
From: Linus Torvalds @ 2015-01-10 3:06 UTC (permalink / raw)
To: linux-arm-kernel
On Fri, Jan 9, 2015 at 6:51 PM, David Lang <david@lang.hm> wrote:
>
> what about a dedicated virtualization host (where your workload is a handful
> of virtual machines), would the file cache issue still be overwelming, even
> though it's the virtual machines accessing things?
How much filesystem caches does the host need or use? It can range
from basically zero ("pure" hypervisor host with no filesystem at all)
to 100% (virtual filesystem in all the clients with the host doing all
the real work).
I dunno. I do know that you definitely don't want to haev a
desktop/workstation with 64kB pages.
Linus
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-10 2:27 ` Linus Torvalds
2015-01-10 2:51 ` David Lang
@ 2015-01-10 3:17 ` Tony Luck
2015-01-10 20:16 ` Arnd Bergmann
2 siblings, 0 replies; 50+ messages in thread
From: Tony Luck @ 2015-01-10 3:17 UTC (permalink / raw)
To: linux-arm-kernel
On Fri, Jan 9, 2015 at 6:27 PM, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
> Big pages are a bad bad bad idea. They work fine for databases, and
> that's pretty much just about it. I'm sure there are some other loads,
> but they are few and far between.
For HPC too. They tend not to do a lot of I/O (and when they do it is
from a few big files). Then they just sit crunching over gigabytes of
memory for seven and a half million years before doing:
printf("Answer is %d\n", 42);
-Tony
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-09 19:43 ` Will Deacon
@ 2015-01-10 3:29 ` Laszlo Ersek
2015-01-10 4:39 ` Linus Torvalds
2015-01-10 15:22 ` Kyle McMartin
0 siblings, 2 replies; 50+ messages in thread
From: Laszlo Ersek @ 2015-01-10 3:29 UTC (permalink / raw)
To: linux-arm-kernel
On 01/09/15 20:43, Will Deacon wrote:
> On Fri, Jan 09, 2015 at 06:37:36PM +0000, Marc Zyngier wrote:
>> On 09/01/15 17:57, Mark Rutland wrote:
>>> On Fri, Jan 09, 2015 at 02:27:06PM +0000, Mark Langsdorf wrote:
>>>> On 01/09/2015 08:19 AM, Steve Capper wrote:
>>>>> On 9 January 2015 at 12:13, Mark Rutland <mark.rutland@arm.com> wrote:
>>>>>> On Thu, Jan 08, 2015 at 12:51:31PM +0000, Mark Langsdorf wrote:
>>>>>>> I'm consistently getting an out of memory killer triggered when
>>>>>>> compiling the kernel (make -j 16 -s) on a 16 core ARM64 system
>>>>>>> with 16 GB of memory. This doesn't happen when running a 3.18
>>>>>>> kernel.
>>>>>>>
>>>>>>> I'm going to start bisecting the failure now, but here's the crash
>>>>>>> log in case someone can see something obvious in it.
>>>>>>
>>>>>> FWIW I've just reproduced this with v3.19-rc3 defconfig +
>>>>>> CONFIG_ARM64_64K_PAGES=y by attempting a git clone of mainline. My
>>>>>> system has 16GB of RAM and 6 CPUs.
>
> [...]
>
>>> I wasn't able to trigger the issue again with git, and the only way I've
>>> managed to trigger the issue is repeatedly building the kernel in a
>>> loop:
>>>
>>> while true; do
>>> git clean -fdx > /dev/null 2>&1;
>>> make defconfig > /dev/null 2>&1;
>>> make > /dev/null > 2>&1;
>>> done
>>>
>>> Which after a while died:
>>>
>>> -bash: fork: Cannot allocate memory
>
> [...]
>
>> Just as another data point: I'm reproducing the exact same thing (it
>> only took a couple of kernel builds to kill the box), with almost all
>> 16GB of RAM stuck in Active(anon). I do *not* have CMA enabled though.
>>
>> I've kicked another run with 4k pages.
>
> The `mallocstress' tool from LTP seems to be a quick way to reproduce
> the memory leak behind this (leaks 5/8GB on my Juno). It spawns a bunch
> of threads, that each call malloc until it returns NULL. I thought maybe
> we're leaking page tables, but 5GB is pretty excessive.
>
> However, I'm unable to reproduce the problem under a 32-bit kernel on my
> TC2 board or on 3.18 + the 3.19 merge window pull for arm64.
>
> I guess we should try to bisect using the above.
>
> Will
>
I've bisected this issue to
> f045bbb9fa1bf6f507ad4de12d4e3471d8f672f1 is the first bad commit
> commit f045bbb9fa1bf6f507ad4de12d4e3471d8f672f1
> Author: Linus Torvalds <torvalds@linux-foundation.org>
> Date: Wed Dec 17 11:59:04 2014 -0800
>
> mmu_gather: fix over-eager tlb_flush_mmu_free() calling
>
> Dave Hansen reports that commit fb7332a9fedf ("mmu_gather: move minimal
> range calculations into generic code") caused a performance problem:
>
> "tlb_finish_mmu() goes up about 9x in the profiles (~0.4%->3.6%) and
> tlb_flush_mmu_free() takes about 3.1% of CPU time with the patch
> applied, but does not show up at all on the commit before"
>
> and the reason is that Will moved the test for whether we need to flush
> from tlb_flush_mmu() into tlb_flush_mmu_tlbonly(). But that meant that
> tlb_flush_mmu_free() basically lost that check.
>
> Move it back into tlb_flush_mmu() where it belongs, so that it covers
> both tlb_flush_mmu_tlbonly() _and_ tlb_flush_mmu_free().
>
> Reported-and-tested-by: Dave Hansen <dave@sr71.net>
> Acked-by: Will Deacon <will.deacon@arm.com>
> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
>
> :040000 040000 a4768484a068b37a43863123ac782fb6d01149b7 75ff0be6a2f3e4caa9a06df6503fa54d25dfa44d M mm
in an aarch64 QEMU/KVM guest running Fedora 21 Server (8GB RAM, 8
VCPUs). I bisected between v3.18 and v3.19-rc3. I used
"config-3.17.4-302.fc21.aarch64" as starting config, on which I kept
running olddefconfig and localmodconfig.
<tangent>
It is beneficial to perform such a bisection in a virtual machine. Super
fast reboot times, and the hot parts of the virtual disk are cached in
host memory. It's easy to boot a test kernel for running the reproducer,
and then reboot the known good distro kernel for building the next step
in the bisection.
</tangent>
As reproducer I used "./mallocstress -t 300" (recommended by Mark
Langsdorf & Kyle McMartin, but also named by Will just above this
thread).
One thing I noticed during the several repro turns is that the OOM
killer never hit while mallocstress was running "normally" (ie. before
the first thread exited). In the broken kernels, the OOM killer always
hit after a few (tens) of the threads had exited. The leak is probably
related to thread exit. (Which is consistent with the kernel build
reproducer, because that causes a lot of threads (processes) to exit
too.)
Bisection log below.
Thanks
Laszlo
git bisect start
# bad: [b1940cd21c0f4abdce101253e860feff547291b0] Linux 3.19-rc3
git bisect bad b1940cd21c0f4abdce101253e860feff547291b0
# good: [b2776bf7149bddd1f4161f14f79520f17fc1d71d] Linux 3.18
git bisect good b2776bf7149bddd1f4161f14f79520f17fc1d71d
# good: [a7cfef21e3d066343bec14d3113a9f9c92d1c2a8] Merge branches 'core', 'cxgb4', 'ipoib', 'iser', 'mlx4', 'ocrdma', 'odp' and 'srp' into for-next
git bisect good a7cfef21e3d066343bec14d3113a9f9c92d1c2a8
# good: [988adfdffdd43cfd841df734664727993076d7cb] Merge branch 'drm-next' of git://people.freedesktop.org/~airlied/linux
git bisect good 988adfdffdd43cfd841df734664727993076d7cb
# good: [eb64c3c6cdb8fa8a4d324eb71a9033b62e150918] Merge tag 'stable/for-linus-3.19-rc0b-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip
git bisect good eb64c3c6cdb8fa8a4d324eb71a9033b62e150918
# bad: [385336e321c41b5174055c0194b60c19a27cc5c5] Merge tag 'platform-drivers-x86-v3.19-1' of git://git.infradead.org/users/dvhart/linux-platform-drivers-x86
git bisect bad 385336e321c41b5174055c0194b60c19a27cc5c5
# bad: [87c31b39abcb6fb6bd7d111200c9627a594bf6a9] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/ebiederm/user-namespace
git bisect bad 87c31b39abcb6fb6bd7d111200c9627a594bf6a9
# good: [0ea90b9e79cff66934119e6dd8fa8e9d0f7d005a] Merge tag 'microblaze-3.19-rc1' of git://git.monstr.eu/linux-2.6-microblaze
git bisect good 0ea90b9e79cff66934119e6dd8fa8e9d0f7d005a
# good: [d797da41b2aceed5daa8cd2eee92cd74b2a0c652] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/dtor/input
git bisect good d797da41b2aceed5daa8cd2eee92cd74b2a0c652
# good: [c89d99546dc5b076ccd6692c48ada9a92820a4ac] Merge branch 'eduardo-soc-thermal' into thermal-soc
git bisect good c89d99546dc5b076ccd6692c48ada9a92820a4ac
# good: [9f3e15129902bca9d8e296c165345f158bac94eb] Merge tag 'virtio-next-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/rusty/linux
git bisect good 9f3e15129902bca9d8e296c165345f158bac94eb
# good: [80dd00a23784b384ccea049bfb3f259d3f973b9d] userns: Check euid no fsuid when establishing an unprivileged uid mapping
git bisect good 80dd00a23784b384ccea049bfb3f259d3f973b9d
# good: [db86da7cb76f797a1a8b445166a15cb922c6ff85] userns: Unbreak the unprivileged remount tests
git bisect good db86da7cb76f797a1a8b445166a15cb922c6ff85
# good: [cc669743a39e3f61c9ca5e786e959bf478ccd197] Merge tag 'vfio-v3.19-rc1' of git://github.com/awilliam/linux-vfio
git bisect good cc669743a39e3f61c9ca5e786e959bf478ccd197
# bad: [f045bbb9fa1bf6f507ad4de12d4e3471d8f672f1] mmu_gather: fix over-eager tlb_flush_mmu_free() calling
git bisect bad f045bbb9fa1bf6f507ad4de12d4e3471d8f672f1
# good: [cf3c0a1579eff90195a791c5f464463c1011ef4a] x86: mm: fix VM_FAULT_RETRY handling
git bisect good cf3c0a1579eff90195a791c5f464463c1011ef4a
# first bad commit: [f045bbb9fa1bf6f507ad4de12d4e3471d8f672f1] mmu_gather: fix over-eager tlb_flush_mmu_free() calling
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-10 3:29 ` Laszlo Ersek
@ 2015-01-10 4:39 ` Linus Torvalds
2015-01-10 13:37 ` Will Deacon
2015-01-10 15:22 ` Kyle McMartin
1 sibling, 1 reply; 50+ messages in thread
From: Linus Torvalds @ 2015-01-10 4:39 UTC (permalink / raw)
To: linux-arm-kernel
On Fri, Jan 9, 2015 at 7:29 PM, Laszlo Ersek <lersek@redhat.com> wrote:
>
> I've bisected this issue to
.. commit f045bbb9fa1b ("mmu_gather: fix over-eager
tlb_flush_mmu_free() calling")
Hmm. That commit literally just undoes something that commit
fb7332a9fedf ("mmu_gather: move minimal range calculations into
generic code") changed, and that was very wrong on x86.
But arm64 did have very different TLB flushing logic, so there may be
some ARM64 reason that Will did that change originally, and then he
forgot that reason when he ack'ed commit f045bbb9fa1b that undid it.
Will?
Before your mmu_gather range calculations commit, we used to have
In tlb_flush_mmu_tlbonly():
tlb->need_flush = 0;
and in tlb_flush_mmu():
if (!tlb->need_flush)
return;
and your commit changed the rule to be
!tlb->need_flush == !tlb->end
so in the current tree we have
In tlb_flush_mmu_tlbonly():
__tlb_reset_range(tlb); // replaces "tlb->need_flush = 0;"
and in tlb_flush_mmu():
if (!tlb->end) // replaces if (!tlb->need_flush)
return;
so we seem to do exactly the same as 3.18.
But in your original patch, you moved that "if (!tlb->end) return;"
check from tlb_flush_mmu() into tlb_flush_mmu_tlbonly(), and that
apparently is actually needed on arm64. But *why*?
Also, looking at that commit fb7332a9fedf, I note that some of the
"need_flush" setting was simply removed. See for example
arch/powerpc/mm/hugetlbpage.c, and also in mm/memory.c:
tlb_remove_table(). Is there something non-obvious that sets tlb->end
there?
The other need_flush removals seem to all be paired with adding a
__tlb_adjust_range() call, which will set ->end.
I'm starting to suspect that you moved the need_flush test into
tlbonly exactly because you removed that
tlb->need_flush = 1;
from mm/memory.c: tlb_remove_table().
x86 doesn't care, because x86 doesn't *use* tlb_remove_table(). But
arm64 does, at least with the RCU freeing.
Any ideas?
Linus
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-10 3:06 ` Linus Torvalds
@ 2015-01-10 10:46 ` Andreas Mohr
2015-01-10 19:42 ` Linus Torvalds
0 siblings, 1 reply; 50+ messages in thread
From: Andreas Mohr @ 2015-01-10 10:46 UTC (permalink / raw)
To: linux-arm-kernel
Linus Torvalds wrote:
> I dunno. I do know that you definitely don't want to haev a
> desktop/workstation with 64kB pages.
Yet that is what any VirtualAlloc() call on Windows does
(well, not exactly *page* granularity but *allocation* granularity there).
Prime example: do a naively specific/custom VirtualAlloc() request
for a simple string Hello World\0 allocation (11+1 bytes),
get one page (4kB, "Private Data") plus "overhead" (60kB, "Unusable").
--> allocation efficiency: 0.01831%(!).
And that does hurt plenty IME, especially on a 32bit address space's
very limited 2GB/3GB total per Win32 process.
http://blogs.microsoft.co.il/sasha/2014/07/22/tracking-unusable-virtual-memory-vmmap/
"Why is address space allocation granularity 64K?"
http://blogs.msdn.com/b/oldnewthing/archive/2003/10/08/55239.aspx
One thing less left to wonder why 'doze is such a performance pig...
Andreas Mohr
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-10 4:39 ` Linus Torvalds
@ 2015-01-10 13:37 ` Will Deacon
2015-01-10 19:47 ` Laszlo Ersek
2015-01-10 19:51 ` Linus Torvalds
0 siblings, 2 replies; 50+ messages in thread
From: Will Deacon @ 2015-01-10 13:37 UTC (permalink / raw)
To: linux-arm-kernel
Hi Linus, Laszlo,
On Sat, Jan 10, 2015 at 04:39:05AM +0000, Linus Torvalds wrote:
> On Fri, Jan 9, 2015 at 7:29 PM, Laszlo Ersek <lersek@redhat.com> wrote:
> > I've bisected this issue to
Thanks for bisecting this!
> .. commit f045bbb9fa1b ("mmu_gather: fix over-eager
> tlb_flush_mmu_free() calling")
>
> Hmm. That commit literally just undoes something that commit
> fb7332a9fedf ("mmu_gather: move minimal range calculations into
> generic code") changed, and that was very wrong on x86.
>
> But arm64 did have very different TLB flushing logic, so there may be
> some ARM64 reason that Will did that change originally, and then he
> forgot that reason when he ack'ed commit f045bbb9fa1b that undid it.
>
> Will?
I'm wondering if this is now broken in the fullmm case, because tlb->end
will be zero and we won't actually free any of the pages being unmapped
on task exit. Does that sound plausible?
> Before your mmu_gather range calculations commit, we used to have
>
> In tlb_flush_mmu_tlbonly():
>
> tlb->need_flush = 0;
>
> and in tlb_flush_mmu():
>
> if (!tlb->need_flush)
> return;
>
> and your commit changed the rule to be
>
> !tlb->need_flush == !tlb->end
>
> so in the current tree we have
>
> In tlb_flush_mmu_tlbonly():
>
> __tlb_reset_range(tlb); // replaces "tlb->need_flush = 0;"
>
> and in tlb_flush_mmu():
>
> if (!tlb->end) // replaces if (!tlb->need_flush)
> return;
>
> so we seem to do exactly the same as 3.18.
>
> But in your original patch, you moved that "if (!tlb->end) return;"
> check from tlb_flush_mmu() into tlb_flush_mmu_tlbonly(), and that
> apparently is actually needed on arm64. But *why*?
My hunch is that when a task exits and sets fullmm, end is zero and so the
old need_flush cases no longer run. With my original patch, we skipped the
TLB invalidation (since the task is exiting and we will invalidate the TLB
for that ASID before the ASID is reallocated) but still did the freeing.
With the current code, we skip the freeing too, which causes us to leak
pages on exit.
> Also, looking at that commit fb7332a9fedf, I note that some of the
> "need_flush" setting was simply removed. See for example
> arch/powerpc/mm/hugetlbpage.c, and also in mm/memory.c:
> tlb_remove_table(). Is there something non-obvious that sets tlb->end
> there?
I figured that need_flush was already set in these cases, so the additional
setting was redundant:
https://lkml.org/lkml/2014/11/10/340
> The other need_flush removals seem to all be paired with adding a
> __tlb_adjust_range() call, which will set ->end.
>
> I'm starting to suspect that you moved the need_flush test into
> tlbonly exactly because you removed that
>
> tlb->need_flush = 1;
>
> from mm/memory.c: tlb_remove_table().
>
> x86 doesn't care, because x86 doesn't *use* tlb_remove_table(). But
> arm64 does, at least with the RCU freeing.
>
> Any ideas?
I guess we can either check need_flush as well as end, or we could set both
start == end == some_nonzero_value in __tlb_adjust_range when need_flush is
set. Unfortunately, I'm away from my h/w right now, so it's not easy to test
this.
What do you reckon?
Will
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-10 3:29 ` Laszlo Ersek
2015-01-10 4:39 ` Linus Torvalds
@ 2015-01-10 15:22 ` Kyle McMartin
1 sibling, 0 replies; 50+ messages in thread
From: Kyle McMartin @ 2015-01-10 15:22 UTC (permalink / raw)
To: linux-arm-kernel
On Sat, Jan 10, 2015 at 04:29:39AM +0100, Laszlo Ersek wrote:
> I've bisected this issue to
>
Awesome, this was on my list of list of suspicious commits to check
before my ARM64 box decided not to come back from reboot on Friday. :)
Thanks for bisecting!
cheers,
--Kyle
> > f045bbb9fa1bf6f507ad4de12d4e3471d8f672f1 is the first bad commit
> > commit f045bbb9fa1bf6f507ad4de12d4e3471d8f672f1
> > Author: Linus Torvalds <torvalds@linux-foundation.org>
> > Date: Wed Dec 17 11:59:04 2014 -0800
> >
> > mmu_gather: fix over-eager tlb_flush_mmu_free() calling
> >
> > Dave Hansen reports that commit fb7332a9fedf ("mmu_gather: move minimal
> > range calculations into generic code") caused a performance problem:
> >
> > "tlb_finish_mmu() goes up about 9x in the profiles (~0.4%->3.6%) and
> > tlb_flush_mmu_free() takes about 3.1% of CPU time with the patch
> > applied, but does not show up at all on the commit before"
> >
> > and the reason is that Will moved the test for whether we need to flush
> > from tlb_flush_mmu() into tlb_flush_mmu_tlbonly(). But that meant that
> > tlb_flush_mmu_free() basically lost that check.
> >
> > Move it back into tlb_flush_mmu() where it belongs, so that it covers
> > both tlb_flush_mmu_tlbonly() _and_ tlb_flush_mmu_free().
> >
> > Reported-and-tested-by: Dave Hansen <dave@sr71.net>
> > Acked-by: Will Deacon <will.deacon@arm.com>
> > Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
> >
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-10 10:46 ` Andreas Mohr
@ 2015-01-10 19:42 ` Linus Torvalds
0 siblings, 0 replies; 50+ messages in thread
From: Linus Torvalds @ 2015-01-10 19:42 UTC (permalink / raw)
To: linux-arm-kernel
On Sat, Jan 10, 2015 at 2:46 AM, Andreas Mohr <andi@lisas.de> wrote:
>
> Yet that is what any VirtualAlloc() call on Windows does
> One thing less left to wonder why 'doze is such a performance pig...
Well, to be fair, you shouldn't use VirtualAlloc() as some 'malloc()'
replacement. It's more of a backing store allocator for malloc() and
friends than anything else.
Linus
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-10 13:37 ` Will Deacon
@ 2015-01-10 19:47 ` Laszlo Ersek
2015-01-10 19:56 ` Linus Torvalds
2015-01-10 19:51 ` Linus Torvalds
1 sibling, 1 reply; 50+ messages in thread
From: Laszlo Ersek @ 2015-01-10 19:47 UTC (permalink / raw)
To: linux-arm-kernel
On 01/10/15 14:37, Will Deacon wrote:
> My hunch is that when a task exits and sets fullmm, end is zero and so the
> old need_flush cases no longer run.
(Disclaimer: I'm completely unfamiliar with this code.)
If you have the following call chain in mind:
exit_mmap()
tlb_gather_mmu()
then I think that (fullmm != 0) precludes (end == 0).
I grepped the tree for "fullmm", and only tlb_gather_mmu() seems to set
it. There are several instances of that function, but each sets fullmm to:
/* Is it from 0 to ~0? */
tlb->fullmm = !(start | (end+1));
So, a nonzero fullmm seems to imply (end == ~0UL).
(And sure enough, exit_mmap() passes it ((unsigned long)-1) as "end").
> With my original patch, we skipped the
> TLB invalidation (since the task is exiting and we will invalidate the TLB
> for that ASID before the ASID is reallocated) but still did the freeing.
> With the current code, we skip the freeing too, which causes us to leak
> pages on exit.
Yes, the new check prevents
tlb_flush_mmu()
tlb_flush_mmu_free() <--- this
free_pages_and_swap_cache()
> I guess we can either check need_flush as well as end, or we could set both
> start == end == some_nonzero_value in __tlb_adjust_range when need_flush is
> set. Unfortunately, I'm away from my h/w right now, so it's not easy to test
> this.
If you have a patch that applies and builds, I'm glad to test it. I got
a few hours now and I'll have some tomorrow as well. (On Monday I guess
you'll have access to your hardware again.)
Thanks!
Laszlo
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-10 13:37 ` Will Deacon
2015-01-10 19:47 ` Laszlo Ersek
@ 2015-01-10 19:51 ` Linus Torvalds
2015-01-12 12:42 ` Will Deacon
1 sibling, 1 reply; 50+ messages in thread
From: Linus Torvalds @ 2015-01-10 19:51 UTC (permalink / raw)
To: linux-arm-kernel
On Sat, Jan 10, 2015 at 5:37 AM, Will Deacon <will.deacon@arm.com> wrote:
>>
>> Will?
>
> I'm wondering if this is now broken in the fullmm case, because tlb->end
> will be zero and we won't actually free any of the pages being unmapped
> on task exit. Does that sound plausible?
But did anything change wrt fullmm? I don't see any changes wrt fullmm
logic in generic code.
The arm64 code changed more, so maybe there was somethinig I missed.
Again, arm64 uses tlb_end_vma() etc, so arm64 certainly triggers code
that x86 does not.
I can revert the commit that causes problems, but considering the
performance impact on x86 (it would be a regression since 3.18), I
would *really* like to fix the arm64 problem instead. So I'll wait
with the revert for at least a week, I think, hoping that the arm64
people figure this out. Sound reasonable?
Linus
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-10 19:47 ` Laszlo Ersek
@ 2015-01-10 19:56 ` Linus Torvalds
2015-01-10 20:08 ` Laszlo Ersek
0 siblings, 1 reply; 50+ messages in thread
From: Linus Torvalds @ 2015-01-10 19:56 UTC (permalink / raw)
To: linux-arm-kernel
On Sat, Jan 10, 2015 at 11:47 AM, Laszlo Ersek <lersek@redhat.com> wrote:
>
> I grepped the tree for "fullmm", and only tlb_gather_mmu() seems to set
> it. There are several instances of that function, but each sets fullmm to:
>
> /* Is it from 0 to ~0? */
> tlb->fullmm = !(start | (end+1));
>
> So, a nonzero fullmm seems to imply (end == ~0UL).
Yes. But note how it imples "end == ~0ul", but it does *not* imply
"tlb->end" having that value.
tlb->end is initialized to zero (not obvious, but it's what the call
to __tlb_reset_range() does). It's then updated by
__tlb_adjust_range() as we actually flush individual pages.
Linus
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-10 19:56 ` Linus Torvalds
@ 2015-01-10 20:08 ` Laszlo Ersek
0 siblings, 0 replies; 50+ messages in thread
From: Laszlo Ersek @ 2015-01-10 20:08 UTC (permalink / raw)
To: linux-arm-kernel
On 01/10/15 20:56, Linus Torvalds wrote:
> On Sat, Jan 10, 2015 at 11:47 AM, Laszlo Ersek <lersek@redhat.com> wrote:
>>
>> I grepped the tree for "fullmm", and only tlb_gather_mmu() seems to set
>> it. There are several instances of that function, but each sets fullmm to:
>>
>> /* Is it from 0 to ~0? */
>> tlb->fullmm = !(start | (end+1));
>>
>> So, a nonzero fullmm seems to imply (end == ~0UL).
>
> Yes. But note how it imples "end == ~0ul", but it does *not* imply
> "tlb->end" having that value.
Ooops! :)
> tlb->end is initialized to zero (not obvious, but it's what the call
> to __tlb_reset_range() does). It's then updated by
> __tlb_adjust_range() as we actually flush individual pages.
Thanks.
Laszlo
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-10 2:27 ` Linus Torvalds
2015-01-10 2:51 ` David Lang
2015-01-10 3:17 ` Tony Luck
@ 2015-01-10 20:16 ` Arnd Bergmann
2015-01-10 21:00 ` Linus Torvalds
2015-01-12 11:53 ` Catalin Marinas
2 siblings, 2 replies; 50+ messages in thread
From: Arnd Bergmann @ 2015-01-10 20:16 UTC (permalink / raw)
To: linux-arm-kernel
On Friday 09 January 2015 18:27:38 Linus Torvalds wrote:
> On Fri, Jan 9, 2015 at 4:35 PM, Kirill A. Shutemov <kirill@shutemov.name> wrote:
> >
> > With bigger page size there's also reduction in number of entities to
> > handle by kernel: less memory occupied by struct pages, fewer pages on
> > lru, etc.
>
> Really, do the math. [...]
>
> With a 64kB page, that means that for caching the kernel tree (what,
> closer to 50k files by now), you are basically wasting 60kB for most
> source files. Say, 60kB * 30k files, or 1.8GB.
On a recent kernel, I get 628 MB for storing all files of the
kernel tree in 4KB pages, and 3141 MB for storing the same data
in 64KB pages, almost exactly factor 5, or 2.45 GiB wasted.
> Maybe things have changed, and maybe I did my math wrong, and people
> can give a more exact number. But it's an example of why 64kB
> granularity is completely unacceptable in any kind of general-purpose
> load.
I'd say it's unacceptable for any file backed mappings in general, but
usually an improvement for anonymous maps, for the same reasons that
transparent huge pages are great. IIRC, AIX works great with 64k
pages, but only because of two reasons that don't apply on Linux:
a) The PowerPC MMU can mix 4KB and 64KB pages in a single process.
Linux doesn't use this feature except for very special cases,
although it could be done on PowerPC but not most other architectures.
b) Linux has a unified page cache page size that is used for both
anonymous and file backed mappings. It's a great feature of the
Linux MM code (it avoids having two copies of each mapped file
in memory), but other OSs can just use 4KB blocks in the file
system cache independent of the page size.
> 4kB works well. 8kB is perfectly acceptable. 16kB is already wasting a
> lot of memory. 32kB and up is complete garbage for general-purpose
> computing.
I was expecting 16KB pages to work better, but you are right:
arnd:~/linux$ for i in 1 2 4 8 16 32 64 128 256 ; do echo -n "$i KiB pages: " ; total=0 ; git ls-files | xargs ls -ld | while read a b c d e f ; do echo $[((e + $i*1024 - 1) / (1024 * $i)) ] ; done | sort -n | uniq -c | while read num size ; do total=$[$total + ($num * $size) * $i] ; echo $[total / 1024] MiB ; done | tail -n 1 ; done
1 KiB pages: 544 MiB
2 KiB pages: 571 MiB
4 KiB pages: 628 MiB
8 KiB pages: 759 MiB
16 KiB pages: 1055 MiB
32 KiB pages: 1717 MiB
64 KiB pages: 3141 MiB
128 KiB pages: 6103 MiB
256 KiB pages: 12125 MiB
Regarding ARM64 in particular, I think it would be nice to investigate
how to extend the THP code to cover 64KB TLBs when running with the 4KB
page size. There is a hint bit in the page table to tell the CPU that
a set of 16 aligned pages can share one TLB, and it would be nice to
use that bit in Linux, and to make this case more common for anonymous
mappings, and possible large file based mappings.
Arnd
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-10 20:16 ` Arnd Bergmann
@ 2015-01-10 21:00 ` Linus Torvalds
2015-01-10 21:36 ` Arnd Bergmann
2015-01-12 11:53 ` Catalin Marinas
1 sibling, 1 reply; 50+ messages in thread
From: Linus Torvalds @ 2015-01-10 21:00 UTC (permalink / raw)
To: linux-arm-kernel
On Sat, Jan 10, 2015 at 12:16 PM, Arnd Bergmann <arnd@arndb.de> wrote:
>
> On a recent kernel, I get 628 MB for storing all files of the
> kernel tree in 4KB pages, and 3141 MB for storing the same data
> in 64KB pages, almost exactly factor 5, or 2.45 GiB wasted.
Ok, so it's even worse than it used to be. Partly because the tree
has grown, partly because I did the math for 16kB and 64kB is just
hugely worse.
I did the math back in the days when the PPC people were talking about
16kB pages (iirc - it's been closer to a decade, so I might
misremember the details).
And back then, with 4kB pages I could cache the kernel tree twice over
in 1GB, and have enough left to run a graphical desktop. So enough
memory to build a tree and also enough to have two kernel trees and do
"diff -urN" between them.
Of course, back then, 1-2GB was the usual desktop memory size, so the
"I can do kernel development in 1GB without excessive IO" mattered to
me in ways it wouldn't today.
And it was before "git", so the whole "two kernel trees and do diffs
between them" was a real concern.
With 16kB pages, I think I had to have twice the memory for the same loads.
> IIRC, AIX works great with 64k pages, but only because of two
> reasons that don't apply on Linux:
.. there's a few other ones:
(c) nobody really runs AIX on dekstops. It's very much a DB load
environment, with historically some HPC.
(d) the powerpc TLB fill/buildup/teardown costs are horrible, so on
AIX the cost of lots of small pages is much higher too.
Now obviously, we *could* try to have a 64kB page size, and then do
lots of tricks to actually allocate file caches in partial pages in
order to avoid the internal fragmentation costs. HOWEVER:
- that obviously doesn't help with the page management overhead (in
fact, it hurts). So it would be purely about trying to optimize for
bad TLB's.
- that adds a *lot* of complexity to the VM. The coherency issues
when you may need to move cached information between partial pages and
full pages (required for mmap, but *most* files don't get mmap'ed)
would actually be pretty horrible.
- all of this cost and complexity wouldn't help at all on x86, so it
would be largely untested and almost inevitably broken crap.
so I feel pretty confident in saying it won't happen. It's just too
much of a bother, for little to no actual upside. It's likely a much
better approach to try to instead use THP for anonymous mappings.
Linus
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-10 21:00 ` Linus Torvalds
@ 2015-01-10 21:36 ` Arnd Bergmann
2015-01-10 21:48 ` Linus Torvalds
` (2 more replies)
0 siblings, 3 replies; 50+ messages in thread
From: Arnd Bergmann @ 2015-01-10 21:36 UTC (permalink / raw)
To: linux-arm-kernel
On Saturday 10 January 2015 13:00:27 Linus Torvalds wrote:
>
> > IIRC, AIX works great with 64k pages, but only because of two
> > reasons that don't apply on Linux:
>
> .. there's a few other ones:
>
> (c) nobody really runs AIX on dekstops. It's very much a DB load
> environment, with historically some HPC.
>
> (d) the powerpc TLB fill/buildup/teardown costs are horrible, so on
> AIX the cost of lots of small pages is much higher too.
I think (d) applies to ARM as well, since it has no hardware
dirty/referenced bit tracking and requires the OS to mark the
pages as invalid/readonly until the first access. ARMv8.1
has a fix for that, but it's optional and we haven't seen any
implementations yet.
> so I feel pretty confident in saying it won't happen. It's just too
> much of a bother, for little to no actual upside. It's likely a much
> better approach to try to instead use THP for anonymous mappings.
arm64 already supports 2MB transparent hugepages. I guess it
wouldn't be too hard to change it so that an existing hugepage
on an anonymous mapping that gets split up into 4KB pages gets
split along 64KB boundaries with the contiguous mapping bit set.
Having full support for multiple hugepage sizes (64KB, 2MB and 32MB
in case of ARM64 with 4KB PAGE_SIZE) would be even better and
probably negate any benefits of 64KB PAGE_SIZE, but requires more
changes to common mm code.
Arnd
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-10 21:36 ` Arnd Bergmann
@ 2015-01-10 21:48 ` Linus Torvalds
2015-01-12 11:37 ` Kirill A. Shutemov
2015-01-12 12:18 ` Catalin Marinas
2 siblings, 0 replies; 50+ messages in thread
From: Linus Torvalds @ 2015-01-10 21:48 UTC (permalink / raw)
To: linux-arm-kernel
On Sat, Jan 10, 2015 at 1:36 PM, Arnd Bergmann <arnd@arndb.de> wrote:
>>
>> (d) the powerpc TLB fill/buildup/teardown costs are horrible, so on
>> AIX the cost of lots of small pages is much higher too.
>
> I think (d) applies to ARM as well, since it has no hardware
> dirty/referenced bit tracking and requires the OS to mark the
> pages as invalid/readonly until the first access. ARMv8.1
> has a fix for that, but it's optional and we haven't seen any
> implementations yet.
Powerpc really makes things worse by having those hashed page tables
that (a) have bad locality and (b) have to be built up and torn down
in software. I don't think ARM ends up coming close, even with the
issues it has.
Now, it's definitely true that the x86 page table handling hadrware
tends to just be superior. Both Intel and AMD had to work really hard
on it, because Windows (in the pre-NT days) used to flush the TLB
absolutely _all_ the time. So x86 hardware really does tend to do
very well on this.
ARM simply doesn't have the same kind of history. TLB issues seldom
show up very much on simple benchmarks or on smaller loads. It's one
of those things that tends to take a couple of generations.
[ Or, sadly, _much_ more, because some hardware designers never get
the memo, and continue to blame software and say "you should use big
pages", because they don't see the problems ]
Of course, it's entirely possible that vendors like AMD coudl transfer
their TLB handling know-how to their ARM64 cores. I have no visibility
into that, maybe some people here do..
Linus
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-10 21:36 ` Arnd Bergmann
2015-01-10 21:48 ` Linus Torvalds
@ 2015-01-12 11:37 ` Kirill A. Shutemov
2015-01-12 12:18 ` Catalin Marinas
2 siblings, 0 replies; 50+ messages in thread
From: Kirill A. Shutemov @ 2015-01-12 11:37 UTC (permalink / raw)
To: linux-arm-kernel
On Sat, Jan 10, 2015 at 10:36:13PM +0100, Arnd Bergmann wrote:
> On Saturday 10 January 2015 13:00:27 Linus Torvalds wrote:
> > so I feel pretty confident in saying it won't happen. It's just too
> > much of a bother, for little to no actual upside. It's likely a much
> > better approach to try to instead use THP for anonymous mappings.
>
> arm64 already supports 2MB transparent hugepages. I guess it
> wouldn't be too hard to change it so that an existing hugepage
> on an anonymous mapping that gets split up into 4KB pages gets
> split along 64KB boundaries with the contiguous mapping bit set.
What you are talking about is in fact multi-level transparent huge page
support: you need to couple 4k pages into 64k to avoid breaking them apart
by compactation or migration or whatever.
That definetely would not make THP code simplier.
> Having full support for multiple hugepage sizes (64KB, 2MB and 32MB
> in case of ARM64 with 4KB PAGE_SIZE) would be even better and
> probably negate any benefits of 64KB PAGE_SIZE, but requires more
> changes to common mm code.
>
> Arnd
--
Kirill A. Shutemov
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-10 20:16 ` Arnd Bergmann
2015-01-10 21:00 ` Linus Torvalds
@ 2015-01-12 11:53 ` Catalin Marinas
2015-01-12 13:15 ` Arnd Bergmann
1 sibling, 1 reply; 50+ messages in thread
From: Catalin Marinas @ 2015-01-12 11:53 UTC (permalink / raw)
To: linux-arm-kernel
On Sat, Jan 10, 2015 at 08:16:02PM +0000, Arnd Bergmann wrote:
> Regarding ARM64 in particular, I think it would be nice to investigate
> how to extend the THP code to cover 64KB TLBs when running with the 4KB
> page size. There is a hint bit in the page table to tell the CPU that
> a set of 16 aligned pages can share one TLB, and it would be nice to
> use that bit in Linux, and to make this case more common for anonymous
> mappings, and possible large file based mappings.
The generic THP code assumes that huge pages are done at the pmd level,
which means 2MB for arm64 with 4KB page configuration. Hugetlb allows
larger ptes which may not necessarily be at the pmd level, though we
haven't implemented this on arm64 and it's not transparent either. As a
first step it would be nice if at least we unify the APIs between
hugetlbfs and THP (set_huge_pte_at vs. set_pmd_at).
I think you could do some arch-only tricks by pretending that you have a
pte with 16 entries only and a dummy pmd (without a corresponding
hardware page table level) that can host a "huge" page (16 consecutive
ptes). But we lose the 2MB transparent huge page as I don't see
mm/huge_memory.c handling huge puds. We also lose the ability of
building 4 real level page tables since we use the pmd as a dummy one.
But it would be a nice investigation. Maybe something simpler like
getting the mm layer to prefer contiguous 64KB ranges and we do the
detection in the arch set_pte_at().
--
Catalin
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-10 21:36 ` Arnd Bergmann
2015-01-10 21:48 ` Linus Torvalds
2015-01-12 11:37 ` Kirill A. Shutemov
@ 2015-01-12 12:18 ` Catalin Marinas
2015-01-12 13:57 ` Arnd Bergmann
2 siblings, 1 reply; 50+ messages in thread
From: Catalin Marinas @ 2015-01-12 12:18 UTC (permalink / raw)
To: linux-arm-kernel
On Sat, Jan 10, 2015 at 09:36:13PM +0000, Arnd Bergmann wrote:
> On Saturday 10 January 2015 13:00:27 Linus Torvalds wrote:
> > > IIRC, AIX works great with 64k pages, but only because of two
> > > reasons that don't apply on Linux:
> >
> > .. there's a few other ones:
> >
> > (c) nobody really runs AIX on dekstops. It's very much a DB load
> > environment, with historically some HPC.
> >
> > (d) the powerpc TLB fill/buildup/teardown costs are horrible, so on
> > AIX the cost of lots of small pages is much higher too.
>
> I think (d) applies to ARM as well, since it has no hardware
> dirty/referenced bit tracking and requires the OS to mark the
> pages as invalid/readonly until the first access. ARMv8.1
> has a fix for that, but it's optional and we haven't seen any
> implementations yet.
Do you happen have any data on how significantly non-hardware
dirty/access bits impact the performance? I think it may affect the user
process start-up time a but at run-time it shouldn't be that bad.
If it is that significant, we could optimise it further in the arch
code. For example, make a fast exception path where we need to mark the
pte dirty. This would be handled by arch code without even calling
handle_pte_fault().
> > so I feel pretty confident in saying it won't happen. It's just too
> > much of a bother, for little to no actual upside. It's likely a much
> > better approach to try to instead use THP for anonymous mappings.
>
> arm64 already supports 2MB transparent hugepages. I guess it
> wouldn't be too hard to change it so that an existing hugepage
> on an anonymous mapping that gets split up into 4KB pages gets
> split along 64KB boundaries with the contiguous mapping bit set.
>
> Having full support for multiple hugepage sizes (64KB, 2MB and 32MB
> in case of ARM64 with 4KB PAGE_SIZE) would be even better and
> probably negate any benefits of 64KB PAGE_SIZE, but requires more
> changes to common mm code.
As I replied to your other email, I don't think that's simple for the
transparent huge pages case.
The main advantage I see with 64KB pages is not the reduced TLB pressure
but the number of levels of page tables. Take the AMD Seattle board for
example, with 4KB pages you need 4 levels but 64KB allow only 2 levels
(42-bit VA). Larger TLBs and improved walk caches (caching VA -> pmd
entry translation rather than all the way to pte/PA) make things better
but you still have the warming up time for any fork/new process as they
don't share the same TLB entries.
But as Linus said already, the trade-off with the memory wastage
is highly dependent on the targeted load.
--
Catalin
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-10 19:51 ` Linus Torvalds
@ 2015-01-12 12:42 ` Will Deacon
2015-01-12 13:22 ` Mark Langsdorf
` (2 more replies)
0 siblings, 3 replies; 50+ messages in thread
From: Will Deacon @ 2015-01-12 12:42 UTC (permalink / raw)
To: linux-arm-kernel
On Sat, Jan 10, 2015 at 07:51:03PM +0000, Linus Torvalds wrote:
> On Sat, Jan 10, 2015 at 5:37 AM, Will Deacon <will.deacon@arm.com> wrote:
> >>
> >> Will?
> >
> > I'm wondering if this is now broken in the fullmm case, because tlb->end
> > will be zero and we won't actually free any of the pages being unmapped
> > on task exit. Does that sound plausible?
>
> But did anything change wrt fullmm? I don't see any changes wrt fullmm
> logic in generic code.
No, and now that I've had a play, the issue is worse than I thought.
> The arm64 code changed more, so maybe there was somethinig I missed.
> Again, arm64 uses tlb_end_vma() etc, so arm64 certainly triggers code
> that x86 does not.
Yup, and I think tlb_end_vma is also problematic. The issue is that we
reset the range in __tlb_end_vma, and therefore the batched pages don't
get freed. We also have an issue in the fullmm case, because
tlb_flush_mmu tests tlb->end != 0 before doing any freeing.
The fundamental problem is that tlb->end *only* indicates whether or not
TLB invalidation is required, *not* whether there are pages to be freed.
The advantage of the old need_flush flag was that it was sticky over
calls to things like tlb_flush_mmu_tlbonly.
> I can revert the commit that causes problems, but considering the
> performance impact on x86 (it would be a regression since 3.18), I
> would *really* like to fix the arm64 problem instead. So I'll wait
> with the revert for at least a week, I think, hoping that the arm64
> people figure this out. Sound reasonable?
Sure. I've put together the following patch which:
(1) Uses tlb->end != 0 only as the predicate for TLB invalidation
(2) Uses tlb->local.nr as the predicate for page freeing in
tlb_flush_mmu_free
(3) Ensures tlb->end != 0 for the fullmm case (I think this was a
benign change introduced by my original patch, but this puts us
back to 3.18 behaviour)
Since this effectively reverts f045bbb9fa1b, I've added Dave to Cc. It
should fix the leak without reintroducing the performance regression.
Linus: this moves the tlb->end check back into tlb_flush_mmu_tlbonly.
How much do you hate that?
Will
--->8
diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h
index 08848050922e..db284bff29dc 100644
--- a/include/asm-generic/tlb.h
+++ b/include/asm-generic/tlb.h
@@ -136,8 +136,12 @@ static inline void __tlb_adjust_range(struct mmu_gather *tlb,
static inline void __tlb_reset_range(struct mmu_gather *tlb)
{
- tlb->start = TASK_SIZE;
- tlb->end = 0;
+ if (tlb->fullmm) {
+ tlb->start = tlb->end = ~0;
+ } else {
+ tlb->start = TASK_SIZE;
+ tlb->end = 0;
+ }
}
/*
diff --git a/mm/memory.c b/mm/memory.c
index c6565f00fb38..54f3a9b00956 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -235,6 +235,9 @@ void tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm, unsigned long
static void tlb_flush_mmu_tlbonly(struct mmu_gather *tlb)
{
+ if (!tlb->end)
+ return;
+
tlb_flush(tlb);
mmu_notifier_invalidate_range(tlb->mm, tlb->start, tlb->end);
#ifdef CONFIG_HAVE_RCU_TABLE_FREE
@@ -247,7 +250,7 @@ static void tlb_flush_mmu_free(struct mmu_gather *tlb)
{
struct mmu_gather_batch *batch;
- for (batch = &tlb->local; batch; batch = batch->next) {
+ for (batch = &tlb->local; batch && batch->nr; batch = batch->next) {
free_pages_and_swap_cache(batch->pages, batch->nr);
batch->nr = 0;
}
@@ -256,9 +259,6 @@ static void tlb_flush_mmu_free(struct mmu_gather *tlb)
void tlb_flush_mmu(struct mmu_gather *tlb)
{
- if (!tlb->end)
- return;
-
tlb_flush_mmu_tlbonly(tlb);
tlb_flush_mmu_free(tlb);
}
^ permalink raw reply related [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-12 11:53 ` Catalin Marinas
@ 2015-01-12 13:15 ` Arnd Bergmann
0 siblings, 0 replies; 50+ messages in thread
From: Arnd Bergmann @ 2015-01-12 13:15 UTC (permalink / raw)
To: linux-arm-kernel
On Monday 12 January 2015 11:53:42 Catalin Marinas wrote:
> On Sat, Jan 10, 2015 at 08:16:02PM +0000, Arnd Bergmann wrote:
> > Regarding ARM64 in particular, I think it would be nice to investigate
> > how to extend the THP code to cover 64KB TLBs when running with the 4KB
> > page size. There is a hint bit in the page table to tell the CPU that
> > a set of 16 aligned pages can share one TLB, and it would be nice to
> > use that bit in Linux, and to make this case more common for anonymous
> > mappings, and possible large file based mappings.
>
> The generic THP code assumes that huge pages are done at the pmd level,
> which means 2MB for arm64 with 4KB page configuration. Hugetlb allows
> larger ptes which may not necessarily be at the pmd level, though we
> haven't implemented this on arm64 and it's not transparent either. As a
> first step it would be nice if at least we unify the APIs between
> hugetlbfs and THP (set_huge_pte_at vs. set_pmd_at).
>
> I think you could do some arch-only tricks by pretending that you have a
> pte with 16 entries only and a dummy pmd (without a corresponding
> hardware page table level) that can host a "huge" page (16 consecutive
> ptes). But we lose the 2MB transparent huge page as I don't see
> mm/huge_memory.c handling huge puds. We also lose the ability of
> building 4 real level page tables since we use the pmd as a dummy one.
Yes, it quickly gets ugly at that point.
> But it would be a nice investigation. Maybe something simpler like
> getting the mm layer to prefer contiguous 64KB ranges and we do the
> detection in the arch set_pte_at().
Doing the detection would be easy enough I guess and immediately
helps with the post-split THP mapping, but I don't think that
by itself would have a noticeable benefit on general workloads.
My first reaction to a change to the mm layer was that it's probably really
hard, but then again if we limit it to anonymous mappings, all we really
need is a modification in do_anonymous_page() to allocate a larger chunk
if possible and install n PTEs at a time or fall back to the current
behavior if anything gets in the way. For completeness, the same thing
could be done in do_wp_page() for the case where an entire block of pages
are either not mapped or point to the zero page. Anything beyond that
probably adds more complexity than it gains.
Do we have someone who code this up and do some benchmarks to find out
the cost in terms of memory consumption and the performance compared to
normal 4k pages and static 64k pages?
Do the Cortex-A53/A57 cores actually implement the necessary hardware
feature?
IIRC some x86 processors are also able to use larger TLBs for contiguous
page table entries even without an architected hint bit, so if one
could show this to perform better on x86, it would be much easier to
merge.
Arnd
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-12 12:42 ` Will Deacon
@ 2015-01-12 13:22 ` Mark Langsdorf
2015-01-12 19:03 ` Dave Hansen
2015-01-12 19:06 ` Linus Torvalds
2 siblings, 0 replies; 50+ messages in thread
From: Mark Langsdorf @ 2015-01-12 13:22 UTC (permalink / raw)
To: linux-arm-kernel
On 01/12/2015 06:42 AM, Will Deacon wrote:
> On Sat, Jan 10, 2015 at 07:51:03PM +0000, Linus Torvalds wrote:
>> I can revert the commit that causes problems, but considering the
>> performance impact on x86 (it would be a regression since 3.18), I
>> would *really* like to fix the arm64 problem instead. So I'll wait
>> with the revert for at least a week, I think, hoping that the arm64
>> people figure this out. Sound reasonable?
>
> Sure. I've put together the following patch which:
>
> (1) Uses tlb->end != 0 only as the predicate for TLB invalidation
>
> (2) Uses tlb->local.nr as the predicate for page freeing in
> tlb_flush_mmu_free
>
> (3) Ensures tlb->end != 0 for the fullmm case (I think this was a
> benign change introduced by my original patch, but this puts us
> back to 3.18 behaviour)
>
> Since this effectively reverts f045bbb9fa1b, I've added Dave to Cc. It
> should fix the leak without reintroducing the performance regression.
>
> Linus: this moves the tlb->end check back into tlb_flush_mmu_tlbonly.
> How much do you hate that?
>
> diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h
> index 08848050922e..db284bff29dc 100644
> --- a/include/asm-generic/tlb.h
> +++ b/include/asm-generic/tlb.h
> @@ -136,8 +136,12 @@ static inline void __tlb_adjust_range(struct mmu_gather *tlb,
>
> static inline void __tlb_reset_range(struct mmu_gather *tlb)
> {
> - tlb->start = TASK_SIZE;
> - tlb->end = 0;
> + if (tlb->fullmm) {
> + tlb->start = tlb->end = ~0;
> + } else {
> + tlb->start = TASK_SIZE;
> + tlb->end = 0;
> + }
> }
>
> /*
> diff --git a/mm/memory.c b/mm/memory.c
> index c6565f00fb38..54f3a9b00956 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -235,6 +235,9 @@ void tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm, unsigned long
>
> static void tlb_flush_mmu_tlbonly(struct mmu_gather *tlb)
> {
> + if (!tlb->end)
> + return;
> +
> tlb_flush(tlb);
> mmu_notifier_invalidate_range(tlb->mm, tlb->start, tlb->end);
> #ifdef CONFIG_HAVE_RCU_TABLE_FREE
> @@ -247,7 +250,7 @@ static void tlb_flush_mmu_free(struct mmu_gather *tlb)
> {
> struct mmu_gather_batch *batch;
>
> - for (batch = &tlb->local; batch; batch = batch->next) {
> + for (batch = &tlb->local; batch && batch->nr; batch = batch->next) {
> free_pages_and_swap_cache(batch->pages, batch->nr);
> batch->nr = 0;
> }
> @@ -256,9 +259,6 @@ static void tlb_flush_mmu_free(struct mmu_gather *tlb)
>
> void tlb_flush_mmu(struct mmu_gather *tlb)
> {
> - if (!tlb->end)
> - return;
> -
> tlb_flush_mmu_tlbonly(tlb);
> tlb_flush_mmu_free(tlb);
> }
>
This fixes the originally reported problem. I have no opinion on
moving the tlb->end check back into tlb_flush_mmu_tlbonly.
Tested-by: Mark Langsdorf <mlangsdo@redhat.com>
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-12 12:18 ` Catalin Marinas
@ 2015-01-12 13:57 ` Arnd Bergmann
2015-01-12 14:23 ` Catalin Marinas
0 siblings, 1 reply; 50+ messages in thread
From: Arnd Bergmann @ 2015-01-12 13:57 UTC (permalink / raw)
To: linux-arm-kernel
On Monday 12 January 2015 12:18:15 Catalin Marinas wrote:
> On Sat, Jan 10, 2015 at 09:36:13PM +0000, Arnd Bergmann wrote:
> > On Saturday 10 January 2015 13:00:27 Linus Torvalds wrote:
> > > > IIRC, AIX works great with 64k pages, but only because of two
> > > > reasons that don't apply on Linux:
> > >
> > > .. there's a few other ones:
> > >
> > > (c) nobody really runs AIX on dekstops. It's very much a DB load
> > > environment, with historically some HPC.
> > >
> > > (d) the powerpc TLB fill/buildup/teardown costs are horrible, so on
> > > AIX the cost of lots of small pages is much higher too.
> >
> > I think (d) applies to ARM as well, since it has no hardware
> > dirty/referenced bit tracking and requires the OS to mark the
> > pages as invalid/readonly until the first access. ARMv8.1
> > has a fix for that, but it's optional and we haven't seen any
> > implementations yet.
>
> Do you happen have any data on how significantly non-hardware
> dirty/access bits impact the performance? I think it may affect the user
> process start-up time a but at run-time it shouldn't be that bad.
>
> If it is that significant, we could optimise it further in the arch
> code. For example, make a fast exception path where we need to mark the
> pte dirty. This would be handled by arch code without even calling
> handle_pte_fault().
If I understand the way that LRU works right, we end up clearing
the referenced bits in shrink_active_list()->page_referenced()->
page_referenced_one()->ptep_clear_flush_young_notify()->pte_mkold()
whenever there is memory pressure, so definitely not just for
startup.
> > > so I feel pretty confident in saying it won't happen. It's just too
> > > much of a bother, for little to no actual upside. It's likely a much
> > > better approach to try to instead use THP for anonymous mappings.
> >
> > arm64 already supports 2MB transparent hugepages. I guess it
> > wouldn't be too hard to change it so that an existing hugepage
> > on an anonymous mapping that gets split up into 4KB pages gets
> > split along 64KB boundaries with the contiguous mapping bit set.
> >
> > Having full support for multiple hugepage sizes (64KB, 2MB and 32MB
> > in case of ARM64 with 4KB PAGE_SIZE) would be even better and
> > probably negate any benefits of 64KB PAGE_SIZE, but requires more
> > changes to common mm code.
>
> As I replied to your other email, I don't think that's simple for the
> transparent huge pages case.
>
> The main advantage I see with 64KB pages is not the reduced TLB pressure
> but the number of levels of page tables. Take the AMD Seattle board for
> example, with 4KB pages you need 4 levels but 64KB allow only 2 levels
> (42-bit VA). Larger TLBs and improved walk caches (caching VA -> pmd
> entry translation rather than all the way to pte/PA) make things better
> but you still have the warming up time for any fork/new process as they
> don't share the same TLB entries.
Not sure I'm following. Does the A57 core cache partial TLBs or not?
Even if not, I would expect the page tables to be hot in dcache most
of the time, possibly with the exception of the last level on
multi-threaded processes, but then you are back to the difference
between the page size and the upper levels almost out of the equation.
Arnd
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-12 13:57 ` Arnd Bergmann
@ 2015-01-12 14:23 ` Catalin Marinas
2015-01-12 15:42 ` Arnd Bergmann
0 siblings, 1 reply; 50+ messages in thread
From: Catalin Marinas @ 2015-01-12 14:23 UTC (permalink / raw)
To: linux-arm-kernel
On Mon, Jan 12, 2015 at 01:57:48PM +0000, Arnd Bergmann wrote:
> On Monday 12 January 2015 12:18:15 Catalin Marinas wrote:
> > On Sat, Jan 10, 2015 at 09:36:13PM +0000, Arnd Bergmann wrote:
> > > On Saturday 10 January 2015 13:00:27 Linus Torvalds wrote:
> > > > > IIRC, AIX works great with 64k pages, but only because of two
> > > > > reasons that don't apply on Linux:
> > > >
> > > > .. there's a few other ones:
> > > >
> > > > (c) nobody really runs AIX on dekstops. It's very much a DB load
> > > > environment, with historically some HPC.
> > > >
> > > > (d) the powerpc TLB fill/buildup/teardown costs are horrible, so on
> > > > AIX the cost of lots of small pages is much higher too.
> > >
> > > I think (d) applies to ARM as well, since it has no hardware
> > > dirty/referenced bit tracking and requires the OS to mark the
> > > pages as invalid/readonly until the first access. ARMv8.1
> > > has a fix for that, but it's optional and we haven't seen any
> > > implementations yet.
> >
> > Do you happen have any data on how significantly non-hardware
> > dirty/access bits impact the performance? I think it may affect the user
> > process start-up time a but at run-time it shouldn't be that bad.
> >
> > If it is that significant, we could optimise it further in the arch
> > code. For example, make a fast exception path where we need to mark the
> > pte dirty. This would be handled by arch code without even calling
> > handle_pte_fault().
>
> If I understand the way that LRU works right, we end up clearing
> the referenced bits in shrink_active_list()->page_referenced()->
> page_referenced_one()->ptep_clear_flush_young_notify()->pte_mkold()
> whenever there is memory pressure, so definitely not just for
> startup.
Yes. Actually, I think the start-up is probably not that bad. For pages
pointing to zero-page, you need to go through the kernel to allocate an
anonymous page and change the pte anyway. If the access was "write", the
page is marked dirty already (same with the access flag, the page is
marked young to avoid a subsequent fault).
So, I guess it's run-time cost of the LRU algorithm, especially under
memory pressure. Harder to benchmark though (we'll see when we get
hardware, though probably not very soon).
> > > > so I feel pretty confident in saying it won't happen. It's just too
> > > > much of a bother, for little to no actual upside. It's likely a much
> > > > better approach to try to instead use THP for anonymous mappings.
> > >
> > > arm64 already supports 2MB transparent hugepages. I guess it
> > > wouldn't be too hard to change it so that an existing hugepage
> > > on an anonymous mapping that gets split up into 4KB pages gets
> > > split along 64KB boundaries with the contiguous mapping bit set.
> > >
> > > Having full support for multiple hugepage sizes (64KB, 2MB and 32MB
> > > in case of ARM64 with 4KB PAGE_SIZE) would be even better and
> > > probably negate any benefits of 64KB PAGE_SIZE, but requires more
> > > changes to common mm code.
> >
> > As I replied to your other email, I don't think that's simple for the
> > transparent huge pages case.
> >
> > The main advantage I see with 64KB pages is not the reduced TLB pressure
> > but the number of levels of page tables. Take the AMD Seattle board for
> > example, with 4KB pages you need 4 levels but 64KB allow only 2 levels
> > (42-bit VA). Larger TLBs and improved walk caches (caching VA -> pmd
> > entry translation rather than all the way to pte/PA) make things better
> > but you still have the warming up time for any fork/new process as they
> > don't share the same TLB entries.
>
> Not sure I'm following. Does the A57 core cache partial TLBs or not?
AFAIK, yes (I think A15 started doing this years ago). Architecturally,
an entry at any page table level could be cached in the TLB if it is
valid, irrespective of whether the full translation is valid.
Implementations, however, usually just cache the VA->pmd translation in
the intermediate TLB. Anyway, populating such cache still requires 3
accesses on a 4-level system.
> Even if not, I would expect the page tables to be hot in dcache most
> of the time, possibly with the exception of the last level on
> multi-threaded processes, but then you are back to the difference
> between the page size and the upper levels almost out of the equation.
As I said, I think it's more important with virtualisation where each
guest page table entry address needs to be translated from the guest
address space (IPA) to the physical address via the stage 2 tables.
BTW, arm64 can differentiate between a leaf TLB invalidation (just
changing the pte) and an all-levels one (which affects the walk cache
for a given VA). I have some hacked code to do this in Linux but I
haven't had time to assess the impact properly.
--
Catalin
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-12 14:23 ` Catalin Marinas
@ 2015-01-12 15:42 ` Arnd Bergmann
0 siblings, 0 replies; 50+ messages in thread
From: Arnd Bergmann @ 2015-01-12 15:42 UTC (permalink / raw)
To: linux-arm-kernel
On Monday 12 January 2015 14:23:32 Catalin Marinas wrote:
>
> So, I guess it's run-time cost of the LRU algorithm, especially under
> memory pressure. Harder to benchmark though (we'll see when we get
> hardware, though probably not very soon).
One thing you could try is to add an access fault handler that gathers
statistics about number of calls (you could trivially get this by
using gcov) and time spent in the handler for a workload that causes
memory pressure.
Arnd
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-12 12:42 ` Will Deacon
2015-01-12 13:22 ` Mark Langsdorf
@ 2015-01-12 19:03 ` Dave Hansen
2015-01-12 19:06 ` Linus Torvalds
2 siblings, 0 replies; 50+ messages in thread
From: Dave Hansen @ 2015-01-12 19:03 UTC (permalink / raw)
To: linux-arm-kernel
On 01/12/2015 04:42 AM, Will Deacon wrote:
> Since this effectively reverts f045bbb9fa1b, I've added Dave to Cc. It
> should fix the leak without reintroducing the performance regression.
I ran this on the big system that showed the earlier issue. Everything
seems OK, at least with the test that showed the issue earlier (the brk1
test from will-it-scale).
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-12 12:42 ` Will Deacon
2015-01-12 13:22 ` Mark Langsdorf
2015-01-12 19:03 ` Dave Hansen
@ 2015-01-12 19:06 ` Linus Torvalds
2015-01-12 19:07 ` Linus Torvalds
2 siblings, 1 reply; 50+ messages in thread
From: Linus Torvalds @ 2015-01-12 19:06 UTC (permalink / raw)
To: linux-arm-kernel
On Tue, Jan 13, 2015 at 1:42 AM, Will Deacon <will.deacon@arm.com> wrote:
>
> Linus: this moves the tlb->end check back into tlb_flush_mmu_tlbonly.
> How much do you hate that?
So I'm ok with it, as long as we don't have a performance regression.
Your "don't bother freeing when the batch is empty" should hopefully
be fine. Dave, does that work for your case?
Linus
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-12 19:06 ` Linus Torvalds
@ 2015-01-12 19:07 ` Linus Torvalds
2015-01-12 19:24 ` Will Deacon
0 siblings, 1 reply; 50+ messages in thread
From: Linus Torvalds @ 2015-01-12 19:07 UTC (permalink / raw)
To: linux-arm-kernel
On Tue, Jan 13, 2015 at 8:06 AM, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> So I'm ok with it, as long as we don't have a performance regression.
>
> Your "don't bother freeing when the batch is empty" should hopefully
> be fine. Dave, does that work for your case?
Oh, and Dave just replied that it's ok. So should I just take it
directly, or expect it through the arm64 tree? Either works for me.
Linus
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-12 19:07 ` Linus Torvalds
@ 2015-01-12 19:24 ` Will Deacon
0 siblings, 0 replies; 50+ messages in thread
From: Will Deacon @ 2015-01-12 19:24 UTC (permalink / raw)
To: linux-arm-kernel
On Mon, Jan 12, 2015 at 07:07:12PM +0000, Linus Torvalds wrote:
> On Tue, Jan 13, 2015 at 8:06 AM, Linus Torvalds
> <torvalds@linux-foundation.org> wrote:
> >
> > So I'm ok with it, as long as we don't have a performance regression.
> >
> > Your "don't bother freeing when the batch is empty" should hopefully
> > be fine. Dave, does that work for your case?
>
> Oh, and Dave just replied that it's ok. So should I just take it
> directly, or expect it through the arm64 tree? Either works for me.
Although I do have a couple of arm64 fixes on the radar, it'd be quicke
if you just take the patch. I added a commit log/SoB below.
Cheers,
Will
--->8
>From bcf792ffc9ce29415261d2055954b883c5bec978 Mon Sep 17 00:00:00 2001
From: Will Deacon <will.deacon@arm.com>
Date: Mon, 12 Jan 2015 19:10:55 +0000
Subject: [PATCH] mm: mmu_gather: use tlb->end != 0 only for TLB invalidation
When batching up address ranges for TLB invalidation, we check tlb->end
!= 0 to indicate that some pages have actually been unmapped.
As of commit f045bbb9fa1b ("mmu_gather: fix over-eager
tlb_flush_mmu_free() calling"), we use the same check for freeing these
pages in order to avoid a performance regression where we call
free_pages_and_swap_cache even when no pages are actually queued up.
Unfortunately, the range could have been reset (tlb->end = 0) by
tlb_end_vma, which has been shown to cause memory leaks on arm64.
Furthermore, investigation into these leaks revealed that the fullmm
case on task exit no longer invalidates the TLB, by virtue of tlb->end
== 0 (in 3.18, need_flush would have been set).
This patch resolves the problem by reverting f045bbb9fa1b, using
tlb->local.nr as the predicate for page freeing in tlb_flush_mmu_free
and ensuring that tlb->end is initialised to a non-zero value in the
fullmm case.
Tested-by: Mark Langsdorf <mlangsdo@redhat.com>
Tested-by: Dave Hansen <dave@sr71.net>
Signed-off-by: Will Deacon <will.deacon@arm.com>
---
include/asm-generic/tlb.h | 8 ++++++--
mm/memory.c | 8 ++++----
2 files changed, 10 insertions(+), 6 deletions(-)
diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h
index 08848050922e..db284bff29dc 100644
--- a/include/asm-generic/tlb.h
+++ b/include/asm-generic/tlb.h
@@ -136,8 +136,12 @@ static inline void __tlb_adjust_range(struct mmu_gather *tlb,
static inline void __tlb_reset_range(struct mmu_gather *tlb)
{
- tlb->start = TASK_SIZE;
- tlb->end = 0;
+ if (tlb->fullmm) {
+ tlb->start = tlb->end = ~0;
+ } else {
+ tlb->start = TASK_SIZE;
+ tlb->end = 0;
+ }
}
/*
diff --git a/mm/memory.c b/mm/memory.c
index c6565f00fb38..54f3a9b00956 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -235,6 +235,9 @@ void tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm, unsigned long
static void tlb_flush_mmu_tlbonly(struct mmu_gather *tlb)
{
+ if (!tlb->end)
+ return;
+
tlb_flush(tlb);
mmu_notifier_invalidate_range(tlb->mm, tlb->start, tlb->end);
#ifdef CONFIG_HAVE_RCU_TABLE_FREE
@@ -247,7 +250,7 @@ static void tlb_flush_mmu_free(struct mmu_gather *tlb)
{
struct mmu_gather_batch *batch;
- for (batch = &tlb->local; batch; batch = batch->next) {
+ for (batch = &tlb->local; batch && batch->nr; batch = batch->next) {
free_pages_and_swap_cache(batch->pages, batch->nr);
batch->nr = 0;
}
@@ -256,9 +259,6 @@ static void tlb_flush_mmu_free(struct mmu_gather *tlb)
void tlb_flush_mmu(struct mmu_gather *tlb)
{
- if (!tlb->end)
- return;
-
tlb_flush_mmu_tlbonly(tlb);
tlb_flush_mmu_free(tlb);
}
--
2.1.4
^ permalink raw reply related [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-10 2:51 ` David Lang
2015-01-10 3:06 ` Linus Torvalds
@ 2015-01-13 3:33 ` Rik van Riel
2015-01-13 10:28 ` Catalin Marinas
1 sibling, 1 reply; 50+ messages in thread
From: Rik van Riel @ 2015-01-13 3:33 UTC (permalink / raw)
To: linux-arm-kernel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 01/09/2015 09:51 PM, David Lang wrote:
> On Fri, 9 Jan 2015, Linus Torvalds wrote:
>
>> Big pages are a bad bad bad idea. They work fine for databases,
>> and that's pretty much just about it. I'm sure there are some
>> other loads, but they are few and far between.
>
> what about a dedicated virtualization host (where your workload is
> a handful of virtual machines), would the file cache issue still
> be overwelming, even though it's the virtual machines accessing
> things?
You would still have page cache inside the guest.
Using large pages in the host, and small pages in the guest
would not give you the TLB benefits, and that is assuming
that different page sizes in host and guest even work...
Using large pages in the guests gets you back to the wasted
memory, except you are now wasting memory in a situation where
you have less memory available in each guest. Density is a real
consideration for virtualization.
- --
All rights reversed
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAEBAgAGBQJUtJH4AAoJEM553pKExN6Dd3QH/ivcIo2n06Czg14/gL61MSHM
uZPOuMGQt51DYtF3s3mtDHqWyZq9hafz+2hoSJDwGIvVE6hJKVJ5rvb/OcN7AEKe
PWfru+bOvID0d4YOy38ax2tZwdItlL/sj1AbTbPXjnkLWm0yP3dYVM40dj47JvPy
+aE3iHB+wPZ+xxUmQ5KIlpRydUS1fl+tdmsiyi41fSFu8X19YDtDSrPylLk3to/w
6RGbHWLxJQZXJk+pkVWuSELmzWRrCaNaE7XBlvP9VS4U8bRg8WYJJXax1FBKLGBO
ygVt2OmqLi9dneN8ePNRUW8x2Y6OqjobDgCkOzTxJB8NrtRDJpSqWCI4W5xReBM=
=WhPN
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 50+ messages in thread
* Linux 3.19-rc3
2015-01-13 3:33 ` Rik van Riel
@ 2015-01-13 10:28 ` Catalin Marinas
0 siblings, 0 replies; 50+ messages in thread
From: Catalin Marinas @ 2015-01-13 10:28 UTC (permalink / raw)
To: linux-arm-kernel
On Tue, Jan 13, 2015 at 03:33:12AM +0000, Rik van Riel wrote:
> On 01/09/2015 09:51 PM, David Lang wrote:
> > On Fri, 9 Jan 2015, Linus Torvalds wrote:
> >
> >> Big pages are a bad bad bad idea. They work fine for databases,
> >> and that's pretty much just about it. I'm sure there are some
> >> other loads, but they are few and far between.
> >
> > what about a dedicated virtualization host (where your workload is
> > a handful of virtual machines), would the file cache issue still
> > be overwelming, even though it's the virtual machines accessing
> > things?
>
> You would still have page cache inside the guest.
>
> Using large pages in the host, and small pages in the guest
> would not give you the TLB benefits, and that is assuming
> that different page sizes in host and guest even work...
This works on ARM. The TLB caching the full VA->PA translation would
indeed stick to the guest page size as that's the input. But, depending
on the TLB implementation, it may also cache the guest PA -> real PA
translation (a TLB with the guest/Intermediate PA as input; ARMv8 also
introduces TLB invalidation ops that take such IPA as input). A miss in
the stage 1 (guest) TLB would be cheaper if it hits in the stage 2 TLB,
especially when it needs to look up the stage 2 for each level in the
stage 1 table.
But when it doesn't hit in any of the stages, it's still beneficial to
have smaller number of levels at stage 2 (host) and that's what 64KB
pages bring on ARM. If you use the maximum 4 levels in both host and
guest, a TLB miss in the guest requires 24 memory accesses to populate
it (each guest page table level entry needs a stage 2 look-up). In
practice, you may get some locality but I think the guest page table
access pattern can get quite sparse. In addition, stage 2 entries are
not as volatile as they are per VM rather than per process as the stage
1 entries.
> Using large pages in the guests gets you back to the wasted
> memory, except you are now wasting memory in a situation where
> you have less memory available in each guest. Density is a real
> consideration for virtualization.
I agree. I think guests should stick to 4KB pages (well, unless all they
need to do is mmap large database files).
--
Catalin
^ permalink raw reply [flat|nested] 50+ messages in thread
end of thread, other threads:[~2015-01-13 10:28 UTC | newest]
Thread overview: 50+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <CA+55aFwsxoyLb9OWMSCL3doe_cz_EQtKsEFCyPUYn_T87pbz0A@mail.gmail.com>
2015-01-08 12:51 ` Linux 3.19-rc3 Mark Langsdorf
2015-01-08 13:45 ` Catalin Marinas
2015-01-08 17:29 ` Mark Langsdorf
2015-01-08 17:34 ` Catalin Marinas
2015-01-08 18:48 ` Mark Langsdorf
2015-01-08 19:21 ` Linus Torvalds
2015-01-09 23:27 ` Catalin Marinas
2015-01-10 0:35 ` Kirill A. Shutemov
2015-01-10 2:27 ` Linus Torvalds
2015-01-10 2:51 ` David Lang
2015-01-10 3:06 ` Linus Torvalds
2015-01-10 10:46 ` Andreas Mohr
2015-01-10 19:42 ` Linus Torvalds
2015-01-13 3:33 ` Rik van Riel
2015-01-13 10:28 ` Catalin Marinas
2015-01-10 3:17 ` Tony Luck
2015-01-10 20:16 ` Arnd Bergmann
2015-01-10 21:00 ` Linus Torvalds
2015-01-10 21:36 ` Arnd Bergmann
2015-01-10 21:48 ` Linus Torvalds
2015-01-12 11:37 ` Kirill A. Shutemov
2015-01-12 12:18 ` Catalin Marinas
2015-01-12 13:57 ` Arnd Bergmann
2015-01-12 14:23 ` Catalin Marinas
2015-01-12 15:42 ` Arnd Bergmann
2015-01-12 11:53 ` Catalin Marinas
2015-01-12 13:15 ` Arnd Bergmann
2015-01-08 15:08 ` Michal Hocko
2015-01-08 16:37 ` Mark Langsdorf
2015-01-09 15:56 ` Michal Hocko
2015-01-09 12:13 ` Mark Rutland
2015-01-09 14:19 ` Steve Capper
2015-01-09 14:27 ` Mark Langsdorf
2015-01-09 17:57 ` Mark Rutland
2015-01-09 18:37 ` Marc Zyngier
2015-01-09 19:43 ` Will Deacon
2015-01-10 3:29 ` Laszlo Ersek
2015-01-10 4:39 ` Linus Torvalds
2015-01-10 13:37 ` Will Deacon
2015-01-10 19:47 ` Laszlo Ersek
2015-01-10 19:56 ` Linus Torvalds
2015-01-10 20:08 ` Laszlo Ersek
2015-01-10 19:51 ` Linus Torvalds
2015-01-12 12:42 ` Will Deacon
2015-01-12 13:22 ` Mark Langsdorf
2015-01-12 19:03 ` Dave Hansen
2015-01-12 19:06 ` Linus Torvalds
2015-01-12 19:07 ` Linus Torvalds
2015-01-12 19:24 ` Will Deacon
2015-01-10 15:22 ` Kyle McMartin
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).