Linux 3.19-rc3

[mark.rutland@arm.com (Mark Rutland)]

On Fri, Jan 09, 2015 at 02:27:06PM +0000, Mark Langsdorf wrote:
> On 01/09/2015 08:19 AM, Steve Capper wrote:
> > On 9 January 2015 at 12:13, Mark Rutland <mark.rutland@arm.com> wrote:
> >> On Thu, Jan 08, 2015 at 12:51:31PM +0000, Mark Langsdorf wrote:
> >>> I'm consistently getting an out of memory killer triggered when
> >>> compiling the kernel (make -j 16 -s) on a 16 core ARM64 system
> >>> with 16 GB of memory. This doesn't happen when running a 3.18
> >>> kernel.
> >>>
> >>> I'm going to start bisecting the failure now, but here's the crash
> >>> log in case someone can see something obvious in it.
> >>
> >> FWIW I've just reproduced this with v3.19-rc3 defconfig +
> >> CONFIG_ARM64_64K_PAGES=y by attempting a git clone of mainline. My
> >> system has 16GB of RAM and 6 CPUs.
> >>
> >> I have a similarly dodgy looking number of pages reserved
> >> (18446744073709544451 A.K.A. -7165). Log below.
> >>
> >
> > I think the negative page reserved count is a consequence of another bug.
> >
> > We have the following reporting code in lib/show_mem.c:
> > #ifdef CONFIG_CMA
> >          printk("%lu pages reserved\n", (reserved - totalcma_pages));
> >          printk("%lu pages cma reserved\n", totalcma_pages);
> > #else
> >
> > With totalcma_pages being reported as 8192, that would account for the
> > -7000ish values reported.
> >
> > That change appears to have come from:
> > 49abd8c lib/show_mem.c: add cma reserved information
> >
> > Is the quickest way to exacerbate this OOM a kernel compile?
> 
> I haven't really tried to characterize this. Compiling a kernel
> on a 64K page machine causes a failure reasonably quickly and
> doesn't require a lot of thought. I think that time spent finding
> a faster reproducer wouldn't pay off.

I wasn't able to trigger the issue again with git, and the only way I've
managed to trigger the issue is repeatedly building the kernel in a
loop:

while true; do
	git clean -fdx > /dev/null 2>&1;
	make defconfig > /dev/null 2>&1;
	make > /dev/null > 2>&1;
done

Which after a while died:

-bash: fork: Cannot allocate memory

I didn't see anything interesting in dmesg, but I was able to get at
/proc/meminfo:

MemTotal:       16695168 kB
MemFree:          998336 kB
MemAvailable:     325568 kB
Buffers:           51200 kB
Cached:           236224 kB
SwapCached:            0 kB
Active:         14970880 kB
Inactive:         580288 kB
Active(anon):   14834496 kB
Inactive(anon):     5760 kB
Active(file):     136384 kB
Inactive(file):   574528 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:               448 kB
Writeback:             0 kB
AnonPages:         22400 kB
Mapped:            10240 kB
Shmem:              8768 kB
Slab:              63744 kB
SReclaimable:      27072 kB
SUnreclaim:        36672 kB
KernelStack:        1824 kB
PageTables:         3776 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     8347584 kB
Committed_AS:      50368 kB
VmallocTotal:   2142764992 kB
VmallocUsed:      283264 kB
VmallocChunk:   2142387200 kB
AnonHugePages:         0 kB
CmaTotal:         524288 kB
CmaFree:             128 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:     524288 kB

And also magic-sysrq m:

SysRq : Show Memory
Mem-Info:
DMA per-cpu:
CPU    0: hi:    6, btch:   1 usd:   1
CPU    1: hi:    6, btch:   1 usd:   1
CPU    2: hi:    6, btch:   1 usd:   1
CPU    3: hi:    6, btch:   1 usd:   3
CPU    4: hi:    6, btch:   1 usd:   5
CPU    5: hi:    6, btch:   1 usd:   5
Normal per-cpu:
CPU    0: hi:    6, btch:   1 usd:   0
CPU    1: hi:    6, btch:   1 usd:   5
CPU    2: hi:    6, btch:   1 usd:   1
CPU    3: hi:    6, btch:   1 usd:   5
CPU    4: hi:    6, btch:   1 usd:   5
CPU    5: hi:    6, btch:   1 usd:   5
active_anon:231780 inactive_anon:90 isolated_anon:0
 active_file:2131 inactive_file:8977 isolated_file:0
 unevictable:0 dirty:8 writeback:0 unstable:0
 free:15601 slab_reclaimable:423 slab_unreclaimable:573
 mapped:160 shmem:137 pagetables:59 bounce:0
 free_cma:2
DMA free:302336kB min:208000kB low:259968kB high:312000kB active_anon:3618432kB inactive_anon:768kB active_file:34432kB inactive_file:131584kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:4177920kB managed:4166528kB mlocked:0kB dirty:192kB writeback:0kB mapped:4736kB shmem:1024kB slab_reclaimable:5184kB slab_unreclaimable:3328kB kernel_stack:0kB pagetables:1600kB unstable:0kB bounce:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:1208448 all_unreclaimable? yes
lowmem_reserve[]: 0 764 764
Normal free:696128kB min:625472kB low:781824kB high:938176kB active_anon:11215488kB inactive_anon:4992kB active_file:101952kB inactive_file:442944kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:12582912kB managed:12528640kB mlocked:0kB dirty:320kB writeback:0kB mapped:5504kB shmem:7744kB slab_reclaimable:21888kB slab_unreclaimable:33344kB kernel_stack:1840kB pagetables:2176kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3331648 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0
DMA: 42*64kB (MRC) 37*128kB (R) 6*256kB (R) 5*512kB (R) 2*1024kB (R) 3*2048kB (R) 1*4096kB (R) 0*8192kB 1*16384kB (R) 0*32768kB 0*65536kB 0*131072kB 1*262144kB (R) 0*524288kB = 302336kB
Normal: 280*64kB (MR) 40*128kB (R) 5*256kB (R) 4*512kB (R) 6*1024kB (R) 4*2048kB (R) 1*4096kB (R) 1*8192kB (R) 1*16384kB (R) 1*32768kB (R) 1*65536kB (R) 0*131072kB 0*262144kB 1*524288kB (R) = 691968kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=524288kB
4492 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap  = 0kB
Total swap = 0kB
261888 pages RAM
0 pages HighMem/MovableOnly
18446744073709544450 pages reserved
8192 pages cma reserved

I also ran ps aux, but I didn't see any stale tasks lying around, nor
did any remaining tasks seem to account for all that active anonymous
memory.

I'll see if I can reproduce on x86.

Thanks,
Mark.