some question about order0 page allocation

some question about order0 page allocation

[yoma sophian]

[-- Attachment #1: Type: text/plain, Size: 4710 bytes --]

hi all:
I got oom log like at the end of mail from my embedded system.
But the log makes me curious are
a. the free memory size is ok.
        (Normal free:50080kB)
    the pcp hot page is also enough.
        (Normal per-cpu:
        CPU    0: hi:  186, btch:  31 usd:  18
        CPU    1: hi:  186, btch:  31 usd:  22)

b. the water mark should be ok as well.
    in __alloc_pages_may_oom, we use ALLOC_WMARK_HIGH|ALLOC_CPUSET for
watermake checking.
   so in this case:
   free-free_cma = 3192KB > (highmark -= highmark/2 = 2130KB)

But why the oom-killer sill be activated?
appreciate your kind help in advance.

(the kernel version is 3.10)
[ 5515.127555] dialog invoked oom-killer: gfp_mask=0x80d0, order=0,
oom_score_adj=0
[ 5515.136841] CPU: 0 PID: 1535 Comm: com.nvt.dialser Tainted: G
    O 3.10.0+ #28
[ 5515.145711] Backtrace:
[ 5515.149716] [<c00129f8>] (dump_backtrace+0x0/0x114) from
[<c0012c68>] (show_stack+0x20/0x24)
[ 5515.160163]  r6:000080d0 r5:00000000 r4:de3c2000 r3:271ae71c
[ 5515.167278] [<c0012c48>] (show_stack+0x0/0x24) from [<c0550f6c>]
(dump_stack+0x24/0x28)
[ 5515.177129] [<c0550f48>] (dump_stack+0x0/0x28) from [<c010de20>]
(dump_header.isra.13+0x78/0x18c)
[ 5515.188266] [<c010dda8>] (dump_header.isra.13+0x0/0x18c) from
[<c010e384>] (oom_kill_process+0x268/0x3a8)
[ 5515.199405]  r9:00000000 r8:00000000 r7:000080d0 r6:00022d82 r5:000080d0
r4:df679f80
[ 5515.208930] [<c010e11c>] (oom_kill_process+0x0/0x3a8) from
[<c010e884>] (out_of_memory+0x20c/0x2d8)
[ 5515.219014] [<c010e678>] (out_of_memory+0x0/0x2d8) from
[<c011319c>] (__alloc_pages_nodemask+0x928/0x940)
[ 5515.229835] [<c0112874>] (__alloc_pages_nodemask+0x0/0x940) from
[<c01131d4>] (__get_free_pages+0x20/0x3c)
[ 5515.240825] [<c01131b4>] (__get_free_pages+0x0/0x3c) from
[<c0113210>] (get_zeroed_page+0x20/0x24)
[ 5515.250145] [<c01131f0>] (get_zeroed_page+0x0/0x24) from
[<c01adaa8>] (sysfs_follow_link+0x24/0x1a0)
[ 5515.260366] [<c01ada84>] (sysfs_follow_link+0x0/0x1a0) from
[<c015a8fc>] (path_lookupat+0x388/0x800)
[ 5515.270740] [<c015a574>] (path_lookupat+0x0/0x800) from
[<c015ada4>] (filename_lookup+0x30/0xcc)
[ 5515.281052] [<c015ad74>] (filename_lookup+0x0/0xcc) from
[<c015d520>] (user_path_at_empty+0x68/0x90)
[ 5515.291374]  r8:ffffff9c r7:de3c3f60 r6:de3c3eb8 r5:00000001 r4:d3a853c0
r3:de3c3eb8
[ 5515.300401] [<c015d4b8>] (user_path_at_empty+0x0/0x90) from
[<c015d56c>] (user_path_at+0x24/0x2c)
[ 5515.310516]  r8:00000010 r7:be29cd9c r6:ffffff9c r5:00000000 r4:00000001
[ 5515.318228] [<c015d548>] (user_path_at+0x0/0x2c) from [<c014ca9c>]
(SyS_faccessat+0xa4/0x1f4)
[ 5515.327702] [<c014c9f8>] (SyS_faccessat+0x0/0x1f4) from
[<c014cc10>] (SyS_access+0x24/0x28)
[ 5515.336850] [<c014cbec>] (SyS_access+0x0/0x28) from [<c000e380>]
(ret_fast_syscall+0x0/0x48)
[ 5515.346494] Mem-info:
[ 5515.348849] Normal per-cpu:
[ 5515.352345] CPU    0: hi:  186, btch:  31 usd:  18
[ 5515.357785] CPU    1: hi:  186, btch:  31 usd:  22
[ 5515.362814] active_anon:109178 inactive_anon:2272 isolated_anon:0
[ 5515.362814]  active_file:260 inactive_file:961 isolated_file:0
[ 5515.362814]  unevictable:0 dirty:0 writeback:0 unstable:0
[ 5515.362814]  free:12118 slab_reclaimable:1271 slab_unreclaimable:5113
[ 5515.362814]  mapped:2653 shmem:2298 pagetables:979 bounce:0
[ 5515.362814]  free_cma:11605
[ 5515.396900] Normal free:50080kB min:2840kB low:3548kB high:4260kB
active_anon:436712kB inactive_anon:9088kB active_file:1316kB
inactive_file:1636kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:585728kB managed:504960kB mlocked:0kB
dirty:0kB writeback:0kB mapped:9624kB shmem:9192kB
slab_reclaimable:5084kB slab_unreclaimable:20452kB kernel_stack:2432kB
pagetables:3916kB unstable:0kB bounce:0kB free_cma:46888kB
writeback_tmp:0kB pages_scanned:24 all_unreclaimable? no
[ 5515.441095] lowmem_reserve[]: 0 0 0
[ 5515.444859] Normal: 4314*4kB (UEMC) 3586*8kB (UMC) 131*16kB (MC)
21*32kB (C) 6*64kB (C) 1*128kB (C) 0*256kB 0*512kB 0*1024kB 0*2048kB
0*4096kB = 49224kB
[ 5515.460587] 3477 total pagecache pages
[ 5515.464648] 0 pages in swap cache
[ 5515.468005] Swap cache stats: add 0, delete 0, find 0/0
[ 5515.473512] Free swap  = 0kB
[ 5515.476665] Total swap = 0kB
[ 5515.497647] 146432 pages of RAM
[ 5515.501295] 13056 free pages
[ 5515.504278] 3712 reserved pages
[ 5515.507864] 4891 slab pages
[ 5515.510899] 396881 pages shared
[ 5515.514303] 0 pages swap cached
[ 5515.668990] Out of memory: Kill process 1260 (app) score 563 or
sacrifice child
[ 5515.678006] Killed process 1277 (idlog) total-vm:511668kB,
anon-rss:61016kB, file-rss:680kB
(*) [Fusion Dispatch  5515.978,466] ( 1260: 1518) SaWMan/Watcher:
Process [0x224b4f00 pid:1277 fusion_id:2 flags:] has exited
ABNORMALLY!

[-- Attachment #2: oom.log.tar.bz2 --]
[-- Type: application/x-bzip2, Size: 1903 bytes --]

Re: some question about order0 page allocation

[Michal Hocko]

On Fri 14-10-16 17:29:34, yoma sophian wrote:
[...]
> [ 5515.127555] dialog invoked oom-killer: gfp_mask=0x80d0, order=0,
> oom_score_adj=0

This looks like a GFP_KERNEL + something allocation

> [ 5515.444859] Normal: 4314*4kB (UEMC) 3586*8kB (UMC) 131*16kB (MC)
> 21*32kB (C) 6*64kB (C) 1*128kB (C) 0*256kB 0*512kB 0*1024kB 0*2048kB
> 0*4096kB = 49224kB

And it seems like CMA blocks are spread in all orders and no unmovable
allocations can fallback in them. It seems that there should be some
movable blocks but I do not have any idea why those are not used. Anyway
this is where I would start investigating.
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: some question about order0 page allocation

[yoma sophian]

 hi michal:

2016-10-14 23:26 GMT+08:00 Michal Hocko <mhocko@kernel.org>:
> On Fri 14-10-16 17:29:34, yoma sophian wrote:
> [...]
>> [ 5515.127555] dialog invoked oom-killer: gfp_mask=0x80d0, order=0,
>> oom_score_adj=0
>
> This looks like a GFP_KERNEL + something allocation
Yes, you are correct.
The page is allocated with GFP as (KERNEL + ZERO) flag
>
>> [ 5515.444859] Normal: 4314*4kB (UEMC) 3586*8kB (UMC) 131*16kB (MC)
>> 21*32kB (C) 6*64kB (C) 1*128kB (C) 0*256kB 0*512kB 0*1024kB 0*2048kB
>> 0*4096kB = 49224kB
>
> And it seems like CMA blocks are spread in all orders and no unmovable
> allocations can fallback in them. It seems that there should be some
> movable blocks but I do not have any idea why those are not used. Anyway
> this is where I would start investigating.
Per your kind hint, I trace pcp page allocation again.(since the order
of allocation is 0 this time)
I found when the list of pcp with unmovable type is empty, it will
call rmqueue_bulk for trying to get batch, 31 order-0 pages.
And rmqueue_bulk will call __rmqueue_smallest and even
__rmqueue_fallback once the buddy of unmovable memory is not enough.

But from below message:
[ 5515.444859] Normal: 4314*4kB (UEMC) 3586*8kB (UMC)
the order 0 of U type in buddy is at least has 1 page free.
That mean even there is not enough 32 order-0 pages with U in buddy
right now, buddy can at least provide 1 page to satisfy this
allocation.
if my conclusion is correct, there is no need for fallback.
Please correct me, if I am wrong.

Sincerely appreciate your kind help,

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: some question about order0 page allocation

[Michal Hocko]

On Sun 16-10-16 19:01:23, yoma sophian wrote:
>  hi michal:
> 
> 2016-10-14 23:26 GMT+08:00 Michal Hocko <mhocko@kernel.org>:
> > On Fri 14-10-16 17:29:34, yoma sophian wrote:
> > [...]
> >> [ 5515.127555] dialog invoked oom-killer: gfp_mask=0x80d0, order=0,
> >> oom_score_adj=0
> >
> > This looks like a GFP_KERNEL + something allocation
> Yes, you are correct.
> The page is allocated with GFP as (KERNEL + ZERO) flag
> >
> >> [ 5515.444859] Normal: 4314*4kB (UEMC) 3586*8kB (UMC) 131*16kB (MC)
> >> 21*32kB (C) 6*64kB (C) 1*128kB (C) 0*256kB 0*512kB 0*1024kB 0*2048kB
> >> 0*4096kB = 49224kB
> >
> > And it seems like CMA blocks are spread in all orders and no unmovable
> > allocations can fallback in them. It seems that there should be some
> > movable blocks but I do not have any idea why those are not used. Anyway
> > this is where I would start investigating.
> Per your kind hint, I trace pcp page allocation again.(since the order
> of allocation is 0 this time)
> I found when the list of pcp with unmovable type is empty, it will
> call rmqueue_bulk for trying to get batch, 31 order-0 pages.
> And rmqueue_bulk will call __rmqueue_smallest and even
> __rmqueue_fallback once the buddy of unmovable memory is not enough.
> 
> But from below message:
> [ 5515.444859] Normal: 4314*4kB (UEMC) 3586*8kB (UMC)
> the order 0 of U type in buddy is at least has 1 page free.
> That mean even there is not enough 32 order-0 pages with U in buddy
> right now, buddy can at least provide 1 page to satisfy this
> allocation.
> if my conclusion is correct, there is no need for fallback.
> Please correct me, if I am wrong.

I am not deeply familiar with the mobility code, more so for an old
kernel, but my general understanding is that that the migrate type
information is not exact and there are races possible. Maybe there are
even accounting bugs in such an old kernel. Do you see the same problem
with the current upstream kernel?
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: some question about order0 page allocation

[yoma sophian]

hi MIchal
> I am not deeply familiar with the mobility code, more so for an old
> kernel, but my general understanding is that that the migrate type
> information is not exact and there are races possible.
When we add more debug message for checking the issue.
so far it is only fail on watermark check.
and it seems there did race condition happen like you mentioned to
make the final memory info is incorrect.

Sincerely appreciate your kind help ^^

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>