From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
To: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: Helmut Buchsbaum <helmut.buchsbaum@gmail.com>,
linux-rt-users@vger.kernel.org, rostedt@goodmis.org
Subject: Re: Resend: 3.10.32-rt31: problem with "rcu: Eliminate softirq processing from rcutree"
Date: Fri, 7 Mar 2014 09:04:19 -0800 [thread overview]
Message-ID: <20140307170419.GI3334@linux.vnet.ibm.com> (raw)
In-Reply-To: <20140307162145.GC18441@linutronix.de>
On Fri, Mar 07, 2014 at 05:21:45PM +0100, Sebastian Andrzej Siewior wrote:
> + paulmck
>
> * Helmut Buchsbaum | 2014-03-06 20:05:13 [+0100]:
>
> >I am working with a 3.10-rt based kernel on a custom device based on
> >Xilinx Zynq Z-7010 (dual Cortex-A9) with 64MB DDR2 RAM.
> >Today I rebased my work (3.10-rt mainly enhanced by Xilinx drivers,
> >but otherwise unchanged) from 3.10.32-rt30 to 3.10.32-rt31. There I
> >ran a dohell derivate (from
> >http://git.xenomai.org/xenomai-2.6.git/plain/src/testsuite/xeno-test/dohell?id=v2.6.3,
> >just using customized hackbench call and cyclictest as the only test)
> >when the OOM killer struck, which definitely did not happen with
> >3.10.32-rt30. Bisecting I detected
> >8cf5b0e982cba4fd0e469989a420a15c8a378fa2 "rcu: Eliminate softirq
> >processing from rcutree" as the change which introduced the unwanted
> >behavior:
> >
> >[ 73.865213] ls invoked oom-killer: gfp_mask=0x2000d0, order=0, oom_score_adj=0
> >[ 73.877229] CPU: 1 PID: 623 Comm: ls Not tainted 3.10.32-rt30+ #1
> >[ 73.877280] [<c0013c80>] (unwind_backtrace+0x0/0x138) from [<c0011d14>] (show_stack+0x10/0x14)
> >[ 73.877310] [<c0011d14>] (show_stack+0x10/0x14) from [<c0394d88>] (dump_header.isra.12+0x6c/0xa8)
> >[ 73.877334] [<c0394d88>] (dump_header.isra.12+0x6c/0xa8) from [<c0394e10>] (oom_kill_process.part.14+0x4c/0x384)
> >[ 73.877357] [<c0394e10>] (oom_kill_process.part.14+0x4c/0x384) from [<c007301c>] (out_of_memory+0x128/0x1c8)
> >[ 73.877380] [<c007301c>] (out_of_memory+0x128/0x1c8) from >[<c0076f10>] (__alloc_pages_nodemask+0x6f4/0x71c)
> >[ 73.877406] [<c0076f10>] (__alloc_pages_nodemask+0x6f4/0x71c) from [<c009fe9c>] (allocate_slab+0xe4/0xfc)
> >[ 73.877426] [<c009fe9c>] (allocate_slab+0xe4/0xfc) from [<c009fee4>] (new_slab+0x30/0x154)
> >[ 73.877447] [<c009fee4>] (new_slab+0x30/0x154) from [<c0396608>] (__slab_alloc.isra.52.constprop.54+0x2f0/0x3c4)
> >[ 73.877467] [<c0396608>] (__slab_alloc.isra.52.constprop.54+0x2f0/0x3c4) from [<c00a0614>] (kmem_cache_alloc+0x130/0x138)
> >[ 73.877489] [<c00a0614>] (kmem_cache_alloc+0x130/0x138) from [<c00a5d84>] (get_empty_filp+0x6c/0x1f4)
> >[ 73.877509] [<c00a5d84>] (get_empty_filp+0x6c/0x1f4) from [<c00b18b8>] (path_openat+0x2c/0x43c)
> >[ 73.877527] [<c00b18b8>] (path_openat+0x2c/0x43c) from [<c00b1f80>] (do_filp_open+0x2c/0x80)
> >[ 73.877544] [<c00b1f80>] (do_filp_open+0x2c/0x80) from [<c00a40a4>] (do_sys_open+0xe8/0x174)
> >[ 73.877564] [<c00a40a4>] (do_sys_open+0xe8/0x174) from [<c000e9c0>] (ret_fast_syscall+0x0/0x30)
> >[ 73.877574] Mem-info:
> >[ 73.877582] Normal per-cpu:
> >[ 73.877593] CPU 0: hi: 6, btch: 1 usd: 0
> >[ 73.877603] CPU 1: hi: 6, btch: 1 usd: 0
> >[ 73.877625] active_anon:1827 inactive_anon:5 isolated_anon:0
> >[ 73.877625] active_file:174 inactive_file:1158 isolated_file:0
> >[ 73.877625] unevictable:2493 dirty:1 writeback:1160 unstable:0
> >[ 73.877625] free:3978 slab_reclaimable:1161 slab_unreclaimable:1345
> >[ 73.877625] mapped:440 shmem:512 pagetables:526 bounce:0
> >[ 73.877625] free_cma:3833
> >[ 73.877669] Normal free:15912kB min:824kB low:1028kB high:1236kB active_anon:7308kB inactive_anon:20kB active_file:696kB inacs
> >[ 73.877679] lowmem_reserve[]: 0 0
> >[ 73.877695] Normal: 234*4kB (ERC) 208*8kB (URC) 200*16kB (C) 189*32kB (RC) 61*64kB (RC) 1*128kB (R) 0*256kB 0*512kB 0*1024kB B
> >[ 73.877769] 2196 total pagecache pages
> >[ 73.877780] 0 pages in swap cache
> >[ 73.877789] Swap cache stats: add 0, delete 0, find 0/0
> >[ 73.877796] Free swap = 0kB
> >[ 73.877804] Total swap = 0kB
> >[ 73.884950] 16384 pages of RAM
> >[ 73.884966] 5008 free pages
> >[ 73.884975] 1558 reserved pages
> >[ 73.884983] 1674 slab pages
> >[ 73.884990] 532633 pages shared
> >[ 73.884998] 0 pages swap cached
> >[ 73.885007] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
> >[ 73.908257] [ 502] 0 502 723 119 5 0 0 klogd
> >[ 73.936201] [ 504] 0 504 723 112 5 0 0 syslogd
> >[ 73.980352] [ 506] 0 506 723 76 5 0 0 telnetd
> >[ 74.028534] [ 566] 0 566 724 166 5 0 0 ash
> >[ 74.053796] [ 606] 0 606 723 57 4 0 0 udhcpc
> >[ 74.081419] [ 607] 0 607 724 161 5 0 0 dohell
> >[ 74.090873] [ 609] 0 609 724 68 5 0 0 dohell
> >[ 74.106407] [ 610] 0 610 724 61 5 0 0 dohell
> >[ 74.118017] [ 611] 0 611 724 58 5 0 0 dohell
> >[ 74.128390] [ 612] 0 612 723 110 5 0 0 nc
> >[ 74.198260] [ 613] 0 613 724 61 5 0 0 dohell
> >[ 74.214133] [ 614] 0 614 724 60 5 0 0 dohell
> >[ 74.227782] [ 615] 0 615 724 60 5 0 0 dohell
> >[ 74.240921] [ 616] 0 616 723 104 4 0 0 dd
> >[ 74.264528] [ 617] 0 617 941 360 4 0 0 dd
> >[ 74.284988] [ 618] 0 618 724 60 5 0 0 dohell
> >[ 74.312808] [ 620] 0 620 2523 2494 9 0 0 cyclictest
> >[ 74.333122] [ 623] 0 623 724 174 5 0 0 ls
> >[ 74.361071] [ 1623] 0 1623 690 95 4 0 0 sleep
> >[ 74.382116] [ 2008] 0 2008 448 133 5 0 0 hackbench
> >[ 74.402224] [ 2011] 0 2011 448 54 5 0 0 hackbench
> >[ 74.426216] [ 2012] 0 2012 448 54 5 0 0 hackbench
> >[ 74.452513] [ 2013] 0 2013 448 54 5 0 0 hackbench
> >[ 74.468437] [ 2014] 0 2014 448 54 5 0 0 hackbench
> >[ 74.481603] [ 2015] 0 2015 448 54 5 0 0 hackbench
> >[ 74.495083] [ 2016] 0 2016 448 54 5 0 0 hackbench
> >[ 74.528757] [ 2017] 0 2017 448 54 5 0 0 hackbench
> >[ 74.542685] [ 2019] 0 2019 448 54 5 0 0 hackbench
> >[ 74.561320] [ 2020] 0 2020 448 54 5 0 0 hackbench
> >[ 74.576906] [ 2021] 0 2021 448 54 5 0 0 hackbench
> >[ 74.586537] [ 2022] 0 2022 448 54 5 0 0 hackbench
> >[ 74.610881] [ 2023] 0 2023 448 54 5 0 0 hackbench
> >[ 74.696010] [ 2026] 0 2026 448 59 5 0 0 hackbench
> >[ 74.746412] [ 2032] 0 2032 448 64 5 0 0 hackbench
> >[ 74.781200] [ 2095] 0 2095 689 36 3 0 0 cat
> >[ 74.795979] [ 2098] 0 2098 246 1 2 0 0 ps
> >[ 74.815133] [ 2099] 0 2099 448 133 4 0 0 hackbench
> >[ 74.829415] [ 2100] 0 2100 690 98 4 0 0 cat
> >[ 74.844691] [ 2101] 0 2101 448 54 4 0 0 hackbench
> >[ 74.855449] [ 2102] 0 2102 448 54 4 0 0 hackbench
> >[ 74.876889] [ 2103] 0 2103 448 54 4 0 0 hackbench
> >[ 74.893808] [ 2104] 0 2104 448 54 4 0 0 hackbench
> >[ 74.915100] [ 2105] 0 2105 448 54 4 0 0 hackbench
> >[ 74.932730] [ 2106] 0 2106 448 54 4 0 0 hackbench
> >[ 74.965216] [ 2107] 0 2107 448 54 4 0 0 hackbench
> >[ 75.027941] [ 2108] 0 2108 448 54 4 0 0 hackbench
> >[ 75.046197] [ 2109] 0 2109 448 54 4 0 0 hackbench
> >[ 75.063038] [ 2110] 0 2110 448 54 4 0 0 hackbench
> >[ 75.079646] [ 2111] 0 2111 448 55 4 0 0 hackbench
> >[ 75.115715] [ 2112] 0 2112 448 55 4 0 0 hackbench
> >[ 75.125629] [ 2113] 0 2113 448 55 4 0 0 hackbench
> >[ 75.136509] [ 2114] 0 2114 448 55 4 0 0 hackbench
> >[ 75.146358] [ 2115] 0 2115 448 55 4 0 0 hackbench
> >[ 75.157808] [ 2116] 0 2116 448 55 4 0 0 hackbench
> >[ 75.167646] [ 2117] 0 2117 448 55 4 0 0 hackbench
> >[ 75.177705] [ 2118] 0 2118 448 55 4 0 0 hackbench
> >[ 75.187688] [ 2119] 0 2119 448 55 4 0 0 hackbench
> >[ 75.202440] [ 2120] 0 2120 448 55 4 0 0 hackbench
> >[ 75.226442] [ 2121] 0 2121 448 54 4 0 0 hackbench
> >[ 75.299323] [ 2190] 0 2190 724 152 5 0 0 ps
> >[ 75.309857] [ 2191] 0 2191 392 31 3 0 0 cat
> >[ 75.320704] [ 2192] 0 2192 116 33 3 0 0 hackbench
> >[ 75.331584] [ 2193] 0 2193 246 1 2 0 0 ps
> >[ 75.341056] Out of memory: Kill process 620 (cyclictest) score 163 or sacrifice child
> >[ 75.350899] Killed process 620 (cyclictest) total-vm:10092kB, anon-rss:8544kB, file-rss:1432kB
> >
> >Unfortunately I'm rather busy at the moment, so I couldn't investigate
> >this issue at all, I'm sorry.
> >Steven, probably this issue also concerns 3.8.13.14-rt28-rc1 (I just
> >discovered your post on the list).
> >
> >Regards,
> >Helmut
> >
> >PS: running this test with hackbench alone (no dohell), I get nearly
> >the same result with kmem_cache_alloc() failing.
> >PPS: I had to resend this post since I didn't get through to the list.
> >Sorry about any noise!
>
> Do you have RCU_BOOST enabled? My guess here is that you *something* is
> running at a higher priority not allowing rcuc/ to run and do its job.
The major effect of that commit is to move processing from softirq to
kthread. If your workload has quite a few interrupts, it is possible
that the softirq version would -usually- make progress, but the kthread
version not. (You could still get this to happen with softirq should
ksoftirqd kick in.)
Sebastian's suggestion of RCU_BOOST makes sense, and you will also need to
set CONFIG_RCU_BOOST_PRIO to be higher priority than your highest-priority
CPU-bound thread.
Thanx, Paul
next prev parent reply other threads:[~2014-03-07 17:04 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-03-06 19:05 Resend: 3.10.32-rt31: problem with "rcu: Eliminate softirq processing from rcutree" Helmut Buchsbaum
2014-03-07 16:21 ` Sebastian Andrzej Siewior
2014-03-07 17:04 ` Paul E. McKenney [this message]
2014-03-08 19:08 ` Helmut Buchsbaum
2014-03-08 19:19 ` Helmut Buchsbaum
2014-03-18 10:58 ` Sebastian Andrzej Siewior
2014-03-08 20:04 ` Paul E. McKenney
2014-03-18 11:05 ` Sebastian Andrzej Siewior
2014-03-20 14:50 ` Steven Rostedt
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20140307170419.GI3334@linux.vnet.ibm.com \
--to=paulmck@linux.vnet.ibm.com \
--cc=bigeasy@linutronix.de \
--cc=helmut.buchsbaum@gmail.com \
--cc=linux-rt-users@vger.kernel.org \
--cc=rostedt@goodmis.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).