linux-mm.kvack.org archive mirror
 help / color / mirror / Atom feed
From: Ying Han <yinghan@google.com>
To: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Cc: "linux-mm@kvack.org" <linux-mm@kvack.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"akpm@linux-foundation.org" <akpm@linux-foundation.org>,
	"nishimura@mxp.nes.nec.co.jp" <nishimura@mxp.nes.nec.co.jp>,
	"balbir@linux.vnet.ibm.com" <balbir@linux.vnet.ibm.com>
Subject: Re: [RFC][PATCH v3 0/10] memcg async reclaim
Date: Thu, 26 May 2011 21:33:32 -0700	[thread overview]
Message-ID: <BANLkTi=Cw8HSTUjNfJzH8GhfwQhUua-h7w@mail.gmail.com> (raw)
In-Reply-To: <20110527111639.22e3e257.kamezawa.hiroyu@jp.fujitsu.com>

On Thu, May 26, 2011 at 7:16 PM, KAMEZAWA Hiroyuki
<kamezawa.hiroyu@jp.fujitsu.com> wrote:
> On Thu, 26 May 2011 18:49:26 -0700
> Ying Han <yinghan@google.com> wrote:
>
>> On Wed, May 25, 2011 at 10:10 PM, KAMEZAWA Hiroyuki
>> <kamezawa.hiroyu@jp.fujitsu.com> wrote:
>> >
>> > It's now merge window...I just dump my patch queue to hear other's idea.
>> > I wonder I should wait until dirty_ratio for memcg is queued to mmotm...
>> > I'll be busy with LinuxCon Japan etc...in the next week.
>> >
>> > This patch is onto mmotm-May-11 + some patches queued in mmotm, as numa_stat.
>> >
>> > This is a patch for memcg to keep margin to the limit in background.
>> > By keeping some margin to the limit in background, application can
>> > avoid foreground memory reclaim at charge() and this will help latency.
>> >
>> > Main changes from v2 is.
>> >  - use SCHED_IDLE.
>> >  - removed most of heuristic codes. Now, code is very simple.
>> >
>> > By using SCHED_IDLE, async memory reclaim can only consume 0.3%? of cpu
>> > if the system is truely busy but can use much CPU if the cpu is idle.
>> > Because my purpose is for reducing latency without affecting other running
>> > applications, SCHED_IDLE fits this work.
>> >
>> > If application need to stop by some I/O or event, background memory reclaim
>> > will cull memory while the system is idle.
>> >
>> > Perforemce:
>> >  Running an httpd (apache) under 300M limit. And access 600MB working set
>> >  with normalized distribution access by apatch-bench.
>> >  apatch bench's concurrency was 4 and did 40960 accesses.
>> >
>> > Without async reclaim:
>> > Connection Times (ms)
>> >              min  mean[+/-sd] median   max
>> > Connect:        0    0   0.0      0       2
>> > Processing:    30   37  28.3     32    1793
>> > Waiting:       28   35  25.5     31    1792
>> > Total:         30   37  28.4     32    1793
>> >
>> > Percentage of the requests served within a certain time (ms)
>> >  50%     32
>> >  66%     32
>> >  75%     33
>> >  80%     34
>> >  90%     39
>> >  95%     60
>> >  98%    100
>> >  99%    133
>> >  100%   1793 (longest request)
>> >
>> > With async reclaim:
>> > Connection Times (ms)
>> >              min  mean[+/-sd] median   max
>> > Connect:        0    0   0.0      0       2
>> > Processing:    30   35  12.3     32     678
>> > Waiting:       28   34  12.0     31     658
>> > Total:         30   35  12.3     32     678
>> >
>> > Percentage of the requests served within a certain time (ms)
>> >  50%     32
>> >  66%     32
>> >  75%     33
>> >  80%     34
>> >  90%     39
>> >  95%     49
>> >  98%     71
>> >  99%     86
>> >  100%    678 (longest request)
>> >
>> >
>> > It seems latency is stabilized by hiding memory reclaim.
>> >
>> > The score for memory reclaim was following.
>> > See patch 10 for meaning of each member.
>> >
>> > == without async reclaim ==
>> > recent_scan_success_ratio 44
>> > limit_scan_pages 388463
>> > limit_freed_pages 162238
>> > limit_elapsed_ns 13852159231
>> > soft_scan_pages 0
>> > soft_freed_pages 0
>> > soft_elapsed_ns 0
>> > margin_scan_pages 0
>> > margin_freed_pages 0
>> > margin_elapsed_ns 0
>> >
>> > == with async reclaim ==
>> > recent_scan_success_ratio 6
>> > limit_scan_pages 0
>> > limit_freed_pages 0
>> > limit_elapsed_ns 0
>> > soft_scan_pages 0
>> > soft_freed_pages 0
>> > soft_elapsed_ns 0
>> > margin_scan_pages 1295556
>> > margin_freed_pages 122450
>> > margin_elapsed_ns 644881521
>> >
>> >
>> > For this case, SCHED_IDLE workqueue can reclaim enough memory to the httpd.
>> >
>> > I may need to dig why scan_success_ratio is far different in the both case.
>> > I guess the difference of epalsed_ns is because several threads enter
>> > memory reclaim when async reclaim doesn't run. But may not...
>> >
>>
>>
>> Hmm.. I noticed a very strange behavior on a simple test w/ the patch set.
>>
>> Test:
>> I created a 4g memcg and start doing cat. Then the memcg being OOM
>> killed as soon as it reaches its hard_limit. We shouldn't hit OOM even
>> w/o async-reclaim.
>>
>> Again, I will read through the patch. But like to post the test result first.
>>
>> $ echo $$ >/dev/cgroup/memory/A/tasks
>> $ cat /dev/cgroup/memory/A/memory.limit_in_bytes
>> 4294967296
>>
>> $ time cat /export/hdc3/dd_A/tf0 > /dev/zero
>> Killed
>>
>
> I did the same kind of test without any problem...but ok, I'll do more test
> later.
>
>
>
>> real  0m53.565s
>> user  0m0.061s
>> sys   0m4.814s
>>
>> Here is the OOM log:
>>
>> May 26 18:43:00  kernel: [  963.489112] cat invoked oom-killer:
>> gfp_mask=0xd0, order=0, oom_adj=0, oom_score_adj=0
>> May 26 18:43:00  kernel: [  963.489121] Pid: 9425, comm: cat Tainted:
>> G        W   2.6.39-mcg-DEV #131
>> May 26 18:43:00  kernel: [  963.489123] Call Trace:
>> May 26 18:43:00  kernel: [  963.489134]  [<ffffffff810e3512>]
>> dump_header+0x82/0x1af
>> May 26 18:43:00  kernel: [  963.489137]  [<ffffffff810e33ca>] ?
>> spin_lock+0xe/0x10
>> May 26 18:43:00  kernel: [  963.489140]  [<ffffffff810e33f9>] ?
>> find_lock_task_mm+0x2d/0x67
>> May 26 18:43:00  kernel: [  963.489143]  [<ffffffff810e38dd>]
>> oom_kill_process+0x50/0x27b
>> May 26 18:43:00  kernel: [  963.489155]  [<ffffffff810e3dc6>]
>> mem_cgroup_out_of_memory+0x9a/0xe4
>> May 26 18:43:00  kernel: [  963.489160]  [<ffffffff811153aa>]
>> mem_cgroup_handle_oom+0x134/0x1fe
>> May 26 18:43:00  kernel: [  963.489163]  [<ffffffff81114a72>] ?
>> __mem_cgroup_insert_exceeded+0x83/0x83
>> May 26 18:43:00  kernel: [  963.489176]  [<ffffffff811166e9>]
>> __mem_cgroup_try_charge.clone.3+0x368/0x43a
>> May 26 18:43:00  kernel: [  963.489179]  [<ffffffff81117586>]
>> mem_cgroup_cache_charge+0x95/0x123
>> May 26 18:43:00  kernel: [  963.489183]  [<ffffffff810e16d8>]
>> add_to_page_cache_locked+0x42/0x114
>> May 26 18:43:00  kernel: [  963.489185]  [<ffffffff810e17db>]
>> add_to_page_cache_lru+0x31/0x5f
>> May 26 18:43:00  kernel: [  963.489189]  [<ffffffff81145636>]
>> mpage_readpages+0xb6/0x132
>> May 26 18:43:00  kernel: [  963.489194]  [<ffffffff8119992f>] ?
>> noalloc_get_block_write+0x24/0x24
>> May 26 18:43:00  kernel: [  963.489197]  [<ffffffff8119992f>] ?
>> noalloc_get_block_write+0x24/0x24
>> May 26 18:43:00  kernel: [  963.489201]  [<ffffffff81036742>] ?
>> __switch_to+0x160/0x212
>> May 26 18:43:00  kernel: [  963.489205]  [<ffffffff811978b2>]
>> ext4_readpages+0x1d/0x1f
>> May 26 18:43:00  kernel: [  963.489209]  [<ffffffff810e8d4b>]
>> __do_page_cache_readahead+0x144/0x1e3
>> May 26 18:43:00  kernel: [  963.489212]  [<ffffffff810e8e0b>]
>> ra_submit+0x21/0x25
>> May 26 18:43:00  kernel: [  963.489215]  [<ffffffff810e9075>]
>> ondemand_readahead+0x18c/0x19f
>> May 26 18:43:00  kernel: [  963.489218]  [<ffffffff810e9105>]
>> page_cache_async_readahead+0x7d/0x86
>> May 26 18:43:00  kernel: [  963.489221]  [<ffffffff810e2b7e>]
>> generic_file_aio_read+0x2d8/0x5fe
>> May 26 18:43:00  kernel: [  963.489225]  [<ffffffff81119626>]
>> do_sync_read+0xcb/0x108
>> May 26 18:43:00  kernel: [  963.489230]  [<ffffffff811f168a>] ?
>> fsnotify_perm+0x66/0x72
>> May 26 18:43:00  kernel: [  963.489233]  [<ffffffff811f16f7>] ?
>> security_file_permission+0x2e/0x33
>> May 26 18:43:00  kernel: [  963.489236]  [<ffffffff8111a0c8>]
>> vfs_read+0xab/0x107
>> May 26 18:43:00  kernel: [  963.489239]  [<ffffffff8111a1e4>] sys_read+0x4a/0x6e
>> May 26 18:43:00  kernel: [  963.489244]  [<ffffffff8140f469>]
>> sysenter_dispatch+0x7/0x27
>> May 26 18:43:00  kernel: [  963.489248] Task in /A killed as a result
>> of limit of /A
>> May 26 18:43:00  kernel: [  963.489251] memory: usage 4194304kB, limit
>> 4194304kB, failcnt 26
>> May 26 18:43:00  kernel: [  963.489253] memory+swap: usage 0kB, limit
>> 9007199254740991kB, failcnt 0
>>
>
> Hmm, why memory+swap usage 0kb here...
>
> In this set, I used mem_cgroup_margin() rather than res_counter_margin().
> Hmm, do you disable swap accounting ? If so, I may miss some.

Yes, I disabled the swap accounting in .config:
# CONFIG_CGROUP_MEM_RES_CTLR_SWAP is not set


Here is how i reproduce it:

$ mkdir /dev/cgroup/memory/D
$ echo 4g >/dev/cgroup/memory/D/memory.limit_in_bytes

$ cat /dev/cgroup/memory/D/memory.limit_in_bytes
4294967296

$ cat /dev/cgroup/memory/D/memory.
memory.async_control             memory.max_usage_in_bytes
memory.soft_limit_in_bytes       memory.use_hierarchy
memory.failcnt                   memory.move_charge_at_immigrate
memory.stat
memory.force_empty               memory.oom_control
memory.swappiness
memory.limit_in_bytes            memory.reclaim_stat
memory.usage_in_bytes

$ cat /dev/cgroup/memory/D/memory.async_control
0
$ echo 1 >/dev/cgroup/memory/D/memory.async_control
$ cat /dev/cgroup/memory/D/memory.async_control
1

$ echo $$ >/dev/cgroup/memory/D/tasks
$ cat /proc/4358/cgroup
3:memory:/D

$ time cat /export/hdc3/dd_A/tf0 > /dev/zero
Killed

--Ying


>
> Thanks,
> -Kame
>
>
>
>
>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

  reply	other threads:[~2011-05-27  4:33 UTC|newest]

Thread overview: 35+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2011-05-26  5:10 [RFC][PATCH v3 0/10] memcg async reclaim KAMEZAWA Hiroyuki
2011-05-26  5:15 ` [RFC][PATCH v3 1/10] check reclaimable in hierarchy walk KAMEZAWA Hiroyuki
2011-05-27  5:47   ` Ying Han
2011-05-26  5:18 ` [RFC][PATCH v3 2/10] memcg: fix cached charge drain ratio KAMEZAWA Hiroyuki
2011-05-26  5:19 ` [RFC][PATCH v3 3/10] memcg: a test whether zone is reclaimable or not KAMEZAWA Hiroyuki
2011-05-27  7:21   ` Ying Han
2011-05-27  8:25     ` KAMEZAWA Hiroyuki
2011-05-26  5:20 ` [RFC][PATCH v3 4/10] memcg: export swappiness KAMEZAWA Hiroyuki
2011-05-26  5:23 ` [RFC][PATCH v3 5/10] memcg keep margin to limit in background KAMEZAWA Hiroyuki
2011-05-26  5:24 ` [RFC][PATCH v3 6/10] memcg : auto keep margin in background , workqueue core KAMEZAWA Hiroyuki
2011-05-26  5:30 ` [RFC][PATCH v3 7/10] workqueue: add WQ_IDLEPRI KAMEZAWA Hiroyuki
2011-05-26  9:38   ` Tejun Heo
2011-05-26 10:30     ` KAMEZAWA Hiroyuki
2011-05-26 10:50       ` KAMEZAWA Hiroyuki
2011-05-26 11:44         ` Tejun Heo
2011-05-26 23:41           ` KAMEZAWA Hiroyuki
2011-05-27 20:20     ` Peter Zijlstra
2011-05-26  5:32 ` [RFC][PATCH v3 8/10] memcg: scan ratio calculation KAMEZAWA Hiroyuki
2011-05-26  5:35 ` [RFC][PATCH v3 9/10] memcg: scan limited memory reclaim KAMEZAWA Hiroyuki
2011-05-26  5:36 ` [RFC][PATCH v3 10/10] memcg : reclaim statistics KAMEZAWA Hiroyuki
2011-05-27  1:13   ` Ying Han
2011-05-27  1:17   ` Ying Han
2011-05-27  1:12     ` KAMEZAWA Hiroyuki
2011-05-27  1:14     ` KAMEZAWA Hiroyuki
2011-05-27  1:22       ` Ying Han
2011-05-27  1:49 ` [RFC][PATCH v3 0/10] memcg async reclaim Ying Han
2011-05-27  2:16   ` KAMEZAWA Hiroyuki
2011-05-27  4:33     ` Ying Han [this message]
2011-05-27  4:34       ` KAMEZAWA Hiroyuki
2011-05-27  4:49         ` Ying Han
2011-05-27  7:20           ` Ying Han
2011-05-31 17:04             ` Ying Han
2011-05-27  2:48   ` KAMEZAWA Hiroyuki
2011-05-27  3:05     ` KAMEZAWA Hiroyuki
2011-05-27  4:34       ` Ying Han

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='BANLkTi=Cw8HSTUjNfJzH8GhfwQhUua-h7w@mail.gmail.com' \
    --to=yinghan@google.com \
    --cc=akpm@linux-foundation.org \
    --cc=balbir@linux.vnet.ibm.com \
    --cc=kamezawa.hiroyu@jp.fujitsu.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=nishimura@mxp.nes.nec.co.jp \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).