From: Vladimir Davydov <vdavydov@virtuozzo.com>
To: Michal Hocko <mhocko@kernel.org>
Cc: Andrew Morton <akpm@linux-foundation.org>,
Johannes Weiner <hannes@cmpxchg.org>,
linux-mm@kvack.org, LKML <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH] memcg: put soft limit reclaim out of way if the excess tree is empty
Date: Mon, 1 Aug 2016 17:26:15 +0300 [thread overview]
Message-ID: <20160801142615.GC19395@esperanza> (raw)
In-Reply-To: <20160801141227.GI13544@dhcp22.suse.cz>
On Mon, Aug 01, 2016 at 04:12:28PM +0200, Michal Hocko wrote:
...
> From: Michal Hocko <mhocko@suse.com>
> Date: Mon, 1 Aug 2016 10:42:06 +0200
> Subject: [PATCH] memcg: put soft limit reclaim out of way if the excess tree
> is empty
>
> We've had a report about soft lockups caused by lock bouncing in the
> soft reclaim path:
>
> [331404.849734] BUG: soft lockup - CPU#0 stuck for 22s! [kav4proxy-kavic:3128]
> [331404.849920] RIP: 0010:[<ffffffff81469798>] [<ffffffff81469798>] _raw_spin_lock+0x18/0x20
> [331404.849997] Call Trace:
> [331404.850010] [<ffffffff811557ea>] mem_cgroup_soft_limit_reclaim+0x25a/0x280
> [331404.850020] [<ffffffff8111041d>] shrink_zones+0xed/0x200
> [331404.850027] [<ffffffff81111a94>] do_try_to_free_pages+0x74/0x320
> [331404.850034] [<ffffffff81112072>] try_to_free_pages+0x112/0x180
> [331404.850042] [<ffffffff81104a6f>] __alloc_pages_slowpath+0x3ff/0x820
> [331404.850049] [<ffffffff81105079>] __alloc_pages_nodemask+0x1e9/0x200
> [331404.850056] [<ffffffff81141e01>] alloc_pages_vma+0xe1/0x290
> [331404.850064] [<ffffffff8112402f>] do_wp_page+0x19f/0x840
> [331404.850071] [<ffffffff811257cd>] handle_pte_fault+0x1cd/0x230
> [331404.850079] [<ffffffff8146d3ed>] do_page_fault+0x1fd/0x4c0
> [331404.850087] [<ffffffff81469ec5>] page_fault+0x25/0x30
>
> There are no memcgs created so there cannot be any in the soft limit
> excess obviously:
> [...]
> memory 0 1 1
>
> so all this just seems to be mem_cgroup_largest_soft_limit_node
> trying to get spin_lock_irq(&mctz->lock) just to find out that the soft
> limit excess tree is empty. This is just pointless waisting of cycles
> and cache line bouncing during heavy parallel reclaim on large machines.
> The particular machine wasn't very healthy and most probably suffering
> from a memory leak which just caused the memory reclaim to trash
> heavily. But bouncing on the lock certainly didn't help...
>
> Introduce soft_limit_tree_empty which does the optimistic lockless check
> and bail out early if the tree is empty. This is theoretically racy but
> that shouldn't matter all that much. First of all soft limit is a best
> effort feature and it is slowly getting deprecated and its usage should
> be really scarce. Bouncing on a lock without a good reason is surely
> much bigger problem, especially on large CPU machines.
>
> Signed-off-by: Michal Hocko <mhocko@suse.com>
Acked-by: Vladimir Davydov <vdavydov@virtuozzo.com>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
WARNING: multiple messages have this Message-ID (diff)
From: Vladimir Davydov <vdavydov@virtuozzo.com>
To: Michal Hocko <mhocko@kernel.org>
Cc: Andrew Morton <akpm@linux-foundation.org>,
Johannes Weiner <hannes@cmpxchg.org>, <linux-mm@kvack.org>,
LKML <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH] memcg: put soft limit reclaim out of way if the excess tree is empty
Date: Mon, 1 Aug 2016 17:26:15 +0300 [thread overview]
Message-ID: <20160801142615.GC19395@esperanza> (raw)
In-Reply-To: <20160801141227.GI13544@dhcp22.suse.cz>
On Mon, Aug 01, 2016 at 04:12:28PM +0200, Michal Hocko wrote:
...
> From: Michal Hocko <mhocko@suse.com>
> Date: Mon, 1 Aug 2016 10:42:06 +0200
> Subject: [PATCH] memcg: put soft limit reclaim out of way if the excess tree
> is empty
>
> We've had a report about soft lockups caused by lock bouncing in the
> soft reclaim path:
>
> [331404.849734] BUG: soft lockup - CPU#0 stuck for 22s! [kav4proxy-kavic:3128]
> [331404.849920] RIP: 0010:[<ffffffff81469798>] [<ffffffff81469798>] _raw_spin_lock+0x18/0x20
> [331404.849997] Call Trace:
> [331404.850010] [<ffffffff811557ea>] mem_cgroup_soft_limit_reclaim+0x25a/0x280
> [331404.850020] [<ffffffff8111041d>] shrink_zones+0xed/0x200
> [331404.850027] [<ffffffff81111a94>] do_try_to_free_pages+0x74/0x320
> [331404.850034] [<ffffffff81112072>] try_to_free_pages+0x112/0x180
> [331404.850042] [<ffffffff81104a6f>] __alloc_pages_slowpath+0x3ff/0x820
> [331404.850049] [<ffffffff81105079>] __alloc_pages_nodemask+0x1e9/0x200
> [331404.850056] [<ffffffff81141e01>] alloc_pages_vma+0xe1/0x290
> [331404.850064] [<ffffffff8112402f>] do_wp_page+0x19f/0x840
> [331404.850071] [<ffffffff811257cd>] handle_pte_fault+0x1cd/0x230
> [331404.850079] [<ffffffff8146d3ed>] do_page_fault+0x1fd/0x4c0
> [331404.850087] [<ffffffff81469ec5>] page_fault+0x25/0x30
>
> There are no memcgs created so there cannot be any in the soft limit
> excess obviously:
> [...]
> memory 0 1 1
>
> so all this just seems to be mem_cgroup_largest_soft_limit_node
> trying to get spin_lock_irq(&mctz->lock) just to find out that the soft
> limit excess tree is empty. This is just pointless waisting of cycles
> and cache line bouncing during heavy parallel reclaim on large machines.
> The particular machine wasn't very healthy and most probably suffering
> from a memory leak which just caused the memory reclaim to trash
> heavily. But bouncing on the lock certainly didn't help...
>
> Introduce soft_limit_tree_empty which does the optimistic lockless check
> and bail out early if the tree is empty. This is theoretically racy but
> that shouldn't matter all that much. First of all soft limit is a best
> effort feature and it is slowly getting deprecated and its usage should
> be really scarce. Bouncing on a lock without a good reason is surely
> much bigger problem, especially on large CPU machines.
>
> Signed-off-by: Michal Hocko <mhocko@suse.com>
Acked-by: Vladimir Davydov <vdavydov@virtuozzo.com>
next prev parent reply other threads:[~2016-08-01 14:26 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-08-01 10:00 [PATCH] memcg: put soft limit reclaim out of way if the excess tree is empty Michal Hocko
2016-08-01 10:00 ` Michal Hocko
2016-08-01 13:57 ` Vladimir Davydov
2016-08-01 13:57 ` Vladimir Davydov
2016-08-01 14:12 ` Michal Hocko
2016-08-01 14:12 ` Michal Hocko
2016-08-01 14:26 ` Vladimir Davydov [this message]
2016-08-01 14:26 ` Vladimir Davydov
2016-08-01 15:03 ` Johannes Weiner
2016-08-01 15:03 ` Johannes Weiner
2016-08-01 15:24 ` Michal Hocko
2016-08-01 15:24 ` Michal Hocko
2016-08-01 17:17 ` Johannes Weiner
2016-08-01 17:17 ` Johannes Weiner
2016-08-01 17:39 ` Michal Hocko
2016-08-01 17:39 ` Michal Hocko
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20160801142615.GC19395@esperanza \
--to=vdavydov@virtuozzo.com \
--cc=akpm@linux-foundation.org \
--cc=hannes@cmpxchg.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mhocko@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.