* Re: [v3 0/9] parallelized "struct page" zeroing
From: David Miller @ 2017-05-12 16:57 UTC (permalink / raw)
To: pasha.tatashin
Cc: mhocko, linux-kernel, sparclinux, linux-mm, linuxppc-dev,
linux-s390, borntraeger, heiko.carstens
In-Reply-To: <65b8a658-76d1-0617-ece8-ff7a3c1c4046@oracle.com>
From: Pasha Tatashin <pasha.tatashin@oracle.com>
Date: Thu, 11 May 2017 16:59:33 -0400
> We should either keep memset() only for deferred struct pages as what
> I have in my patches.
>
> Another option is to add a new function struct_page_clear() which
> would default to memset() and to something else on platforms that
> decide to optimize it.
>
> On SPARC it would call STBIs, and we would do one membar call after
> all "struct pages" are initialized.
No membars will be performed for single individual page struct clear,
the cutoff to use the STBI is larger than that.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [patch 2/2] MM: allow per-cpu vmstat_threshold and vmstat_worker configuration
From: Christoph Lameter @ 2017-05-12 16:57 UTC (permalink / raw)
To: Marcelo Tosatti
Cc: Luiz Capitulino, linux-kernel, linux-mm, Rik van Riel,
Linux RT Users, cmetcalf
In-Reply-To: <20170512161915.GA4185@amt.cnet>
On Fri, 12 May 2017, Marcelo Tosatti wrote:
> > What exactly is the issue you are seeing and want to address? I think we
> > have similar aims and as far as I know the current situation is already
> > good enough for what you may need. You may just not be aware of how to
> > configure this.
>
> I want to disable vmstat worker thread completly from an isolated CPU.
> Because it adds overhead to a latency target, target which
> the lower the better.
NOHZ already does that. I wanted to know what your problem is that you
see. The latency issue has already been solved as far as I can tell .
Please tell me why the existing solutions are not sufficient for you.
> > I doubt that doing inline updates will do much good compared to what we
> > already have and what the dataplan mode can do.
>
> Can the dataplan mode disable vmstat worker thread completly on a given
> CPU?
That already occurs when you call quiet_vmstat() and is used by the NOHZ
logic. Configure that correctly and you should be fine.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [v3 0/9] parallelized "struct page" zeroing
From: Pasha Tatashin @ 2017-05-12 17:24 UTC (permalink / raw)
To: David Miller
Cc: mhocko, linux-kernel, sparclinux, linux-mm, linuxppc-dev,
linux-s390, borntraeger, heiko.carstens
In-Reply-To: <20170512.125708.475573831936972365.davem@davemloft.net>
On 05/12/2017 12:57 PM, David Miller wrote:
> From: Pasha Tatashin <pasha.tatashin@oracle.com>
> Date: Thu, 11 May 2017 16:59:33 -0400
>
>> We should either keep memset() only for deferred struct pages as what
>> I have in my patches.
>>
>> Another option is to add a new function struct_page_clear() which
>> would default to memset() and to something else on platforms that
>> decide to optimize it.
>>
>> On SPARC it would call STBIs, and we would do one membar call after
>> all "struct pages" are initialized.
>
> No membars will be performed for single individual page struct clear,
> the cutoff to use the STBI is larger than that.
>
Right now it is larger, but what I suggested is to add a new optimized
routine just for this case, which would do STBI for 64-bytes but without
membar (do membar at the end of memmap_init_zone() and
deferred_init_memmap()
#define struct_page_clear(page) \
__asm__ __volatile__( \
"stxa %%g0, [%0]%2\n" \
"stxa %%xg0, [%0 + %1]%2\n" \
: /* No output */ \
: "r" (page), "r" (0x20), "i"(ASI_BLK_INIT_QUAD_LDD_P))
And insert it into __init_single_page() instead of memset()
The final result is 4.01s/T which is even faster compared to current 4.97s/T
Pasha
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [v3 0/9] parallelized "struct page" zeroing
From: David Miller @ 2017-05-12 17:37 UTC (permalink / raw)
To: pasha.tatashin
Cc: mhocko, linux-kernel, sparclinux, linux-mm, linuxppc-dev,
linux-s390, borntraeger, heiko.carstens
In-Reply-To: <6da8d4a6-3332-8331-c329-b05efd88a70d@oracle.com>
From: Pasha Tatashin <pasha.tatashin@oracle.com>
Date: Fri, 12 May 2017 13:24:52 -0400
> Right now it is larger, but what I suggested is to add a new optimized
> routine just for this case, which would do STBI for 64-bytes but
> without membar (do membar at the end of memmap_init_zone() and
> deferred_init_memmap()
>
> #define struct_page_clear(page) \
> __asm__ __volatile__( \
> "stxa %%g0, [%0]%2\n" \
> "stxa %%xg0, [%0 + %1]%2\n" \
> : /* No output */ \
> : "r" (page), "r" (0x20), "i"(ASI_BLK_INIT_QUAD_LDD_P))
>
> And insert it into __init_single_page() instead of memset()
>
> The final result is 4.01s/T which is even faster compared to current
> 4.97s/T
Ok, indeed, that would work.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* [RFC] [PATCH 0/1] ksm: fix use after free with merge_across_nodes = 0
From: Andrea Arcangeli @ 2017-05-12 19:38 UTC (permalink / raw)
To: Andrew Morton, linux-mm
Cc: Evgheni Dereveanchin, Andrey Ryabinin, Petr Holasek, Hugh Dickins,
Arjan van de Ven, Davidlohr Bueso, Gavin Guo, Jay Vosburgh,
Mel Gorman
Hello,
The KSMscale patch in -mm (not yet upstream) is fundamental for
enterprise use and in turn it's included in -mm, RHEL, CentoOS and
Ubuntu and it'd be great if it could be merged upstream (especially
after solving this problem with merge_across_nodes = 0 ...).
https://marc.info/?l=linux-mm&m=149265809928003&w=2
http://kernel.ubuntu.com/~gavinguo/sf00131845/numa-131845.svg
http://kernel.ubuntu.com/~gavinguo/sf00131845/virtual_appliances_loading.png
A few weeks ago I got a report that with merge_across_nodes set to 0
KSM would eventually crash with an user after free (I assumed it was
an use after free because the kindly provided crashdump showed a
corrupted stable_node). Everything was again rock solid after setting
merge_across_nodes back to 1.
merge_across_nodes set to 0 is a tuning performance optimization
for NUMA that creates a different copy of KSM pages for each NUMA node
with a KSM stable_tree for each node (instead of sharing the same
equal memory across the whole system with a single stable_tree).
I couldn't reproduce this bug so far but there's a definitive use
after free in the merge_across_nodes = 0 path, so it would help if who
can reproduce already can give this a spin (untested... or better
tested but only in a NUMA balancing environment that never reproduced the use
after free in the first place so it's inconclusive).
In production I recommend to leave the merge_across_nodes default
value set to 1 if running with the KSMscale patch applied for the time
being, until this is confirmed fixed.
Again this fix should be considered untested so it should be run in testing
environment only.
Thanks,
Andrea
Andrea Arcangeli (1):
ksm: fix use after free with merge_across_nodes = 0
mm/ksm.c | 66 +++++++++++++++++++++++++++++++++++++++++++++++++++++-----------
1 file changed, 55 insertions(+), 11 deletions(-)
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* [PATCH 1/1] ksm: fix use after free with merge_across_nodes = 0
From: Andrea Arcangeli @ 2017-05-12 19:38 UTC (permalink / raw)
To: Andrew Morton, linux-mm
Cc: Evgheni Dereveanchin, Andrey Ryabinin, Petr Holasek, Hugh Dickins,
Arjan van de Ven, Davidlohr Bueso, Gavin Guo, Jay Vosburgh,
Mel Gorman
In-Reply-To: <20170512193805.8807-1-aarcange@redhat.com>
If merge_across_nodes was manually set to 0 (not the default value) by
the admin or a tuned profile on NUMA systems triggering cross-NODE
page migrations, a stable_node use after free could materialize.
If the chain is collapsed stable_node would point to the old chain
that was already freed. stable_node_dup would be the stable_node dup
now converted to a regular stable_node and indexed in the rbtree in
replacement of the freed stable_node chain (not anymore a dup).
This special case where the chain is collapsed in the NUMA replacement
path, is now detected by setting stable_node to NULL by the
chain_prune callee if it decides to collapse the chain. This tells the
NUMA replacement code that even if stable_node and stable_node_dup are
different, this is not a chain if stable_node is NULL, as the
stable_node_dup was converted to a regular stable_node and the chain
was collapsed.
It is generally safer for the callee to force the caller stable_node
to NULL the moment it become stale so any other mistake like this
would result in an instant Oops easier to debug than an use after free.
Otherwise the replace logic would act like if stable_node was a valid
chain, when in fact it was freed. Notably
stable_node_chain_add_dup(page_node, stable_node) would run on a
stable stable_node.
Andrey Ryabinin found the source of the use after free in
chain_prune().
Reported-by: Andrey Ryabinin <aryabinin@virtuozzo.com>
Reported-by: Evgheni Dereveanchin <ederevea@redhat.com>
Signed-off-by: Andrea Arcangeli <aarcange@redhat.com>
---
mm/ksm.c | 66 +++++++++++++++++++++++++++++++++++++++++++++++++++++-----------
1 file changed, 55 insertions(+), 11 deletions(-)
diff --git a/mm/ksm.c b/mm/ksm.c
index 44a1b2a..b53fd58 100644
--- a/mm/ksm.c
+++ b/mm/ksm.c
@@ -338,6 +338,7 @@ static inline void stable_node_chain_add_dup(struct stable_node *dup,
static inline void __stable_node_dup_del(struct stable_node *dup)
{
+ VM_BUG_ON(!is_stable_node_dup(dup));
hlist_del(&dup->hlist_dup);
ksm_stable_node_dups--;
}
@@ -1315,12 +1316,12 @@ bool is_page_sharing_candidate(struct stable_node *stable_node)
return __is_page_sharing_candidate(stable_node, 0);
}
-static struct stable_node *stable_node_dup(struct stable_node *stable_node,
+static struct stable_node *stable_node_dup(struct stable_node **_stable_node,
struct page **tree_page,
struct rb_root *root,
bool prune_stale_stable_nodes)
{
- struct stable_node *dup, *found = NULL;
+ struct stable_node *dup, *found = NULL, *stable_node = *_stable_node;
struct hlist_node *hlist_safe;
struct page *_tree_page;
int nr = 0;
@@ -1393,6 +1394,15 @@ static struct stable_node *stable_node_dup(struct stable_node *stable_node,
free_stable_node(stable_node);
ksm_stable_node_chains--;
ksm_stable_node_dups--;
+ /*
+ * NOTE: the caller depends on the
+ * *_stable_node to become NULL if the chain
+ * was collapsed. Enforce that if anything
+ * uses a stale (freed) stable_node chain a
+ * visible crash will materialize (instead of
+ * an use after free).
+ */
+ *_stable_node = stable_node = NULL;
} else if (__is_page_sharing_candidate(found, 1)) {
/*
* Refile our candidate at the head
@@ -1422,11 +1432,12 @@ static struct stable_node *stable_node_dup_any(struct stable_node *stable_node,
typeof(*stable_node), hlist_dup);
}
-static struct stable_node *__stable_node_chain(struct stable_node *stable_node,
+static struct stable_node *__stable_node_chain(struct stable_node **_stable_node,
struct page **tree_page,
struct rb_root *root,
bool prune_stale_stable_nodes)
{
+ struct stable_node *stable_node = *_stable_node;
if (!is_stable_node_chain(stable_node)) {
if (is_page_sharing_candidate(stable_node)) {
*tree_page = get_ksm_page(stable_node, false);
@@ -1434,11 +1445,11 @@ static struct stable_node *__stable_node_chain(struct stable_node *stable_node,
}
return NULL;
}
- return stable_node_dup(stable_node, tree_page, root,
+ return stable_node_dup(_stable_node, tree_page, root,
prune_stale_stable_nodes);
}
-static __always_inline struct stable_node *chain_prune(struct stable_node *s_n,
+static __always_inline struct stable_node *chain_prune(struct stable_node **s_n,
struct page **t_p,
struct rb_root *root)
{
@@ -1449,7 +1460,7 @@ static __always_inline struct stable_node *chain(struct stable_node *s_n,
struct page **t_p,
struct rb_root *root)
{
- return __stable_node_chain(s_n, t_p, root, false);
+ return __stable_node_chain(&s_n, t_p, root, false);
}
/*
@@ -1490,7 +1501,15 @@ static struct page *stable_tree_search(struct page *page)
cond_resched();
stable_node = rb_entry(*new, struct stable_node, node);
stable_node_any = NULL;
- stable_node_dup = chain_prune(stable_node, &tree_page, root);
+ stable_node_dup = chain_prune(&stable_node, &tree_page, root);
+ /*
+ * NOTE: stable_node may have been freed by
+ * chain_prune() if the returned stable_node_dup is
+ * not NULL. stable_node_dup may have been inserted in
+ * the rbtree instead as a regular stable_node (in
+ * order to collapse the stable_node chain if a single
+ * stable_node dup was found in it).
+ */
if (!stable_node_dup) {
/*
* Either all stable_node dups were full in
@@ -1605,20 +1624,33 @@ static struct page *stable_tree_search(struct page *page)
return NULL;
replace:
- if (stable_node_dup == stable_node) {
+ /*
+ * If stable_node was a chain and chain_prune collapsed it,
+ * stable_node will be NULL here. In that case the
+ * stable_node_dup is the regular stable_node that has
+ * replaced the chain. If stable_node is not NULL and equal to
+ * stable_node_dup there was no chain and stable_node_dup is
+ * the regular stable_node in the stable rbtree. Otherwise
+ * stable_node is the chain and stable_node_dup is the dup to
+ * replace.
+ */
+ if (!stable_node || stable_node_dup == stable_node) {
+ VM_BUG_ON(is_stable_node_chain(stable_node_dup));
+ VM_BUG_ON(is_stable_node_dup(stable_node_dup));
/* there is no chain */
if (page_node) {
VM_BUG_ON(page_node->head != &migrate_nodes);
list_del(&page_node->list);
DO_NUMA(page_node->nid = nid);
- rb_replace_node(&stable_node->node, &page_node->node,
+ rb_replace_node(&stable_node_dup->node,
+ &page_node->node,
root);
if (is_page_sharing_candidate(page_node))
get_page(page);
else
page = NULL;
} else {
- rb_erase(&stable_node->node, root);
+ rb_erase(&stable_node_dup->node, root);
page = NULL;
}
} else {
@@ -1645,7 +1677,17 @@ static struct page *stable_tree_search(struct page *page)
/* stable_node_dup could be null if it reached the limit */
if (!stable_node_dup)
stable_node_dup = stable_node_any;
- if (stable_node_dup == stable_node) {
+ /*
+ * If stable_node was a chain and chain_prune collapsed it,
+ * stable_node will be NULL here. In that case the
+ * stable_node_dup is the regular stable_node that has
+ * replaced the chain. If stable_node is not NULL and equal to
+ * stable_node_dup there was no chain and stable_node_dup is
+ * the regular stable_node in the stable rbtree.
+ */
+ if (!stable_node || stable_node_dup == stable_node) {
+ VM_BUG_ON(is_stable_node_chain(stable_node_dup));
+ VM_BUG_ON(is_stable_node_dup(stable_node_dup));
/* chain is missing so create it */
stable_node = alloc_stable_node_chain(stable_node_dup,
root);
@@ -1658,6 +1700,8 @@ static struct page *stable_tree_search(struct page *page)
* of the current nid for this page
* content.
*/
+ VM_BUG_ON(!is_stable_node_chain(stable_node));
+ VM_BUG_ON(!is_stable_node_dup(stable_node_dup));
VM_BUG_ON(page_node->head != &migrate_nodes);
list_del(&page_node->list);
DO_NUMA(page_node->nid = nid);
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* Re: [RFC] [PATCH 0/1] ksm: fix use after free with merge_across_nodes = 0
From: Andrew Morton @ 2017-05-12 20:37 UTC (permalink / raw)
To: Andrea Arcangeli
Cc: linux-mm, Evgheni Dereveanchin, Andrey Ryabinin, Petr Holasek,
Hugh Dickins, Arjan van de Ven, Davidlohr Bueso, Gavin Guo,
Jay Vosburgh, Mel Gorman
In-Reply-To: <20170512193805.8807-1-aarcange@redhat.com>
On Fri, 12 May 2017 21:38:04 +0200 Andrea Arcangeli <aarcange@redhat.com> wrote:
> The KSMscale patch in -mm (not yet upstream) is fundamental for
> enterprise use and in turn it's included in -mm, RHEL, CentoOS and
> Ubuntu and it'd be great if it could be merged upstream (especially
> after solving this problem with merge_across_nodes = 0 ...).
In April 2016 I folded two fixes against
ksm-introduce-ksm_max_page_sharing-per-page-deduplication-limit.patch
into that patch and disabled the patch from -mm. I think/assume
because it was causing crashes. The patch then got forgotten about.
I have resurrected it (below) and merged this patch after it.
From: Andrea Arcangeli <aarcange@redhat.com>
Subject: ksm: introduce ksm_max_page_sharing per page deduplication limit
Without a max deduplication limit for each KSM page, the list of the
rmap_items associated to each stable_node can grow infinitely large.
During the rmap walk each entry can take up to ~10usec to process because
of IPIs for the TLB flushing (both for the primary MMU and the secondary
MMUs with the MMU notifier). With only 16GB of address space shared in
the same KSM page, that would amount to dozens of seconds of kernel
runtime.
A ~256 max deduplication factor will reduce the latencies of the rmap
walks on KSM pages to order of a few msec. Just doing the cond_resched()
during the rmap walks is not enough, the list size must have a limit too,
otherwise the caller could get blocked in (schedule friendly) kernel
computations for seconds, unexpectedly.
There's room for optimization to significantly reduce the IPI delivery
cost during the page_referenced(), but at least for page_migration in the
KSM case (used by hard NUMA bindings, compaction and NUMA balancing) it
may be inevitable to send lots of IPIs if each rmap_item->mm is active on
a different CPU and there are lots of CPUs. Even if we ignore the IPI
delivery cost, we've still to walk the whole KSM rmap list, so we can't
allow millions or billions (ulimited) number of entries in the KSM
stable_node rmap_item lists.
The limit is enforced efficiently by adding a second dimension to the
stable rbtree. So there are three types of stable_nodes: the regular ones
(identical as before, living in the first flat dimension of the stable
rbtree), the "chains" and the "dups".
Every "chain" and all "dups" linked into a "chain" enforce the invariant
that they represent the same write protected memory content, even if each
"dup" will be pointed by a different KSM page copy of that content. This
way the stable rbtree lookup computational complexity is unaffected if
compared to an unlimited max_sharing_limit. It is still enforced that
there cannot be KSM page content duplicates in the stable rbtree itself.
Adding the second dimension to the stable rbtree only after the
max_page_sharing limit hits, provides for a zero memory footprint increase
on 64bit archs. The memory overhead of the per-KSM page stable_tree and
per virtual mapping rmap_item is unchanged. Only after the
max_page_sharing limit hits, we need to allocate a stable_tree "chain" and
rb_replace() the "regular" stable_node with the newly allocated
stable_node "chain". After that we simply add the "regular" stable_node
to the chain as a stable_node "dup" by linking hlist_dup in the
stable_node_chain->hlist. This way the "regular" (flat) stable_node is
converted to a stable_node "dup" living in the second dimension of the
stable rbtree.
During stable rbtree lookups the stable_node "chain" is identified as
stable_node->rmap_hlist_len == STABLE_NODE_CHAIN (aka
is_stable_node_chain()).
When dropping stable_nodes, the stable_node "dup" is identified as
stable_node->head == STABLE_NODE_DUP_HEAD (aka is_stable_node_dup()).
The STABLE_NODE_DUP_HEAD must be an unique valid pointer never used
elsewhere in any stable_node->head/node to avoid a clashes with the
stable_node->node.rb_parent_color pointer, and different from
&migrate_nodes. So the second field of &migrate_nodes is picked and
verified as always safe with a BUILD_BUG_ON in case the list_head
implementation changes in the future.
The STABLE_NODE_DUP is picked as a random negative value in
stable_node->rmap_hlist_len. rmap_hlist_len cannot become negative when
it's a "regular" stable_node or a stable_node "dup".
The stable_node_chain->nid is irrelevant. The stable_node_chain->kpfn is
aliased in a union with a time field used to rate limit the
stable_node_chain->hlist prunes.
The garbage collection of the stable_node_chain happens lazily during
stable rbtree lookups (as for all other kind of stable_nodes), or while
disabling KSM with "echo 2 >/sys/kernel/mm/ksm/run" while collecting the
entire stable rbtree.
While the "regular" stable_nodes and the stable_node "dups" must wait for
their underlying tree_page to be freed before they can be freed
themselves, the stable_node "chains" can be freed immediately if the
stable_node->hlist turns empty. This is because the "chains" are never
pointed by any page->mapping and they're effectively stable rbtree KSM
self contained metadata.
[akpm@linux-foundation.org: fix non-NUMA build]
Signed-off-by: Andrea Arcangeli <aarcange@redhat.com>
Tested-by: Petr Holasek <pholasek@redhat.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: Arjan van de Ven <arjan@linux.intel.com>
Cc: Evgheni Dereveanchin <ederevea@redhat.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Gavin Guo <gavin.guo@canonical.com>
Cc: Jay Vosburgh <jay.vosburgh@canonical.com>
Cc: Mel Gorman <mgorman@techsingularity.net>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
Documentation/vm/ksm.txt | 63 +++
mm/ksm.c | 733 +++++++++++++++++++++++++++++++++----
2 files changed, 730 insertions(+), 66 deletions(-)
diff -puN Documentation/vm/ksm.txt~ksm-introduce-ksm_max_page_sharing-per-page-deduplication-limit Documentation/vm/ksm.txt
--- a/Documentation/vm/ksm.txt~ksm-introduce-ksm_max_page_sharing-per-page-deduplication-limit
+++ a/Documentation/vm/ksm.txt
@@ -98,6 +98,50 @@ use_zero_pages - specifies whether emp
it is only effective for pages merged after the change.
Default: 0 (normal KSM behaviour as in earlier releases)
+max_page_sharing - Maximum sharing allowed for each KSM page. This
+ enforces a deduplication limit to avoid the virtual
+ memory rmap lists to grow too large. The minimum
+ value is 2 as a newly created KSM page will have at
+ least two sharers. The rmap walk has O(N)
+ complexity where N is the number of rmap_items
+ (i.e. virtual mappings) that are sharing the page,
+ which is in turn capped by max_page_sharing. So
+ this effectively spread the the linear O(N)
+ computational complexity from rmap walk context
+ over different KSM pages. The ksmd walk over the
+ stable_node "chains" is also O(N), but N is the
+ number of stable_node "dups", not the number of
+ rmap_items, so it has not a significant impact on
+ ksmd performance. In practice the best stable_node
+ "dup" candidate will be kept and found at the head
+ of the "dups" list. The higher this value the
+ faster KSM will merge the memory (because there
+ will be fewer stable_node dups queued into the
+ stable_node chain->hlist to check for pruning) and
+ the higher the deduplication factor will be, but
+ the slowest the worst case rmap walk could be for
+ any given KSM page. Slowing down the rmap_walk
+ means there will be higher latency for certain
+ virtual memory operations happening during
+ swapping, compaction, NUMA balancing and page
+ migration, in turn decreasing responsiveness for
+ the caller of those virtual memory operations. The
+ scheduler latency of other tasks not involved with
+ the VM operations doing the rmap walk is not
+ affected by this parameter as the rmap walks are
+ always schedule friendly themselves.
+
+stable_node_chains_prune_millisecs - How frequently to walk the whole
+ list of stable_node "dups" linked in the
+ stable_node "chains" in order to prune stale
+ stable_nodes. Smaller milllisecs values will free
+ up the KSM metadata with lower latency, but they
+ will make ksmd use more CPU during the scan. This
+ only applies to the stable_node chains so it's a
+ noop if not a single KSM page hit the
+ max_page_sharing yet (there would be no stable_node
+ chains in such case).
+
The effectiveness of KSM and MADV_MERGEABLE is shown in /sys/kernel/mm/ksm/:
pages_shared - how many shared pages are being used
@@ -106,10 +150,29 @@ pages_unshared - how many pages unique
pages_volatile - how many pages changing too fast to be placed in a tree
full_scans - how many times all mergeable areas have been scanned
+stable_node_chains - number of stable node chains allocated, this is
+ effectively the number of KSM pages that hit the
+ max_page_sharing limit
+stable_node_dups - number of stable node dups queued into the
+ stable_node chains
+
A high ratio of pages_sharing to pages_shared indicates good sharing, but
a high ratio of pages_unshared to pages_sharing indicates wasted effort.
pages_volatile embraces several different kinds of activity, but a high
proportion there would also indicate poor use of madvise MADV_MERGEABLE.
+The maximum possible page_sharing/page_shared ratio is limited by the
+max_page_sharing tunable. To increase the ratio max_page_sharing must
+be increased accordingly.
+
+The stable_node_dups/stable_node_chains ratio is also affected by the
+max_page_sharing tunable, and an high ratio may indicate fragmentation
+in the stable_node dups, which could be solved by introducing
+fragmentation algorithms in ksmd which would refile rmap_items from
+one stable_node dup to another stable_node dup, in order to freeup
+stable_node "dups" with few rmap_items in them, but that may increase
+the ksmd CPU usage and possibly slowdown the readonly computations on
+the KSM pages of the applications.
+
Izik Eidus,
Hugh Dickins, 17 Nov 2009
diff -puN mm/ksm.c~ksm-introduce-ksm_max_page_sharing-per-page-deduplication-limit mm/ksm.c
--- a/mm/ksm.c~ksm-introduce-ksm_max_page_sharing-per-page-deduplication-limit
+++ a/mm/ksm.c
@@ -128,9 +128,12 @@ struct ksm_scan {
* struct stable_node - node of the stable rbtree
* @node: rb node of this ksm page in the stable tree
* @head: (overlaying parent) &migrate_nodes indicates temporarily on that list
+ * @hlist_dup: linked into the stable_node->hlist with a stable_node chain
* @list: linked into migrate_nodes, pending placement in the proper node tree
* @hlist: hlist head of rmap_items using this ksm page
* @kpfn: page frame number of this ksm page (perhaps temporarily on wrong nid)
+ * @chain_prune_time: time of the last full garbage collection
+ * @rmap_hlist_len: number of rmap_item entries in hlist or STABLE_NODE_CHAIN
* @nid: NUMA node id of stable tree in which linked (may not match kpfn)
*/
struct stable_node {
@@ -138,11 +141,24 @@ struct stable_node {
struct rb_node node; /* when node of stable tree */
struct { /* when listed for migration */
struct list_head *head;
- struct list_head list;
+ struct {
+ struct hlist_node hlist_dup;
+ struct list_head list;
+ };
};
};
struct hlist_head hlist;
- unsigned long kpfn;
+ union {
+ unsigned long kpfn;
+ unsigned long chain_prune_time;
+ };
+ /*
+ * STABLE_NODE_CHAIN can be any negative number in
+ * rmap_hlist_len negative range, but better not -1 to be able
+ * to reliably detect underflows.
+ */
+#define STABLE_NODE_CHAIN -1024
+ int rmap_hlist_len;
#ifdef CONFIG_NUMA
int nid;
#endif
@@ -192,6 +208,7 @@ static struct rb_root *root_unstable_tre
/* Recently migrated nodes of stable tree, pending proper placement */
static LIST_HEAD(migrate_nodes);
+#define STABLE_NODE_DUP_HEAD ((struct list_head *)&migrate_nodes.prev)
#define MM_SLOTS_HASH_BITS 10
static DEFINE_HASHTABLE(mm_slots_hash, MM_SLOTS_HASH_BITS);
@@ -219,6 +236,18 @@ static unsigned long ksm_pages_unshared;
/* The number of rmap_items in use: to calculate pages_volatile */
static unsigned long ksm_rmap_items;
+/* The number of stable_node chains */
+static unsigned long ksm_stable_node_chains;
+
+/* The number of stable_node dups linked to the stable_node chains */
+static unsigned long ksm_stable_node_dups;
+
+/* Delay in pruning stale stable_node_dups in the stable_node_chains */
+static int ksm_stable_node_chains_prune_millisecs = 2000;
+
+/* Maximum number of page slots sharing a stable node */
+static int ksm_max_page_sharing = 256;
+
/* Number of pages ksmd should scan in one batch */
static unsigned int ksm_thread_pages_to_scan = 100;
@@ -287,6 +316,44 @@ static void __init ksm_slab_free(void)
mm_slot_cache = NULL;
}
+static __always_inline bool is_stable_node_chain(struct stable_node *chain)
+{
+ return chain->rmap_hlist_len == STABLE_NODE_CHAIN;
+}
+
+static __always_inline bool is_stable_node_dup(struct stable_node *dup)
+{
+ return dup->head == STABLE_NODE_DUP_HEAD;
+}
+
+static inline void stable_node_chain_add_dup(struct stable_node *dup,
+ struct stable_node *chain)
+{
+ VM_BUG_ON(is_stable_node_dup(dup));
+ dup->head = STABLE_NODE_DUP_HEAD;
+ VM_BUG_ON(!is_stable_node_chain(chain));
+ hlist_add_head(&dup->hlist_dup, &chain->hlist);
+ ksm_stable_node_dups++;
+}
+
+static inline void __stable_node_dup_del(struct stable_node *dup)
+{
+ hlist_del(&dup->hlist_dup);
+ ksm_stable_node_dups--;
+}
+
+static inline void stable_node_dup_del(struct stable_node *dup)
+{
+ VM_BUG_ON(is_stable_node_chain(dup));
+ if (is_stable_node_dup(dup))
+ __stable_node_dup_del(dup);
+ else
+ rb_erase(&dup->node, root_stable_tree + NUMA(dup->nid));
+#ifdef CONFIG_DEBUG_VM
+ dup->head = NULL;
+#endif
+}
+
static inline struct rmap_item *alloc_rmap_item(void)
{
struct rmap_item *rmap_item;
@@ -317,6 +384,8 @@ static inline struct stable_node *alloc_
static inline void free_stable_node(struct stable_node *stable_node)
{
+ VM_BUG_ON(stable_node->rmap_hlist_len &&
+ !is_stable_node_chain(stable_node));
kmem_cache_free(stable_node_cache, stable_node);
}
@@ -498,25 +567,82 @@ static inline int get_kpfn_nid(unsigned
return ksm_merge_across_nodes ? 0 : NUMA(pfn_to_nid(kpfn));
}
+static struct stable_node *alloc_stable_node_chain(struct stable_node *dup,
+ struct rb_root *root)
+{
+ struct stable_node *chain = alloc_stable_node();
+ VM_BUG_ON(is_stable_node_chain(dup));
+ if (likely(chain)) {
+ INIT_HLIST_HEAD(&chain->hlist);
+ chain->chain_prune_time = jiffies;
+ chain->rmap_hlist_len = STABLE_NODE_CHAIN;
+#if defined (CONFIG_DEBUG_VM) && defined(CONFIG_NUMA)
+ chain->nid = -1; /* debug */
+#endif
+ ksm_stable_node_chains++;
+
+ /*
+ * Put the stable node chain in the first dimension of
+ * the stable tree and at the same time remove the old
+ * stable node.
+ */
+ rb_replace_node(&dup->node, &chain->node, root);
+
+ /*
+ * Move the old stable node to the second dimension
+ * queued in the hlist_dup. The invariant is that all
+ * dup stable_nodes in the chain->hlist point to pages
+ * that are wrprotected and have the exact same
+ * content.
+ */
+ stable_node_chain_add_dup(dup, chain);
+ }
+ return chain;
+}
+
+static inline void free_stable_node_chain(struct stable_node *chain,
+ struct rb_root *root)
+{
+ rb_erase(&chain->node, root);
+ free_stable_node(chain);
+ ksm_stable_node_chains--;
+}
+
static void remove_node_from_stable_tree(struct stable_node *stable_node)
{
struct rmap_item *rmap_item;
+ /* check it's not STABLE_NODE_CHAIN or negative */
+ BUG_ON(stable_node->rmap_hlist_len < 0);
+
hlist_for_each_entry(rmap_item, &stable_node->hlist, hlist) {
if (rmap_item->hlist.next)
ksm_pages_sharing--;
else
ksm_pages_shared--;
+ VM_BUG_ON(stable_node->rmap_hlist_len <= 0);
+ stable_node->rmap_hlist_len--;
put_anon_vma(rmap_item->anon_vma);
rmap_item->address &= PAGE_MASK;
cond_resched();
}
+ /*
+ * We need the second aligned pointer of the migrate_nodes
+ * list_head to stay clear from the rb_parent_color union
+ * (aligned and different than any node) and also different
+ * from &migrate_nodes. This will verify that future list.h changes
+ * don't break STABLE_NODE_DUP_HEAD.
+ */
+#if GCC_VERSION >= 40903 /* only recent gcc can handle it */
+ BUILD_BUG_ON(STABLE_NODE_DUP_HEAD <= &migrate_nodes);
+ BUILD_BUG_ON(STABLE_NODE_DUP_HEAD >= &migrate_nodes + 1);
+#endif
+
if (stable_node->head == &migrate_nodes)
list_del(&stable_node->list);
else
- rb_erase(&stable_node->node,
- root_stable_tree + NUMA(stable_node->nid));
+ stable_node_dup_del(stable_node);
free_stable_node(stable_node);
}
@@ -635,6 +761,8 @@ static void remove_rmap_item_from_tree(s
ksm_pages_sharing--;
else
ksm_pages_shared--;
+ VM_BUG_ON(stable_node->rmap_hlist_len <= 0);
+ stable_node->rmap_hlist_len--;
put_anon_vma(rmap_item->anon_vma);
rmap_item->address &= PAGE_MASK;
@@ -743,6 +871,31 @@ static int remove_stable_node(struct sta
return err;
}
+static int remove_stable_node_chain(struct stable_node *stable_node,
+ struct rb_root *root)
+{
+ struct stable_node *dup;
+ struct hlist_node *hlist_safe;
+
+ if (!is_stable_node_chain(stable_node)) {
+ VM_BUG_ON(is_stable_node_dup(stable_node));
+ if (remove_stable_node(stable_node))
+ return true;
+ else
+ return false;
+ }
+
+ hlist_for_each_entry_safe(dup, hlist_safe,
+ &stable_node->hlist, hlist_dup) {
+ VM_BUG_ON(!is_stable_node_dup(dup));
+ if (remove_stable_node(dup))
+ return true;
+ }
+ BUG_ON(!hlist_empty(&stable_node->hlist));
+ free_stable_node_chain(stable_node, root);
+ return false;
+}
+
static int remove_all_stable_nodes(void)
{
struct stable_node *stable_node, *next;
@@ -753,7 +906,8 @@ static int remove_all_stable_nodes(void)
while (root_stable_tree[nid].rb_node) {
stable_node = rb_entry(root_stable_tree[nid].rb_node,
struct stable_node, node);
- if (remove_stable_node(stable_node)) {
+ if (remove_stable_node_chain(stable_node,
+ root_stable_tree + nid)) {
err = -EBUSY;
break; /* proceed to next nid */
}
@@ -1139,6 +1293,163 @@ static struct page *try_to_merge_two_pag
return err ? NULL : page;
}
+static __always_inline
+bool __is_page_sharing_candidate(struct stable_node *stable_node, int offset)
+{
+ VM_BUG_ON(stable_node->rmap_hlist_len < 0);
+ /*
+ * Check that at least one mapping still exists, otherwise
+ * there's no much point to merge and share with this
+ * stable_node, as the underlying tree_page of the other
+ * sharer is going to be freed soon.
+ */
+ return stable_node->rmap_hlist_len &&
+ stable_node->rmap_hlist_len + offset < ksm_max_page_sharing;
+}
+
+static __always_inline
+bool is_page_sharing_candidate(struct stable_node *stable_node)
+{
+ return __is_page_sharing_candidate(stable_node, 0);
+}
+
+static struct stable_node *stable_node_dup(struct stable_node *stable_node,
+ struct page **tree_page,
+ struct rb_root *root,
+ bool prune_stale_stable_nodes)
+{
+ struct stable_node *dup, *found = NULL;
+ struct hlist_node *hlist_safe;
+ struct page *_tree_page;
+ int nr = 0;
+ int found_rmap_hlist_len;
+
+ if (!prune_stale_stable_nodes ||
+ time_before(jiffies, stable_node->chain_prune_time +
+ msecs_to_jiffies(
+ ksm_stable_node_chains_prune_millisecs)))
+ prune_stale_stable_nodes = false;
+ else
+ stable_node->chain_prune_time = jiffies;
+
+ hlist_for_each_entry_safe(dup, hlist_safe,
+ &stable_node->hlist, hlist_dup) {
+ cond_resched();
+ /*
+ * We must walk all stable_node_dup to prune the stale
+ * stable nodes during lookup.
+ *
+ * get_ksm_page can drop the nodes from the
+ * stable_node->hlist if they point to freed pages
+ * (that's why we do a _safe walk). The "dup"
+ * stable_node parameter itself will be freed from
+ * under us if it returns NULL.
+ */
+ _tree_page = get_ksm_page(dup, false);
+ if (!_tree_page)
+ continue;
+ nr += 1;
+ if (is_page_sharing_candidate(dup)) {
+ if (!found ||
+ dup->rmap_hlist_len > found_rmap_hlist_len) {
+ if (found)
+ put_page(*tree_page);
+ found = dup;
+ found_rmap_hlist_len = found->rmap_hlist_len;
+ *tree_page = _tree_page;
+
+ if (!prune_stale_stable_nodes)
+ break;
+ /* skip put_page */
+ continue;
+ }
+ }
+ put_page(_tree_page);
+ }
+
+ /*
+ * nr is relevant only if prune_stale_stable_nodes is true,
+ * otherwise we may break the loop at nr == 1 even if there
+ * are multiple entries.
+ */
+ if (prune_stale_stable_nodes && found) {
+ if (nr == 1) {
+ /*
+ * If there's not just one entry it would
+ * corrupt memory, better BUG_ON. In KSM
+ * context with no lock held it's not even
+ * fatal.
+ */
+ BUG_ON(stable_node->hlist.first->next);
+
+ /*
+ * There's just one entry and it is below the
+ * deduplication limit so drop the chain.
+ */
+ rb_replace_node(&stable_node->node, &found->node,
+ root);
+ free_stable_node(stable_node);
+ ksm_stable_node_chains--;
+ ksm_stable_node_dups--;
+ } else if (__is_page_sharing_candidate(found, 1)) {
+ /*
+ * Refile our candidate at the head
+ * after the prune if our candidate
+ * can accept one more future sharing
+ * in addition to the one underway.
+ */
+ hlist_del(&found->hlist_dup);
+ hlist_add_head(&found->hlist_dup,
+ &stable_node->hlist);
+ }
+ }
+
+ return found;
+}
+
+static struct stable_node *stable_node_dup_any(struct stable_node *stable_node,
+ struct rb_root *root)
+{
+ if (!is_stable_node_chain(stable_node))
+ return stable_node;
+ if (hlist_empty(&stable_node->hlist)) {
+ free_stable_node_chain(stable_node, root);
+ return NULL;
+ }
+ return hlist_entry(stable_node->hlist.first,
+ typeof(*stable_node), hlist_dup);
+}
+
+static struct stable_node *__stable_node_chain(struct stable_node *stable_node,
+ struct page **tree_page,
+ struct rb_root *root,
+ bool prune_stale_stable_nodes)
+{
+ if (!is_stable_node_chain(stable_node)) {
+ if (is_page_sharing_candidate(stable_node)) {
+ *tree_page = get_ksm_page(stable_node, false);
+ return stable_node;
+ }
+ return NULL;
+ }
+ return stable_node_dup(stable_node, tree_page, root,
+ prune_stale_stable_nodes);
+}
+
+static __always_inline struct stable_node *chain_prune(struct stable_node *s_n,
+ struct page **t_p,
+ struct rb_root *root)
+{
+ return __stable_node_chain(s_n, t_p, root, true);
+}
+
+static __always_inline struct stable_node *chain(struct stable_node *s_n,
+ struct page **t_p,
+ struct rb_root *root)
+{
+ return __stable_node_chain(s_n, t_p, root, false);
+}
+
/*
* stable_tree_search - search for page inside the stable tree
*
@@ -1154,7 +1465,7 @@ static struct page *stable_tree_search(s
struct rb_root *root;
struct rb_node **new;
struct rb_node *parent;
- struct stable_node *stable_node;
+ struct stable_node *stable_node, *stable_node_dup, *stable_node_any;
struct stable_node *page_node;
page_node = page_stable_node(page);
@@ -1176,7 +1487,32 @@ again:
cond_resched();
stable_node = rb_entry(*new, struct stable_node, node);
- tree_page = get_ksm_page(stable_node, false);
+ stable_node_any = NULL;
+ stable_node_dup = chain_prune(stable_node, &tree_page, root);
+ if (!stable_node_dup) {
+ /*
+ * Either all stable_node dups were full in
+ * this stable_node chain, or this chain was
+ * empty and should be rb_erased.
+ */
+ stable_node_any = stable_node_dup_any(stable_node,
+ root);
+ if (!stable_node_any) {
+ /* rb_erase just run */
+ goto again;
+ }
+ /*
+ * Take any of the stable_node dups page of
+ * this stable_node chain to let the tree walk
+ * continue. All KSM pages belonging to the
+ * stable_node dups in a stable_node chain
+ * have the same content and they're
+ * wrprotected at all times. Any will work
+ * fine to continue the walk.
+ */
+ tree_page = get_ksm_page(stable_node_any, false);
+ }
+ VM_BUG_ON(!stable_node_dup ^ !!stable_node_any);
if (!tree_page) {
/*
* If we walked over a stale stable_node,
@@ -1199,6 +1535,34 @@ again:
else if (ret > 0)
new = &parent->rb_right;
else {
+ if (page_node) {
+ VM_BUG_ON(page_node->head != &migrate_nodes);
+ /*
+ * Test if the migrated page should be merged
+ * into a stable node dup. If the mapcount is
+ * 1 we can migrate it with another KSM page
+ * without adding it to the chain.
+ */
+ if (page_mapcount(page) > 1)
+ goto chain_append;
+ }
+
+ if (!stable_node_dup) {
+ /*
+ * If the stable_node is a chain and
+ * we got a payload match in memcmp
+ * but we cannot merge the scanned
+ * page in any of the existing
+ * stable_node dups because they're
+ * all full, we need to wait the
+ * scanned page to find itself a match
+ * in the unstable tree to create a
+ * brand new KSM page to add later to
+ * the dups of this stable_node.
+ */
+ return NULL;
+ }
+
/*
* Lock and unlock the stable_node's page (which
* might already have been migrated) so that page
@@ -1206,23 +1570,21 @@ again:
* It would be more elegant to return stable_node
* than kpage, but that involves more changes.
*/
- tree_page = get_ksm_page(stable_node, true);
- if (tree_page) {
- unlock_page(tree_page);
- if (get_kpfn_nid(stable_node->kpfn) !=
- NUMA(stable_node->nid)) {
- put_page(tree_page);
- goto replace;
- }
- return tree_page;
- }
- /*
- * There is now a place for page_node, but the tree may
- * have been rebalanced, so re-evaluate parent and new.
- */
- if (page_node)
+ tree_page = get_ksm_page(stable_node_dup, true);
+ if (unlikely(!tree_page))
+ /*
+ * The tree may have been rebalanced,
+ * so re-evaluate parent and new.
+ */
goto again;
- return NULL;
+ unlock_page(tree_page);
+
+ if (get_kpfn_nid(stable_node_dup->kpfn) !=
+ NUMA(stable_node_dup->nid)) {
+ put_page(tree_page);
+ goto replace;
+ }
+ return tree_page;
}
}
@@ -1233,22 +1595,72 @@ again:
DO_NUMA(page_node->nid = nid);
rb_link_node(&page_node->node, parent, new);
rb_insert_color(&page_node->node, root);
- get_page(page);
- return page;
+out:
+ if (is_page_sharing_candidate(page_node)) {
+ get_page(page);
+ return page;
+ } else
+ return NULL;
replace:
- if (page_node) {
- list_del(&page_node->list);
- DO_NUMA(page_node->nid = nid);
- rb_replace_node(&stable_node->node, &page_node->node, root);
- get_page(page);
+ if (stable_node_dup == stable_node) {
+ /* there is no chain */
+ if (page_node) {
+ VM_BUG_ON(page_node->head != &migrate_nodes);
+ list_del(&page_node->list);
+ DO_NUMA(page_node->nid = nid);
+ rb_replace_node(&stable_node->node, &page_node->node,
+ root);
+ if (is_page_sharing_candidate(page_node))
+ get_page(page);
+ else
+ page = NULL;
+ } else {
+ rb_erase(&stable_node->node, root);
+ page = NULL;
+ }
} else {
- rb_erase(&stable_node->node, root);
- page = NULL;
+ VM_BUG_ON(!is_stable_node_chain(stable_node));
+ __stable_node_dup_del(stable_node_dup);
+ if (page_node) {
+ VM_BUG_ON(page_node->head != &migrate_nodes);
+ list_del(&page_node->list);
+ DO_NUMA(page_node->nid = nid);
+ stable_node_chain_add_dup(page_node, stable_node);
+ if (is_page_sharing_candidate(page_node))
+ get_page(page);
+ else
+ page = NULL;
+ } else {
+ page = NULL;
+ }
}
- stable_node->head = &migrate_nodes;
- list_add(&stable_node->list, stable_node->head);
+ stable_node_dup->head = &migrate_nodes;
+ list_add(&stable_node_dup->list, stable_node_dup->head);
return page;
+
+chain_append:
+ /* stable_node_dup could be null if it reached the limit */
+ if (!stable_node_dup)
+ stable_node_dup = stable_node_any;
+ if (stable_node_dup == stable_node) {
+ /* chain is missing so create it */
+ stable_node = alloc_stable_node_chain(stable_node_dup,
+ root);
+ if (!stable_node)
+ return NULL;
+ }
+ /*
+ * Add this stable_node dup that was
+ * migrated to the stable_node chain
+ * of the current nid for this page
+ * content.
+ */
+ VM_BUG_ON(page_node->head != &migrate_nodes);
+ list_del(&page_node->list);
+ DO_NUMA(page_node->nid = nid);
+ stable_node_chain_add_dup(page_node, stable_node);
+ goto out;
}
/*
@@ -1265,7 +1677,8 @@ static struct stable_node *stable_tree_i
struct rb_root *root;
struct rb_node **new;
struct rb_node *parent;
- struct stable_node *stable_node;
+ struct stable_node *stable_node, *stable_node_dup, *stable_node_any;
+ bool need_chain = false;
kpfn = page_to_pfn(kpage);
nid = get_kpfn_nid(kpfn);
@@ -1280,7 +1693,32 @@ again:
cond_resched();
stable_node = rb_entry(*new, struct stable_node, node);
- tree_page = get_ksm_page(stable_node, false);
+ stable_node_any = NULL;
+ stable_node_dup = chain(stable_node, &tree_page, root);
+ if (!stable_node_dup) {
+ /*
+ * Either all stable_node dups were full in
+ * this stable_node chain, or this chain was
+ * empty and should be rb_erased.
+ */
+ stable_node_any = stable_node_dup_any(stable_node,
+ root);
+ if (!stable_node_any) {
+ /* rb_erase just run */
+ goto again;
+ }
+ /*
+ * Take any of the stable_node dups page of
+ * this stable_node chain to let the tree walk
+ * continue. All KSM pages belonging to the
+ * stable_node dups in a stable_node chain
+ * have the same content and they're
+ * wrprotected at all times. Any will work
+ * fine to continue the walk.
+ */
+ tree_page = get_ksm_page(stable_node_any, false);
+ }
+ VM_BUG_ON(!stable_node_dup ^ !!stable_node_any);
if (!tree_page) {
/*
* If we walked over a stale stable_node,
@@ -1303,27 +1741,37 @@ again:
else if (ret > 0)
new = &parent->rb_right;
else {
- /*
- * It is not a bug that stable_tree_search() didn't
- * find this node: because at that time our page was
- * not yet write-protected, so may have changed since.
- */
- return NULL;
+ need_chain = true;
+ break;
}
}
- stable_node = alloc_stable_node();
- if (!stable_node)
+ stable_node_dup = alloc_stable_node();
+ if (!stable_node_dup)
return NULL;
- INIT_HLIST_HEAD(&stable_node->hlist);
- stable_node->kpfn = kpfn;
- set_page_stable_node(kpage, stable_node);
- DO_NUMA(stable_node->nid = nid);
- rb_link_node(&stable_node->node, parent, new);
- rb_insert_color(&stable_node->node, root);
+ INIT_HLIST_HEAD(&stable_node_dup->hlist);
+ stable_node_dup->kpfn = kpfn;
+ set_page_stable_node(kpage, stable_node_dup);
+ stable_node_dup->rmap_hlist_len = 0;
+ DO_NUMA(stable_node_dup->nid = nid);
+ if (!need_chain) {
+ rb_link_node(&stable_node_dup->node, parent, new);
+ rb_insert_color(&stable_node_dup->node, root);
+ } else {
+ if (!is_stable_node_chain(stable_node)) {
+ struct stable_node *orig = stable_node;
+ /* chain is missing so create it */
+ stable_node = alloc_stable_node_chain(orig, root);
+ if (!stable_node) {
+ free_stable_node(stable_node_dup);
+ return NULL;
+ }
+ }
+ stable_node_chain_add_dup(stable_node_dup, stable_node);
+ }
- return stable_node;
+ return stable_node_dup;
}
/*
@@ -1413,8 +1861,27 @@ struct rmap_item *unstable_tree_search_i
* the same ksm page.
*/
static void stable_tree_append(struct rmap_item *rmap_item,
- struct stable_node *stable_node)
+ struct stable_node *stable_node,
+ bool max_page_sharing_bypass)
{
+ /*
+ * rmap won't find this mapping if we don't insert the
+ * rmap_item in the right stable_node
+ * duplicate. page_migration could break later if rmap breaks,
+ * so we can as well crash here. We really need to check for
+ * rmap_hlist_len == STABLE_NODE_CHAIN, but we can as well check
+ * for other negative values as an undeflow if detected here
+ * for the first time (and not when decreasing rmap_hlist_len)
+ * would be sign of memory corruption in the stable_node.
+ */
+ BUG_ON(stable_node->rmap_hlist_len < 0);
+
+ stable_node->rmap_hlist_len++;
+ if (!max_page_sharing_bypass)
+ /* possibly non fatal but unexpected overflow, only warn */
+ WARN_ON_ONCE(stable_node->rmap_hlist_len >
+ ksm_max_page_sharing);
+
rmap_item->head = stable_node;
rmap_item->address |= STABLE_FLAG;
hlist_add_head(&rmap_item->hlist, &stable_node->hlist);
@@ -1442,19 +1909,26 @@ static void cmp_and_merge_page(struct pa
struct page *kpage;
unsigned int checksum;
int err;
+ bool max_page_sharing_bypass = false;
stable_node = page_stable_node(page);
if (stable_node) {
if (stable_node->head != &migrate_nodes &&
- get_kpfn_nid(stable_node->kpfn) != NUMA(stable_node->nid)) {
- rb_erase(&stable_node->node,
- root_stable_tree + NUMA(stable_node->nid));
+ get_kpfn_nid(READ_ONCE(stable_node->kpfn)) !=
+ NUMA(stable_node->nid)) {
+ stable_node_dup_del(stable_node);
stable_node->head = &migrate_nodes;
list_add(&stable_node->list, stable_node->head);
}
if (stable_node->head != &migrate_nodes &&
rmap_item->head == stable_node)
return;
+ /*
+ * If it's a KSM fork, allow it to go over the sharing limit
+ * without warnings.
+ */
+ if (!is_page_sharing_candidate(stable_node))
+ max_page_sharing_bypass = true;
}
/* We first start with searching the page inside the stable tree */
@@ -1474,7 +1948,8 @@ static void cmp_and_merge_page(struct pa
* add its rmap_item to the stable tree.
*/
lock_page(kpage);
- stable_tree_append(rmap_item, page_stable_node(kpage));
+ stable_tree_append(rmap_item, page_stable_node(kpage),
+ max_page_sharing_bypass);
unlock_page(kpage);
}
put_page(kpage);
@@ -1524,8 +1999,10 @@ static void cmp_and_merge_page(struct pa
lock_page(kpage);
stable_node = stable_tree_insert(kpage);
if (stable_node) {
- stable_tree_append(tree_rmap_item, stable_node);
- stable_tree_append(rmap_item, stable_node);
+ stable_tree_append(tree_rmap_item, stable_node,
+ false);
+ stable_tree_append(rmap_item, stable_node,
+ false);
}
unlock_page(kpage);
@@ -2029,6 +2506,48 @@ static void wait_while_offlining(void)
}
}
+static bool stable_node_dup_remove_range(struct stable_node *stable_node,
+ unsigned long start_pfn,
+ unsigned long end_pfn)
+{
+ if (stable_node->kpfn >= start_pfn &&
+ stable_node->kpfn < end_pfn) {
+ /*
+ * Don't get_ksm_page, page has already gone:
+ * which is why we keep kpfn instead of page*
+ */
+ remove_node_from_stable_tree(stable_node);
+ return true;
+ }
+ return false;
+}
+
+static bool stable_node_chain_remove_range(struct stable_node *stable_node,
+ unsigned long start_pfn,
+ unsigned long end_pfn,
+ struct rb_root *root)
+{
+ struct stable_node *dup;
+ struct hlist_node *hlist_safe;
+
+ if (!is_stable_node_chain(stable_node)) {
+ VM_BUG_ON(is_stable_node_dup(stable_node));
+ return stable_node_dup_remove_range(stable_node, start_pfn,
+ end_pfn);
+ }
+
+ hlist_for_each_entry_safe(dup, hlist_safe,
+ &stable_node->hlist, hlist_dup) {
+ VM_BUG_ON(!is_stable_node_dup(dup));
+ stable_node_dup_remove_range(dup, start_pfn, end_pfn);
+ }
+ if (hlist_empty(&stable_node->hlist)) {
+ free_stable_node_chain(stable_node, root);
+ return true; /* notify caller that tree was rebalanced */
+ } else
+ return false;
+}
+
static void ksm_check_stable_tree(unsigned long start_pfn,
unsigned long end_pfn)
{
@@ -2040,15 +2559,12 @@ static void ksm_check_stable_tree(unsign
node = rb_first(root_stable_tree + nid);
while (node) {
stable_node = rb_entry(node, struct stable_node, node);
- if (stable_node->kpfn >= start_pfn &&
- stable_node->kpfn < end_pfn) {
- /*
- * Don't get_ksm_page, page has already gone:
- * which is why we keep kpfn instead of page*
- */
- remove_node_from_stable_tree(stable_node);
+ if (stable_node_chain_remove_range(stable_node,
+ start_pfn, end_pfn,
+ root_stable_tree +
+ nid))
node = rb_first(root_stable_tree + nid);
- } else
+ else
node = rb_next(node);
cond_resched();
}
@@ -2294,6 +2810,47 @@ static ssize_t use_zero_pages_store(stru
}
KSM_ATTR(use_zero_pages);
+static ssize_t max_page_sharing_show(struct kobject *kobj,
+ struct kobj_attribute *attr, char *buf)
+{
+ return sprintf(buf, "%u\n", ksm_max_page_sharing);
+}
+
+static ssize_t max_page_sharing_store(struct kobject *kobj,
+ struct kobj_attribute *attr,
+ const char *buf, size_t count)
+{
+ int err;
+ int knob;
+
+ err = kstrtoint(buf, 10, &knob);
+ if (err)
+ return err;
+ /*
+ * When a KSM page is created it is shared by 2 mappings. This
+ * being a signed comparison, it implicitly verifies it's not
+ * negative.
+ */
+ if (knob < 2)
+ return -EINVAL;
+
+ if (READ_ONCE(ksm_max_page_sharing) == knob)
+ return count;
+
+ mutex_lock(&ksm_thread_mutex);
+ wait_while_offlining();
+ if (ksm_max_page_sharing != knob) {
+ if (ksm_pages_shared || remove_all_stable_nodes())
+ err = -EBUSY;
+ else
+ ksm_max_page_sharing = knob;
+ }
+ mutex_unlock(&ksm_thread_mutex);
+
+ return err ? err : count;
+}
+KSM_ATTR(max_page_sharing);
+
static ssize_t pages_shared_show(struct kobject *kobj,
struct kobj_attribute *attr, char *buf)
{
@@ -2332,6 +2889,46 @@ static ssize_t pages_volatile_show(struc
}
KSM_ATTR_RO(pages_volatile);
+static ssize_t stable_node_dups_show(struct kobject *kobj,
+ struct kobj_attribute *attr, char *buf)
+{
+ return sprintf(buf, "%lu\n", ksm_stable_node_dups);
+}
+KSM_ATTR_RO(stable_node_dups);
+
+static ssize_t stable_node_chains_show(struct kobject *kobj,
+ struct kobj_attribute *attr, char *buf)
+{
+ return sprintf(buf, "%lu\n", ksm_stable_node_chains);
+}
+KSM_ATTR_RO(stable_node_chains);
+
+static ssize_t
+stable_node_chains_prune_millisecs_show(struct kobject *kobj,
+ struct kobj_attribute *attr,
+ char *buf)
+{
+ return sprintf(buf, "%u\n", ksm_stable_node_chains_prune_millisecs);
+}
+
+static ssize_t
+stable_node_chains_prune_millisecs_store(struct kobject *kobj,
+ struct kobj_attribute *attr,
+ const char *buf, size_t count)
+{
+ unsigned long msecs;
+ int err;
+
+ err = kstrtoul(buf, 10, &msecs);
+ if (err || msecs > UINT_MAX)
+ return -EINVAL;
+
+ ksm_stable_node_chains_prune_millisecs = msecs;
+
+ return count;
+}
+KSM_ATTR(stable_node_chains_prune_millisecs);
+
static ssize_t full_scans_show(struct kobject *kobj,
struct kobj_attribute *attr, char *buf)
{
@@ -2351,6 +2948,10 @@ static struct attribute *ksm_attrs[] = {
#ifdef CONFIG_NUMA
&merge_across_nodes_attr.attr,
#endif
+ &max_page_sharing_attr.attr,
+ &stable_node_chains_attr.attr,
+ &stable_node_dups_attr.attr,
+ &stable_node_chains_prune_millisecs_attr.attr,
&use_zero_pages_attr.attr,
NULL,
};
_
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH V2] mm/madvise: Enable (soft|hard) offline of HugeTLB pages at PGD level
From: Andrew Morton @ 2017-05-12 21:35 UTC (permalink / raw)
To: Anshuman Khandual; +Cc: linux-kernel, linux-mm, n-horiguchi, aneesh.kumar
In-Reply-To: <20170426035731.6924-1-khandual@linux.vnet.ibm.com>
On Wed, 26 Apr 2017 09:27:31 +0530 Anshuman Khandual <khandual@linux.vnet.ibm.com> wrote:
> Though migrating gigantic HugeTLB pages does not sound much like real
> world use case, they can be affected by memory errors. Hence migration
> at the PGD level HugeTLB pages should be supported just to enable soft
> and hard offline use cases.
>
> While allocating the new gigantic HugeTLB page, it should not matter
> whether new page comes from the same node or not. There would be very
> few gigantic pages on the system afterall, we should not be bothered
> about node locality when trying to save a big page from crashing.
>
> This introduces a new HugeTLB allocator called alloc_huge_page_nonid()
> which will scan over all online nodes on the system and allocate a
> single HugeTLB page.
>
> ...
>
> --- a/mm/hugetlb.c
> +++ b/mm/hugetlb.c
> @@ -1669,6 +1669,23 @@ struct page *__alloc_buddy_huge_page_with_mpol(struct hstate *h,
> return __alloc_buddy_huge_page(h, vma, addr, NUMA_NO_NODE);
> }
>
> +struct page *alloc_huge_page_nonid(struct hstate *h)
> +{
> + struct page *page = NULL;
> + int nid = 0;
> +
> + spin_lock(&hugetlb_lock);
> + if (h->free_huge_pages - h->resv_huge_pages > 0) {
> + for_each_online_node(nid) {
> + page = dequeue_huge_page_node(h, nid);
> + if (page)
> + break;
> + }
> + }
> + spin_unlock(&hugetlb_lock);
> + return page;
> +}
> +
> /*
> * This allocation function is useful in the context where vma is irrelevant.
> * E.g. soft-offlining uses this function because it only cares physical
> diff --git a/mm/memory-failure.c b/mm/memory-failure.c
> index fe64d7729a8e..d4f5710cf3f7 100644
> --- a/mm/memory-failure.c
> +++ b/mm/memory-failure.c
> @@ -1481,11 +1481,15 @@ EXPORT_SYMBOL(unpoison_memory);
> static struct page *new_page(struct page *p, unsigned long private, int **x)
> {
> int nid = page_to_nid(p);
> - if (PageHuge(p))
> + if (PageHuge(p)) {
> + if (hstate_is_gigantic(page_hstate(compound_head(p))))
> + return alloc_huge_page_nonid(page_hstate(compound_head(p)));
> +
> return alloc_huge_page_node(page_hstate(compound_head(p)),
> nid);
> - else
> + } else {
> return __alloc_pages_node(nid, GFP_HIGHUSER_MOVABLE, 0);
> + }
> }
Rather than adding alloc_huge_page_nonid(), would it be neater to teach
alloc_huge_page_node() (actually dequeue_huge_page_node()) to understand
nid==NUMA_NO_NODE?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* mmotm 2017-05-12-15-53 uploaded
From: akpm @ 2017-05-12 22:54 UTC (permalink / raw)
To: mm-commits, linux-kernel, linux-mm, linux-fsdevel, linux-next,
sfr, mhocko, broonie
The mm-of-the-moment snapshot 2017-05-12-15-53 has been uploaded to
http://www.ozlabs.org/~akpm/mmotm/
mmotm-readme.txt says
README for mm-of-the-moment:
http://www.ozlabs.org/~akpm/mmotm/
This is a snapshot of my -mm patch queue. Uploaded at random hopefully
more than once a week.
You will need quilt to apply these patches to the latest Linus release (4.x
or 4.x-rcY). The series file is in broken-out.tar.gz and is duplicated in
http://ozlabs.org/~akpm/mmotm/series
The file broken-out.tar.gz contains two datestamp files: .DATE and
.DATE-yyyy-mm-dd-hh-mm-ss. Both contain the string yyyy-mm-dd-hh-mm-ss,
followed by the base kernel version against which this patch series is to
be applied.
This tree is partially included in linux-next. To see which patches are
included in linux-next, consult the `series' file. Only the patches
within the #NEXT_PATCHES_START/#NEXT_PATCHES_END markers are included in
linux-next.
A git tree which contains the memory management portion of this tree is
maintained at git://git.kernel.org/pub/scm/linux/kernel/git/mhocko/mm.git
by Michal Hocko. It contains the patches which are between the
"#NEXT_PATCHES_START mm" and "#NEXT_PATCHES_END" markers, from the series
file, http://www.ozlabs.org/~akpm/mmotm/series.
A full copy of the full kernel tree with the linux-next and mmotm patches
already applied is available through git within an hour of the mmotm
release. Individual mmotm releases are tagged. The master branch always
points to the latest release, so it's constantly rebasing.
http://git.cmpxchg.org/cgit.cgi/linux-mmotm.git/
To develop on top of mmotm git:
$ git remote add mmotm git://git.kernel.org/pub/scm/linux/kernel/git/mhocko/mm.git
$ git remote update mmotm
$ git checkout -b topic mmotm/master
<make changes, commit>
$ git send-email mmotm/master.. [...]
To rebase a branch with older patches to a new mmotm release:
$ git remote update mmotm
$ git rebase --onto mmotm/master <topic base> topic
The directory http://www.ozlabs.org/~akpm/mmots/ (mm-of-the-second)
contains daily snapshots of the -mm tree. It is updated more frequently
than mmotm, and is untested.
A git copy of this tree is available at
http://git.cmpxchg.org/cgit.cgi/linux-mmots.git/
and use of this tree is similar to
http://git.cmpxchg.org/cgit.cgi/linux-mmotm.git/, described above.
This mmotm tree contains the following patches against 4.11:
(patches marked "*" will be included in linux-next)
origin.patch
i-need-old-gcc.patch
* hwpoison-memcg-forcibly-uncharge-lru-pages.patch
* time-delete-current_fs_time-function.patch
* mm-vmstat-remove-spurious-warn-during-zoneinfo-print.patch
* gcov-support-gcc-71.patch
* mm-khugepaged-add-missed-tracepoint-for-collapse_huge_page_swapin.patch
* mm-vmalloc-fix-vmalloc-users-tracking-properly.patch
* tigran-has-moved.patch
* dax-prevent-invalidation-of-mapped-dax-entries.patch
* mm-fix-data-corruption-due-to-stale-mmap-reads.patch
* ext4-return-back-to-starting-transaction-in-ext4_dax_huge_fault.patch
* dax-fix-data-corruption-when-fault-races-with-write.patch
* dax-fix-pmd-data-corruption-when-fault-races-with-write.patch
* mm-thp-copying-user-pages-must-schedule-on-collapse.patch
* mm-vmscan-scan-until-it-founds-eligible-pages.patch
* mm-docs-update-memorystat-description-with-workingset-entries.patch
* mm-skip-hwpoisoned-pages-when-onlining-pages.patch
* arm-arch-arm-include-asm-pageh-needs-personalityh.patch
* teach-initramfs_root_uid-and-initramfs_root_gid-that-1-means-current-user.patch
* clarify-help-text-that-compression-applies-to-ramfs-as-well-as-legacy-ramdisk.patch
* ocfs2-old-mle-put-and-release-after-the-function-dlm_add_migration_mle-called.patch
* ocfs2-old-mle-put-and-release-after-the-function-dlm_add_migration_mle-called-fix.patch
* ocfs2-dlm-optimization-of-code-while-free-dead-node-locks.patch
* ocfs2-dlm-optimization-of-code-while-free-dead-node-locks-checkpatch-fixes.patch
* block-restore-proc-partitions-to-not-display-non-partitionable-removable-devices.patch
mm.patch
* mm-slub-remove-a-redundant-assignment-in-___slab_alloc.patch
* mm-slub-reset-cpu_slabs-pointer-in-deactivate_slab.patch
* mm-slub-pack-red_left_pad-with-another-int-to-save-a-word.patch
* mm-slub-wrap-cpu_slab-partial-in-config_slub_cpu_partial.patch
* mm-slub-wrap-cpu_slab-partial-in-config_slub_cpu_partial-fix.patch
* mm-slub-wrap-kmem_cache-cpu_partial-in-config-config_slub_cpu_partial.patch
* mm-sparsemem-break-out-of-loops-early.patch
* mark-protection_map-as-__ro_after_init.patch
* mm-vmscan-fix-unsequenced-modification-and-access-warning.patch
* mm-nobootmem-return-0-when-start_pfn-equals-end_pfn.patch
* ksm-introduce-ksm_max_page_sharing-per-page-deduplication-limit.patch
* ksm-fix-use-after-free-with-merge_across_nodes-=-0.patch
* zram-introduce-zram_entry-to-prepare-dedup-functionality.patch
* zram-implement-deduplication-in-zram.patch
* zram-make-deduplication-feature-optional.patch
* zram-compare-all-the-entries-with-same-checksum-for-deduplication.patch
* mm-vmstat-standardize-file-operations-variable-names.patch
* mm-page_alloc-return-0-in-case-this-node-has-no-page-within-the-zone.patch
* mm-vmscan-do-not-pass-reclaimed-slab-to-vmpressure.patch
* mm-page_owner-align-with-pageblock_nr-pages.patch
* mm-walk-the-zone-in-pageblock_nr_pages-steps.patch
* fs-epoll-short-circuit-fetching-events-if-thread-has-been-killed.patch
* kexec-move-vmcoreinfo-out-of-the-kernels-bss-section.patch
* powerpc-fadump-use-the-correct-vmcoreinfo_note_size-for-phdr.patch
* kdump-protect-vmcoreinfo-data-under-the-crash-memory.patch
* kdump-vmcoreinfo-report-actual-value-of-phys_base.patch
* uapi-fix-linux-sysctlh-userspace-compilation-errors.patch
* bfs-fix-sanity-checks-for-empty-files.patch
* scripts-gdb-add-lx-fdtdump-command.patch
* procfs-fdinfo-extend-information-about-epoll-target-files.patch
* kcmp-add-kcmp_epoll_tfd-mode-to-compare-epoll-target-files.patch
* kernel-reboot-add-devm_register_reboot_notifier.patch
* kernel-reboot-add-devm_register_reboot_notifier-fix.patch
* fault-inject-support-systematic-fault-injection.patch
* fault-inject-support-systematic-fault-injection-fix.patch
* fault-inject-automatically-detect-the-number-base-for-fail-nth-write-interface.patch
* fault-inject-parse-as-natural-1-based-value-for-fail-nth-write-interface.patch
* fault-inject-make-fail-nth-read-write-interface-symmetric.patch
* fault-inject-simplify-access-check-for-fail-nth.patch
* fault-inject-simplify-access-check-for-fail-nth-fix.patch
* fault-inject-add-proc-pid-fail-nth.patch
* ipc-sem-avoid-indexing-past-end-of-sem_array.patch
linux-next.patch
linux-next-git-rejects.patch
* imx7-fix-kconfig-warning-and-build-errors.patch
* sparc64-ng4-memset-32-bits-overflow.patch
* mm-zeroing-hash-tables-in-allocator.patch
* mm-updated-callers-to-use-hash_zero-flag.patch
* mm-adaptive-hash-table-scaling.patch
* lib-crc-ccitt-add-ccitt-false-crc16-variant.patch
mm-add-strictlimit-knob-v2.patch
make-sure-nobodys-leaking-resources.patch
releasing-resources-with-children.patch
kernel-forkc-export-kernel_thread-to-modules.patch
mutex-subsystem-synchro-test-module.patch
slab-leaks3-default-y.patch
workaround-for-a-pci-restoring-bug.patch
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: 8 Gigabytes and constantly swapping
From: Arthur Marsh @ 2017-05-13 4:15 UTC (permalink / raw)
To: Tetsuo Handa; +Cc: linux-mm
In-Reply-To: <b64f3ec1-0839-3051-d030-164625620712@internode.on.net>
Arthur Marsh wrote on 12/05/17 23:09:
> It does seem to be related to chromium starting up several processes
> when opening extra windows/tabs.
I tried using ionice -c3 on chromium, which meant that disk I/O by
chromium should only occur when disk I/O was otherwise idle, and this
made a major difference.
It appears that chromium's use of multiple processes and threads means
that it effectively hogs all the disk I/O but ends up not being able to
do much as different processes/threads are making different demands on
disk I/O, leading to kswapd0 having the greatest CPU usage of any
process but still under 1 percent of available CPU time and the system
spending over 99 percent of the time in wait. All while about 1 GiB of 4
GiB available swap is being used.
Is there any existing way to limit the chromium process tree to have the
same amount of access to disk I/O as say firefox running as a single
process?
Arthur.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* [PATCH 1/1] ksm: prevent crash after write_protect_page fails
From: Andrea Arcangeli @ 2017-05-13 13:10 UTC (permalink / raw)
To: Andrew Morton, linux-mm
Cc: Hugh Dickins, Federico Simoncelli, Kirill A. Shutemov
"err" needs to be left set to -EFAULT if split_huge_page
succeeds. Otherwise if "err" gets clobbered with zero and
write_protect_page fails, try_to_merge_one_page() will succeed instead
of returning -EFAULT and then try_to_merge_with_ksm_page() will
continue thinking kpage is a PageKsm when in fact it's still an
anonymous page. Eventually it'll crash in page_add_anon_rmap.
This has been reproduced on Fedora25 kernel but I can reproduce with
upstream too.
The bug was introduced in commit
f765f540598a129dc4bb6f698fd4acc92f296c14 in v4.5.
page:fffff67546ce1cc0 count:4 mapcount:2 mapping:ffffa094551e36e1 index:0x7f0f46673
flags: 0x2ffffc0004007c(referenced|uptodate|dirty|lru|active|swapbacked)
page dumped because: VM_BUG_ON_PAGE(!PageLocked(page))
page->mem_cgroup:ffffa09674bf0000
------------[ cut here ]------------
kernel BUG at mm/rmap.c:1222!
invalid opcode: 0000 [#1] SMP
CPU: 1 PID: 76 Comm: ksmd Not tainted 4.9.3-200.fc25.x86_64 #1
task: ffffa0968be65b80 task.stack: ffffc0e941b3c000
RIP: 0010:[<ffffffff9a20ac94>] [<ffffffff9a20ac94>] do_page_add_anon_rmap+0x1c4/0x240
RSP: 0018:ffffc0e941b3fd48 EFLAGS: 00010282
RAX: 0000000000000021 RBX: fffff67546ce1cc0 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: ffffa0969ec4e0a0
RBP: ffffc0e941b3fd70 R08: 0000000000018a84 R09: 0000000000000005
R10: fffff6754f87de00 R11: 000000000000049f R12: 00007fe6f5f67000
R13: ffffa094990ecc00 R14: 0000000000000000 R15: ffffa093c369c480
FS: 0000000000000000(0000) GS:ffffa0969ec40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f4c21c221f0 CR3: 0000000152e07000 CR4: 00000000000426e0
Stack:
fffff675470fd9c0 00007fe6f5f67000 fffff67540f8a900 ffffa094990ecc00
ffffa093c369c480 ffffc0e941b3fd80 ffffffff9a20ad28 ffffc0e941b3fe00
ffffffff9a228b4b 80000001c3f67807 0000000000000000 80000001c3f67805
Call Trace:
[<ffffffff9a20ad28>] page_add_anon_rmap+0x18/0x20
[<ffffffff9a228b4b>] try_to_merge_with_ksm_page+0x50b/0x780
[<ffffffff9a229fd1>] ksm_scan_thread+0x1211/0x1410
[<ffffffff9a0e7270>] ? prepare_to_wait_event+0x100/0x100
[<ffffffff9a228dc0>] ? try_to_merge_with_ksm_page+0x780/0x780
[<ffffffff9a0c2569>] kthread+0xd9/0xf0
[<ffffffff9a0c2490>] ? kthread_park+0x60/0x60
[<ffffffff9a81be55>] ret_from_fork+0x25/0x30
Reported-by: Federico Simoncelli <fsimonce@redhat.com>
Signed-off-by: Andrea Arcangeli <aarcange@redhat.com>
---
mm/ksm.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/mm/ksm.c b/mm/ksm.c
index b53fd58..fc0c73b 100644
--- a/mm/ksm.c
+++ b/mm/ksm.c
@@ -1185,8 +1185,7 @@ static int try_to_merge_one_page(struct vm_area_struct *vma,
goto out;
if (PageTransCompound(page)) {
- err = split_huge_page(page);
- if (err)
+ if (split_huge_page(page))
goto out_unlock;
}
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* Re: [v3 3/9] mm: add "zero" argument to vmemmap allocators
From: kbuild test robot @ 2017-05-13 19:17 UTC (permalink / raw)
To: Pavel Tatashin
Cc: kbuild-all, linux-kernel, sparclinux, linux-mm, linuxppc-dev,
linux-s390, borntraeger, heiko.carstens, davem
In-Reply-To: <1494003796-748672-4-git-send-email-pasha.tatashin@oracle.com>
[-- Attachment #1: Type: text/plain, Size: 1793 bytes --]
Hi Pavel,
[auto build test ERROR on mmotm/master]
[also build test ERROR on next-20170512]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Pavel-Tatashin/parallelized-struct-page-zeroing/20170507-061412
base: git://git.cmpxchg.org/linux-mmotm.git master
config: ia64-gensparse_defconfig (attached as .config)
compiler: ia64-linux-gcc (GCC) 6.2.0
reproduce:
wget https://raw.githubusercontent.com/01org/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=ia64
All error/warnings (new ones prefixed by >>):
mm/sparse-vmemmap.c: In function '__earlyonly_bootmem_alloc':
>> mm/sparse-vmemmap.c:45:14: error: implicit declaration of function 'memblock_virt_alloc_try_nid_raw' [-Werror=implicit-function-declaration]
void *mem = memblock_virt_alloc_try_nid_raw(size, align, goal,
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> mm/sparse-vmemmap.c:45:14: warning: initialization makes pointer from integer without a cast [-Wint-conversion]
cc1: some warnings being treated as errors
vim +/memblock_virt_alloc_try_nid_raw +45 mm/sparse-vmemmap.c
39 static void * __ref __earlyonly_bootmem_alloc(int node,
40 unsigned long size,
41 unsigned long align,
42 unsigned long goal,
43 bool zero)
44 {
> 45 void *mem = memblock_virt_alloc_try_nid_raw(size, align, goal,
46 BOOTMEM_ALLOC_ACCESSIBLE,
47 node);
48 if (!mem) {
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 17766 bytes --]
^ permalink raw reply
* Re: [PATCH] mm/madvise: Dont poison entire HugeTLB page for single page errors
From: Anshuman Khandual @ 2017-05-14 2:34 UTC (permalink / raw)
To: Naoya Horiguchi, Anshuman Khandual
Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org,
akpm@linux-foundation.org, aneesh.kumar@linux.vnet.ibm.com
In-Reply-To: <20170512081001.GA13069@hori1.linux.bs1.fc.nec.co.jp>
On 05/12/2017 01:40 PM, Naoya Horiguchi wrote:
> On Thu, Apr 20, 2017 at 04:36:27PM +0530, Anshuman Khandual wrote:
>> Currently soft_offline_page() migrates the entire HugeTLB page, then
>> dequeues it from the active list by making it a dangling HugeTLB page
>> which ofcourse can not be used further and marks the entire HugeTLB
>> page as poisoned. This might be a costly waste of memory if the error
>> involved affects only small section of the entire page.
>>
>> This changes the behaviour so that only the affected page is marked
>> poisoned and then the HugeTLB page is released back to buddy system.
> Hi Anshuman,
>
> This is a good catch, and we can solve this issue now because freeing
> hwpoisoned page (previously forbidden) is available now.
>
> And I'm thinking that the same issue for hard/soft-offline on free
> hugepages can be solved, so I'll submit a patchset which includes
> updated version of your patch.
Sure, thanks Naoya.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH V2] mm/madvise: Enable (soft|hard) offline of HugeTLB pages at PGD level
From: Anshuman Khandual @ 2017-05-14 4:11 UTC (permalink / raw)
To: Andrew Morton, Anshuman Khandual
Cc: linux-kernel, linux-mm, n-horiguchi, aneesh.kumar
In-Reply-To: <20170512143503.81e0de2ae3d88a53168c601a@linux-foundation.org>
On 05/13/2017 03:05 AM, Andrew Morton wrote:
> On Wed, 26 Apr 2017 09:27:31 +0530 Anshuman Khandual <khandual@linux.vnet.ibm.com> wrote:
>
>> Though migrating gigantic HugeTLB pages does not sound much like real
>> world use case, they can be affected by memory errors. Hence migration
>> at the PGD level HugeTLB pages should be supported just to enable soft
>> and hard offline use cases.
>>
>> While allocating the new gigantic HugeTLB page, it should not matter
>> whether new page comes from the same node or not. There would be very
>> few gigantic pages on the system afterall, we should not be bothered
>> about node locality when trying to save a big page from crashing.
>>
>> This introduces a new HugeTLB allocator called alloc_huge_page_nonid()
>> which will scan over all online nodes on the system and allocate a
>> single HugeTLB page.
>>
>> ...
>>
>> --- a/mm/hugetlb.c
>> +++ b/mm/hugetlb.c
>> @@ -1669,6 +1669,23 @@ struct page *__alloc_buddy_huge_page_with_mpol(struct hstate *h,
>> return __alloc_buddy_huge_page(h, vma, addr, NUMA_NO_NODE);
>> }
>>
>> +struct page *alloc_huge_page_nonid(struct hstate *h)
>> +{
>> + struct page *page = NULL;
>> + int nid = 0;
>> +
>> + spin_lock(&hugetlb_lock);
>> + if (h->free_huge_pages - h->resv_huge_pages > 0) {
>> + for_each_online_node(nid) {
>> + page = dequeue_huge_page_node(h, nid);
>> + if (page)
>> + break;
>> + }
>> + }
>> + spin_unlock(&hugetlb_lock);
>> + return page;
>> +}
>> +
>> /*
>> * This allocation function is useful in the context where vma is irrelevant.
>> * E.g. soft-offlining uses this function because it only cares physical
>> diff --git a/mm/memory-failure.c b/mm/memory-failure.c
>> index fe64d7729a8e..d4f5710cf3f7 100644
>> --- a/mm/memory-failure.c
>> +++ b/mm/memory-failure.c
>> @@ -1481,11 +1481,15 @@ EXPORT_SYMBOL(unpoison_memory);
>> static struct page *new_page(struct page *p, unsigned long private, int **x)
>> {
>> int nid = page_to_nid(p);
>> - if (PageHuge(p))
>> + if (PageHuge(p)) {
>> + if (hstate_is_gigantic(page_hstate(compound_head(p))))
>> + return alloc_huge_page_nonid(page_hstate(compound_head(p)));
>> +
>> return alloc_huge_page_node(page_hstate(compound_head(p)),
>> nid);
>> - else
>> + } else {
>> return __alloc_pages_node(nid, GFP_HIGHUSER_MOVABLE, 0);
>> + }
>> }
>
> Rather than adding alloc_huge_page_nonid(), would it be neater to teach
> alloc_huge_page_node() (actually dequeue_huge_page_node()) to understand
> nid==NUMA_NO_NODE?
Sure, will change dequeue_huge_page_node() to accommodate NUMA_NO_NODE and
let soft offline call with NUMA_NO_NODE in case of gigantic huge pages.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* [patch 15/18] mm/vmscan: Adjust system_state checks
From: Thomas Gleixner @ 2017-05-14 18:27 UTC (permalink / raw)
To: LKML
Cc: Peter Zijlstra, Ingo Molnar, Steven Rostedt, Mark Rutland,
Andrew Morton, Johannes Weiner, Mel Gorman, Michal Hocko,
Vlastimil Babka, linux-mm
In-Reply-To: <20170514182716.347236777@linutronix.de>
[-- Attachment #1: mm-vmscan--Adjust-system_state-checks.patch --]
[-- Type: text/plain, Size: 1212 bytes --]
To enable smp_processor_id() and might_sleep() debug checks earlier, it's
required to add system states between SYSTEM_BOOTING and SYSTEM_RUNNING.
Adjust the system_state check in kswapd_run() to handle the extra states.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Mel Gorman <mgorman@techsingularity.net>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: linux-mm@kvack.org
---
mm/vmscan.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3643,7 +3643,7 @@ int kswapd_run(int nid)
pgdat->kswapd = kthread_run(kswapd, pgdat, "kswapd%d", nid);
if (IS_ERR(pgdat->kswapd)) {
/* failure at boot is fatal */
- BUG_ON(system_state == SYSTEM_BOOTING);
+ BUG_ON(system_state < SYSTEM_RUNNING);
pr_err("Failed to start kswapd on node %d\n", nid);
ret = PTR_ERR(pgdat->kswapd);
pgdat->kswapd = NULL;
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH v7 0/7] Introduce ZONE_CMA
From: Joonsoo Kim @ 2017-05-15 3:57 UTC (permalink / raw)
To: Michal Hocko
Cc: Andrew Morton, Rik van Riel, Johannes Weiner, mgorman,
Laura Abbott, Minchan Kim, Marek Szyprowski, Michal Nazarewicz,
Aneesh Kumar K . V, Vlastimil Babka, Russell King, Will Deacon,
linux-mm, linux-kernel, kernel-team
In-Reply-To: <20170512063815.GC6803@dhcp22.suse.cz>
On Fri, May 12, 2017 at 08:38:15AM +0200, Michal Hocko wrote:
> On Fri 12-05-17 11:00:48, Joonsoo Kim wrote:
> > On Thu, May 11, 2017 at 11:13:04AM +0200, Michal Hocko wrote:
> > > On Thu 11-05-17 11:12:43, Joonsoo Kim wrote:
> > > > Sorry for the late response. I was on a vacation.
> > > >
> > > > On Tue, May 02, 2017 at 03:32:29PM +0200, Michal Hocko wrote:
> > > > > On Tue 02-05-17 13:01:32, Joonsoo Kim wrote:
> > > > > > On Thu, Apr 27, 2017 at 05:06:36PM +0200, Michal Hocko wrote:
> > > > > [...]
> > > > > > > I see this point and I agree that using a specific zone might be a
> > > > > > > _nicer_ solution in the end but you have to consider another aspects as
> > > > > > > well. The main one I am worried about is a long term maintainability.
> > > > > > > We are really out of page flags and consuming one for a rather specific
> > > > > > > usecase is not good. Look at ZONE_DMA. I am pretty sure that almost
> > > > > > > no sane HW needs 16MB zone anymore, yet we have hard time to get rid
> > > > > > > of it and so we have that memory laying around unused all the time
> > > > > > > and blocking one page flag bit. CMA falls into a similar category
> > > > > > > AFAIU. I wouldn't be all that surprised if a future HW will not need CMA
> > > > > > > allocations in few years, yet we will have to fight to get rid of it
> > > > > > > like we do with ZONE_DMA. And not only that. We will also have to fight
> > > > > > > finding page flags for other more general usecases in the meantime.
> > > > > >
> > > > > > This maintenance problem is inherent. This problem exists even if we
> > > > > > uses MIGRATETYPE approach. We cannot remove many hooks for CMA if a
> > > > > > future HW will not need CMA allocation in few years. The only
> > > > > > difference is that one takes single zone bit only for CMA user and the
> > > > > > other approach takes many hooks that we need to take care about it all
> > > > > > the time.
> > > > >
> > > > > And I consider this a big difference. Because while hooks are not nice
> > > > > they will affect CMA users (in a sense of bugs/performance etc.). While
> > > > > an additional bit consumed will affect potential future and more generic
> > > > > features.
> > > >
> > > > Because these hooks are so tricky and are spread on many places,
> > > > bugs about these hooks can be made by *non-CMA* user and they hurt
> > > > *CMA* user. These hooks could also delay non-CMA user's development speed
> > > > since there are many hooks about CMA and understanding how CMA is managed
> > > > is rather difficult.
> > >
> > > Than make those hooks easier to maintain. Seriously this is a
> > > non-argument.
> >
> > I can't understand what you said here.
>
> I wanted to say that you can make those hooks so non-intrusive that
> nobody outside of the CMA has to even care that CMA exists.
I guess that current code is the result of such effort and it would be
intrusive.
>
> > With zone approach, someone who
> > isn't related to CMA don't need to understand how CMA is managed.
> >
> > >
> > > [...]
> > >
> > > > > And all this can be isolated to CMA specific hooks with mostly minimum
> > > > > impact to most users. I hear you saying that zone approach is more natural
> > > > > and I would agree if we wouldn't have to care about the number of zones.
> > > >
> > > > I attach a solution about one more bit in page flags although I don't
> > > > agree with your opinion that additional bit is no-go approach. Just
> > > > note that we have already used three bits for zone encoding in some
> > > > configuration due to ZONE_DEVICE.
> > >
> > > I am absolutely not happy about ZONE_DEVICE but there is _no_ other
> > > viable solution right now. I know that people behind this are still
> > > considering struct page over direct pfn usage but they are not in the
> > > same situation as CMA which _can_ work without additional zone.
> >
> > IIUC, ZONE_DEVICE can reuse the other zone and migratetype.
>
> They are not going to migrate anything or define any allocation fallback
> policy because those pages are outside of the page allocator completely.
> And that is why a zone approach is a reasonable approach. There are
> probably other ways and I will certainly push going that way.
I have a different opinion but it's not a main issue here so I don't
argue anymore.
> [...]
>
> > > If you _really_ insist on using zone for CMA then reuse ZONE_MOVABLE.
> > > I absolutely miss why do you push a specialized zone so hard.
> >
> > As I said before, there is no fundamental issue to reuse ZONE_MOVABLE.
> > I just don't want to reuse it because they have a different
> > characteristic. In MM subsystem context, their characteristic is the same.
> > However, CMA memory should be used for the device in runtime so more
> > allocation guarantee is needed. See the offline_pages() in
> > mm/memory_hotplug.c. They can bear in 120 sec to offline the
> > memory but CMA memory can't.
>
> This is just an implementation detail. Pinned pages in the CMA ranges
> should be easilly checked. Moreover memory hotplug cares only about
> hotplugable memory and placing CMA ranges there could be seen as a
> configuration bug.
I just wanted to say that there is a difference and I'm worry about
that it would cause the unexpected problem in the future. We can
easily distinguish them by adding another explicit check but I don't
think that this is the best.
> > And, this is a design issue. I don't want to talk about why should we
> > pursuit the good design. Originally, ZONE exists to manage different
> > type of memory. Migratetype is not for that purpose. Using separate
> > zone fits the original purpose. Mixing them would be a bad design and
> > we would esaily encounter the unexpected problem in the future.
>
> As I've said earlier. Abusing ZONE_MOVABLE is not ideal either. I would
> rather keep the status quo and fix the cluttered code and make it easier
> to follow. But if you absolutely insist that a specialized zone is
> necessary then ZONE_MOVABLE a) already exists and we do not need to
> consume another bit b) most of the CMA zone characteristics overlap
> with MOVABLE. So it is the least painful zone to use with the current
> restrictions we have.
I also think that using ZONE_MOVALBE for CMA memory is a good
candidate to consider. I will consider it more deeply.
> > > [...]
> > > > > No, but I haven't heard any single argument that those bugs are
> > > > > impossible to fix with the current approach. They might be harder to fix
> > > > > but if I can chose between harder for CMA and harder for other more
> > > > > generic HW independent features I will go for the first one. And do not
> > > > > take me wrong, I have nothing against CMA as such. It solves a real life
> > > > > problem. I just believe it doesn't deserve to consume a new bit in page
> > > > > flags because that is just too scarce resource.
> > > >
> > > > As I mentioned above, I think that maintenance overhead due to CMA
> > > > deserves to consume a new bit in page flags. It also provide us
> > > > extendability to introduce more zones in the future.
> > > >
> > > > Anyway, this value-judgement is subjective so I guess that we
> > > > cannot agree with each other. To solve your concern,
> > > > I make following solution. Please let me know your opinion about this.
> > > > This patch can be applied on top of my ZONE_CMA series.
> > >
> > > I don not think this makes situation any easier or more acceptable for
> > > merging.
> >
> > Please say the reason. This implementation don't use additional bit in
> > page flags that you concerned about. And, there is no performance
> > regression at least in my simple test.
>
> I really do not want to question your "simple test" but page_zonenum is
> used in many performance sensitive paths and proving it doesn't regress
> would require testing many different workload. Are you going to do that?
In fact, I don't think that we need to take care about this
performance problem seriously. The reasons are that:
1. Currently, there is a usable bit in the page flags.
2. Even if others consume one usable bit, there still exists spare bit
in 64b kernel. And, for 32b kernel, the number of the zone can be five
if both, ZONE_CMA and ZONE_HIGHMEM, are used. And, using ZONE_HIGHMEM
in 32b system is out of the trend.
3. Even if we fall into the latter category, I can optimize it not to
regress if both the zone, ZONE_MOVABLE and ZONE_CMA, aren't used
simultaneously with two zone bits in page flags. However, using both
zones is not usual case.
4. This performance problem only affects CMA users and there is also a
benefit due to removal of many hooks in MM subsystem so net result would
not be worse.
So, I think that performance would be better in most of cases. It
would be magianlly worse in rare cases and they could bear with it. Do
you still think that using ZONE_MOVABLE for CMA memory is
necessary rather than separate zone, ZONE_CMA?
> > > But I feel we are looping without much progress. So let me NAK this
> > > until it is _proven_ that the current code is unfixable nor ZONE_MOVABLE
> > > can be reused
> >
> > I want to open all the possibilty so could you check that ZONE_MOVABLE
> > can be overlapped with other zones? IIRC, your rework doesn't allow
> > it.
>
> My rework keeps the status quo, which is based on the assumption that
> zones cannot overlap. A longer term plan is that this restriction is
> removed. As I've said earlier overlapping zones is an interesting
> concept which is definitely worth pursuing.
Okay. We did a lot of discussion so it's better to summarise it.
1. ZONE_CMA might be a nicer solution than MIGRATETYPE.
2. Additional bit in page flags would cause another kind of
maintenance problem so it's better to avoid it as much as possible.
3. Abusing ZONE_MOVABLE looks better than introducing ZONE_CMA since
it doesn't need additional bit in page flag.
4. (Not-yet-finished) If ZONE_CMA doesn't need extra bit in page
flags with hacky magic and it has no performance regression,
??? (it's okay to use separate zone for CMA?)
If you clarify your opinion on forth argument, I will try to think the
best approach with considering these arguements.
Thanks.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Low memory killer problem
From: zhiyuan_zhu @ 2017-05-15 3:58 UTC (permalink / raw)
To: vinmenon; +Cc: linux-mm
In-Reply-To: <AF7C0ADF1FEABA4DABABB97411952A2EDD0A004D@CN-MBX05.HTC.COM.TW>
[-- Attachment #1: Type: text/plain, Size: 1487 bytes --]
Loop Linux-mm, thanks.
I want to know, why lowmemory killer not included ION free memory?
Thnanks.
From: Zhiyuan Zhu(朱志遠)
Sent: Friday, May 12, 2017 4:56 PM
To: vinmenon@codeaurora.org
Cc: Kyle Lin(林昭凱); Zhiyuan Zhu(朱志遠)
Subject: Low memory killer problem
Dear Vinmenon,
I found a part of ION memory will be return to system in android platform,
But these memorys can’t accounted in low-memory-killer strategy.
…
And I also found ION memory comes from, kmalloc/vmalloc/alloc pages/reserved memory.
What affect if account these memorys for free?
Many thanks.
…
Lowmemory killer strategy
af6c02b83 (Vinayak Menon 2015-08-27 16:29:37 +0530 418) global_page_state(NR_FILE_PAGES) + zcache_pages())
af6c02b83 (Vinayak Menon 2015-08-27 16:29:37 +0530 419) other_file = global_page_state(NR_FILE_PAGES) + zcache_pages() -
058dbde92 (Vinayak Menon 2014-02-27 00:36:22 +0530 420) global_page_state(NR_SHMEM) -
058dbde92 (Vinayak Menon 2014-02-27 00:36:22 +0530 422) total_swapcache_pages();
Meminfo example
$ adb shell cat /proc/meminfo
MemTotal: 3805312 kB
MemFree: 1446220 kB
MemAvailable: 2388384 kB
Buffers: 16796 kB
Cached: 1190868 kB
…
IonTotal: 224252 kB
IonInUse: 199108 kB
Thanks
BR
Zhiyuan zhu
[-- Attachment #2: Type: text/html, Size: 8056 bytes --]
^ permalink raw reply
* [PATCH] mm/kasan: use kasan_zero_pud for p4d table
From: js1304 @ 2017-05-15 6:20 UTC (permalink / raw)
To: Andrew Morton
Cc: Kirill A . Shutemov, Andrey Ryabinin, Alexander Potapenko,
Dmitry Vyukov, kasan-dev, linux-mm, linux-kernel, kernel-team,
Joonsoo Kim
From: Joonsoo Kim <iamjoonsoo.kim@lge.com>
There is missing optimization in zero_p4d_populate() that can save
some memory when mapping zero shadow. Implement it like as others.
Signed-off-by: Joonsoo Kim <iamjoonsoo.kim@lge.com>
---
mm/kasan/kasan_init.c | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/mm/kasan/kasan_init.c b/mm/kasan/kasan_init.c
index b96a5f7..554e4c0 100644
--- a/mm/kasan/kasan_init.c
+++ b/mm/kasan/kasan_init.c
@@ -118,6 +118,18 @@ static void __init zero_p4d_populate(pgd_t *pgd, unsigned long addr,
do {
next = p4d_addr_end(addr, end);
+ if (IS_ALIGNED(addr, P4D_SIZE) && end - addr >= P4D_SIZE) {
+ pud_t *pud;
+ pmd_t *pmd;
+
+ p4d_populate(&init_mm, p4d, lm_alias(kasan_zero_pud));
+ pud = pud_offset(p4d, addr);
+ pud_populate(&init_mm, pud, lm_alias(kasan_zero_pmd));
+ pmd = pmd_offset(pud, addr);
+ pmd_populate_kernel(&init_mm, pmd,
+ lm_alias(kasan_zero_pte));
+ continue;
+ }
if (p4d_none(*p4d)) {
p4d_populate(&init_mm, p4d,
--
2.7.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* RE: Low memory killer problem
From: zhiyuan_zhu @ 2017-05-15 7:25 UTC (permalink / raw)
To: vinmenon, linux-mm, skhiani, gregkh; +Cc: torvalds
In-Reply-To: <AF7C0ADF1FEABA4DABABB97411952A2EDD0A004D@CN-MBX05.HTC.COM.TW>
[-- Attachment #1: Type: text/plain, Size: 1746 bytes --]
Loop MM maintainers,
Dear All,
Who can talk about this problem? Thanks.
Maybe this is lowmemory killer’s bug ?
ION memory is complex now, we need to have a breakdown for them, I think.
Thanks a lot
Zhiyuan zhu
From: Zhiyuan Zhu(朱志遠)
Sent: Friday, May 12, 2017 4:56 PM
To: vinmenon@codeaurora.org
Cc: Kyle Lin(林昭凱); Zhiyuan Zhu(朱志遠)
Subject: Low memory killer problem
Dear Vinmenon,
I found a part of ION memory will be return to system in android platform,
But these memorys can’t accounted in low-memory-killer strategy.
…
And I also found ION memory comes from, kmalloc/vmalloc/alloc pages/reserved memory.
What affect if account these memorys for free?
Many thanks.
…
Lowmemory killer strategy
af6c02b83 (Vinayak Menon 2015-08-27 16:29:37 +0530 418) global_page_state(NR_FILE_PAGES) + zcache_pages())
af6c02b83 (Vinayak Menon 2015-08-27 16:29:37 +0530 419) other_file = global_page_state(NR_FILE_PAGES) + zcache_pages() -
058dbde92 (Vinayak Menon 2014-02-27 00:36:22 +0530 420) global_page_state(NR_SHMEM) -
5c4698e38 (kyle_lin 2015-04-16 16:04:22 +0800 421) global_page_state(NR_MLOCK) -
058dbde92 (Vinayak Menon 2014-02-27 00:36:22 +0530 422) total_swapcache_pages();
Meminfo example
$ adb shell cat /proc/meminfo
MemTotal: 3805312 kB
MemFree: 1446220 kB
MemAvailable: 2388384 kB
Buffers: 16796 kB
Cached: 1190868 kB
…
IonTotal: 224252 kB
IonInUse: 199108 kB
Thanks
BR
Zhiyuan zhu
[-- Attachment #2: Type: text/html, Size: 9170 bytes --]
^ permalink raw reply
* Re: mm: page allocation failures in swap_duplicate -> add_swap_count_continuation
From: Michal Hocko @ 2017-05-15 8:03 UTC (permalink / raw)
To: Christian Borntraeger; +Cc: linux-mm, Linux Kernel Mailing List
In-Reply-To: <772d81b0-df36-8644-41ca-dc13d0c0f2b5@de.ibm.com>
On Fri 12-05-17 11:18:42, Christian Borntraeger wrote:
> Folks,
>
> recently I have seen page allocation failures during
> paging in the paging code:
> e.g.
>
> May 05 21:36:53 kernel: Call Trace:
> May 05 21:36:53 kernel: ([<0000000000112f62>] show_trace+0x62/0x78)
> May 05 21:36:53 kernel: [<0000000000113050>] show_stack+0x68/0xe0
> May 05 21:36:53 kernel: [<00000000004fb97e>] dump_stack+0x7e/0xb0
> May 05 21:36:53 kernel: [<0000000000299262>] warn_alloc+0xf2/0x190
> May 05 21:36:53 kernel: [<000000000029a25a>] __alloc_pages_nodemask+0xeda/0xfe0
> May 05 21:36:53 kernel: [<00000000002fa570>] alloc_pages_current+0xb8/0x170
> May 05 21:36:53 kernel: [<00000000002f03fc>] add_swap_count_continuation+0x3c/0x280
> May 05 21:36:53 kernel: [<00000000002f068c>] swap_duplicate+0x4c/0x80
> May 05 21:36:53 kernel: [<00000000002dfbfa>] try_to_unmap_one+0x372/0x578
> May 05 21:36:53 kernel: [<000000000030131a>] rmap_walk_ksm+0x14a/0x1d8
> May 05 21:36:53 kernel: [<00000000002e0d60>] try_to_unmap+0x140/0x170
> May 05 21:36:53 kernel: [<00000000002abc9c>] shrink_page_list+0x944/0xad8
> May 05 21:36:53 kernel: [<00000000002ac720>] shrink_inactive_list+0x1e0/0x5b8
> May 05 21:36:53 kernel: [<00000000002ad642>] shrink_node_memcg+0x5e2/0x800
> May 05 21:36:53 kernel: [<00000000002ad954>] shrink_node+0xf4/0x360
> May 05 21:36:53 kernel: [<00000000002aeb00>] kswapd+0x330/0x810
> May 05 21:36:53 kernel: [<0000000000189f14>] kthread+0x144/0x168
> May 05 21:36:53 kernel: [<00000000008011ea>] kernel_thread_starter+0x6/0xc
> May 05 21:36:53 kernel: [<00000000008011e4>] kernel_thread_starter+0x0/0xc
>
> This seems to be new in 4.11 but the relevant code did not seem to have
> changed.
>
> Something like this
>
> diff --git a/mm/swapfile.c b/mm/swapfile.c
> index 1781308..b2dd53e 100644
> --- a/mm/swapfile.c
> +++ b/mm/swapfile.c
> @@ -3039,7 +3039,7 @@ int swap_duplicate(swp_entry_t entry)
> int err = 0;
>
> while (!err && __swap_duplicate(entry, 1) == -ENOMEM)
> - err = add_swap_count_continuation(entry, GFP_ATOMIC);
> + err = add_swap_count_continuation(entry, GFP_ATOMIC | __GFP_NOWARN);
> return err;
> }
>
>
> seems not appropriate, because this code does not know if the caller can
> handle returned errors.
>
> Would something like the following (white space damaged cut'n'paste be ok?
> (the try_to_unmap_one change looks fine, not sure if copy_one_pte does the
> right thing)
No, it won't. If you want to silent the warning then explain _why_ it is
a good approach. It is not immediatelly clear to me.
>
> diff --git a/mm/memory.c b/mm/memory.c
> index 235ba51..3ae6f33 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -898,7 +898,7 @@ copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,
> swp_entry_t entry = pte_to_swp_entry(pte);
>
> if (likely(!non_swap_entry(entry))) {
> - if (swap_duplicate(entry) < 0)
> + if (swap_duplicate(entry, __GFP_NOWARN) < 0)
> return entry.val;
Moreover if you add a gfp_mask argument then the _full_ mask should be
given rather than just one of the modifiers.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: Low memory killer problem
From: Greg KH @ 2017-05-15 8:05 UTC (permalink / raw)
To: zhiyuan_zhu; +Cc: vinmenon, linux-mm, skhiani, torvalds
In-Reply-To: <AF7C0ADF1FEABA4DABABB97411952A2EDD0A4F06@CN-MBX03.HTC.COM.TW>
On Mon, May 15, 2017 at 07:25:20AM +0000, zhiyuan_zhu@htc.com wrote:
> Loop MM maintainers,
>
>
>
> Dear All,
>
>
>
> Who can talk about this problem? Thanks.
What problem?
> Maybe this is lowmemory killera??s bug ?
This code is now removed from the kernel, so I doubt there could be a
bug in it :)
> ION memory is complex now, we need to have a breakdown for them, I think.
What do you mean by this?
thanks,
greg k-h
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: 8 Gigabytes and constantly swapping
From: Michal Hocko @ 2017-05-15 8:09 UTC (permalink / raw)
To: Arthur Marsh; +Cc: linux-mm
In-Reply-To: <171e8fa1-3f14-dc18-09b5-48399b250a30@internode.on.net>
On Fri 12-05-17 18:21:27, Arthur Marsh wrote:
> I've been building the Linus git head kernels as the source gets updated and
> the one built about 3 hours ago managed to get stuck with kswapd0 as the
> highest consumer of CPU cycles (but still under 1 percent) of processes
> listed by top for over 15 minutes, after which I hit the power switch and
> rebooted with a Debian 4.11.0 kernel.
>
> The previous kernel built less than 24 hours earlier did not have this
> problem.
>
> CPU is an Athlon64 (Athlon II X4, 4 cores), RAM is 8GiB, swap is 4GiB, load
> was mainly firefox and chromium. Opening a new window in chromium seemed to
> help trigger the problem.
>
> It's not much information to go on, just wondered if anyone else had
> experienced similar issues?
>
> I'm happy to supply more configuration information and run tests including
> with kernels built with test patches applied.
Is this 32b or 64b kernel? Could you take /proc/vmstat snapshots ever
second while the kswapd is active?
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: mm: page allocation failures in swap_duplicate -> add_swap_count_continuation
From: Christian Borntraeger @ 2017-05-15 8:10 UTC (permalink / raw)
To: Michal Hocko; +Cc: linux-mm, Linux Kernel Mailing List
In-Reply-To: <20170515080323.GD6056@dhcp22.suse.cz>
On 05/15/2017 10:03 AM, Michal Hocko wrote:
> On Fri 12-05-17 11:18:42, Christian Borntraeger wrote:
>> Folks,
>>
>> recently I have seen page allocation failures during
>> paging in the paging code:
>> e.g.
>>
>> May 05 21:36:53 kernel: Call Trace:
>> May 05 21:36:53 kernel: ([<0000000000112f62>] show_trace+0x62/0x78)
>> May 05 21:36:53 kernel: [<0000000000113050>] show_stack+0x68/0xe0
>> May 05 21:36:53 kernel: [<00000000004fb97e>] dump_stack+0x7e/0xb0
>> May 05 21:36:53 kernel: [<0000000000299262>] warn_alloc+0xf2/0x190
>> May 05 21:36:53 kernel: [<000000000029a25a>] __alloc_pages_nodemask+0xeda/0xfe0
>> May 05 21:36:53 kernel: [<00000000002fa570>] alloc_pages_current+0xb8/0x170
>> May 05 21:36:53 kernel: [<00000000002f03fc>] add_swap_count_continuation+0x3c/0x280
>> May 05 21:36:53 kernel: [<00000000002f068c>] swap_duplicate+0x4c/0x80
>> May 05 21:36:53 kernel: [<00000000002dfbfa>] try_to_unmap_one+0x372/0x578
>> May 05 21:36:53 kernel: [<000000000030131a>] rmap_walk_ksm+0x14a/0x1d8
>> May 05 21:36:53 kernel: [<00000000002e0d60>] try_to_unmap+0x140/0x170
>> May 05 21:36:53 kernel: [<00000000002abc9c>] shrink_page_list+0x944/0xad8
>> May 05 21:36:53 kernel: [<00000000002ac720>] shrink_inactive_list+0x1e0/0x5b8
>> May 05 21:36:53 kernel: [<00000000002ad642>] shrink_node_memcg+0x5e2/0x800
>> May 05 21:36:53 kernel: [<00000000002ad954>] shrink_node+0xf4/0x360
>> May 05 21:36:53 kernel: [<00000000002aeb00>] kswapd+0x330/0x810
>> May 05 21:36:53 kernel: [<0000000000189f14>] kthread+0x144/0x168
>> May 05 21:36:53 kernel: [<00000000008011ea>] kernel_thread_starter+0x6/0xc
>> May 05 21:36:53 kernel: [<00000000008011e4>] kernel_thread_starter+0x0/0xc
>>
>> This seems to be new in 4.11 but the relevant code did not seem to have
>> changed.
>>
>> Something like this
>>
>> diff --git a/mm/swapfile.c b/mm/swapfile.c
>> index 1781308..b2dd53e 100644
>> --- a/mm/swapfile.c
>> +++ b/mm/swapfile.c
>> @@ -3039,7 +3039,7 @@ int swap_duplicate(swp_entry_t entry)
>> int err = 0;
>>
>> while (!err && __swap_duplicate(entry, 1) == -ENOMEM)
>> - err = add_swap_count_continuation(entry, GFP_ATOMIC);
>> + err = add_swap_count_continuation(entry, GFP_ATOMIC | __GFP_NOWARN);
>> return err;
>> }
>>
>>
>> seems not appropriate, because this code does not know if the caller can
>> handle returned errors.
>>
>> Would something like the following (white space damaged cut'n'paste be ok?
>> (the try_to_unmap_one change looks fine, not sure if copy_one_pte does the
>> right thing)
>
> No, it won't. If you want to silent the warning then explain _why_ it is
> a good approach. It is not immediatelly clear to me.
Consider my mail a bug report, not a proper fix. As far as I can tell, try_to_unmap_one
can handle allocation failure gracefully, so not warn here _looks_ fine to me.
>
>>
>> diff --git a/mm/memory.c b/mm/memory.c
>> index 235ba51..3ae6f33 100644
>> --- a/mm/memory.c
>> +++ b/mm/memory.c
>> @@ -898,7 +898,7 @@ copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,
>> swp_entry_t entry = pte_to_swp_entry(pte);
>>
>> if (likely(!non_swap_entry(entry))) {
>> - if (swap_duplicate(entry) < 0)
>> + if (swap_duplicate(entry, __GFP_NOWARN) < 0)
>> return entry.val;
This code has special casing for the allocation failure path, but I cannot
decide if it does the right thing here.
>
> Moreover if you add a gfp_mask argument then the _full_ mask should be
> given rather than just one of the modifiers.
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* RE: Low memory killer problem
From: zhiyuan_zhu @ 2017-05-15 8:22 UTC (permalink / raw)
To: gregkh; +Cc: vinmenon, linux-mm, skhiani, torvalds, Jet_Li
In-Reply-To: <20170515080535.GA22076@kroah.com>
Dear Greg,
Very sorry my mail history is lost.
I found a part of ION memory will be return to system in android platform,
But these memorys can’t accounted in low-memory-killer strategy.
…
And I also found ION memory comes from, kmalloc/vmalloc/alloc pages/reserved memory.
I understand reserved memory shouldn't accounted to free memory.
But the memory which alloced by kmalloc/vmalloc/alloc pages, can be reclaimed.
But the low-memory killer can't accounted this part,
Many thanks.
Code location,
---> drivers/staging/android/lowmemorykiller.c -> lowmem_scan
Code review,
415 other_free = global_page_state(NR_FREE_PAGES);
416
417 if (global_page_state(NR_SHMEM) + global_page_state(NR_MLOCK) + total_swapcache_pages() <
418 global_page_state(NR_FILE_PAGES) + zcache_pages())
419 other_file = global_page_state(NR_FILE_PAGES) + zcache_pages() -
420 global_page_state(NR_SHMEM) -
422 total_swapcache_pages();
So, other_free means: MemFree,
other_file means: Buffers + Cached - ShMEM - SwapCache.
And part of ( IonInUse - IonTotal ) should be reclaimed but not accounted.
Thanks.
Meminfo example
$ adb shell cat /proc/meminfo
MemTotal: 3805312 kB
MemFree: 1446220 kB
MemAvailable: 2388384 kB
Buffers: 16796 kB
Cached: 1190868 kB
…
IonTotal: 224252 kB
IonInUse: 199108 kB
Thanks
BR
Zhiyuan zhu
-----Original Message-----
From: Greg KH [mailto:gregkh@linuxfoundation.org]
Sent: Monday, May 15, 2017 4:06 PM
To: Zhiyuan Zhu(朱志遠)
Cc: vinmenon@codeaurora.org; linux-mm@kvack.org; skhiani@codeaurora.org; torvalds@linux-foundation.org
Subject: Re: Low memory killer problem
On Mon, May 15, 2017 at 07:25:20AM +0000, zhiyuan_zhu@htc.com wrote:
> Loop MM maintainers,
>
>
>
> Dear All,
>
>
>
> Who can talk about this problem? Thanks.
What problem?
> Maybe this is lowmemory killer’s bug ?
This code is now removed from the kernel, so I doubt there could be a bug in it :)
> ION memory is complex now, we need to have a breakdown for them, I think.
What do you mean by this?
thanks,
greg k-h
^ permalink raw reply
* Re: [PATCH] mm/kasan: use kasan_zero_pud for p4d table
From: Andrey Ryabinin @ 2017-05-15 8:29 UTC (permalink / raw)
To: js1304, Andrew Morton
Cc: Kirill A . Shutemov, Alexander Potapenko, Dmitry Vyukov,
kasan-dev, linux-mm, linux-kernel, kernel-team, Joonsoo Kim
In-Reply-To: <1494829255-23946-1-git-send-email-iamjoonsoo.kim@lge.com>
On 05/15/2017 09:20 AM, js1304@gmail.com wrote:
> From: Joonsoo Kim <iamjoonsoo.kim@lge.com>
>
> There is missing optimization in zero_p4d_populate() that can save
> some memory when mapping zero shadow. Implement it like as others.
>
> Signed-off-by: Joonsoo Kim <iamjoonsoo.kim@lge.com>
Acked-by: Andrey Ryabinin <aryabinin@virtuozzo.com>
> ---
> mm/kasan/kasan_init.c | 12 ++++++++++++
> 1 file changed, 12 insertions(+)
>
> diff --git a/mm/kasan/kasan_init.c b/mm/kasan/kasan_init.c
> index b96a5f7..554e4c0 100644
> --- a/mm/kasan/kasan_init.c
> +++ b/mm/kasan/kasan_init.c
> @@ -118,6 +118,18 @@ static void __init zero_p4d_populate(pgd_t *pgd, unsigned long addr,
>
> do {
> next = p4d_addr_end(addr, end);
> + if (IS_ALIGNED(addr, P4D_SIZE) && end - addr >= P4D_SIZE) {
> + pud_t *pud;
> + pmd_t *pmd;
> +
> + p4d_populate(&init_mm, p4d, lm_alias(kasan_zero_pud));
> + pud = pud_offset(p4d, addr);
> + pud_populate(&init_mm, pud, lm_alias(kasan_zero_pmd));
> + pmd = pmd_offset(pud, addr);
> + pmd_populate_kernel(&init_mm, pmd,
> + lm_alias(kasan_zero_pte));
> + continue;
> + }
>
> if (p4d_none(*p4d)) {
> p4d_populate(&init_mm, p4d,
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox