* Re: zswap: Delete an error message for a failed memory allocation in zswap_dstmem_prepare()
From: SF Markus Elfring @ 2017-05-21 9:40 UTC (permalink / raw)
To: Wolfram Sang, linux-mm
Cc: Dan Streetman, Seth Jennings, LKML, kernel-janitors
In-Reply-To: <20170521084734.GB1456@katana>
> Markus, can you please stop CCing me on every of those patches?
Yes, of course.
>> Link: http://events.linuxfoundation.org/sites/events/files/slides/LCJ16-Refactor_Strings-WSang_0.pdf
Did I interpret any information from your presentation slides in an
inappropriate way?
> And why do you create a patch for every occasion in the same file?
This can occasionally happen when I am more unsure about the change acceptance
for a specific place.
> Do you want to increase your patch count?
This can also happen as a side effect if such a source code search pattern
will point hundreds of places out for further software development considerations.
How would you prefer to clarify the remaining update candidates there?
Regards,
Markus
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: zswap: Delete an error message for a failed memory allocation in zswap_dstmem_prepare()
From: Wolfram Sang @ 2017-05-21 9:56 UTC (permalink / raw)
To: SF Markus Elfring
Cc: linux-mm, Dan Streetman, Seth Jennings, LKML, kernel-janitors
In-Reply-To: <7bd4b458-6f6e-416b-97a9-b1b3d0840144@users.sourceforge.net>
[-- Attachment #1: Type: text/plain, Size: 983 bytes --]
Markus,
> > Markus, can you please stop CCing me on every of those patches?
>
> Yes, of course.
Thanks!
> >> Link: http://events.linuxfoundation.org/sites/events/files/slides/LCJ16-Refactor_Strings-WSang_0.pdf
>
> Did I interpret any information from your presentation slides in an
> inappropriate way?
Have you read my LWN article "Best practices for a big patch series"?
https://lwn.net/Articles/585782/
> > And why do you create a patch for every occasion in the same file?
>
> This can occasionally happen when I am more unsure about the change acceptance
> for a specific place.
Why were you unsure here?
> This can also happen as a side effect if such a source code search pattern
> will point hundreds of places out for further software development considerations.
> How would you prefer to clarify the remaining update candidates there?
Maybe the article mentioned can provice further guidance?
Have a nice sunday,
Wolfram
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply
* Re: Using best practices for big software change possibilities
From: SF Markus Elfring @ 2017-05-21 10:12 UTC (permalink / raw)
To: Wolfram Sang
Cc: linux-mm, Dan Streetman, Seth Jennings, LKML, kernel-janitors
In-Reply-To: <20170521095654.bzpaa2obfszrajgb@ninjato>
> Have you read my LWN article "Best practices for a big patch series"?
>
> https://lwn.net/Articles/585782/
Yes.
>> This can also happen as a side effect if such a source code search pattern
>> will point hundreds of places out for further software development considerations.
>> How would you prefer to clarify the remaining update candidates there?
>
> Maybe the article mentioned can provice further guidance?
Partly, yes.
I am trying to achieve some software improvements also for special change patterns.
This approach can trigger corresponding communication difficulties.
How do you think about to resolve them by additional means besides mail exchange?
Regards,
Markus
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: Using best practices for big software change possibilities
From: Wolfram Sang @ 2017-05-21 10:27 UTC (permalink / raw)
To: SF Markus Elfring
Cc: linux-mm, Dan Streetman, Seth Jennings, LKML, kernel-janitors
In-Reply-To: <82cfcf3e-0089-0629-f10c-e01346487f6a@users.sourceforge.net>
[-- Attachment #1: Type: text/plain, Size: 300 bytes --]
> How do you think about to resolve them by additional means besides mail exchange?
That can work. E.g. meeting at conferences often solved mail
communication problems.
For now, I still wonder why you were unsure about grouping the changes
into one patch? Maybe there is something to be learned?
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply
* Re: Using best practices for big software change possibilities
From: SF Markus Elfring @ 2017-05-21 10:45 UTC (permalink / raw)
To: Wolfram Sang
Cc: linux-mm, Dan Streetman, Seth Jennings, LKML, kernel-janitors
In-Reply-To: <20170521102750.ljgvdw2btuks3tqf@ninjato>
>> How do you think about to resolve them by additional means besides mail exchange?
>
> That can work.
I am curious to find out which other communication means could really help here.
> E.g. meeting at conferences often solved mail communication problems.
I find my resources too limited at the moment to attend conferences on site.
How are the chances for further clarification by ordinary telephone calls?
> For now, I still wonder why you were unsure about grouping the changes
> into one patch?
I am varying the patch granularity for affected software areas to some degree.
But I came also places along where I got an impression for higher uncertainty.
> Maybe there is something to be learned?
This is also generally possible.
Would you like to extend the scope for the change pattern around questionable
error messages from a single source file to whole subsystem trees in Linux?
Regards,
Markus
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* [PATCH] LSM: Make security_hook_heads a local variable.
From: Tetsuo Handa @ 2017-05-21 11:14 UTC (permalink / raw)
To: linux-security-module
Cc: linux-mm, kernel-hardening, linux-kernel, Tetsuo Handa,
Casey Schaufler, Greg KH, Igor Stoppa, James Morris, Kees Cook,
Paul Moore, Stephen Smalley
In-Reply-To: <20170520085147.GA4619@kroah.com>
A sealable memory allocator patch was proposed at
http://lkml.kernel.org/r/20170519103811.2183-1-igor.stoppa@huawei.com ,
and is waiting for a follow-on patch showing how any of the kernel
can be changed to use this new subsystem. So, here it is for LSM hooks.
The LSM hooks ("struct security_hook_heads security_hook_heads" and
"struct security_hook_list ...[]") will benefit from this allocator via
protection using set_memory_ro()/set_memory_rw(), and it will remove
CONFIG_SECURITY_WRITABLE_HOOKS config option.
This means that these structures will be allocated at run time using
smalloc(), and therefore the address of these structures will be
determined at run time rather than compile time.
But currently, LSM_HOOK_INIT() macro depends on the address of
security_hook_heads being known at compile time. But we already
initialize security_hook_heads as an array of "struct list_head".
Therefore, let's use index number (or relative offset from the head
of security_hook_heads) instead of absolute address of
security_hook_heads so that LSM_HOOK_INIT() macro does not need to
know absolute address of security_hook_heads. Then, security_add_hooks()
will be able to allocate and copy "struct security_hook_list ...[]" using
smalloc().
Signed-off-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
Cc: Kees Cook <keescook@chromium.org>
Cc: Paul Moore <paul@paul-moore.com>
Cc: Stephen Smalley <sds@tycho.nsa.gov>
Cc: Casey Schaufler <casey@schaufler-ca.com>
Cc: James Morris <james.l.morris@oracle.com>
Cc: Igor Stoppa <igor.stoppa@huawei.com>
Cc: Greg KH <gregkh@linuxfoundation.org>
---
include/linux/lsm_hooks.h | 6 +++---
security/security.c | 10 ++++++++--
2 files changed, 11 insertions(+), 5 deletions(-)
diff --git a/include/linux/lsm_hooks.h b/include/linux/lsm_hooks.h
index 080f34e..865c11d 100644
--- a/include/linux/lsm_hooks.h
+++ b/include/linux/lsm_hooks.h
@@ -1884,8 +1884,8 @@ struct security_hook_heads {
*/
struct security_hook_list {
struct list_head list;
- struct list_head *head;
union security_list_options hook;
+ const unsigned int idx;
char *lsm;
};
@@ -1896,9 +1896,9 @@ struct security_hook_list {
* text involved.
*/
#define LSM_HOOK_INIT(HEAD, HOOK) \
- { .head = &security_hook_heads.HEAD, .hook = { .HEAD = HOOK } }
+ { .idx = offsetof(struct security_hook_heads, HEAD) / \
+ sizeof(struct list_head), .hook = { .HEAD = HOOK } }
-extern struct security_hook_heads security_hook_heads;
extern char *lsm_names;
extern void security_add_hooks(struct security_hook_list *hooks, int count,
diff --git a/security/security.c b/security/security.c
index 54b1e39..d6883ce 100644
--- a/security/security.c
+++ b/security/security.c
@@ -33,7 +33,7 @@
/* Maximum number of letters for an LSM name string */
#define SECURITY_NAME_MAX 10
-struct security_hook_heads security_hook_heads __lsm_ro_after_init;
+static struct security_hook_heads security_hook_heads __lsm_ro_after_init;
char *lsm_names;
/* Boot-time LSM user choice */
static __initdata char chosen_lsm[SECURITY_NAME_MAX + 1] =
@@ -152,10 +152,16 @@ void __init security_add_hooks(struct security_hook_list *hooks, int count,
char *lsm)
{
int i;
+ struct list_head *list = (struct list_head *) &security_hook_heads;
for (i = 0; i < count; i++) {
+ const unsigned int idx = hooks[i].idx;
+
+ if (WARN_ON(idx >= sizeof(security_hook_heads) /
+ sizeof(struct list_head)))
+ continue;
hooks[i].lsm = lsm;
- list_add_tail_rcu(&hooks[i].list, hooks[i].head);
+ list_add_tail_rcu(&hooks[i].list, &list[idx]);
}
if (lsm_append(lsm, &lsm_names) < 0)
panic("%s - Cannot get early memory.\n", __func__);
--
1.8.3.1
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* Re: [v4 1/1] mm: Adaptive hash table scaling
From: Pasha Tatashin @ 2017-05-21 12:58 UTC (permalink / raw)
To: Andi Kleen; +Cc: akpm, linux-kernel, linux-mm, mhocko
In-Reply-To: <87h90faroe.fsf@firstfloor.org>
Hi Andi,
Thank you for looking at this. I mentioned earlier, I would not want to
impose a cap. However, if you think that for example dcache needs a cap,
there is already a mechanism for that via high_limit argument, so the
client can be changed to provide that cap. However, this particular
patch addresses scaling problem for everyone by making it scale with
memory at a slower pace.
Thank you,
Pasha
On 05/20/2017 10:07 PM, Andi Kleen wrote:
> Pavel Tatashin <pasha.tatashin@oracle.com> writes:
>
>> Allow hash tables to scale with memory but at slower pace, when HASH_ADAPT
>> is provided every time memory quadruples the sizes of hash tables will only
>> double instead of quadrupling as well. This algorithm starts working only
>> when memory size reaches a certain point, currently set to 64G.
>>
>> This is example of dentry hash table size, before and after four various
>> memory configurations:
>
> IMHO the scale is still too aggressive. I find it very unlikely
> that a 1TB machine really needs 256MB of hash table because
> number of used files are unlikely to directly scale with memory.
>
> Perhaps should just cap it at some large size, e.g. 32M
>
> -Andi
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org. For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* [PATCH] Patch for remapping pages around the fault page
From: Sarunya Pumma @ 2017-05-21 15:12 UTC (permalink / raw)
To: rppt
Cc: linux-mm, akpm, kirill.shutemov, jack, ross.zwisler, mhocko,
aneesh.kumar, lstoakes, dave.jiang, Sarunya Pumma
After the fault handler performs the __do_fault function to read a fault
page when a page fault occurs, it does not map other pages that have been
read together with the fault page. This can cause a number of minor page
faults to be large. Therefore, this patch is developed to remap pages
around the fault page by aiming to map the pages that have been read
synchronously or asynchronously with the fault page.
The major function of this patch is the redo_fault_around function. This
function computes the start and end offsets of the pages to be mapped,
determines whether to do the page remapping, remaps pages using the
map_pages function, and returns. In the redo_fault_around function, the
start and end offsets are computed the same way as the do_fault_around
function. To determine whether to do the remapping, we determine if the
pages around the fault page are already mapped. If they are, the remapping
will not be performed.
As checking every page can be inefficient if a number of pages to be mapped
is large, we have added a threshold called "vm_nr_rempping" to consider
whether to check the status of every page around the fault page or just
some pages. Note that the vm_nr_rempping parameter can be adjusted via the
Sysctl interface. In the case that a number of pages to be mapped is
smaller than the vm_nr_rempping threshold, we check all pages around the
fault page (within the start and end offsets). Otherwise, we check only the
adjacent pages (left and right).
The page remapping is beneficial when performing the "almost sequential"
page accesses, where pages are accessed in order but some pages are
skipped.
The following is one example scenario that we can reduce one page fault
every 16 page:
Assume that we want to access pages sequentially and skip every page that
marked as PG_readahead. Assume that the read-ahead size is 32 pages and the
number of pages to be mapped each time (fault_around_pages) is 16.
When accessing a page at offset 0, a major page fault occurs, so pages from
page 0 to page 31 is read from the disk to the page cache. With this, page
24 is marked as a read-ahead page (PG_readahead). Then only page 0 is
mapped to the virtual memory space.
When accessing a page at offset 1, a minor page fault occurs, pages from
page 0 to page 15 will be mapped.
We keep accessing pages until page 31. Note that we skip page 24.
When accessing a page at offset 32, a major page fault occurs. The same
process will be repeated. The other 32 pages will be read from the disk.
Only page 32 is mapped. Then a minor page fault at the next page (page
33) will occur.
>From this example, two page faults occur every 16 page. With this patch, we
can eliminate the minor page fault in every 16 page.
Thank you very much for your time for reviewing the patch.
Signed-off-by: Sarunya Pumma <sarunya@vt.edu>
---
include/linux/mm.h | 2 ++
kernel/sysctl.c | 8 +++++
mm/memory.c | 90 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 100 insertions(+)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 7cb17c6..2d533a3 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -34,6 +34,8 @@ struct bdi_writeback;
void init_mm_internals(void);
+extern unsigned long vm_nr_remapping;
+
#ifndef CONFIG_NEED_MULTIPLE_NODES /* Don't use mapnrs, do it properly */
extern unsigned long max_mapnr;
diff --git a/kernel/sysctl.c b/kernel/sysctl.c
index 4dfba1a..16c7efe 100644
--- a/kernel/sysctl.c
+++ b/kernel/sysctl.c
@@ -1332,6 +1332,14 @@ static struct ctl_table vm_table[] = {
.extra1 = &zero,
.extra2 = &one_hundred,
},
+ {
+ .procname = "nr_remapping",
+ .data = &vm_nr_remapping,
+ .maxlen = sizeof(vm_nr_remapping),
+ .mode = 0644,
+ .proc_handler = proc_doulongvec_minmax,
+ .extra1 = &zero,
+ },
#ifdef CONFIG_HUGETLB_PAGE
{
.procname = "nr_hugepages",
diff --git a/mm/memory.c b/mm/memory.c
index 6ff5d72..3d0dca9 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -83,6 +83,9 @@
#warning Unfortunate NUMA and NUMA Balancing config, growing page-frame for last_cpupid.
#endif
+/* A preset threshold for considering page remapping */
+unsigned long vm_nr_remapping = 32;
+
#ifndef CONFIG_NEED_MULTIPLE_NODES
/* use the per-pgdat data instead for discontigmem - mbligh */
unsigned long max_mapnr;
@@ -3374,6 +3377,82 @@ static int do_fault_around(struct vm_fault *vmf)
return ret;
}
+static int redo_fault_around(struct vm_fault *vmf)
+{
+ unsigned long address = vmf->address, nr_pages, mask;
+ pgoff_t start_pgoff = vmf->pgoff;
+ pgoff_t end_pgoff;
+ pte_t *lpte, *rpte;
+ int off, ret = 0, is_mapped = 0;
+
+ nr_pages = READ_ONCE(fault_around_bytes) >> PAGE_SHIFT;
+ mask = ~(nr_pages * PAGE_SIZE - 1) & PAGE_MASK;
+
+ vmf->address = max(address & mask, vmf->vma->vm_start);
+ off = ((address - vmf->address) >> PAGE_SHIFT) & (PTRS_PER_PTE - 1);
+ start_pgoff -= off;
+
+ /*
+ * end_pgoff is either end of page table or end of vma
+ * or fault_around_pages() from start_pgoff, depending what is nearest.
+ */
+ end_pgoff = start_pgoff -
+ ((vmf->address >> PAGE_SHIFT) & (PTRS_PER_PTE - 1)) +
+ PTRS_PER_PTE - 1;
+ end_pgoff = min3(end_pgoff, vma_pages(vmf->vma) + vmf->vma->vm_pgoff - 1,
+ start_pgoff + nr_pages - 1);
+
+ if (nr_pages < vm_nr_remapping) {
+ int i, start_off = 0, end_off = 0;
+
+ lpte = vmf->pte - off;
+ for (i = 0; i < nr_pages; i++) {
+ if (!pte_none(*lpte)) {
+ is_mapped++;
+ } else {
+ if (!start_off)
+ start_off = i;
+ end_off = i;
+ }
+ lpte++;
+ }
+ if (is_mapped != nr_pages) {
+ is_mapped = 0;
+ end_pgoff = start_pgoff + end_off;
+ start_pgoff += start_off;
+ vmf->pte += start_off;
+ }
+ lpte = NULL;
+ } else {
+ lpte = vmf->pte - 1;
+ rpte = vmf->pte + 1;
+ if (!pte_none(*lpte) && !pte_none(*rpte))
+ is_mapped = 1;
+ lpte = NULL;
+ rpte = NULL;
+ }
+
+ if (!is_mapped) {
+ vmf->pte -= off;
+ vmf->vma->vm_ops->map_pages(vmf, start_pgoff, end_pgoff);
+ vmf->pte -= (vmf->address >> PAGE_SHIFT) - (address >> PAGE_SHIFT);
+ }
+
+ /* Huge page is mapped? Page fault is solved */
+ if (pmd_trans_huge(*vmf->pmd)) {
+ ret = VM_FAULT_NOPAGE;
+ goto out;
+ }
+
+ if (vmf->pte)
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
+
+out:
+ vmf->address = address;
+ vmf->pte = NULL;
+ return ret;
+}
+
static int do_read_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
@@ -3394,6 +3473,17 @@ static int do_read_fault(struct vm_fault *vmf)
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
return ret;
+ /*
+ * Remap pages after read
+ */
+ if (!(vma->vm_flags & VM_RAND_READ) && vma->vm_ops->map_pages
+ && fault_around_bytes >> PAGE_SHIFT > 1) {
+ ret |= alloc_set_pte(vmf, vmf->memcg, vmf->page);
+ unlock_page(vmf->page);
+ redo_fault_around(vmf);
+ return ret;
+ }
+
ret |= finish_fault(vmf);
unlock_page(vmf->page);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
--
2.7.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* Re: [v4 1/1] mm: Adaptive hash table scaling
From: Andi Kleen @ 2017-05-21 16:35 UTC (permalink / raw)
To: Pasha Tatashin; +Cc: Andi Kleen, akpm, linux-kernel, linux-mm, mhocko
In-Reply-To: <a09bba26-8461-653d-6b43-2df897a238f0@oracle.com>
On Sun, May 21, 2017 at 08:58:25AM -0400, Pasha Tatashin wrote:
> Hi Andi,
>
> Thank you for looking at this. I mentioned earlier, I would not want to
> impose a cap. However, if you think that for example dcache needs a cap,
> there is already a mechanism for that via high_limit argument, so the client
Lots of arguments are not the solution. Today this only affects a few
highend systems, but we'll see much more large memory systems in the
future. We don't want to have all these users either waste their memory,
or apply magic arguments.
> can be changed to provide that cap. However, this particular patch addresses
> scaling problem for everyone by making it scale with memory at a slower
> pace.
Yes your patch goes in the right direction and should be applied.
Just could be even more aggressive.
Long term probably all these hash tables need to be converted to rhash
to dynamically resize.
-Andi
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH] slub/memcg: Cure the brainless abuse of sysfs attributes
From: Thomas Gleixner @ 2017-05-21 20:01 UTC (permalink / raw)
To: Christoph Hellwig
Cc: LKML, linux-mm, Johannes Weiner, Michal Hocko, Christoph Lameter,
Andrew Morton, Steven Rostedt, Peter Zijlstra
In-Reply-To: <20170520131645.GA5058@infradead.org>
On Sat, 20 May 2017, Christoph Hellwig wrote:
> On Sat, May 20, 2017 at 12:52:03PM +0200, Thomas Gleixner wrote:
> > This should be rewritten proper by adding a propagate() callback to those
> > slub_attributes which must be propagated and avoid that insane conversion
> > to and from ASCII
>
> Exactly..
>
> >, but that's too large for a hot fix.
>
> What made this such a hot fix? Looks like this crap has been in
> for quite a while.
Well, having something in tree which uses stale or uninitialized buffers
does justify a hot fix which can be easily backported to stable.
Thanks,
tglx
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH] slub/memcg: Cure the brainless abuse of sysfs attributes
From: David Rientjes @ 2017-05-22 1:25 UTC (permalink / raw)
To: Thomas Gleixner
Cc: LKML, linux-mm, Johannes Weiner, Michal Hocko, Christoph Lameter,
Andrew Morton, Steven Rostedt, Peter Zijlstra
In-Reply-To: <alpine.DEB.2.20.1705201244540.2255@nanos>
On Sat, 20 May 2017, Thomas Gleixner wrote:
> memcg_propagate_slab_attrs() abuses the sysfs attribute file functions to
> propagate settings from the root kmem_cache to a newly created
> kmem_cache. It does that with:
>
> attr->show(root, buf);
> attr->store(new, buf, strlen(bug);
>
> Aside of being a lazy and absurd hackery this is broken because it does not
> check the return value of the show() function.
>
> Some of the show() functions return 0 w/o touching the buffer. That means in
> such a case the store function is called with the stale content of the
> previous show(). That causes nonsense like invoking kmem_cache_shrink() on
> a newly created kmem_cache. In the worst case it would cause handing in an
> uninitialized buffer.
>
> This should be rewritten proper by adding a propagate() callback to those
> slub_attributes which must be propagated and avoid that insane conversion
> to and from ASCII, but that's too large for a hot fix.
>
> Check at least the return value of the show() function, so calling store()
> with stale content is prevented.
>
> Reported-by: Steven Rostedt <rostedt@goodmis.org>
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
> Cc: stable@vger.kernel.org
Acked-by: David Rientjes <rientjes@google.com>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* [PATCH] mm, THP, swap: Check whether CONFIG_THP_SWAP enabled earlier
From: Huang, Ying @ 2017-05-22 2:18 UTC (permalink / raw)
To: Andrew Morton
Cc: linux-mm, linux-kernel, Huang Ying, Johannes Weiner, Minchan Kim
From: Huang Ying <ying.huang@intel.com>
This patch is only a code clean up patch without functionality
changes. It moves CONFIG_THP_SWAP checking from inside swap slot
allocation to before we start swapping the THP. This makes the code
path a little easier to be followed and understood.
Signed-off-by: "Huang, Ying" <ying.huang@intel.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Minchan Kim <minchan@kernel.org>
---
mm/swap_slots.c | 3 +--
mm/vmscan.c | 3 ++-
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/mm/swap_slots.c b/mm/swap_slots.c
index 90c1032a8ac3..14c2a91289e5 100644
--- a/mm/swap_slots.c
+++ b/mm/swap_slots.c
@@ -310,8 +310,7 @@ swp_entry_t get_swap_page(struct page *page)
entry.val = 0;
if (PageTransHuge(page)) {
- if (IS_ENABLED(CONFIG_THP_SWAP))
- get_swap_pages(1, true, &entry);
+ get_swap_pages(1, true, &entry);
return entry;
}
diff --git a/mm/vmscan.c b/mm/vmscan.c
index f7e949ac9756..90722afd4916 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -1134,7 +1134,8 @@ static unsigned long shrink_page_list(struct list_head *page_list,
* away. Chances are some or all of the
* tail pages can be freed without IO.
*/
- if (!compound_mapcount(page) &&
+ if ((!IS_ENABLED(CONFIG_THP_SWAP) ||
+ !compound_mapcount(page)) &&
split_huge_page_to_list(page, page_list))
goto activate_locked;
}
--
2.11.0
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* Re: [PATCH v1 00/11] mm/kasan: support per-page shadow memory to reduce memory consumption
From: Dmitry Vyukov @ 2017-05-22 6:02 UTC (permalink / raw)
To: Joonsoo Kim
Cc: Andrey Ryabinin, Andrew Morton, Alexander Potapenko, kasan-dev,
linux-mm@kvack.org, LKML, Thomas Gleixner, Ingo Molnar,
H . Peter Anvin, kernel-team
In-Reply-To: <20170519015348.GA1763@js1304-desktop>
On Fri, May 19, 2017 at 3:53 AM, Joonsoo Kim <js1304@gmail.com> wrote:
> On Wed, May 17, 2017 at 03:17:13PM +0300, Andrey Ryabinin wrote:
>> On 05/16/2017 04:16 AM, js1304@gmail.com wrote:
>> > From: Joonsoo Kim <iamjoonsoo.kim@lge.com>
>> >
>> > Hello, all.
>> >
>> > This is an attempt to recude memory consumption of KASAN. Please see
>> > following description to get the more information.
>> >
>> > 1. What is per-page shadow memory
>> >
>> > This patch introduces infrastructure to support per-page shadow memory.
>> > Per-page shadow memory is the same with original shadow memory except
>> > the granualarity. It's one byte shows the shadow value for the page.
>> > The purpose of introducing this new shadow memory is to save memory
>> > consumption.
>> >
>> > 2. Problem of current approach
>> >
>> > Until now, KASAN needs shadow memory for all the range of the memory
>> > so the amount of statically allocated memory is so large. It causes
>> > the problem that KASAN cannot run on the system with hard memory
>> > constraint. Even if KASAN can run, large memory consumption due to
>> > KASAN changes behaviour of the workload so we cannot validate
>> > the moment that we want to check.
>> >
>> > 3. How does this patch fix the problem
>> >
>> > This patch tries to fix the problem by reducing memory consumption for
>> > the shadow memory. There are two observations.
>> >
>>
>>
>> I think that the best way to deal with your problem is to increase shadow scale size.
>>
>> You'll need to add tunable to gcc to control shadow size. I expect that gcc has some
>> places where 8-shadow scale size is hardcoded, but it should be fixable.
>>
>> The kernel also have some small amount of code written with KASAN_SHADOW_SCALE_SIZE == 8 in mind,
>> which should be easy to fix.
>>
>> Note that bigger shadow scale size requires bigger alignment of allocated memory and variables.
>> However, according to comments in gcc/asan.c gcc already aligns stack and global variables and at
>> 32-bytes boundary.
>> So we could bump shadow scale up to 32 without increasing current stack consumption.
>>
>> On a small machine (1Gb) 1/32 of shadow is just 32Mb which is comparable to yours 30Mb, but I expect it to be
>> much faster. More importantly, this will require only small amount of simple changes in code, which will be
>> a *lot* more easier to maintain.
Interesting option. We never considered increasing scale in user space
due to performance implications. But the algorithm always supported up
to 128x scale. Definitely worth considering as an option.
> I agree that it is also a good option to reduce memory consumption.
> Nevertheless, there are two reasons that justifies this patchset.
>
> 1) With this patchset, memory consumption isn't increased in
> proportional to total memory size. Please consider my 4Gb system
> example on the below. With increasing shadow scale size to 32, memory
> would be consumed by 128M. However, this patchset consumed 50MB. This
> difference can be larger if we run KASAN with bigger machine.
>
> 2) These two optimization can be applied simulatenously. It is just an
> orthogonal feature. If shadow scale size is increased to 32, memory
> consumption will be decreased in case of my patchset, too.
>
> Therefore, I think that this patchset is useful in any case.
It is definitely useful all else being equal. But it does considerably
increase code size and complexity, which is an important aspect.
Also note that there is also fixed size quarantine (1/32 of RAM) and
redzones. Reducing shadow overhead beyond some threshold has
diminishing returns, because overall overhead will be just dominated
by quarantine/redzones.
What's your target devices and constraints? We run KASAN on phones
today without any issues.
> Note that increasing shadow scale has it's own trade-off. It requires
> that the size of slab object is aligned to shadow scale. It will
> increase memory consumption due to slab.
I've tried to retest your latest change on top of
http://git.cmpxchg.org/cgit.cgi/linux-mmots.git
d9cd9c95cc3b2fed0f04d233ebf2f7056741858c, but now this version
https://codereview.appspot.com/325780043 always crashes during boot
for me. Report points to zero shadow.
[ 0.123434] ==================================================================
[ 0.125153] BUG: KASAN: double-free or invalid-free in
cleanup_uevent_env+0x2c/0x40
[ 0.126900]
[ 0.127318] CPU: 1 PID: 226 Comm: kworker/u8:0 Not tainted
4.12.0-rc1-mm1+ #376
[ 0.128995] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS Bochs 01/01/2011
[ 0.130896] Call Trace:
[ 0.131202] kworker/u8:0 (277) used greatest stack depth: 22976 bytes left
[ 0.133129] dump_stack+0xb0/0x13d
[ 0.133958] ? _atomic_dec_and_lock+0x1e3/0x1e3
[ 0.135020] ? load_image_and_restore+0xf6/0xf6
[ 0.136083] ? kmemdup+0x31/0x40
[ 0.136143] kworker/u8:0 (320) used greatest stack depth: 22112 bytes left
[ 0.138294] ? cleanup_uevent_env+0x2c/0x40
[ 0.139255] print_address_description+0x6a/0x270
[ 0.140285] ? cleanup_uevent_env+0x2c/0x40
[ 0.141224] ? cleanup_uevent_env+0x2c/0x40
[ 0.142168] kasan_report_double_free+0x55/0x80
[ 0.143162] kasan_slab_free+0xa4/0xc0
[ 0.143934] ? cleanup_uevent_env+0x2c/0x40
[ 0.144882] kfree+0x8f/0x190
[ 0.145561] cleanup_uevent_env+0x2c/0x40
[ 0.146455] umh_complete+0x3c/0x60
[ 0.147180] call_usermodehelper_exec_async+0x671/0x950
[ 0.148334] ? __asan_report_store_n_noabort+0x12/0x20
[ 0.149460] ? native_load_sp0+0xa3/0xb0
[ 0.150213] ? umh_complete+0x60/0x60
[ 0.150990] ? kasan_end_report+0x20/0x50
[ 0.151829] ? finish_task_switch+0x510/0x7d0
[ 0.152760] ? copy_user_overflow+0x20/0x20
[ 0.153565] ? umh_complete+0x60/0x60
[ 0.154341] ? umh_complete+0x60/0x60
[ 0.155125] ret_from_fork+0x2c/0x40
[ 0.155888]
[ 0.156190] Allocated by task 1:
[ 0.156890] save_stack_trace+0x16/0x20
[ 0.157629] save_stack+0x43/0xd0
[ 0.158299] kasan_kmalloc+0xad/0xe0
[ 0.159068] kmem_cache_alloc_trace+0x61/0x170
[ 0.159920] kobject_uevent_env+0x1b2/0xa20
[ 0.160819] kobject_uevent+0xb/0x10
[ 0.161551] param_sysfs_init+0x28e/0x2d2
[ 0.162375] do_one_initcall+0x8c/0x290
[ 0.163083] kernel_init_freeable+0x4a2/0x554
[ 0.163958] kernel_init+0xe/0x120
[ 0.164669] ret_from_fork+0x2c/0x40
[ 0.165393]
[ 0.165685] Freed by task 0:
[ 0.166232] (stack is not available)
[ 0.166954]
[ 0.167247] The buggy address belongs to the object at ffff88007b45e818
[ 0.167247] which belongs to the cache kmalloc-4096 of size 4096
[ 0.169709] The buggy address is located 0 bytes inside of
[ 0.169709] 4096-byte region [ffff88007b45e818, ffff88007b45f818)
[ 0.171897] The buggy address belongs to the page:
[ 0.172833] page:ffffea0001ed1600 count:1 mapcount:0 mapping:
(null) index:0x0 compound_mapcount: 0
[ 0.174560] flags: 0x100000000008100(slab|head)
[ 0.175410] raw: 0100000000008100 0000000000000000 0000000000000000
0000000100070007
[ 0.176819] raw: ffffea0001ed0c20 ffffea0001ed3c20 ffff88007c80ed40
0000000000000000
[ 0.178250] page dumped because: kasan: bad access detected
[ 0.179312]
[ 0.179586] Memory state around the buggy address:
[ 0.180488] ffff88007b45e700: fc fc fc fc fc fc fc fc fc fc fc fc
fc fc fc fc
[ 0.181801] ffff88007b45e780: fc fc fc fc fc fc fc fc fc fc fc fc
fc fc fc fc
[ 0.183112] >ffff88007b45e800: fc fc fc 00 00 00 00 00 00 00 00 00
00 00 00 00
[ 0.184518] ^
[ 0.185177] ffff88007b45e880: 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00
[ 0.186420] ffff88007b45e900: 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00
[ 0.187723] ==================================================================
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* [PATCH] mm: introduce MADV_CLR_HUGEPAGE
From: Mike Rapoport @ 2017-05-22 6:12 UTC (permalink / raw)
To: Andrew Morton
Cc: Arnd Bergmann, Kirill A. Shutemov, Andrea Arcangeli, linux-mm,
lkml, Mike Rapoport
Currently applications can explicitly enable or disable THP for a memory
region using MADV_HUGEPAGE or MADV_NOHUGEPAGE. However, once either of
these advises is used, the region will always have
VM_HUGEPAGE/VM_NOHUGEPAGE flag set in vma->vm_flags.
The MADV_CLR_HUGEPAGE resets both these flags and allows managing THP in
the region according to system-wide settings.
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
---
include/uapi/asm-generic/mman-common.h | 3 +++
mm/khugepaged.c | 7 +++++++
mm/madvise.c | 5 +++++
3 files changed, 15 insertions(+)
diff --git a/include/uapi/asm-generic/mman-common.h b/include/uapi/asm-generic/mman-common.h
index 8c27db0..3201712 100644
--- a/include/uapi/asm-generic/mman-common.h
+++ b/include/uapi/asm-generic/mman-common.h
@@ -58,6 +58,9 @@
overrides the coredump filter bits */
#define MADV_DODUMP 17 /* Clear the MADV_DONTDUMP flag */
+#define MADV_CLR_HUGEPAGE 18 /* Clear flags controlling backing with
+ hugepages */
+
/* compatibility flags */
#define MAP_FILE 0
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 945fd1c..b9ee9bb 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -336,6 +336,13 @@ int hugepage_madvise(struct vm_area_struct *vma,
* it got registered before VM_NOHUGEPAGE was set.
*/
break;
+ case MADV_CLR_HUGEPAGE:
+ *vm_flags &= ~(VM_HUGEPAGE | VM_NOHUGEPAGE);
+ /*
+ * The vma will be treated according to the
+ * system-wide settings in transparent_hugepage_flags
+ */
+ break;
}
return 0;
diff --git a/mm/madvise.c b/mm/madvise.c
index 25b78ee..ae650a3 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -105,6 +105,7 @@ static long madvise_behavior(struct vm_area_struct *vma,
break;
case MADV_HUGEPAGE:
case MADV_NOHUGEPAGE:
+ case MADV_CLR_HUGEPAGE:
error = hugepage_madvise(vma, &new_flags, behavior);
if (error) {
/*
@@ -684,6 +685,7 @@ madvise_behavior_valid(int behavior)
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
case MADV_HUGEPAGE:
case MADV_NOHUGEPAGE:
+ case MADV_CLR_HUGEPAGE:
#endif
case MADV_DONTDUMP:
case MADV_DODUMP:
@@ -739,6 +741,9 @@ madvise_behavior_valid(int behavior)
* MADV_NOHUGEPAGE - mark the given range as not worth being backed by
* transparent huge pages so the existing pages will not be
* coalesced into THP and new pages will not be allocated as THP.
+ * MADV_CLR_HUGEPAGE - clear MADV_HUGEPAGE/MADV_NOHUGEPAGE marking;
+ * the range will be treated by khugepaged according to the
+ * system wide settings
* MADV_DONTDUMP - the application wants to prevent pages in the given range
* from being included in its core dump.
* MADV_DODUMP - cancel MADV_DONTDUMP: no longer exclude from core dump.
--
2.7.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* Re: [v4 1/1] mm: Adaptive hash table scaling
From: Michael Ellerman @ 2017-05-22 6:17 UTC (permalink / raw)
To: Pavel Tatashin, akpm, linux-kernel, linux-mm, mhocko
In-Reply-To: <1495300013-653283-2-git-send-email-pasha.tatashin@oracle.com>
Pavel Tatashin <pasha.tatashin@oracle.com> writes:
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 8afa63e81e73..15bba5c325a5 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -7169,6 +7169,17 @@ static unsigned long __init arch_reserved_kernel_pages(void)
> #endif
>
> /*
> + * Adaptive scale is meant to reduce sizes of hash tables on large memory
> + * machines. As memory size is increased the scale is also increased but at
> + * slower pace. Starting from ADAPT_SCALE_BASE (64G), every time memory
> + * quadruples the scale is increased by one, which means the size of hash table
> + * only doubles, instead of quadrupling as well.
> + */
> +#define ADAPT_SCALE_BASE (64ull << 30)
> +#define ADAPT_SCALE_SHIFT 2
> +#define ADAPT_SCALE_NPAGES (ADAPT_SCALE_BASE >> PAGE_SHIFT)
> +
> +/*
> * allocate a large system hash table from bootmem
> * - it is assumed that the hash table must contain an exact power-of-2
> * quantity of entries
> @@ -7199,6 +7210,14 @@ void *__init alloc_large_system_hash(const char *tablename,
> if (PAGE_SHIFT < 20)
> numentries = round_up(numentries, (1<<20)/PAGE_SIZE);
>
> + if (!high_limit) {
> + unsigned long long adapt;
> +
> + for (adapt = ADAPT_SCALE_NPAGES; adapt < numentries;
> + adapt <<= ADAPT_SCALE_SHIFT)
> + scale++;
> + }
This still doesn't work for me. The scale++ is overflowing according to
UBSAN (line 7221).
It looks like numentries is 194560.
00000950 68 0a 50 49 44 20 68 61 73 68 20 74 61 62 6c 65 |h.PID hash table|
00000960 20 65 6e 74 72 69 65 73 3a 20 34 30 39 36 20 28 | entries: 4096 (|
00000970 6f 72 64 65 72 3a 20 32 2c 20 31 36 33 38 34 20 |order: 2, 16384 |
00000980 62 79 74 65 73 29 0a 61 6c 6c 6f 63 5f 6c 61 72 |bytes).alloc_lar|
00000990 67 65 5f 73 79 73 74 65 6d 5f 68 61 73 68 3a 20 |ge_system_hash: |
000009a0 6e 75 6d 65 6e 74 72 69 65 73 20 31 39 34 35 36 |numentries 19456|
000009b0 30 0a 61 6c 6c 6f 63 5f 6c 61 72 67 65 5f 73 79 |0.alloc_large_sy|
000009c0 73 74 65 6d 5f 68 61 73 68 3a 20 61 64 61 70 74 |stem_hash: adapt|
000009d0 20 30 0a 3d 3d 3d 3d 3d 3d 3d 3d 3d 3d 3d 3d 3d | 0.=============|
000009e0 3d 3d 3d 3d 3d 3d 3d 3d 3d 3d 3d 3d 3d 3d 3d 3d |================|
*
00000a20 3d 3d 3d 0a 55 42 53 41 4e 3a 20 55 6e 64 65 66 |===.UBSAN: Undef|
00000a30 69 6e 65 64 20 62 65 68 61 76 69 6f 75 72 20 69 |ined behaviour i|
00000a40 6e 20 2e 2e 2f 6d 6d 2f 70 61 67 65 5f 61 6c 6c |n ../mm/page_all|
00000a50 6f 63 2e 63 3a 37 32 32 31 3a 31 30 0a 73 69 67 |oc.c:7221:10.sig|
00000a60 6e 65 64 20 69 6e 74 65 67 65 72 20 6f 76 65 72 |ned integer over|
00000a70 66 6c 6f 77 3a 0a 32 31 34 37 34 38 33 36 34 37 |flow:.2147483647|
00000a80 20 2b 20 31 20 63 61 6e 6e 6f 74 20 62 65 20 72 | + 1 cannot be r|
00000a90 65 70 72 65 73 65 6e 74 65 64 20 69 6e 20 74 79 |epresented in ty|
00000aa0 70 65 20 27 69 6e 74 20 5b 34 5d 27 0a 43 50 55 |pe 'int [4]'.CPU|
00000ab0 3a 20 30 20 50 49 44 3a 20 30 20 43 6f 6d 6d 3a |: 0 PID: 0 Comm:|
00000ac0 20 73 77 61 70 70 65 72 20 4e 6f 74 20 74 61 69 | swapper Not tai|
00000ad0 6e 74 65 64 20 34 2e 31 32 2e 30 2d 72 63 31 2d |nted 4.12.0-rc1-|
00000ae0 67 63 63 2d 36 2e 33 2e 31 2d 30 30 31 38 32 2d |gcc-6.3.1-00182-|
00000af0 67 36 37 64 30 36 38 37 32 32 34 61 39 2d 64 69 |g67d0687224a9-di|
00000b00 72 74 79 20 23 38 0a 43 61 6c 6c 20 54 72 61 63 |rty #8.Call Trac|
00000b10 65 3a 0a 5b 63 30 65 30 35 65 61 30 5d 20 5b 63 |e:.[c0e05ea0] [c|
00000b20 30 34 37 38 38 63 34 5d 20 75 62 73 61 6e 5f 65 |04788c4] ubsan_e|
00000b30 70 69 6c 6f 67 75 65 2b 30 78 31 38 2f 30 78 34 |pilogue+0x18/0x4|
00000b40 63 20 28 75 6e 72 65 6c 69 61 62 6c 65 29 0a 5b |c (unreliable).[|
00000b50 63 30 65 30 35 65 62 30 5d 20 5b 63 30 34 37 39 |c0e05eb0] [c0479|
00000b60 32 36 30 5d 20 68 61 6e 64 6c 65 5f 6f 76 65 72 |260] handle_over|
00000b70 66 6c 6f 77 2b 30 78 62 63 2f 30 78 64 63 0a 5b |flow+0xbc/0xdc.[|
00000b80 63 30 65 30 35 66 33 30 5d 20 5b 63 30 61 62 39 |c0e05f30] [c0ab9|
00000b90 38 66 38 5d 20 61 6c 6c 6f 63 5f 6c 61 72 67 65 |8f8] alloc_large|
00000ba0 5f 73 79 73 74 65 6d 5f 68 61 73 68 2b 30 78 65 |_system_hash+0xe|
00000bb0 34 2f 30 78 35 65 63 0a 5b 63 30 65 30 35 66 39 |4/0x5ec.[c0e05f9|
00000bc0 30 5d 20 5b 63 30 61 62 65 30 30 30 5d 20 76 66 |0] [c0abe000] vf|
00000bd0 73 5f 63 61 63 68 65 73 5f 69 6e 69 74 5f 65 61 |s_caches_init_ea|
00000be0 72 6c 79 2b 30 78 34 63 2f 30 78 36 34 0a 5b 63 |rly+0x4c/0x64.[c|
00000bf0 30 65 30 35 66 62 30 5d 20 5b 63 30 61 61 35 32 |0e05fb0] [c0aa52|
00000c00 31 38 5d 20 73 74 61 72 74 5f 6b 65 72 6e 65 6c |18] start_kernel|
00000c10 2b 30 78 32 33 63 2f 30 78 33 63 34 0a 5b 63 30 |+0x23c/0x3c4.[c0|
00000c20 65 30 35 66 66 30 5d 20 5b 30 30 30 30 33 34 34 |e05ff0] [0000344|
00000c30 63 5d 20 30 78 33 34 34 63 0a 3d 3d 3d 3d 3d 3d |c] 0x344c.======|
00000c40 3d 3d 3d 3d 3d 3d 3d 3d 3d 3d 3d 3d 3d 3d 3d 3d |================|
cheers
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH] mm: introduce MADV_CLR_HUGEPAGE
From: Anshuman Khandual @ 2017-05-22 7:26 UTC (permalink / raw)
To: Mike Rapoport, Andrew Morton
Cc: Arnd Bergmann, Kirill A. Shutemov, Andrea Arcangeli, linux-mm,
lkml
In-Reply-To: <1495433562-26625-1-git-send-email-rppt@linux.vnet.ibm.com>
On 05/22/2017 11:42 AM, Mike Rapoport wrote:
> Currently applications can explicitly enable or disable THP for a memory
> region using MADV_HUGEPAGE or MADV_NOHUGEPAGE. However, once either of
> these advises is used, the region will always have
> VM_HUGEPAGE/VM_NOHUGEPAGE flag set in vma->vm_flags.
> The MADV_CLR_HUGEPAGE resets both these flags and allows managing THP in
> the region according to system-wide settings.
Invoking madvise() for the first time with either MADV_HUGEPAGE or
MADV_NOHUGEPAGE on the buffer will unsubscribe it from the system
wide behavior for good. I am not saying we should not have a way
to put it back into system wide mode but are there no other functions
through madvise() or any other interface which may have the same
situation.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH] mm: introduce MADV_CLR_HUGEPAGE
From: Mike Rapoport @ 2017-05-22 8:12 UTC (permalink / raw)
To: Anshuman Khandual
Cc: Andrew Morton, Arnd Bergmann, Kirill A. Shutemov,
Andrea Arcangeli, linux-mm, lkml
In-Reply-To: <8b21bb9a-4efc-288b-933d-be7e6a5e4a0a@linux.vnet.ibm.com>
On Mon, May 22, 2017 at 12:56:45PM +0530, Anshuman Khandual wrote:
> On 05/22/2017 11:42 AM, Mike Rapoport wrote:
> > Currently applications can explicitly enable or disable THP for a memory
> > region using MADV_HUGEPAGE or MADV_NOHUGEPAGE. However, once either of
> > these advises is used, the region will always have
> > VM_HUGEPAGE/VM_NOHUGEPAGE flag set in vma->vm_flags.
> > The MADV_CLR_HUGEPAGE resets both these flags and allows managing THP in
> > the region according to system-wide settings.
>
> Invoking madvise() for the first time with either MADV_HUGEPAGE or
> MADV_NOHUGEPAGE on the buffer will unsubscribe it from the system
> wide behavior for good. I am not saying we should not have a way
> to put it back into system wide mode but are there no other functions
> through madvise() or any other interface which may have the same
> situation.
There are madvise() interfaces that set or clear some of the vma->vm_flags,
e.g MADV_*FORK or MADV_*DUMP. The difference with MADV_*HUGEPAGE is that
it is using two flags and with current madvise() interface either of them
has to be set, but there is no interface to clear them both.
--
Sincerely yours,
Mike.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH] mm/oom_kill: count global and memory cgroup oom kills
From: Konstantin Khlebnikov @ 2017-05-22 9:11 UTC (permalink / raw)
To: Roman Guschin
Cc: linux-mm, Andrew Morton, Tejun Heo, cgroups, linux-kernel,
Vlastimil Babka, Michal Hocko, hannes
In-Reply-To: <CALo0P1123MROxgveCdX6YFpWDwG4qrAyHu3Xd1F+ckaFBnF4dQ@mail.gmail.com>
On 19.05.2017 19:34, Roman Guschin wrote:
> 2017-05-19 15:22 GMT+01:00 Konstantin Khlebnikov <khlebnikov@yandex-team.ru>:
>> Show count of global oom killer invocations in /proc/vmstat and
>> count of oom kills inside memory cgroup in knob "memory.events"
>> (in memory.oom_control for v1 cgroup).
>>
>> Also describe difference between "oom" and "oom_kill" in memory
>> cgroup documentation. Currently oom in memory cgroup kills tasks
>> iff shortage has happened inside page fault.
>>
>> These counters helps in monitoring oom kills - for now
>> the only way is grepping for magic words in kernel log.
>>
>> Signed-off-by: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>
>> ---
>> Documentation/cgroup-v2.txt | 12 +++++++++++-
>> include/linux/memcontrol.h | 1 +
>> include/linux/vm_event_item.h | 1 +
>> mm/memcontrol.c | 2 ++
>> mm/oom_kill.c | 6 ++++++
>> mm/vmstat.c | 1 +
>> 6 files changed, 22 insertions(+), 1 deletion(-)
>>
>> diff --git a/Documentation/cgroup-v2.txt b/Documentation/cgroup-v2.txt
>> index dc5e2dcdbef4..a742008d76aa 100644
>> --- a/Documentation/cgroup-v2.txt
>> +++ b/Documentation/cgroup-v2.txt
>> @@ -830,9 +830,19 @@ PAGE_SIZE multiple when read back.
>>
>> oom
>>
>> + The number of time the cgroup's memory usage was
>> + reached the limit and allocation was about to fail.
>> + Result could be oom kill, -ENOMEM from any syscall or
>> + completely ignored in cases like disk readahead.
>> + For now oom in memory cgroup kills tasks iff shortage
>> + has happened inside page fault.
>
> From a user's point of view the difference between "oom" and "max"
> becomes really vague here,
> assuming that "max" is described almost in the same words:
>
> "The number of times the cgroup's memory usage was
> about to go over the max boundary. If direct reclaim
> fails to bring it down, the OOM killer is invoked."
>
> I wonder, if it's better to fix the existing "oom" value to show what
> it has to show, according to docs,
> rather than to introduce a new one?
>
Nope, they are different. I think we should rephase documentation somehow
low - count of reclaims below low level
high - count of post-allocation reclaims above high level
max - count of direct reclaims
oom - count of failed direct reclaims
oom_kill - count of oom killer invocations and killed processes
>> +
>> + oom_kill
>> +
>> The number of times the OOM killer has been invoked in
>> the cgroup. This may not exactly match the number of
>> - processes killed but should generally be close.
>> + processes killed but should generally be close: each
>> + invocation could kill several processes at once.
>>
>> memory.stat
>>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [v4 1/1] mm: Adaptive hash table scaling
From: Michal Hocko @ 2017-05-22 9:29 UTC (permalink / raw)
To: Pavel Tatashin; +Cc: akpm, linux-kernel, linux-mm
In-Reply-To: <1495300013-653283-2-git-send-email-pasha.tatashin@oracle.com>
On Sat 20-05-17 13:06:53, Pavel Tatashin wrote:
[...]
> /*
> + * Adaptive scale is meant to reduce sizes of hash tables on large memory
> + * machines. As memory size is increased the scale is also increased but at
> + * slower pace. Starting from ADAPT_SCALE_BASE (64G), every time memory
> + * quadruples the scale is increased by one, which means the size of hash table
> + * only doubles, instead of quadrupling as well.
> + */
> +#define ADAPT_SCALE_BASE (64ull << 30)
I have only noticed this email today because my incoming emails stopped
syncing since Friday. But this is _definitely_ not the right approachh.
64G for 32b systems is _way_ off. We have only ~1G for the kernel. I've
already proposed scaling up to 32M for 32b systems and Andi seems to be
suggesting the same. So can we fold or apply the following instead?
---
^ permalink raw reply
* Re: [RFC PATCH 2/2] mm, oom: do not trigger out_of_memory from the #PF
From: Michal Hocko @ 2017-05-22 9:31 UTC (permalink / raw)
To: Tetsuo Handa; +Cc: akpm, hannes, guro, vdavydov.dev, linux-mm, linux-kernel
In-Reply-To: <201705200843.HAI95393.FQSFLOHVMJtOFO@I-love.SAKURA.ne.jp>
On Sat 20-05-17 08:43:29, Tetsuo Handa wrote:
> Michal Hocko wrote:
[...]
> > Why would looping inside an allocator with a restricted context be any
> > better than retrying the whole thing?
>
> I'm not suggesting you to loop inside an allocator nor retry the whole thing.
> I'm suggesting you to avoid returning VM_FAULT_OOM by making allocations succeed
> (by e.g. calling oom_kill_process()) regardless of restricted context if you
> want to remove out_of_memory() from pagefault_out_of_memory(), for situation
> will not improve until memory is allocated (e.g. somebody else calls
> oom_kill_process() via a __GFP_FS allocation request).
And again for the hundred and so many times I will only repeat that
triggering OOM from those restricted contexts is just too dangerous
without other changes.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH] dm ioctl: Restore __GFP_HIGH in copy_params()
From: Michal Hocko @ 2017-05-22 9:37 UTC (permalink / raw)
To: Mikulas Patocka
Cc: Junaid Shahid, David Rientjes, Alasdair Kergon, Mike Snitzer,
Andrew Morton, linux-mm, andreslc, gthelen, vbabka, linux-kernel
In-Reply-To: <alpine.LRH.2.02.1705191934340.17646@file01.intranet.prod.int.rdu2.redhat.com>
On Fri 19-05-17 19:43:23, Mikulas Patocka wrote:
>
>
> On Fri, 19 May 2017, Michal Hocko wrote:
>
> > On Thu 18-05-17 19:50:46, Junaid Shahid wrote:
> > > (Adding back the correct linux-mm email address and also adding linux-kernel.)
> > >
> > > On Thursday, May 18, 2017 01:41:33 PM David Rientjes wrote:
> > [...]
> > > > Let's ask Mikulas, who changed this from PF_MEMALLOC to __GFP_HIGH,
> > > > assuming there was a reason to do it in the first place in two different
> > > > ways.
> >
> > Hmm, the old PF_MEMALLOC used to have the following comment
> > /*
> > * Trying to avoid low memory issues when a device is
> > * suspended.
> > */
> >
> > I am not really sure what that means but __GFP_HIGH certainly have a
> > different semantic than PF_MEMALLOC. The later grants the full access to
> > the memory reserves while the prior on partial access. If this is _really_
> > needed then it deserves a comment explaining why.
> > --
> > Michal Hocko
> > SUSE Labs
>
> Sometimes, I/O to a device mapper device is blocked until the userspace
> daemon dmeventd does some action (for example, when dm-mirror leg fails,
> dmeventd needs to mark the leg as failed in the lvm metadata and then
> reload the device).
>
> The dmeventd daemon mlocks itself in memory so that it doesn't generate
> any I/O. But it must be able to call ioctls. __GFP_HIGH is there so that
> the ioctls issued by dmeventd have higher chance of succeeding if some I/O
> is blocked, waiting for dmeventd action. It reduces the possibility of
> low-memory-deadlock, though it doesn't eliminate it entirely.
So what happens if the memory reserves are depleted. Do we deadlock? Why
is OOM killer insufficient to allow the further progress?
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: mm, something wring in page_lock_anon_vma_read()?
From: Xishi Qiu @ 2017-05-22 9:48 UTC (permalink / raw)
To: Hugh Dickins
Cc: Andrew Morton, Tejun Heo, Michal Hocko, Johannes Weiner,
Mel Gorman, Michal Hocko, Vlastimil Babka, Minchan Kim,
David Rientjes, Joonsoo Kim, aarcange, sumeet.keswani,
Rik van Riel, Linux MM, LKML, zhong jiang
In-Reply-To: <alpine.LSU.2.11.1705191935220.11750@eggly.anvils>
On 2017/5/20 10:40, Hugh Dickins wrote:
> On Sat, 20 May 2017, Xishi Qiu wrote:
>>
>> Here is a bug report form redhat: https://bugzilla.redhat.com/show_bug.cgi?id=1305620
>> And I meet the bug too. However it is hard to reproduce, and
>> 624483f3ea82598("mm: rmap: fix use-after-free in __put_anon_vma") is not help.
>>
>> From the vmcore, it seems that the page is still mapped(_mapcount=0 and _count=2),
>> and the value of mapping is a valid address(mapping = 0xffff8801b3e2a101),
>> but anon_vma has been corrupted.
>>
>> Any ideas?
>
> Sorry, no. I assume that _mapcount has been misaccounted, for example
> a pte mapped in on top of another pte; but cannot begin tell you where
Hi Hugh,
What does "a pte mapped in on top of another pte" mean? Could you give more info?
Thanks,
Xishi Qiu
> in Red Hat's kernel-3.10.0-229.4.2.el7 that might happen.
>
> Hugh
>
> .
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* [PATCH] mm: Define KB, MB, GB, TB in core VM
From: Anshuman Khandual @ 2017-05-22 11:17 UTC (permalink / raw)
To: linux-mm, linux-kernel; +Cc: akpm
There are many places where we define size either left shifting integers
or multiplying 1024s without any generic definition to fall back on. But
there are couples of (powerpc and lz4) attempts to define these standard
memory sizes. Lets move these definitions to core VM to make sure that
all new usage come from these definitions eventually standardizing it
across all places.
Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
---
arch/powerpc/mm/hash_utils_64.c | 4 ----
include/linux/mm.h | 5 +++++
lib/lz4/lz4defs.h | 5 +----
3 files changed, 6 insertions(+), 8 deletions(-)
diff --git a/arch/powerpc/mm/hash_utils_64.c b/arch/powerpc/mm/hash_utils_64.c
index f2095ce..ef64040 100644
--- a/arch/powerpc/mm/hash_utils_64.c
+++ b/arch/powerpc/mm/hash_utils_64.c
@@ -74,10 +74,6 @@
#define DBG_LOW(fmt...)
#endif
-#define KB (1024)
-#define MB (1024*KB)
-#define GB (1024L*MB)
-
/*
* Note: pte --> Linux PTE
* HPTE --> PowerPC Hashed Page Table Entry
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 7cb17c6..9f5779f 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2549,5 +2549,10 @@ static inline bool page_is_guard(struct page *page)
static inline void setup_nr_node_ids(void) {}
#endif
+#define KB (1UL << 10)
+#define MB (1UL << 20)
+#define GB (1UL << 30)
+#define TB (1UL << 40)
+
#endif /* __KERNEL__ */
#endif /* _LINUX_MM_H */
diff --git a/lib/lz4/lz4defs.h b/lib/lz4/lz4defs.h
index 00a0b58..67a0f6d 100644
--- a/lib/lz4/lz4defs.h
+++ b/lib/lz4/lz4defs.h
@@ -37,6 +37,7 @@
#include <asm/unaligned.h>
#include <linux/string.h> /* memset, memcpy */
+#include <linux/mm.h>
#define FORCE_INLINE __always_inline
@@ -81,10 +82,6 @@
#define HASH_UNIT sizeof(size_t)
-#define KB (1 << 10)
-#define MB (1 << 20)
-#define GB (1U << 30)
-
#define MAXD_LOG 16
#define MAX_DISTANCE ((1 << MAXD_LOG) - 1)
#define STEPSIZE sizeof(size_t)
--
1.8.5.2
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* Re: [PATCH] mm: introduce MADV_CLR_HUGEPAGE
From: Kirill A. Shutemov @ 2017-05-22 11:42 UTC (permalink / raw)
To: Mike Rapoport
Cc: Andrew Morton, Arnd Bergmann, Kirill A. Shutemov,
Andrea Arcangeli, linux-mm, lkml
In-Reply-To: <1495433562-26625-1-git-send-email-rppt@linux.vnet.ibm.com>
On Mon, May 22, 2017 at 09:12:42AM +0300, Mike Rapoport wrote:
> Currently applications can explicitly enable or disable THP for a memory
> region using MADV_HUGEPAGE or MADV_NOHUGEPAGE. However, once either of
> these advises is used, the region will always have
> VM_HUGEPAGE/VM_NOHUGEPAGE flag set in vma->vm_flags.
> The MADV_CLR_HUGEPAGE resets both these flags and allows managing THP in
> the region according to system-wide settings.
Seems reasonable. But could you describe an use-case when it's useful in
real world.
And the name is bad. But I don't have better suggestion. At least do not
abbreviate CLEAR. Saving two letters doesn't worth it.
Maybe RESET instead?
--
Kirill A. Shutemov
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH] dm ioctl: Restore __GFP_HIGH in copy_params()
From: Mikulas Patocka @ 2017-05-22 12:00 UTC (permalink / raw)
To: Michal Hocko
Cc: Junaid Shahid, David Rientjes, Alasdair Kergon, Mike Snitzer,
Andrew Morton, linux-mm, andreslc, gthelen, vbabka, linux-kernel
In-Reply-To: <20170522093725.GF8509@dhcp22.suse.cz>
On Mon, 22 May 2017, Michal Hocko wrote:
> On Fri 19-05-17 19:43:23, Mikulas Patocka wrote:
> >
> >
> > On Fri, 19 May 2017, Michal Hocko wrote:
> >
> > > On Thu 18-05-17 19:50:46, Junaid Shahid wrote:
> > > > (Adding back the correct linux-mm email address and also adding linux-kernel.)
> > > >
> > > > On Thursday, May 18, 2017 01:41:33 PM David Rientjes wrote:
> > > [...]
> > > > > Let's ask Mikulas, who changed this from PF_MEMALLOC to __GFP_HIGH,
> > > > > assuming there was a reason to do it in the first place in two different
> > > > > ways.
> > >
> > > Hmm, the old PF_MEMALLOC used to have the following comment
> > > /*
> > > * Trying to avoid low memory issues when a device is
> > > * suspended.
> > > */
> > >
> > > I am not really sure what that means but __GFP_HIGH certainly have a
> > > different semantic than PF_MEMALLOC. The later grants the full access to
> > > the memory reserves while the prior on partial access. If this is _really_
> > > needed then it deserves a comment explaining why.
> > > --
> > > Michal Hocko
> > > SUSE Labs
> >
> > Sometimes, I/O to a device mapper device is blocked until the userspace
> > daemon dmeventd does some action (for example, when dm-mirror leg fails,
> > dmeventd needs to mark the leg as failed in the lvm metadata and then
> > reload the device).
> >
> > The dmeventd daemon mlocks itself in memory so that it doesn't generate
> > any I/O. But it must be able to call ioctls. __GFP_HIGH is there so that
> > the ioctls issued by dmeventd have higher chance of succeeding if some I/O
> > is blocked, waiting for dmeventd action. It reduces the possibility of
> > low-memory-deadlock, though it doesn't eliminate it entirely.
>
> So what happens if the memory reserves are depleted. Do we deadlock?
Yes, it will deadlock.
> Why is OOM killer insufficient to allow the further progress?
I don't know if the OOM killer will or won't be triggered in this
situation, it depends on the people who wrote the OOM killer.
> --
> Michal Hocko
> SUSE Labs
Mikulas
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox