* Re: [PATCH] mm: per-cgroup memory reclaim stats
From: Michal Hocko @ 2017-05-19 10:47 UTC (permalink / raw)
To: Roman Gushchin
Cc: Johannes Weiner, Tejun Heo, Li Zefan, Vladimir Davydov, cgroups,
linux-doc, linux-kernel, linux-mm
In-Reply-To: <1494530183-30808-1-git-send-email-guro@fb.com>
On Thu 11-05-17 20:16:23, Roman Gushchin wrote:
> Track the following reclaim counters for every memory cgroup:
> PGREFILL, PGSCAN, PGSTEAL, PGACTIVATE, PGDEACTIVATE, PGLAZYFREE and
> PGLAZYFREED.
>
> These values are exposed using the memory.stats interface of cgroup v2.
>
> The meaning of each value is the same as for global counters,
> available using /proc/vmstat.
>
> Also, for consistency, rename mem_cgroup_count_vm_event() to
> count_memcg_event_mm().
>
> Signed-off-by: Roman Gushchin <guro@fb.com>
> Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
> Cc: Johannes Weiner <hannes@cmpxchg.org>
> Cc: Tejun Heo <tj@kernel.org>
> Cc: Li Zefan <lizefan@huawei.com>
> Cc: Michal Hocko <mhocko@kernel.org>
> Cc: Vladimir Davydov <vdavydov.dev@gmail.com>
> Cc: cgroups@vger.kernel.org
> Cc: linux-doc@vger.kernel.org
> Cc: linux-kernel@vger.kernel.org
> Cc: linux-mm@kvack.org
Acked-by: Michal Hocko <mhocko@suse.com>
Thanks!
> ---
> Documentation/cgroup-v2.txt | 28 ++++++++++++++++++++++++++
> fs/dax.c | 2 +-
> fs/ncpfs/mmap.c | 2 +-
> include/linux/memcontrol.h | 48 ++++++++++++++++++++++++++++++++++++++++++---
> mm/filemap.c | 2 +-
> mm/memcontrol.c | 10 ++++++++++
> mm/memory.c | 4 ++--
> mm/shmem.c | 3 +--
> mm/swap.c | 1 +
> mm/vmscan.c | 30 +++++++++++++++++++++-------
> 10 files changed, 113 insertions(+), 17 deletions(-)
>
> diff --git a/Documentation/cgroup-v2.txt b/Documentation/cgroup-v2.txt
> index e50b95c..5804355 100644
> --- a/Documentation/cgroup-v2.txt
> +++ b/Documentation/cgroup-v2.txt
> @@ -918,6 +918,34 @@ PAGE_SIZE multiple when read back.
>
> Number of major page faults incurred
>
> + pgrefill
> +
> + Amount of scanned pages (in an active LRU list)
> +
> + pgscan
> +
> + Amount of scanned pages (in an inactive LRU list)
> +
> + pgsteal
> +
> + Amount of reclaimed pages
> +
> + pgactivate
> +
> + Amount of pages moved to the active LRU list
> +
> + pgdeactivate
> +
> + Amount of pages moved to the inactive LRU lis
> +
> + pglazyfree
> +
> + Amount of pages postponed to be freed under memory pressure
> +
> + pglazyfreed
> +
> + Amount of reclaimed lazyfree pages
> +
> memory.swap.current
>
> A read-only single value file which exists on non-root
> diff --git a/fs/dax.c b/fs/dax.c
> index 66d7906..9aac521d 100644
> --- a/fs/dax.c
> +++ b/fs/dax.c
> @@ -1230,7 +1230,7 @@ static int dax_iomap_pte_fault(struct vm_fault *vmf,
> case IOMAP_MAPPED:
> if (iomap.flags & IOMAP_F_NEW) {
> count_vm_event(PGMAJFAULT);
> - mem_cgroup_count_vm_event(vmf->vma->vm_mm, PGMAJFAULT);
> + count_memcg_event_mm(vmf->vma->vm_mm, PGMAJFAULT);
> major = VM_FAULT_MAJOR;
> }
> error = dax_insert_mapping(mapping, iomap.bdev, iomap.dax_dev,
> diff --git a/fs/ncpfs/mmap.c b/fs/ncpfs/mmap.c
> index 0c3905e..6719c0b 100644
> --- a/fs/ncpfs/mmap.c
> +++ b/fs/ncpfs/mmap.c
> @@ -89,7 +89,7 @@ static int ncp_file_mmap_fault(struct vm_fault *vmf)
> * -- nyc
> */
> count_vm_event(PGMAJFAULT);
> - mem_cgroup_count_vm_event(vmf->vma->vm_mm, PGMAJFAULT);
> + count_memcg_event_mm(vmf->vma->vm_mm, PGMAJFAULT);
> return VM_FAULT_MAJOR;
> }
>
> diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
> index 899949b..b2a5b1c 100644
> --- a/include/linux/memcontrol.h
> +++ b/include/linux/memcontrol.h
> @@ -357,6 +357,17 @@ static inline unsigned short mem_cgroup_id(struct mem_cgroup *memcg)
> }
> struct mem_cgroup *mem_cgroup_from_id(unsigned short id);
>
> +static inline struct mem_cgroup *lruvec_memcg(struct lruvec *lruvec)
> +{
> + struct mem_cgroup_per_node *mz;
> +
> + if (mem_cgroup_disabled())
> + return NULL;
> +
> + mz = container_of(lruvec, struct mem_cgroup_per_node, lruvec);
> + return mz->memcg;
> +}
> +
> /**
> * parent_mem_cgroup - find the accounting parent of a memcg
> * @memcg: memcg whose parent to find
> @@ -546,8 +557,23 @@ unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order,
> gfp_t gfp_mask,
> unsigned long *total_scanned);
>
> -static inline void mem_cgroup_count_vm_event(struct mm_struct *mm,
> - enum vm_event_item idx)
> +static inline void count_memcg_events(struct mem_cgroup *memcg,
> + enum vm_event_item idx,
> + unsigned long count)
> +{
> + if (!mem_cgroup_disabled())
> + this_cpu_add(memcg->stat->events[idx], count);
> +}
> +
> +static inline void count_memcg_page_event(struct page *page,
> + enum memcg_stat_item idx)
> +{
> + if (page->mem_cgroup)
> + count_memcg_events(page->mem_cgroup, idx, 1);
> +}
> +
> +static inline void count_memcg_event_mm(struct mm_struct *mm,
> + enum vm_event_item idx)
> {
> struct mem_cgroup *memcg;
>
> @@ -675,6 +701,11 @@ static inline struct mem_cgroup *mem_cgroup_from_id(unsigned short id)
> return NULL;
> }
>
> +static inline struct mem_cgroup *lruvec_memcg(struct lruvec *lruvec)
> +{
> + return NULL;
> +}
> +
> static inline bool mem_cgroup_online(struct mem_cgroup *memcg)
> {
> return true;
> @@ -789,8 +820,19 @@ static inline void mem_cgroup_split_huge_fixup(struct page *head)
> {
> }
>
> +static inline void count_memcg_events(struct mem_cgroup *memcg,
> + enum vm_event_item idx,
> + unsigned long count)
> +{
> +}
> +
> +static inline void count_memcg_page_event(struct page *page,
> + enum memcg_stat_item idx)
> +{
> +}
> +
> static inline
> -void mem_cgroup_count_vm_event(struct mm_struct *mm, enum vm_event_item idx)
> +void count_memcg_event_mm(struct mm_struct *mm, enum vm_event_item idx)
> {
> }
> #endif /* CONFIG_MEMCG */
> diff --git a/mm/filemap.c b/mm/filemap.c
> index b7b973b..d640613 100644
> --- a/mm/filemap.c
> +++ b/mm/filemap.c
> @@ -2226,7 +2226,7 @@ int filemap_fault(struct vm_fault *vmf)
> /* No page in the page cache at all */
> do_sync_mmap_readahead(vmf->vma, ra, file, offset);
> count_vm_event(PGMAJFAULT);
> - mem_cgroup_count_vm_event(vmf->vma->vm_mm, PGMAJFAULT);
> + count_memcg_event_mm(vmf->vma->vm_mm, PGMAJFAULT);
> ret = VM_FAULT_MAJOR;
> retry_find:
> page = find_get_page(mapping, offset);
> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> index ff73899..0cfa0aa 100644
> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -5231,6 +5231,16 @@ static int memory_stat_show(struct seq_file *m, void *v)
> seq_printf(m, "pgfault %lu\n", events[PGFAULT]);
> seq_printf(m, "pgmajfault %lu\n", events[PGMAJFAULT]);
>
> + seq_printf(m, "pgrefill %lu\n", events[PGREFILL]);
> + seq_printf(m, "pgscan %lu\n", events[PGSCAN_KSWAPD] +
> + events[PGSCAN_DIRECT]);
> + seq_printf(m, "pgsteal %lu\n", events[PGSTEAL_KSWAPD] +
> + events[PGSTEAL_DIRECT]);
> + seq_printf(m, "pgactivate %lu\n", events[PGACTIVATE]);
> + seq_printf(m, "pgdeactivate %lu\n", events[PGDEACTIVATE]);
> + seq_printf(m, "pglazyfree %lu\n", events[PGLAZYFREE]);
> + seq_printf(m, "pglazyfreed %lu\n", events[PGLAZYFREED]);
> +
> seq_printf(m, "workingset_refault %lu\n",
> stat[WORKINGSET_REFAULT]);
> seq_printf(m, "workingset_activate %lu\n",
> diff --git a/mm/memory.c b/mm/memory.c
> index 6ff5d72..5aa1348 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -2719,7 +2719,7 @@ int do_swap_page(struct vm_fault *vmf)
> /* Had to read the page from swap area: Major fault */
> ret = VM_FAULT_MAJOR;
> count_vm_event(PGMAJFAULT);
> - mem_cgroup_count_vm_event(vma->vm_mm, PGMAJFAULT);
> + count_memcg_event_mm(vma->vm_mm, PGMAJFAULT);
> } else if (PageHWPoison(page)) {
> /*
> * hwpoisoned dirty swapcache pages are kept for killing
> @@ -3855,7 +3855,7 @@ int handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
> __set_current_state(TASK_RUNNING);
>
> count_vm_event(PGFAULT);
> - mem_cgroup_count_vm_event(vma->vm_mm, PGFAULT);
> + count_memcg_event_mm(vma->vm_mm, PGFAULT);
>
> /* do counter updates before entering really critical section. */
> check_sync_rss_stat(current);
> diff --git a/mm/shmem.c b/mm/shmem.c
> index e67d6ba..8cf16fb 100644
> --- a/mm/shmem.c
> +++ b/mm/shmem.c
> @@ -1645,8 +1645,7 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index,
> if (fault_type) {
> *fault_type |= VM_FAULT_MAJOR;
> count_vm_event(PGMAJFAULT);
> - mem_cgroup_count_vm_event(charge_mm,
> - PGMAJFAULT);
> + count_memcg_event_mm(charge_mm, PGMAJFAULT);
> }
> /* Here we actually start the io */
> page = shmem_swapin(swap, gfp, info, index);
> diff --git a/mm/swap.c b/mm/swap.c
> index 98d08b4..4f44dbd 100644
> --- a/mm/swap.c
> +++ b/mm/swap.c
> @@ -591,6 +591,7 @@ static void lru_lazyfree_fn(struct page *page, struct lruvec *lruvec,
> add_page_to_lru_list(page, lruvec, LRU_INACTIVE_FILE);
>
> __count_vm_events(PGLAZYFREE, hpage_nr_pages(page));
> + count_memcg_page_event(page, PGLAZYFREE);
> update_page_reclaim_stat(lruvec, 1, 0);
> }
> }
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 5ebf468..76e98b5 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -1266,6 +1266,7 @@ static unsigned long shrink_page_list(struct list_head *page_list,
> }
>
> count_vm_event(PGLAZYFREED);
> + count_memcg_page_event(page, PGLAZYFREED);
> } else if (!mapping || !__remove_mapping(mapping, page, true))
> goto keep_locked;
> /*
> @@ -1295,6 +1296,7 @@ static unsigned long shrink_page_list(struct list_head *page_list,
> if (!PageMlocked(page)) {
> SetPageActive(page);
> pgactivate++;
> + count_memcg_page_event(page, PGACTIVATE);
> }
> keep_locked:
> unlock_page(page);
> @@ -1725,11 +1727,16 @@ shrink_inactive_list(unsigned long nr_to_scan, struct lruvec *lruvec,
> __mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, nr_taken);
> reclaim_stat->recent_scanned[file] += nr_taken;
>
> - if (global_reclaim(sc)) {
> - if (current_is_kswapd())
> + if (current_is_kswapd()) {
> + if (global_reclaim(sc))
> __count_vm_events(PGSCAN_KSWAPD, nr_scanned);
> - else
> + count_memcg_events(lruvec_memcg(lruvec), PGSCAN_KSWAPD,
> + nr_scanned);
> + } else {
> + if (global_reclaim(sc))
> __count_vm_events(PGSCAN_DIRECT, nr_scanned);
> + count_memcg_events(lruvec_memcg(lruvec), PGSCAN_DIRECT,
> + nr_scanned);
> }
> spin_unlock_irq(&pgdat->lru_lock);
>
> @@ -1741,11 +1748,16 @@ shrink_inactive_list(unsigned long nr_to_scan, struct lruvec *lruvec,
>
> spin_lock_irq(&pgdat->lru_lock);
>
> - if (global_reclaim(sc)) {
> - if (current_is_kswapd())
> + if (current_is_kswapd()) {
> + if (global_reclaim(sc))
> __count_vm_events(PGSTEAL_KSWAPD, nr_reclaimed);
> - else
> + count_memcg_events(lruvec_memcg(lruvec), PGSTEAL_KSWAPD,
> + nr_reclaimed);
> + } else {
> + if (global_reclaim(sc))
> __count_vm_events(PGSTEAL_DIRECT, nr_reclaimed);
> + count_memcg_events(lruvec_memcg(lruvec), PGSTEAL_DIRECT,
> + nr_reclaimed);
> }
>
> putback_inactive_pages(lruvec, &page_list);
> @@ -1890,8 +1902,11 @@ static unsigned move_active_pages_to_lru(struct lruvec *lruvec,
> }
> }
>
> - if (!is_active_lru(lru))
> + if (!is_active_lru(lru)) {
> __count_vm_events(PGDEACTIVATE, nr_moved);
> + count_memcg_events(lruvec_memcg(lruvec), PGDEACTIVATE,
> + nr_moved);
> + }
>
> return nr_moved;
> }
> @@ -1929,6 +1944,7 @@ static void shrink_active_list(unsigned long nr_to_scan,
> reclaim_stat->recent_scanned[file] += nr_taken;
>
> __count_vm_events(PGREFILL, nr_scanned);
> + count_memcg_events(lruvec_memcg(lruvec), PGREFILL, nr_scanned);
>
> spin_unlock_irq(&pgdat->lru_lock);
>
> --
> 2.7.4
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org. For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* [PATCH 1/1] Sealable memory support
From: Igor Stoppa @ 2017-05-19 10:38 UTC (permalink / raw)
To: mhocko, dave.hansen, labbott
Cc: linux-mm, kernel-hardening, linux-kernel, Igor Stoppa
In-Reply-To: <20170519103811.2183-1-igor.stoppa@huawei.com>
Dynamically allocated variables can be made read only,
after they have been initialized, provided that they reside in memory
pages devoid of any RW data.
The implementation supplies means to create independent pools of memory,
which can be individually created, sealed/unsealed and destroyed.
A global pool is made available for those kernel modules that do not
need to manage an independent pool.
Signed-off-by: Igor Stoppa <igor.stoppa@huawei.com>
---
mm/Makefile | 2 +-
mm/smalloc.c | 200 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
mm/smalloc.h | 61 ++++++++++++++++++
3 files changed, 262 insertions(+), 1 deletion(-)
create mode 100644 mm/smalloc.c
create mode 100644 mm/smalloc.h
diff --git a/mm/Makefile b/mm/Makefile
index 026f6a8..737c42a 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -39,7 +39,7 @@ obj-y := filemap.o mempool.o oom_kill.o \
mm_init.o mmu_context.o percpu.o slab_common.o \
compaction.o vmacache.o swap_slots.o \
interval_tree.o list_lru.o workingset.o \
- debug.o $(mmu-y)
+ debug.o smalloc.o $(mmu-y)
obj-y += init-mm.o
diff --git a/mm/smalloc.c b/mm/smalloc.c
new file mode 100644
index 0000000..fa04cc5
--- /dev/null
+++ b/mm/smalloc.c
@@ -0,0 +1,200 @@
+/*
+ * smalloc.c: Sealable Memory Allocator
+ *
+ * (C) Copyright 2017 Huawei Technologies Co. Ltd.
+ * Author: Igor Stoppa <igor.stoppa@huawei.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; version 2
+ * of the License.
+ */
+
+#include <linux/module.h>
+#include <linux/printk.h>
+#include <linux/kobject.h>
+#include <linux/sysfs.h>
+#include <linux/init.h>
+#include <linux/fs.h>
+#include <linux/string.h>
+
+
+#include <linux/vmalloc.h>
+#include <asm/cacheflush.h>
+#include "smalloc.h"
+
+#define page_roundup(size) (((size) + !(size) - 1 + PAGE_SIZE) & PAGE_MASK)
+
+#define pages_nr(size) (page_roundup(size) / PAGE_SIZE)
+
+static struct smalloc_pool *global_pool;
+
+struct smalloc_node *__smalloc_create_node(unsigned long words)
+{
+ struct smalloc_node *node;
+ unsigned long size;
+
+ /* Calculate the size to ask from vmalloc, page aligned. */
+ size = page_roundup(NODE_HEADER_SIZE + words * sizeof(align_t));
+ node = vmalloc(size);
+ if (!node) {
+ pr_err("No memory for allocating smalloc node.");
+ return NULL;
+ }
+ /* Initialize the node.*/
+ INIT_LIST_HEAD(&node->list);
+ node->free = node->data;
+ node->available_words = (size - NODE_HEADER_SIZE) / sizeof(align_t);
+ return node;
+}
+
+static __always_inline
+void *node_alloc(struct smalloc_node *node, unsigned long words)
+{
+ register align_t *old_free = node->free;
+
+ node->available_words -= words;
+ node->free += words;
+ return old_free;
+}
+
+void *smalloc(unsigned long size, struct smalloc_pool *pool)
+{
+ struct list_head *pos;
+ struct smalloc_node *node;
+ void *ptr;
+ unsigned long words;
+
+ /* If no pool specified, use the global one. */
+ if (!pool)
+ pool = global_pool;
+
+ mutex_lock(&pool->lock);
+
+ /* If the pool is sealed, then return NULL. */
+ if (pool->seal == SMALLOC_SEALED) {
+ mutex_unlock(&pool->lock);
+ return NULL;
+ }
+
+ /* Calculate minimum number of words required. */
+ words = (size + sizeof(align_t) - 1) / sizeof(align_t);
+
+ /* Look for slot that is large enough, in the existing pool.*/
+ list_for_each(pos, &pool->list) {
+ node = list_entry(pos, struct smalloc_node, list);
+ if (node->available_words >= words) {
+ ptr = node_alloc(node, words);
+ mutex_unlock(&pool->lock);
+ return ptr;
+ }
+ }
+
+ /* No slot found, get a new chunk of virtual memory. */
+ node = __smalloc_create_node(words);
+ if (!node) {
+ mutex_unlock(&pool->lock);
+ return NULL;
+ }
+
+ list_add(&node->list, &pool->list);
+ ptr = node_alloc(node, words);
+ mutex_unlock(&pool->lock);
+ return ptr;
+}
+
+static __always_inline
+unsigned long get_node_size(struct smalloc_node *node)
+{
+ if (!node)
+ return 0;
+ return page_roundup((((void *)node->free) - (void *)node) +
+ node->available_words * sizeof(align_t));
+}
+
+static __always_inline
+unsigned long get_node_pages_nr(struct smalloc_node *node)
+{
+ return pages_nr(get_node_size(node));
+}
+void smalloc_seal_set(enum seal_t seal, struct smalloc_pool *pool)
+{
+ struct list_head *pos;
+ struct smalloc_node *node;
+
+ if (!pool)
+ pool = global_pool;
+ mutex_lock(&pool->lock);
+ if (pool->seal == seal) {
+ mutex_unlock(&pool->lock);
+ return;
+ }
+ list_for_each(pos, &pool->list) {
+ node = list_entry(pos, struct smalloc_node, list);
+ if (seal == SMALLOC_SEALED)
+ set_memory_ro((unsigned long)node,
+ get_node_pages_nr(node));
+ else if (seal == SMALLOC_UNSEALED)
+ set_memory_rw((unsigned long)node,
+ get_node_pages_nr(node));
+ }
+ pool->seal = seal;
+ mutex_unlock(&pool->lock);
+}
+
+int smalloc_initialize(struct smalloc_pool *pool)
+{
+ if (!pool)
+ return -EINVAL;
+ INIT_LIST_HEAD(&pool->list);
+ pool->seal = SMALLOC_UNSEALED;
+ mutex_init(&pool->lock);
+ return 0;
+}
+
+struct smalloc_pool *smalloc_create(void)
+{
+ struct smalloc_pool *pool = vmalloc(sizeof(struct smalloc_pool));
+
+ if (!pool) {
+ pr_err("No memory for allocating pool.");
+ return NULL;
+ }
+ smalloc_initialize(pool);
+ return pool;
+}
+
+int smalloc_destroy(struct smalloc_pool *pool)
+{
+ struct list_head *pos, *q;
+ struct smalloc_node *node;
+
+ if (!pool)
+ return -EINVAL;
+ list_for_each_safe(pos, q, &pool->list) {
+ node = list_entry(pos, struct smalloc_node, list);
+ list_del(pos);
+ vfree(node);
+ }
+ return 0;
+}
+
+static int __init smalloc_init(void)
+{
+ global_pool = smalloc_create();
+ if (!global_pool) {
+ pr_err("Module smalloc initialization failed: no memory.\n");
+ return -ENOMEM;
+ }
+ pr_info("Module smalloc initialized successfully.\n");
+ return 0;
+}
+
+static void __exit smalloc_exit(void)
+{
+ pr_info("Module smalloc un initialized successfully.\n");
+}
+
+module_init(smalloc_init);
+module_exit(smalloc_exit);
+MODULE_LICENSE("GPL");
diff --git a/mm/smalloc.h b/mm/smalloc.h
new file mode 100644
index 0000000..344d962
--- /dev/null
+++ b/mm/smalloc.h
@@ -0,0 +1,61 @@
+/*
+ * smalloc.h: Header for Sealable Memory Allocator
+ *
+ * (C) Copyright 2017 Huawei Technologies Co. Ltd.
+ * Author: Igor Stoppa <igor.stoppa@huawei.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; version 2
+ * of the License.
+ */
+
+#ifndef _SMALLOC_H
+#define _SMALLOC_H
+
+#include <linux/list.h>
+#include <linux/mutex.h>
+
+typedef uint64_t align_t;
+
+enum seal_t {
+ SMALLOC_UNSEALED,
+ SMALLOC_SEALED,
+};
+
+#define __SMALLOC_ALIGNED__ __aligned(sizeof(align_t))
+
+#define NODE_HEADER \
+ struct { \
+ __SMALLOC_ALIGNED__ struct { \
+ struct list_head list; \
+ align_t *free; \
+ unsigned long available_words; \
+ }; \
+ }
+
+#define NODE_HEADER_SIZE sizeof(NODE_HEADER)
+
+struct smalloc_pool {
+ struct list_head list;
+ struct mutex lock;
+ enum seal_t seal;
+};
+
+struct smalloc_node {
+ NODE_HEADER;
+ __SMALLOC_ALIGNED__ align_t data[];
+};
+
+#define smalloc_seal(pool) \
+ smalloc_seal_set(SMALLOC_SEALED, pool)
+
+#define smalloc_unseal(pool) \
+ smalloc_seal_set(SMALLOC_UNSEALED, pool)
+
+struct smalloc_pool *smalloc_create(void);
+int smalloc_destroy(struct smalloc_pool *pool);
+int smalloc_initialize(struct smalloc_pool *pool);
+void *smalloc(unsigned long size, struct smalloc_pool *pool);
+void smalloc_seal_set(enum seal_t seal, struct smalloc_pool *pool);
+#endif
--
2.9.3
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [RFC v3]mm: ro protection for data allocated dynamically
From: Igor Stoppa @ 2017-05-19 10:38 UTC (permalink / raw)
To: mhocko, dave.hansen, labbott
Cc: linux-mm, kernel-hardening, linux-kernel, Igor Stoppa
Not all the data allocated dynamically needs to be altered frequently.
In some cases, it might be written just once, at initialization.
This RFC has the goal of improving memory integrity, by explicitly
making said data write-protected.
A reference implementation is provided.
During the previous 2 rounds, some concerns/questions were risen.
This iteration should address msot of them, if not all.
Basic idea behind the implementation: on systems with MMU, the MMU
supports associating various types of attribute to memory pages.
One of them is being read-only.
The MMU will cause an exception upon attempts to alter a read-only page.
This mechanism is already in use for protecting: kernel text and
constant data.
Relatively recently, it has become possible to have also statically
allocated data to become read-only, with the __ro_after_init annotation.
However nothing is done for variables allocated dynamically.
The catch for re-using the same mechanism, is that soon-to-be read only
variables must be grouped in dedicated memory pages, without any rw data
falling in the same range.
This can be achieved with a dedicated allocator.
The implementation proposed allows to create memory pools.
Each pool can be treated independently from the others, allowing fine
grained control about what data can be overwritten.
A pool is a kernel linked list, where the head contains a mutex used for
accessing the list, and the elements are nodes, providing the memory
actually used.
When a pool receives an allocation request for which it doesn't have
enough memory already available, it obtains a set of contiguous virtual
pages (node) that is large enough to cover the request being processed.
Such memory is likely to be significantly larger than what was required.
The slack is used for fulfilling further allocation requests, provided
that they fit the space available.
The pool ends up being a list of nodes, where each node contains a
request that, at the time it was received, could not be satisfied by
using the exisitng nodes, plus other requests that happened to fit in the
slack. Such requests handle each node as an individual linear pool.
When it's time to seal/unseal a pool, each element (node) of the list is
visited and the range of pages it comprises is passed ot set_memory_ro/rw.
Freeing memory is supported at pool level: if for some reason one or more
memory requests must be discarded, at some point, they are simply ignored.
Upon the pool tear down, then nodes are removed one by one and the
corresponding memory range freed for good with vfree.
This approach avoids the extra coplexity of tracking individual
allocations, yet it allows to control claim back pages when not needed
anymore (i.e. module unloading.)
The same design also supports isolation between different kernel modules:
each module can allocae one or more pools, to obtain the desired level of
granularity when managing portions of its data that need different handling.
The price for this flexibility is that some more slack is produced.
The exact amount depends on the sizes of allocations performed and in
which order they arrive.
Modules that do not want/need all of this flexibility can use the default
global pool provided by the allocator.
This pool is intended to provide consistency with __ro_after_init and
therefore would be sealed at the same time.
Some observations/questions:
* the backend of the memory allocation is done by using vmalloc.
Is here any better way? the bpf uses module_alloc but that seems not
exactly its purpose.
* because of the vmalloc backend, this is not suitable for cases where
it is really needed to have physically contiguous memory regions,
however the type of data that would use this interface is likely to
not require interaction with HW devices that could rise such need.
* the allocator supports defining a preferred alignment (currently set
to 8 bytes, using uint64_t) - is it useful/desirable?
If yes, is it the correct granularity (global)?
* to get the size of the padded header of a node, the current code uses
__align(align_t) and it seems to work, but is it correct?
* examples of uses for this new allcoator:
- LSM Hooks
- policy database of SE Linux (several different structure types)
Igor Stoppa (1):
Sealable memory support
mm/Makefile | 2 +-
mm/smalloc.c | 200 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
mm/smalloc.h | 61 ++++++++++++++++++
3 files changed, 262 insertions(+), 1 deletion(-)
create mode 100644 mm/smalloc.c
create mode 100644 mm/smalloc.h
--
2.9.3
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH v6 05/15] lockdep: Implement crossrelease feature
From: Peter Zijlstra @ 2017-05-19 10:30 UTC (permalink / raw)
To: Byungchul Park
Cc: mingo, tglx, walken, boqun.feng, kirill, linux-kernel, linux-mm,
iamjoonsoo.kim, akpm, willy, npiggin, kernel-team
In-Reply-To: <20170519080708.GG28017@X58A-UD3R>
On Fri, May 19, 2017 at 05:07:08PM +0900, Byungchul Park wrote:
> On Tue, Mar 14, 2017 at 05:18:52PM +0900, Byungchul Park wrote:
> > Lockdep is a runtime locking correctness validator that detects and
> > reports a deadlock or its possibility by checking dependencies between
> > locks. It's useful since it does not report just an actual deadlock but
> > also the possibility of a deadlock that has not actually happened yet.
> > That enables problems to be fixed before they affect real systems.
> >
> > However, this facility is only applicable to typical locks, such as
> > spinlocks and mutexes, which are normally released within the context in
> > which they were acquired. However, synchronization primitives like page
> > locks or completions, which are allowed to be released in any context,
> > also create dependencies and can cause a deadlock. So lockdep should
> > track these locks to do a better job. The 'crossrelease' implementation
> > makes these primitives also be tracked.
>
> Excuse me but I have a question...
>
> Only for maskable irq, can I assume that hardirq are prevented within
> hardirq context? I remember that nested interrupts were allowed in the
> past but not recommanded. But what about now? I'm curious about the
> overall direction of kernel and current status. It would be very
> appriciated if you answer it.
So you're right. In general enabling IRQs from hardirq context is
discouraged but allowed. However, if you were to do that with a lock
held that would instantly make lockdep report a deadlock, as the lock is
then both used from IRQ context and has IRQs enabled.
So from a locking perspective you can assume no nesting, but from a
state tracking pov we have to deal with the nesting I think (although it
is very rare).
You're asking this in relation to the rollback thing, right? I think we
should only save the state when hardirq_context goes from 0->1 and
restore on 1->0.
If you're asking this for another reason, please clarify.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: mm, something wring in page_lock_anon_vma_read()?
From: Xishi Qiu @ 2017-05-19 9:44 UTC (permalink / raw)
To: Andrew Morton, Tejun Heo, Michal Hocko, Johannes Weiner,
Mel Gorman, Michal Hocko, Vlastimil Babka, Minchan Kim,
David Rientjes, Joonsoo Kim, aarcange, sumeet.keswani,
Rik van Riel, Hugh Dickins
Cc: Linux MM, LKML, zhong jiang
In-Reply-To: <591EB25C.9080901@huawei.com>
On 2017/5/19 16:52, Xishi Qiu wrote:
> On 2017/5/18 17:46, Xishi Qiu wrote:
>
>> Hi, my system triggers this bug, and the vmcore shows the anon_vma seems be freed.
>> The kernel is RHEL 7.2, and the bug is hard to reproduce, so I don't know if it
>> exists in mainline, any reply is welcome!
>>
>
> When we alloc anon_vma, we will init the value of anon_vma->root,
> so can we set anon_vma->root to NULL when calling
> anon_vma_free -> kmem_cache_free(anon_vma_cachep, anon_vma);
>
> anon_vma_free()
> ...
> anon_vma->root = NULL;
> kmem_cache_free(anon_vma_cachep, anon_vma);
>
> I find if we do this above, system boot failed, why?
>
If anon_vma was freed, we should not to access the root_anon_vma, because it maybe also
freed(e.g. anon_vma == root_anon_vma), right?
page_lock_anon_vma_read()
...
anon_vma = (struct anon_vma *) (anon_mapping - PAGE_MAPPING_ANON);
root_anon_vma = ACCESS_ONCE(anon_vma->root);
if (down_read_trylock(&root_anon_vma->rwsem)) { // it's not safe
...
if (!atomic_inc_not_zero(&anon_vma->refcount)) { // check anon_vma was not freed
...
anon_vma_lock_read(anon_vma); // it's safe
...
> Thanks,
> Xishi Qiu
>
>> [35030.332666] general protection fault: 0000 [#1] SMP
>> [35030.333016] Modules linked in: veth ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack bridge stp llc dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c rtos_kbox_panic(OE) ipmi_devintf ipmi_si ipmi_msghandler signo_catch(O) cirrus syscopyarea sysfillrect sysimgblt ttm crc32_pclmul ghash_clmulni_intel drm_kms_helper aesni_intel ppdev drm lrw gf128mul parport_pc glue_helper ablk_helper serio_raw cryptd i2c_piix4 parport pcspkr sg floppy i2c_core dm_mod sha512_generic ip_tables sd_mod crc_t10dif crct10dif_generic sr_mod cdrom virtio_console virtio_scsi virtio_net ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel virtio_pci virtio_ring virtio ata_piix libata ext4 mbcache
>> [35030.333016] jbd2
>> [35030.333016] CPU: 3 PID: 48 Comm: kswapd0 Tainted: G OE ---- ------- 3.10.0-327.36.58.4.x86_64 #1
>> [35030.333016] Hardware name: OpenStack Foundation OpenStack Nova, BIOS rel-1.8.1-0-g4adadbd-20160826_044443-hghoulaslx112 04/01/2014
>> [35030.333016] task: ffff8801b2d20000 ti: ffff8801b4c38000 task.ti: ffff8801b4c38000
>> [35030.333016] RIP: 0010:[<ffffffff810acac5>] [<ffffffff810acac5>] down_read_trylock+0x5/0x50
>> [35030.333016] RSP: 0000:ffff8801b4c3ba90 EFLAGS: 00010282
>> [35030.333016] RAX: 0000000000000000 RBX: ffff8801b3e2a100 RCX: 0000000000000000
>> [35030.333016] RDX: 0000000000000000 RSI: 0000000000000000 RDI: deb604d497705c5d
>> [35030.333016] RBP: ffff8801b4c3bab8 R08: ffffea0002c34460 R09: ffff8801b3d7e8a0
>> [35030.333016] R10: 0000000000000004 R11: fff00000fe000000 R12: ffff8801b3e2a101
>> [35030.333016] R13: ffffea0002c34440 R14: deb604d497705c5d R15: ffffea0002c34440
>> [35030.333016] FS: 0000000000000000(0000) GS:ffff8801bed80000(0000) knlGS:0000000000000000
>> [35030.333016] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [35030.333016] CR2: 000000c422011080 CR3: 0000000001976000 CR4: 00000000001407e0
>> [35030.333016] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> [35030.333016] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> [35030.333016] Stack:
>> [35030.333016] ffffffff811b2795 ffffea0002c34440 0000000000000000 000000000000000f
>> [35030.333016] 0000000000000001 ffff8801b4c3bb30 ffffffff811b2a17 ffff8800a712d640
>> [35030.333016] 000000000c4229e2 ffff8801b4c3bb80 0000000100000000 000000000c41fe38
>> [35030.333016] Call Trace:
>> [35030.333016] [<ffffffff811b2795>] ? page_lock_anon_vma_read+0x55/0x110
>> [35030.333016] [<ffffffff811b2a17>] page_referenced+0x1c7/0x350
>> [35030.333016] [<ffffffff8118d9b4>] shrink_active_list+0x1e4/0x400
>> [35030.333016] [<ffffffff8118e08d>] shrink_lruvec+0x4bd/0x770
>> [35030.333016] [<ffffffff8118e3b6>] shrink_zone+0x76/0x1a0
>> [35030.333016] [<ffffffff8118f6cc>] balance_pgdat+0x49c/0x610
>> [35030.333016] [<ffffffff8118f9b3>] kswapd+0x173/0x450
>> [35030.333016] [<ffffffff810a8a00>] ? wake_up_atomic_t+0x30/0x30
>> [35030.333016] [<ffffffff8118f840>] ? balance_pgdat+0x610/0x610
>> [35030.333016] [<ffffffff810a79bf>] kthread+0xcf/0xe0
>> [35030.333016] [<ffffffff810a78f0>] ? kthread_create_on_node+0x120/0x120
>> [35030.333016] [<ffffffff81665bd8>] ret_from_fork+0x58/0x90
>> [35030.333016] [<ffffffff810a78f0>] ? kthread_create_on_node+0x120/0x120
>> [35030.333016] Code: 00 ba ff ff ff ff 48 89 d8 f0 48 0f c1 10 79 05 e8 31 06 27 00 5b 5d c3 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7
>> [35030.333016] RIP [<ffffffff810acac5>] down_read_trylock+0x5/0x50
>> [35030.333016] RSP <ffff8801b4c3ba90>
>> [35030.333016] ------------[ cut here ]------------
>>
>> struct page {
>> flags = 9007194960298056,
>> mapping = 0xffff8801b3e2a101,
>> {
>> {
>> index = 34324593617,
>> freelist = 0x7fde7bbd1,
>> pfmemalloc = 209,
>> thp_mmu_gather = {
>> counter = -35144751
>> },
>> pmd_huge_pte = 0x7fde7bbd1
>> },
>> {
>> counters = 8589934592,
>> {
>> {
>> _mapcount = {
>> counter = 0
>> },
>> {
>> inuse = 0,
>> objects = 0,
>> frozen = 0
>> },
>> units = 0
>> },
>> _count = {
>> counter = 2
>> }
>> }
>> }
>> },
>> {
>> lru = {
>> next = 0xdead000000100100,
>> prev = 0xdead000000200200
>> },
>> {
>> next = 0xdead000000100100,
>> pages = 2097664,
>> pobjects = -559087616
>> },
>> list = {
>> next = 0xdead000000100100,
>> prev = 0xdead000000200200
>> },
>> slab_page = 0xdead000000100100
>> },
>> {
>> private = 0,
>> ptl = {
>> {
>> rlock = {
>> raw_lock = {
>> {
>> head_tail = 0,
>> tickets = {
>> head = 0,
>> tail = 0
>> }
>> }
>> }
>> }
>> }
>> },
>> slab_cache = 0x0,
>> first_page = 0x0
>> }
>> }
>>
>>
>>
>> crash> struct anon_vma 0xffff8801b3e2a100
>> struct anon_vma {
>> root = 0xdeb604d497705c55,
>> rwsem = {
>> count = -8192007903225070328,
>> wait_lock = {
>> raw_lock = {
>> {
>> head_tail = 2955503940,
>> tickets = {
>> head = 26948,
>> tail = 45097
>> }
>> }
>> }
>> },
>> wait_list = {
>> next = 0x559f9107c1b47439,
>> prev = 0x3de13f709bfa043b
>> }
>> },
>> refcount = {
>> counter = -13243516
>> },
>> rb_root = {
>> rb_node = 0x11dd18f9ce0bb2e9
>> }
>> }
>>
>> This address 0xffff8801b3e2a100 can not find in "kmem -S anon_vma"
>>
>> The page flags is
>> crash> kmem -g 0x1FFFFF00080048
>> FLAGS: 1fffff00080048
>> PAGE-FLAG BIT VALUE
>> PG_uptodate 3 0000008
>> PG_active 6 0000040
>> PG_swapbacked 19 0080000
>>
>>
>> .
>>
>
>
>
>
> .
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH] mm/vmstat: add oom_kill counter
From: Konstantin Khlebnikov @ 2017-05-19 9:10 UTC (permalink / raw)
To: Vlastimil Babka, linux-mm, Andrew Morton, linux-kernel,
Michal Hocko
In-Reply-To: <7683805d-e0ac-3ab9-0a73-47eea945436d@suse.cz>
On 19.05.2017 12:05, Vlastimil Babka wrote:
> On 05/19/2017 08:59 AM, Konstantin Khlebnikov wrote:
>> Show count of global oom killer invocations in /proc/vmstat
>
> Maybe some more rationale why is that useful?
Currently the only way to detect oom kill is grepping kernel logs.
This counter makes such monitoring much easier.
>
> Vlastimil
>
>> Signed-off-by: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>
>> ---
>> include/linux/vm_event_item.h | 1 +
>> mm/oom_kill.c | 1 +
>> mm/vmstat.c | 1 +
>> 3 files changed, 3 insertions(+)
>>
>> diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h
>> index d84ae90ccd5c..1707e0a7d943 100644
>> --- a/include/linux/vm_event_item.h
>> +++ b/include/linux/vm_event_item.h
>> @@ -41,6 +41,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
>> KSWAPD_LOW_WMARK_HIT_QUICKLY, KSWAPD_HIGH_WMARK_HIT_QUICKLY,
>> PAGEOUTRUN, PGROTATED,
>> DROP_PAGECACHE, DROP_SLAB,
>> + OOM_KILL,
>> #ifdef CONFIG_NUMA_BALANCING
>> NUMA_PTE_UPDATES,
>> NUMA_HUGE_PTE_UPDATES,
>> diff --git a/mm/oom_kill.c b/mm/oom_kill.c
>> index 04c9143a8625..c734c42826cf 100644
>> --- a/mm/oom_kill.c
>> +++ b/mm/oom_kill.c
>> @@ -883,6 +883,7 @@ static void oom_kill_process(struct oom_control *oc, const char *message)
>> */
>> do_send_sig_info(SIGKILL, SEND_SIG_FORCED, victim, true);
>> mark_oom_victim(victim);
>> + count_vm_event(OOM_KILL);
>> pr_err("Killed process %d (%s) total-vm:%lukB, anon-rss:%lukB, file-rss:%lukB, shmem-rss:%lukB\n",
>> task_pid_nr(victim), victim->comm, K(victim->mm->total_vm),
>> K(get_mm_counter(victim->mm, MM_ANONPAGES)),
>> diff --git a/mm/vmstat.c b/mm/vmstat.c
>> index 76f73670200a..fe80b81a86e0 100644
>> --- a/mm/vmstat.c
>> +++ b/mm/vmstat.c
>> @@ -1018,6 +1018,7 @@ const char * const vmstat_text[] = {
>>
>> "drop_pagecache",
>> "drop_slab",
>> + "oom_kill",
>>
>> #ifdef CONFIG_NUMA_BALANCING
>> "numa_pte_updates",
>>
>> --
>> To unsubscribe, send a message with 'unsubscribe linux-mm' in
>> the body to majordomo@kvack.org. For more info on Linux MM,
>> see: http://www.linux-mm.org/ .
>> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
>>
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH] mm/vmstat: add oom_kill counter
From: Vlastimil Babka @ 2017-05-19 9:05 UTC (permalink / raw)
To: Konstantin Khlebnikov, linux-mm, Andrew Morton, linux-kernel,
Michal Hocko
In-Reply-To: <149517718482.32770.939520643229572472.stgit@buzz>
On 05/19/2017 08:59 AM, Konstantin Khlebnikov wrote:
> Show count of global oom killer invocations in /proc/vmstat
Maybe some more rationale why is that useful?
Vlastimil
> Signed-off-by: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>
> ---
> include/linux/vm_event_item.h | 1 +
> mm/oom_kill.c | 1 +
> mm/vmstat.c | 1 +
> 3 files changed, 3 insertions(+)
>
> diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h
> index d84ae90ccd5c..1707e0a7d943 100644
> --- a/include/linux/vm_event_item.h
> +++ b/include/linux/vm_event_item.h
> @@ -41,6 +41,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
> KSWAPD_LOW_WMARK_HIT_QUICKLY, KSWAPD_HIGH_WMARK_HIT_QUICKLY,
> PAGEOUTRUN, PGROTATED,
> DROP_PAGECACHE, DROP_SLAB,
> + OOM_KILL,
> #ifdef CONFIG_NUMA_BALANCING
> NUMA_PTE_UPDATES,
> NUMA_HUGE_PTE_UPDATES,
> diff --git a/mm/oom_kill.c b/mm/oom_kill.c
> index 04c9143a8625..c734c42826cf 100644
> --- a/mm/oom_kill.c
> +++ b/mm/oom_kill.c
> @@ -883,6 +883,7 @@ static void oom_kill_process(struct oom_control *oc, const char *message)
> */
> do_send_sig_info(SIGKILL, SEND_SIG_FORCED, victim, true);
> mark_oom_victim(victim);
> + count_vm_event(OOM_KILL);
> pr_err("Killed process %d (%s) total-vm:%lukB, anon-rss:%lukB, file-rss:%lukB, shmem-rss:%lukB\n",
> task_pid_nr(victim), victim->comm, K(victim->mm->total_vm),
> K(get_mm_counter(victim->mm, MM_ANONPAGES)),
> diff --git a/mm/vmstat.c b/mm/vmstat.c
> index 76f73670200a..fe80b81a86e0 100644
> --- a/mm/vmstat.c
> +++ b/mm/vmstat.c
> @@ -1018,6 +1018,7 @@ const char * const vmstat_text[] = {
>
> "drop_pagecache",
> "drop_slab",
> + "oom_kill",
>
> #ifdef CONFIG_NUMA_BALANCING
> "numa_pte_updates",
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org. For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: mm, something wring in page_lock_anon_vma_read()?
From: Xishi Qiu @ 2017-05-19 8:52 UTC (permalink / raw)
To: Andrew Morton, Tejun Heo, Michal Hocko, Johannes Weiner,
Mel Gorman, Michal Hocko, Vlastimil Babka, Minchan Kim,
David Rientjes, Joonsoo Kim, aarcange, sumeet.keswani,
Rik van Riel, Hugh Dickins
Cc: Linux MM, LKML, zhong jiang
In-Reply-To: <591D6D79.7030704@huawei.com>
On 2017/5/18 17:46, Xishi Qiu wrote:
> Hi, my system triggers this bug, and the vmcore shows the anon_vma seems be freed.
> The kernel is RHEL 7.2, and the bug is hard to reproduce, so I don't know if it
> exists in mainline, any reply is welcome!
>
When we alloc anon_vma, we will init the value of anon_vma->root,
so can we set anon_vma->root to NULL when calling
anon_vma_free -> kmem_cache_free(anon_vma_cachep, anon_vma);
anon_vma_free()
...
anon_vma->root = NULL;
kmem_cache_free(anon_vma_cachep, anon_vma);
I find if we do this above, system boot failed, why?
Thanks,
Xishi Qiu
> [35030.332666] general protection fault: 0000 [#1] SMP
> [35030.333016] Modules linked in: veth ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack bridge stp llc dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c rtos_kbox_panic(OE) ipmi_devintf ipmi_si ipmi_msghandler signo_catch(O) cirrus syscopyarea sysfillrect sysimgblt ttm crc32_pclmul ghash_clmulni_intel drm_kms_helper aesni_intel ppdev drm lrw gf128mul parport_pc glue_helper ablk_helper serio_raw cryptd i2c_piix4 parport pcspkr sg floppy i2c_core dm_mod sha512_generic ip_tables sd_mod crc_t10dif crct10dif_generic sr_mod cdrom virtio_console virtio_scsi virtio_net ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel virtio_pci virtio_ring virtio ata_piix libata ext4 mbcache
> [35030.333016] jbd2
> [35030.333016] CPU: 3 PID: 48 Comm: kswapd0 Tainted: G OE ---- ------- 3.10.0-327.36.58.4.x86_64 #1
> [35030.333016] Hardware name: OpenStack Foundation OpenStack Nova, BIOS rel-1.8.1-0-g4adadbd-20160826_044443-hghoulaslx112 04/01/2014
> [35030.333016] task: ffff8801b2d20000 ti: ffff8801b4c38000 task.ti: ffff8801b4c38000
> [35030.333016] RIP: 0010:[<ffffffff810acac5>] [<ffffffff810acac5>] down_read_trylock+0x5/0x50
> [35030.333016] RSP: 0000:ffff8801b4c3ba90 EFLAGS: 00010282
> [35030.333016] RAX: 0000000000000000 RBX: ffff8801b3e2a100 RCX: 0000000000000000
> [35030.333016] RDX: 0000000000000000 RSI: 0000000000000000 RDI: deb604d497705c5d
> [35030.333016] RBP: ffff8801b4c3bab8 R08: ffffea0002c34460 R09: ffff8801b3d7e8a0
> [35030.333016] R10: 0000000000000004 R11: fff00000fe000000 R12: ffff8801b3e2a101
> [35030.333016] R13: ffffea0002c34440 R14: deb604d497705c5d R15: ffffea0002c34440
> [35030.333016] FS: 0000000000000000(0000) GS:ffff8801bed80000(0000) knlGS:0000000000000000
> [35030.333016] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [35030.333016] CR2: 000000c422011080 CR3: 0000000001976000 CR4: 00000000001407e0
> [35030.333016] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [35030.333016] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [35030.333016] Stack:
> [35030.333016] ffffffff811b2795 ffffea0002c34440 0000000000000000 000000000000000f
> [35030.333016] 0000000000000001 ffff8801b4c3bb30 ffffffff811b2a17 ffff8800a712d640
> [35030.333016] 000000000c4229e2 ffff8801b4c3bb80 0000000100000000 000000000c41fe38
> [35030.333016] Call Trace:
> [35030.333016] [<ffffffff811b2795>] ? page_lock_anon_vma_read+0x55/0x110
> [35030.333016] [<ffffffff811b2a17>] page_referenced+0x1c7/0x350
> [35030.333016] [<ffffffff8118d9b4>] shrink_active_list+0x1e4/0x400
> [35030.333016] [<ffffffff8118e08d>] shrink_lruvec+0x4bd/0x770
> [35030.333016] [<ffffffff8118e3b6>] shrink_zone+0x76/0x1a0
> [35030.333016] [<ffffffff8118f6cc>] balance_pgdat+0x49c/0x610
> [35030.333016] [<ffffffff8118f9b3>] kswapd+0x173/0x450
> [35030.333016] [<ffffffff810a8a00>] ? wake_up_atomic_t+0x30/0x30
> [35030.333016] [<ffffffff8118f840>] ? balance_pgdat+0x610/0x610
> [35030.333016] [<ffffffff810a79bf>] kthread+0xcf/0xe0
> [35030.333016] [<ffffffff810a78f0>] ? kthread_create_on_node+0x120/0x120
> [35030.333016] [<ffffffff81665bd8>] ret_from_fork+0x58/0x90
> [35030.333016] [<ffffffff810a78f0>] ? kthread_create_on_node+0x120/0x120
> [35030.333016] Code: 00 ba ff ff ff ff 48 89 d8 f0 48 0f c1 10 79 05 e8 31 06 27 00 5b 5d c3 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7
> [35030.333016] RIP [<ffffffff810acac5>] down_read_trylock+0x5/0x50
> [35030.333016] RSP <ffff8801b4c3ba90>
> [35030.333016] ------------[ cut here ]------------
>
> struct page {
> flags = 9007194960298056,
> mapping = 0xffff8801b3e2a101,
> {
> {
> index = 34324593617,
> freelist = 0x7fde7bbd1,
> pfmemalloc = 209,
> thp_mmu_gather = {
> counter = -35144751
> },
> pmd_huge_pte = 0x7fde7bbd1
> },
> {
> counters = 8589934592,
> {
> {
> _mapcount = {
> counter = 0
> },
> {
> inuse = 0,
> objects = 0,
> frozen = 0
> },
> units = 0
> },
> _count = {
> counter = 2
> }
> }
> }
> },
> {
> lru = {
> next = 0xdead000000100100,
> prev = 0xdead000000200200
> },
> {
> next = 0xdead000000100100,
> pages = 2097664,
> pobjects = -559087616
> },
> list = {
> next = 0xdead000000100100,
> prev = 0xdead000000200200
> },
> slab_page = 0xdead000000100100
> },
> {
> private = 0,
> ptl = {
> {
> rlock = {
> raw_lock = {
> {
> head_tail = 0,
> tickets = {
> head = 0,
> tail = 0
> }
> }
> }
> }
> }
> },
> slab_cache = 0x0,
> first_page = 0x0
> }
> }
>
>
>
> crash> struct anon_vma 0xffff8801b3e2a100
> struct anon_vma {
> root = 0xdeb604d497705c55,
> rwsem = {
> count = -8192007903225070328,
> wait_lock = {
> raw_lock = {
> {
> head_tail = 2955503940,
> tickets = {
> head = 26948,
> tail = 45097
> }
> }
> }
> },
> wait_list = {
> next = 0x559f9107c1b47439,
> prev = 0x3de13f709bfa043b
> }
> },
> refcount = {
> counter = -13243516
> },
> rb_root = {
> rb_node = 0x11dd18f9ce0bb2e9
> }
> }
>
> This address 0xffff8801b3e2a100 can not find in "kmem -S anon_vma"
>
> The page flags is
> crash> kmem -g 0x1FFFFF00080048
> FLAGS: 1fffff00080048
> PAGE-FLAG BIT VALUE
> PG_uptodate 3 0000008
> PG_active 6 0000040
> PG_swapbacked 19 0080000
>
>
> .
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH 11/14] mm, memory_hotplug: do not associate hotadded memory to zones until online
From: Vlastimil Babka @ 2017-05-19 8:56 UTC (permalink / raw)
To: Michal Hocko, Andrew Morton
Cc: linux-mm, Mel Gorman, Andrea Arcangeli, Jerome Glisse, Reza Arbab,
Yasuaki Ishimatsu, qiuxishi, Kani Toshimitsu, slaoub, Joonsoo Kim,
Andi Kleen, David Rientjes, Daniel Kiper, Igor Mammedov,
Vitaly Kuznetsov, LKML, Michal Hocko, Dan Williams,
Heiko Carstens, Martin Schwidefsky
In-Reply-To: <20170515085827.16474-12-mhocko@kernel.org>
On 05/15/2017 10:58 AM, Michal Hocko wrote:
> From: Michal Hocko <mhocko@suse.com>
>
> The current memory hotplug implementation relies on having all the
> struct pages associate with a zone/node during the physical hotplug phase
> (arch_add_memory->__add_pages->__add_section->__add_zone). In the vast
> majority of cases this means that they are added to ZONE_NORMAL. This
> has been so since 9d99aaa31f59 ("[PATCH] x86_64: Support memory hotadd
> without sparsemem") and it wasn't a big deal back then because movable
> onlining didn't exist yet.
>
> Much later memory hotplug wanted to (ab)use ZONE_MOVABLE for movable
> onlining 511c2aba8f07 ("mm, memory-hotplug: dynamic configure movable
> memory and portion memory") and then things got more complicated. Rather
> than reconsidering the zone association which was no longer needed
> (because the memory hotplug already depended on SPARSEMEM) a convoluted
> semantic of zone shifting has been developed. Only the currently last
> memblock or the one adjacent to the zone_movable can be onlined movable.
> This essentially means that the online type changes as the new memblocks
> are added.
>
> Let's simulate memory hot online manually
> $ echo 0x100000000 > /sys/devices/system/memory/probe
> $ grep . /sys/devices/system/memory/memory32/valid_zones
> Normal Movable
>
> $ echo $((0x100000000+(128<<20))) > /sys/devices/system/memory/probe
> $ grep . /sys/devices/system/memory/memory3?/valid_zones
> /sys/devices/system/memory/memory32/valid_zones:Normal
> /sys/devices/system/memory/memory33/valid_zones:Normal Movable
>
> $ echo $((0x100000000+2*(128<<20))) > /sys/devices/system/memory/probe
> $ grep . /sys/devices/system/memory/memory3?/valid_zones
> /sys/devices/system/memory/memory32/valid_zones:Normal
> /sys/devices/system/memory/memory33/valid_zones:Normal
> /sys/devices/system/memory/memory34/valid_zones:Normal Movable
>
> $ echo online_movable > /sys/devices/system/memory/memory34/state
> $ grep . /sys/devices/system/memory/memory3?/valid_zones
> /sys/devices/system/memory/memory32/valid_zones:Normal
> /sys/devices/system/memory/memory33/valid_zones:Normal Movable
> /sys/devices/system/memory/memory34/valid_zones:Movable Normal
>
> This is an awkward semantic because an udev event is sent as soon as the
> block is onlined and an udev handler might want to online it based on
> some policy (e.g. association with a node) but it will inherently race
> with new blocks showing up.
>
> This patch changes the physical online phase to not associate pages
> with any zone at all. All the pages are just marked reserved and wait
> for the onlining phase to be associated with the zone as per the online
> request. There are only two requirements
> - existing ZONE_NORMAL and ZONE_MOVABLE cannot overlap
> - ZONE_NORMAL precedes ZONE_MOVABLE in physical addresses
> the later on is not an inherent requirement and can be changed in the
> future. It preserves the current behavior and made the code slightly
> simpler. This is subject to change in future.
>
> This means that the same physical online steps as above will lead to the
> following state:
> Normal Movable
>
> /sys/devices/system/memory/memory32/valid_zones:Normal Movable
> /sys/devices/system/memory/memory33/valid_zones:Normal Movable
>
> /sys/devices/system/memory/memory32/valid_zones:Normal Movable
> /sys/devices/system/memory/memory33/valid_zones:Normal Movable
> /sys/devices/system/memory/memory34/valid_zones:Normal Movable
>
> /sys/devices/system/memory/memory32/valid_zones:Normal Movable
> /sys/devices/system/memory/memory33/valid_zones:Normal Movable
> /sys/devices/system/memory/memory34/valid_zones:Movable
>
> Implementation:
> The current move_pfn_range is reimplemented to check the above
> requirements (allow_online_pfn_range) and then updates the respective
> zone (move_pfn_range_to_zone), the pgdat and links all the pages in the
> pfn range with the zone/node. __add_pages is updated to not require the
> zone and only initializes sections in the range. This allowed to
> simplify the arch_add_memory code (s390 could get rid of quite some
> of code).
>
> devm_memremap_pages is the only user of arch_add_memory which relies
> on the zone association because it only hooks into the memory hotplug
> only half way. It uses it to associate the new memory with ZONE_DEVICE
> but doesn't allow it to be {on,off}lined via sysfs. This means that this
> particular code path has to call move_pfn_range_to_zone explicitly.
>
> The original zone shifting code is kept in place and will be removed in
> the follow up patch for an easier review.
>
> Please note that this patch also changes the original behavior when
> offlining a memory block adjacent to another zone (Normal vs. Movable)
> used to allow to change its movable type. This will be handled later.
>
> Changes since v1
> - we have to associate the page with the node early (in __add_section),
> because pfn_to_node depends on struct page containing this
> information - based on testing by Reza Arbab
> - resize_{zone,pgdat}_range has to check whether they are popoulated -
> Reza Arbab
> - fix devm_memremap_pages to use pfn rather than physical address -
> JA(C)rA'me Glisse
> - move_pfn_range has to check for intersection with zone_movable rather
> than to rely on allow_online_pfn_range(MMOP_ONLINE_MOVABLE) for
> MMOP_ONLINE_KEEP
>
> Changes since v2
> - fix show_valid_zones nr_pages calculation
> - allow_online_pfn_range has to check managed pages rather than present
> - zone_intersects fix bogus check
> - fix zone_intersects + few nits as per Vlastimil
>
> Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
> Cc: linux-arch@vger.kernel.org
> Tested-by: Dan Williams <dan.j.williams@intel.com>
> Acked-by: Heiko Carstens <heiko.carstens@de.ibm.com> # For s390 bits
> Tested-by: Reza Arbab <arbab@linux.vnet.ibm.com>
> Signed-off-by: Michal Hocko <mhocko@suse.com>
Acked-by: Vlastimil Babka <vbabka@suse.cz>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH] mm/vmstat: add oom_kill counter
From: Konstantin Khlebnikov @ 2017-05-19 8:22 UTC (permalink / raw)
To: linux-mm, Andrew Morton, linux-kernel
In-Reply-To: <149517718482.32770.939520643229572472.stgit@buzz>
On 19.05.2017 09:59, Konstantin Khlebnikov wrote:
> Show count of global oom killer invocations in /proc/vmstat
Oops, this actually counts memcg kills too. Will redo.
>
> Signed-off-by: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>
> ---
> include/linux/vm_event_item.h | 1 +
> mm/oom_kill.c | 1 +
> mm/vmstat.c | 1 +
> 3 files changed, 3 insertions(+)
>
> diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h
> index d84ae90ccd5c..1707e0a7d943 100644
> --- a/include/linux/vm_event_item.h
> +++ b/include/linux/vm_event_item.h
> @@ -41,6 +41,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
> KSWAPD_LOW_WMARK_HIT_QUICKLY, KSWAPD_HIGH_WMARK_HIT_QUICKLY,
> PAGEOUTRUN, PGROTATED,
> DROP_PAGECACHE, DROP_SLAB,
> + OOM_KILL,
> #ifdef CONFIG_NUMA_BALANCING
> NUMA_PTE_UPDATES,
> NUMA_HUGE_PTE_UPDATES,
> diff --git a/mm/oom_kill.c b/mm/oom_kill.c
> index 04c9143a8625..c734c42826cf 100644
> --- a/mm/oom_kill.c
> +++ b/mm/oom_kill.c
> @@ -883,6 +883,7 @@ static void oom_kill_process(struct oom_control *oc, const char *message)
> */
> do_send_sig_info(SIGKILL, SEND_SIG_FORCED, victim, true);
> mark_oom_victim(victim);
> + count_vm_event(OOM_KILL);
> pr_err("Killed process %d (%s) total-vm:%lukB, anon-rss:%lukB, file-rss:%lukB, shmem-rss:%lukB\n",
> task_pid_nr(victim), victim->comm, K(victim->mm->total_vm),
> K(get_mm_counter(victim->mm, MM_ANONPAGES)),
> diff --git a/mm/vmstat.c b/mm/vmstat.c
> index 76f73670200a..fe80b81a86e0 100644
> --- a/mm/vmstat.c
> +++ b/mm/vmstat.c
> @@ -1018,6 +1018,7 @@ const char * const vmstat_text[] = {
>
> "drop_pagecache",
> "drop_slab",
> + "oom_kill",
>
> #ifdef CONFIG_NUMA_BALANCING
> "numa_pte_updates",
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH 10/14] mm, vmstat: skip reporting offline pages in pagetypeinfo
From: Vlastimil Babka @ 2017-05-19 8:17 UTC (permalink / raw)
To: Michal Hocko, Andrew Morton
Cc: linux-mm, Mel Gorman, Andrea Arcangeli, Jerome Glisse, Reza Arbab,
Yasuaki Ishimatsu, qiuxishi, Kani Toshimitsu, slaoub, Joonsoo Kim,
Andi Kleen, David Rientjes, Daniel Kiper, Igor Mammedov,
Vitaly Kuznetsov, LKML, Michal Hocko
In-Reply-To: <20170515085827.16474-11-mhocko@kernel.org>
On 05/15/2017 10:58 AM, Michal Hocko wrote:
> From: Michal Hocko <mhocko@suse.com>
>
> pagetypeinfo_showblockcount_print skips over invalid pfns but it would
> report pages which are offline because those have a valid pfn. Their
> migrate type is misleading at best. Now that we have pfn_to_online_page()
> we can use it instead of pfn_valid() and fix this.
>
> Noticed-by: Joonsoo Kim <js1304@gmail.com>
> Signed-off-by: Michal Hocko <mhocko@suse.com>
(with the followup fix)
Acked-by: Vlastimil Babka <vbabka@suse.cz>
> ---
> mm/vmstat.c | 4 +---
> 1 file changed, 1 insertion(+), 3 deletions(-)
>
> diff --git a/mm/vmstat.c b/mm/vmstat.c
> index 571d3ec05566..c432e581f9a9 100644
> --- a/mm/vmstat.c
> +++ b/mm/vmstat.c
> @@ -1223,11 +1223,9 @@ static void pagetypeinfo_showblockcount_print(struct seq_file *m,
> for (pfn = start_pfn; pfn < end_pfn; pfn += pageblock_nr_pages) {
> struct page *page;
>
> - if (!pfn_valid(pfn))
> + if (!pfn_to_online_page(pfn))
> continue;
>
> - page = pfn_to_page(pfn);
> -
> /* Watch for unexpected holes punched in the memmap */
> if (!memmap_valid_within(pfn, page, zone))
> continue;
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH v6 05/15] lockdep: Implement crossrelease feature
From: Byungchul Park @ 2017-05-19 8:07 UTC (permalink / raw)
To: peterz, mingo
Cc: tglx, walken, boqun.feng, kirill, linux-kernel, linux-mm,
iamjoonsoo.kim, akpm, willy, npiggin, kernel-team
In-Reply-To: <1489479542-27030-6-git-send-email-byungchul.park@lge.com>
On Tue, Mar 14, 2017 at 05:18:52PM +0900, Byungchul Park wrote:
> Lockdep is a runtime locking correctness validator that detects and
> reports a deadlock or its possibility by checking dependencies between
> locks. It's useful since it does not report just an actual deadlock but
> also the possibility of a deadlock that has not actually happened yet.
> That enables problems to be fixed before they affect real systems.
>
> However, this facility is only applicable to typical locks, such as
> spinlocks and mutexes, which are normally released within the context in
> which they were acquired. However, synchronization primitives like page
> locks or completions, which are allowed to be released in any context,
> also create dependencies and can cause a deadlock. So lockdep should
> track these locks to do a better job. The 'crossrelease' implementation
> makes these primitives also be tracked.
Excuse me but I have a question...
Only for maskable irq, can I assume that hardirq are prevented within
hardirq context? I remember that nested interrupts were allowed in the
past but not recommanded. But what about now? I'm curious about the
overall direction of kernel and current status. It would be very
appriciated if you answer it.
Thank you.
Byungchul
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH 09/14] mm: __first_valid_page skip over offline pages
From: Vlastimil Babka @ 2017-05-19 8:05 UTC (permalink / raw)
To: Michal Hocko, Andrew Morton
Cc: linux-mm, Mel Gorman, Andrea Arcangeli, Jerome Glisse, Reza Arbab,
Yasuaki Ishimatsu, qiuxishi, Kani Toshimitsu, slaoub, Joonsoo Kim,
Andi Kleen, David Rientjes, Daniel Kiper, Igor Mammedov,
Vitaly Kuznetsov, LKML, Michal Hocko
In-Reply-To: <20170515085827.16474-10-mhocko@kernel.org>
On 05/15/2017 10:58 AM, Michal Hocko wrote:
> From: Michal Hocko <mhocko@suse.com>
>
> __first_valid_page skips over invalid pfns in the range but it might
> still stumble over offline pages. At least start_isolate_page_range
> will mark those set_migratetype_isolate. This doesn't represent
> any immediate AFAICS because alloc_contig_range will fail to isolate
> those pages but it relies on not fully initialized page which will
> become a problem later when we stop associating offline pages to zones.
> Use pfn_to_online_page to handle this.
>
> This is more a preparatory patch than a fix.
>
> Signed-off-by: Michal Hocko <mhocko@suse.com>
Iterating over single pages when the whole section is offline seems
rather wasteful, but it should be really rare, so whatever.
Acked-by: Vlastimil Babka <vbabka@suse.cz>
> ---
> mm/page_isolation.c | 26 ++++++++++++++++++--------
> 1 file changed, 18 insertions(+), 8 deletions(-)
>
> diff --git a/mm/page_isolation.c b/mm/page_isolation.c
> index 5092e4ef00c8..3606104893e0 100644
> --- a/mm/page_isolation.c
> +++ b/mm/page_isolation.c
> @@ -138,12 +138,18 @@ static inline struct page *
> __first_valid_page(unsigned long pfn, unsigned long nr_pages)
> {
> int i;
> - for (i = 0; i < nr_pages; i++)
> - if (pfn_valid_within(pfn + i))
> - break;
> - if (unlikely(i == nr_pages))
> - return NULL;
> - return pfn_to_page(pfn + i);
> +
> + for (i = 0; i < nr_pages; i++) {
> + struct page *page;
> +
> + if (!pfn_valid_within(pfn + i))
> + continue;
> + page = pfn_to_online_page(pfn + i);
> + if (!page)
> + continue;
> + return page;
> + }
> + return NULL;
> }
>
> /*
> @@ -184,8 +190,12 @@ int start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn,
> undo:
> for (pfn = start_pfn;
> pfn < undo_pfn;
> - pfn += pageblock_nr_pages)
> - unset_migratetype_isolate(pfn_to_page(pfn), migratetype);
> + pfn += pageblock_nr_pages) {
> + struct page *page = pfn_to_online_page(pfn);
> + if (!page)
> + continue;
> + unset_migratetype_isolate(page, migratetype);
> + }
>
> return -EBUSY;
> }
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [RFC PATCH] mm, oom: cgroup-aware OOM-killer
From: Michal Hocko @ 2017-05-19 8:02 UTC (permalink / raw)
To: Johannes Weiner
Cc: Roman Gushchin, Tejun Heo, Li Zefan, Vladimir Davydov,
Tetsuo Handa, kernel-team, cgroups, linux-doc, linux-kernel,
linux-mm
In-Reply-To: <20170518181117.GA27689@cmpxchg.org>
On Thu 18-05-17 14:11:17, Johannes Weiner wrote:
> On Thu, May 18, 2017 at 07:30:04PM +0200, Michal Hocko wrote:
> > On Thu 18-05-17 17:28:04, Roman Gushchin wrote:
> > > Traditionally, the OOM killer is operating on a process level.
> > > Under oom conditions, it finds a process with the highest oom score
> > > and kills it.
> > >
> > > This behavior doesn't suit well the system with many running
> > > containers. There are two main issues:
> > >
> > > 1) There is no fairness between containers. A small container with
> > > a few large processes will be chosen over a large one with huge
> > > number of small processes.
> > >
> > > 2) Containers often do not expect that some random process inside
> > > will be killed. So, in general, a much safer behavior is
> > > to kill the whole cgroup. Traditionally, this was implemented
> > > in userspace, but doing it in the kernel has some advantages,
> > > especially in a case of a system-wide OOM.
> > >
> > > To address these issues, cgroup-aware OOM killer is introduced.
> > > Under OOM conditions, it looks for a memcg with highest oom score,
> > > and kills all processes inside.
> > >
> > > Memcg oom score is calculated as a size of active and inactive
> > > anon LRU lists, unevictable LRU list and swap size.
> > >
> > > For a cgroup-wide OOM, only cgroups belonging to the subtree of
> > > the OOMing cgroup are considered.
> >
> > While this might make sense for some workloads/setups it is not a
> > generally acceptable policy IMHO. We have discussed that different OOM
> > policies might be interesting few years back at LSFMM but there was no
> > real consensus on how to do that. One possibility was to allow bpf like
> > mechanisms. Could you explore that path?
>
> OOM policy is an orthogonal discussion, though.
>
> The OOM killer's job is to pick a memory consumer to kill. Per default
> the unit of the memory consumer is a process, but cgroups allow
> grouping processes into compound consumers. Extending the OOM killer
> to respect the new definition of "consumer" is not a new policy.
I do not want to play word games here but picking a task or more tasks
is a policy from my POV but that is not all that important. My primary
point is that this new "implementation" is most probably not what people
who use memory cgroups outside of containers want. Why? Mostly because
they do not care that only a part of the memcg is still alive pretty
much like the current global OOM behavior when a single task (or its
children) are gone all of the sudden. Why should I kill the whole user
slice just because one of its processes went wild?
> I don't think it's reasonable to ask the person who's trying to make
> the OOM killer support group-consumers to design a dynamic OOM policy
> framework instead.
>
> All we want is the OOM policy, whatever it is, applied to cgroups.
And I am not dismissing this usecase. I believe it is valid but not
universally applicable when memory cgroups are deployed. That is why
I think that we need a way to define those policies in some sane way.
Our current oom policies are basically random -
/proc/sys/vm/oom_kill_allocating_task resp. /proc/sys/vm/panic_on_oom.
I am not really sure we want another hardcoded one e.g.
/proc/sys/vm/oom_kill_container because even that might turn out not the
great fit for different container usecases. Do we want to kill the
largest container or the one with the largest memory hog? Should some
containers have a higher priority over others? I am pretty sure more
criterion would pop up with more usecases.
That's why I think that the current OOM killer implementation should
stay as a last resort and be process oriented and we should think about
a way to override it for particular usecases. The exact mechanism is not
completely clear to me to be honest.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH] dm ioctl: Restore __GFP_HIGH in copy_params()
From: Michal Hocko @ 2017-05-19 7:46 UTC (permalink / raw)
To: Junaid Shahid
Cc: David Rientjes, Alasdair Kergon, Mike Snitzer, Andrew Morton,
linux-mm, andreslc, gthelen, mpatocka, vbabka, linux-kernel
In-Reply-To: <1508444.i5EqlA1upv@js-desktop.svl.corp.google.com>
On Thu 18-05-17 19:50:46, Junaid Shahid wrote:
> (Adding back the correct linux-mm email address and also adding linux-kernel.)
>
> On Thursday, May 18, 2017 01:41:33 PM David Rientjes wrote:
[...]
> > Let's ask Mikulas, who changed this from PF_MEMALLOC to __GFP_HIGH,
> > assuming there was a reason to do it in the first place in two different
> > ways.
Hmm, the old PF_MEMALLOC used to have the following comment
/*
* Trying to avoid low memory issues when a device is
* suspended.
*/
I am not really sure what that means but __GFP_HIGH certainly have a
different semantic than PF_MEMALLOC. The later grants the full access to
the memory reserves while the prior on partial access. If this is _really_
needed then it deserves a comment explaining why.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [RFC 1/6] mm, page_alloc: fix more premature OOM due to race with cpuset update
From: Michal Hocko @ 2017-05-19 7:37 UTC (permalink / raw)
To: Christoph Lameter
Cc: Vlastimil Babka, linux-mm, linux-kernel, cgroups, Li Zefan,
Mel Gorman, David Rientjes, Hugh Dickins, Andrea Arcangeli,
Anshuman Khandual, Kirill A. Shutemov, linux-api
In-Reply-To: <alpine.DEB.2.20.1705181351120.29348@east.gentwo.org>
On Thu 18-05-17 14:07:45, Cristopher Lameter wrote:
> On Thu, 18 May 2017, Michal Hocko wrote:
>
> > > See above. OOM Kill in a cpuset does not kill an innocent task but a task
> > > that does an allocation in that specific context meaning a task in that
> > > cpuset that also has a memory policty.
> >
> > No, the oom killer will chose the largest task in the specific NUMA
> > domain. If you just fail such an allocation then a page fault would get
> > VM_FAULT_OOM and pagefault_out_of_memory would kill a task regardless of
> > the cpusets.
>
> Ok someone screwed up that code. There still is the determination that we
> have a constrained alloc:
It would be much more easier if you read emails more carefully. In order
to have a constrained OOM you have to have either a non-null nodemask or
zonelist which. And as I've said above you do not have them from the
pagefault_out_of_memory context. The whole point of this discussion is
_that_ failing allocations will not work currently!
> oom_kill:
> /*
> * Check if there were limitations on the allocation (only relevant for
> * NUMA and memcg) that may require different handling.
> */
> constraint = constrained_alloc(oc);
> if (constraint != CONSTRAINT_MEMORY_POLICY)
> oc->nodemask = NULL;
> check_panic_on_oom(oc, constraint);
>
> -- Ok. A constrained failing alloc used to terminate the allocating
> process here. But it falls through to selecting a "bad process"
This behavior is there for ~10 years.
[...]
> Can we restore the old behavior? If I just specify the right memory policy
> I can cause other processes to just be terminated?
Not normally. Because out_of_memory called from the page allocator
context makes sure to kill tasks from the same NUMA domain (see
oom_unkillable_task).
> > > Regardless of that the point earlier was that the moving logic can avoid
> > > creating temporary situations of empty sets of nodes by analysing the
> > > memory policies etc and only performing moves when doing so is safe.
> >
> > How are you going to do that in a raceless way? Moreover the whole
> > discussion is about _failing_ allocations on an empty cpuset and
> > mempolicy intersection.
>
> Again this is only working for processes that are well behaved and it
> never worked in a different way before. There was always the assumption
> that a process does not allocate in the areas that have allocation
> constraints and that the process does not change memory policies nor
> store them somewhere for late etc etc. HPC apps typically allocate memory
> on startup and then go through long times of processing and I/O.
I would call it a bad design which then triggered a lot of work to make
it semi-working over years. This is what Vlastimil tries to address now.
And yes that might mean we would have to do some restrictions on the
semantics. But as you know this is a user visible API and changing
something that has been fundamentally underdefined initially is quite
hard to fix.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH 08/14] mm, compaction: skip over holes in __reset_isolation_suitable
From: Vlastimil Babka @ 2017-05-19 7:23 UTC (permalink / raw)
To: Michal Hocko, Andrew Morton
Cc: linux-mm, Mel Gorman, Andrea Arcangeli, Jerome Glisse, Reza Arbab,
Yasuaki Ishimatsu, qiuxishi, Kani Toshimitsu, slaoub, Joonsoo Kim,
Andi Kleen, David Rientjes, Daniel Kiper, Igor Mammedov,
Vitaly Kuznetsov, LKML, Michal Hocko
In-Reply-To: <20170515085827.16474-9-mhocko@kernel.org>
On 05/15/2017 10:58 AM, Michal Hocko wrote:
> From: Michal Hocko <mhocko@suse.com>
>
> __reset_isolation_suitable walks the whole zone pfn range and it tries
> to jump over holes by checking the zone for each page. It might still
> stumble over offline pages, though. Skip those by checking
> pfn_to_online_page()
>
> Signed-off-by: Michal Hocko <mhocko@suse.com>
Acked-by: Vlastimil Babka <vbabka@suse.cz>
> ---
> mm/compaction.c | 5 ++---
> 1 file changed, 2 insertions(+), 3 deletions(-)
>
> diff --git a/mm/compaction.c b/mm/compaction.c
> index 613c59e928cb..fb548e4c7bd4 100644
> --- a/mm/compaction.c
> +++ b/mm/compaction.c
> @@ -236,10 +236,9 @@ static void __reset_isolation_suitable(struct zone *zone)
>
> cond_resched();
>
> - if (!pfn_valid(pfn))
> + page = pfn_to_online_page(pfn);
> + if (!page)
> continue;
> -
> - page = pfn_to_page(pfn);
> if (zone != page_zone(page))
> continue;
>
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [mmotm:master 61/139] include/asm-generic/memory_model.h:54:52: warning: 'page' is used uninitialized in this function
From: Michal Hocko @ 2017-05-19 7:22 UTC (permalink / raw)
To: Andrew Morton
Cc: kbuild-all, Johannes Weiner, kbuild test robot,
Linux Memory Management List
In-Reply-To: <201705190738.Nkd5Ar0X%fengguang.wu@intel.com>
Ups, sorry about that. Andrew could you fold the following into
mm-vmstat-skip-reporting-offline-pages-in-pagetypeinfo.patch
---
^ permalink raw reply
* Re: [PATCH 07/14] mm: consider zone which is not fully populated to have holes
From: Vlastimil Babka @ 2017-05-19 7:21 UTC (permalink / raw)
To: Michal Hocko
Cc: Andrew Morton, linux-mm, Mel Gorman, Andrea Arcangeli,
Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML
In-Reply-To: <20170518164210.GD18333@dhcp22.suse.cz>
On 05/18/2017 06:42 PM, Michal Hocko wrote:
> On Thu 18-05-17 18:14:39, Vlastimil Babka wrote:
>> On 05/15/2017 10:58 AM, Michal Hocko wrote:
> [...]
>>> #ifdef CONFIG_MEMORY_HOTPLUG
>>> +/*
>>> + * Return page for the valid pfn only if the page is online. All pfn
>>> + * walkers which rely on the fully initialized page->flags and others
>>> + * should use this rather than pfn_valid && pfn_to_page
>>> + */
>>> +#define pfn_to_online_page(pfn) \
>>> +({ \
>>> + struct page *___page = NULL; \
>>> + \
>>> + if (online_section_nr(pfn_to_section_nr(pfn))) \
>>> + ___page = pfn_to_page(pfn); \
>>> + ___page; \
>>> +})
>>
>> This seems to be already assuming pfn_valid() to be true. There's no
>> "pfn_to_section_nr(pfn) >= NR_MEM_SECTIONS" check and the comment
>> suggests as such, but...
>
> Yes, we should check the validity of the section number. We do not have
> to check whether the section is valid because online sections are a
> subset of those that are valid.
>
>>> diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
>>> index 05796ee974f7..c3a146028ba6 100644
>>> --- a/mm/memory_hotplug.c
>>> +++ b/mm/memory_hotplug.c
>>> @@ -929,6 +929,9 @@ static int online_pages_range(unsigned long start_pfn, unsigned long nr_pages,
>>> unsigned long i;
>>> unsigned long onlined_pages = *(unsigned long *)arg;
>>> struct page *page;
>>> +
>>> + online_mem_sections(start_pfn, start_pfn + nr_pages);
>>
>> Shouldn't this be moved *below* the loop that initializes struct pages?
>> In the offline case you do mark sections offline before "tearing" struct
>> pages, so that should be symmetric.
>
> You are right! Andrew, could you fold the following intot the patch?
> ---
> From 0550b61203d6970b47fd79f5e6372dccd143cbec Mon Sep 17 00:00:00 2001
> From: Michal Hocko <mhocko@suse.com>
> Date: Thu, 18 May 2017 18:38:24 +0200
> Subject: [PATCH] fold me "mm: consider zone which is not fully populated to
> have holes"
>
> - check valid section number in pfn_to_online_page - Vlastimil
> - mark sections online after all struct pages are initialized in
> online_pages_range - Vlastimil
>
> Signed-off-by: Michal Hocko <mhocko@suse.com>
Both the patch and fix:
Acked-by: Vlastimil Babka <vbabka@suse.cz>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* [PATCH] ext4: handle the rest of ext4_mb_load_buddy() ENOMEM errors
From: Konstantin Khlebnikov @ 2017-05-19 7:09 UTC (permalink / raw)
To: Andreas Dilger, linux-ext4, Theodore Ts'o, linux-kernel; +Cc: linux-mm
I've got another report about breaking ext4 by ENOMEM error returned from
ext4_mb_load_buddy() caused by memory shortage in memory cgroup.
This time inside ext4_discard_preallocations().
This patch replaces ext4_error() with ext4_warning() where errors returned
from ext4_mb_load_buddy() are not fatal and handled by caller:
* ext4_mb_discard_group_preallocations() - called before generating ENOSPC,
we'll try to discard other group or return ENOSPC into user-space.
* ext4_trim_all_free() - just stop trimming and return ENOMEM from ioctl.
Some callers cannot handle errors, thus __GFP_NOFAIL is used for them:
* ext4_discard_preallocations()
* ext4_mb_discard_lg_preallocations()
The only unclear case is ext4_group_add_blocks(), probably ext4_std_error()
should handle ENOMEM as warning and don't break filesystem.
Fixes: adb7ef600cc9 ("ext4: use __GFP_NOFAIL in ext4_free_blocks()")
Signed-off-by: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>
---
fs/ext4/mballoc.c | 23 ++++++++++++++---------
1 file changed, 14 insertions(+), 9 deletions(-)
diff --git a/fs/ext4/mballoc.c b/fs/ext4/mballoc.c
index 5083bce20ac4..b7928cddd539 100644
--- a/fs/ext4/mballoc.c
+++ b/fs/ext4/mballoc.c
@@ -3887,7 +3887,8 @@ ext4_mb_discard_group_preallocations(struct super_block *sb,
err = ext4_mb_load_buddy(sb, group, &e4b);
if (err) {
- ext4_error(sb, "Error loading buddy information for %u", group);
+ ext4_warning(sb, "Error %d loading buddy information for %u",
+ err, group);
put_bh(bitmap_bh);
return 0;
}
@@ -4044,10 +4045,11 @@ void ext4_discard_preallocations(struct inode *inode)
BUG_ON(pa->pa_type != MB_INODE_PA);
group = ext4_get_group_number(sb, pa->pa_pstart);
- err = ext4_mb_load_buddy(sb, group, &e4b);
+ err = ext4_mb_load_buddy_gfp(sb, group, &e4b,
+ GFP_NOFS|__GFP_NOFAIL);
if (err) {
- ext4_error(sb, "Error loading buddy information for %u",
- group);
+ ext4_error(sb, "Error %d loading buddy information for %u",
+ err, group);
continue;
}
@@ -4303,11 +4305,14 @@ ext4_mb_discard_lg_preallocations(struct super_block *sb,
spin_unlock(&lg->lg_prealloc_lock);
list_for_each_entry_safe(pa, tmp, &discard_list, u.pa_tmp_list) {
+ int err;
group = ext4_get_group_number(sb, pa->pa_pstart);
- if (ext4_mb_load_buddy(sb, group, &e4b)) {
- ext4_error(sb, "Error loading buddy information for %u",
- group);
+ err = ext4_mb_load_buddy_gfp(sb, group, &e4b,
+ GFP_NOFS|__GFP_NOFAIL);
+ if (err) {
+ ext4_error(sb, "Error %d loading buddy information for %u",
+ err, group);
continue;
}
ext4_lock_group(sb, group);
@@ -5127,8 +5132,8 @@ ext4_trim_all_free(struct super_block *sb, ext4_group_t group,
ret = ext4_mb_load_buddy(sb, group, &e4b);
if (ret) {
- ext4_error(sb, "Error in loading buddy "
- "information for %u", group);
+ ext4_warning(sb, "Error %d loading buddy information for %u",
+ ret, group);
return ret;
}
bitmap = e4b.bd_bitmap;
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [PATCH] mm/vmstat: add oom_kill counter
From: Konstantin Khlebnikov @ 2017-05-19 6:59 UTC (permalink / raw)
To: linux-mm, Andrew Morton, linux-kernel
Show count of global oom killer invocations in /proc/vmstat
Signed-off-by: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>
---
include/linux/vm_event_item.h | 1 +
mm/oom_kill.c | 1 +
mm/vmstat.c | 1 +
3 files changed, 3 insertions(+)
diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h
index d84ae90ccd5c..1707e0a7d943 100644
--- a/include/linux/vm_event_item.h
+++ b/include/linux/vm_event_item.h
@@ -41,6 +41,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
KSWAPD_LOW_WMARK_HIT_QUICKLY, KSWAPD_HIGH_WMARK_HIT_QUICKLY,
PAGEOUTRUN, PGROTATED,
DROP_PAGECACHE, DROP_SLAB,
+ OOM_KILL,
#ifdef CONFIG_NUMA_BALANCING
NUMA_PTE_UPDATES,
NUMA_HUGE_PTE_UPDATES,
diff --git a/mm/oom_kill.c b/mm/oom_kill.c
index 04c9143a8625..c734c42826cf 100644
--- a/mm/oom_kill.c
+++ b/mm/oom_kill.c
@@ -883,6 +883,7 @@ static void oom_kill_process(struct oom_control *oc, const char *message)
*/
do_send_sig_info(SIGKILL, SEND_SIG_FORCED, victim, true);
mark_oom_victim(victim);
+ count_vm_event(OOM_KILL);
pr_err("Killed process %d (%s) total-vm:%lukB, anon-rss:%lukB, file-rss:%lukB, shmem-rss:%lukB\n",
task_pid_nr(victim), victim->comm, K(victim->mm->total_vm),
K(get_mm_counter(victim->mm, MM_ANONPAGES)),
diff --git a/mm/vmstat.c b/mm/vmstat.c
index 76f73670200a..fe80b81a86e0 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -1018,6 +1018,7 @@ const char * const vmstat_text[] = {
"drop_pagecache",
"drop_slab",
+ "oom_kill",
#ifdef CONFIG_NUMA_BALANCING
"numa_pte_updates",
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* Re: [PATCH v4 05/27] btrfs: btrfs_wait_tree_block_writeback can be void return
From: Liu Bo @ 2017-05-19 4:07 UTC (permalink / raw)
To: Jeff Layton
Cc: linux-fsdevel, linux-kernel, linux-btrfs, linux-ext4, linux-cifs,
linux-nfs, linux-mm, jfs-discussion, linux-xfs, cluster-devel,
linux-f2fs-devel, v9fs-developer, linux-nilfs, linux-block,
dhowells, akpm, hch, ross.zwisler, mawilcox, jack, viro, corbet,
neilb, clm, tytso, axboe, josef, hubcap, rpeterso
In-Reply-To: <20170509154930.29524-6-jlayton@redhat.com>
On Tue, May 09, 2017 at 11:49:08AM -0400, Jeff Layton wrote:
> Nothing checks its return value.
Reviewed-by: Liu Bo <bo.li.liu@oracle.com>
-liubo
>
> Signed-off-by: Jeff Layton <jlayton@redhat.com>
> ---
> fs/btrfs/disk-io.c | 6 +++---
> fs/btrfs/disk-io.h | 2 +-
> 2 files changed, 4 insertions(+), 4 deletions(-)
>
> diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c
> index eb1ee7b6f532..8c479bd5534a 100644
> --- a/fs/btrfs/disk-io.c
> +++ b/fs/btrfs/disk-io.c
> @@ -1222,10 +1222,10 @@ int btrfs_write_tree_block(struct extent_buffer *buf)
> buf->start + buf->len - 1);
> }
>
> -int btrfs_wait_tree_block_writeback(struct extent_buffer *buf)
> +void btrfs_wait_tree_block_writeback(struct extent_buffer *buf)
> {
> - return filemap_fdatawait_range(buf->pages[0]->mapping,
> - buf->start, buf->start + buf->len - 1);
> + filemap_fdatawait_range(buf->pages[0]->mapping,
> + buf->start, buf->start + buf->len - 1);
> }
>
> struct extent_buffer *read_tree_block(struct btrfs_fs_info *fs_info, u64 bytenr,
> diff --git a/fs/btrfs/disk-io.h b/fs/btrfs/disk-io.h
> index 2e0ec29bfd69..9cc87835abb5 100644
> --- a/fs/btrfs/disk-io.h
> +++ b/fs/btrfs/disk-io.h
> @@ -127,7 +127,7 @@ int btrfs_wq_submit_bio(struct btrfs_fs_info *fs_info, struct inode *inode,
> extent_submit_bio_hook_t *submit_bio_done);
> unsigned long btrfs_async_submit_limit(struct btrfs_fs_info *info);
> int btrfs_write_tree_block(struct extent_buffer *buf);
> -int btrfs_wait_tree_block_writeback(struct extent_buffer *buf);
> +void btrfs_wait_tree_block_writeback(struct extent_buffer *buf);
> int btrfs_init_log_root_tree(struct btrfs_trans_handle *trans,
> struct btrfs_fs_info *fs_info);
> int btrfs_add_log_tree(struct btrfs_trans_handle *trans,
> --
> 2.9.3
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH] dm ioctl: Restore __GFP_HIGH in copy_params()
From: Junaid Shahid @ 2017-05-19 2:50 UTC (permalink / raw)
To: David Rientjes
Cc: Michal Hocko, Alasdair Kergon, Mike Snitzer, Andrew Morton,
linux-mm, andreslc, gthelen, mpatocka, vbabka, linux-kernel
In-Reply-To: <alpine.DEB.2.10.1705181338090.132717@chino.kir.corp.google.com>
(Adding back the correct linux-mm email address and also adding linux-kernel.)
On Thursday, May 18, 2017 01:41:33 PM David Rientjes wrote:
> On Thu, 18 May 2017, Michal Hocko wrote:
>
> > On Thu 18-05-17 11:50:40, Junaid Shahid wrote:
> > > d224e9381897 (drivers/md/dm-ioctl.c: use kvmalloc rather than opencoded
> > > variant) left out the __GFP_HIGH flag when converting from __vmalloc to
> > > kvmalloc. This can cause the IOCTL to fail in some low memory situations
> > > where it wouldn't have failed earlier. This patch adds it back to avoid
> > > any potential regression.
> >
> > The code previously used __GFP_HIGH only for the vmalloc fallback and
> > that doesn't make that much sense with the current implementation
> > because vmalloc does order-0 pages and those do not really fail and the
> > oom killer is invoked to free memory.
> >
>
> Order-0 pages certainly do fail, there is not an infinite amount of memory
> nor is there a specific exemption to allow order-0 memory to be alloctable
> below watermarks without this gfp flag. OOM kill is the last thing we
> want for these allocations since they are very temporary.
>
> > There is no reason to access memory reserves from this context.
> >
>
> Let's ask Mikulas, who changed this from PF_MEMALLOC to __GFP_HIGH,
> assuming there was a reason to do it in the first place in two different
> ways.
>
> This decision is up to the device mapper maintainers.
>
> > > Signed-off-by: Junaid Shahid <junaids@google.com>
> > > ---
> > > drivers/md/dm-ioctl.c | 2 +-
> > > 1 file changed, 1 insertion(+), 1 deletion(-)
> > >
> > > diff --git a/drivers/md/dm-ioctl.c b/drivers/md/dm-ioctl.c
> > > index 0555b4410e05..bacad7637a56 100644
> > > --- a/drivers/md/dm-ioctl.c
> > > +++ b/drivers/md/dm-ioctl.c
> > > @@ -1715,7 +1715,7 @@ static int copy_params(struct dm_ioctl __user *user, struct dm_ioctl *param_kern
> > > */
> > > dmi = NULL;
> > > noio_flag = memalloc_noio_save();
> > > - dmi = kvmalloc(param_kernel->data_size, GFP_KERNEL);
> > > + dmi = kvmalloc(param_kernel->data_size, GFP_KERNEL | __GFP_HIGH);
> > > memalloc_noio_restore(noio_flag);
> > >
> > > if (!dmi) {
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* [mmotm:master] BUILD REGRESSION ecfddce33be51ebedf97f1e03d954c14e575afb6
From: kbuild test robot @ 2017-05-19 2:08 UTC (permalink / raw)
To: Andrew Morton; +Cc: Johannes Weiner, Linux Memory Management List
git://git.cmpxchg.org/linux-mmotm.git master
ecfddce33be51ebedf97f1e03d954c14e575afb6 pci: test for unexpectedly disabled bridges
include/asm-generic/memory_model.h:54:52: warning: 'page' is used uninitialized in this function [-Wuninitialized]
include/asm-generic/memory_model.h:54:52: warning: 'page' may be used uninitialized in this function [-Wmaybe-uninitialized]
mm/vmstat.c:1224:16: warning: 'page' may be used uninitialized in this function [-Wmaybe-uninitialized]
mm/vmstat.c:1230:8: warning: 'page' may be used uninitialized in this function [-Wmaybe-uninitialized]
Error ids grouped by kconfigs:
recent_errors
a??a??a?? alpha-allyesconfig
a??A A a??a??a?? mm-vmstat.c:warning:page-may-be-used-uninitialized-in-this-function
a??a??a?? arm64-defconfig
a??A A a??a??a?? mm-vmstat.c:warning:page-may-be-used-uninitialized-in-this-function
a??a??a?? arm-exynos_defconfig
a??A A a??a??a?? mm-vmstat.c:warning:page-may-be-used-uninitialized-in-this-function
a??a??a?? arm-multi_v7_defconfig
a??A A a??a??a?? mm-vmstat.c:warning:page-may-be-used-uninitialized-in-this-function
a??a??a?? i386-allmodconfig
a??A A a??a??a?? mm-vmstat.c:warning:page-may-be-used-uninitialized-in-this-function
a??a??a?? i386-defconfig
a??A A a??a??a?? mm-vmstat.c:warning:page-may-be-used-uninitialized-in-this-function
a??a??a?? x86_64-allmodconfig
a??A A a??a??a?? mm-vmstat.c:warning:page-may-be-used-uninitialized-in-this-function
a??a??a?? x86_64-kexec
a??A A a??a??a?? include-asm-generic-memory_model.h:warning:page-is-used-uninitialized-in-this-function
a??a??a?? x86_64-rhel
a??a??a?? include-asm-generic-memory_model.h:warning:page-may-be-used-uninitialized-in-this-function
elapsed time: 238m
configs tested: 112
i386 tinyconfig
x86_64 allmodconfig
arm at91_dt_defconfig
arm allnoconfig
arm efm32_defconfig
arm64 defconfig
arm multi_v5_defconfig
arm sunxi_defconfig
arm64 allnoconfig
arm exynos_defconfig
arm shmobile_defconfig
arm multi_v7_defconfig
powerpc defconfig
s390 default_defconfig
powerpc ppc64_defconfig
powerpc allnoconfig
x86_64 acpi-redef
x86_64 allyesdebian
x86_64 nfsroot
ia64 allnoconfig
ia64 defconfig
ia64 alldefconfig
x86_64 kexec
x86_64 rhel
x86_64 rhel-7.2
i386 randconfig-a0-05181312
mn10300 asb2364_defconfig
openrisc or1ksim_defconfig
um x86_64_defconfig
um i386_defconfig
frv defconfig
tile tilegx_defconfig
c6x evmc6678_defconfig
xtensa common_defconfig
m32r m32104ut_defconfig
xtensa iss_defconfig
m32r opsput_defconfig
m32r usrv_defconfig
m32r mappi3.smp_defconfig
nios2 10m50_defconfig
h8300 h8300h-sim_defconfig
parisc c3000_defconfig
parisc b180_defconfig
parisc defconfig
alpha defconfig
parisc allnoconfig
cris etrax-100lx_v2_defconfig
blackfin TCM-BF537_defconfig
blackfin BF561-EZKIT-SMP_defconfig
blackfin BF533-EZKIT_defconfig
blackfin BF526-EZBRD_defconfig
x86_64 randconfig-x019-201720
x86_64 randconfig-x010-201720
x86_64 randconfig-x015-201720
x86_64 randconfig-x014-201720
x86_64 randconfig-x012-201720
x86_64 randconfig-x018-201720
x86_64 randconfig-x017-201720
x86_64 randconfig-x016-201720
x86_64 randconfig-x013-201720
x86_64 randconfig-x011-201720
m68k sun3_defconfig
m68k multi_defconfig
m68k m5475evb_defconfig
mips jz4740
mips malta_kvm_defconfig
mips 64r6el_defconfig
mips 32r2_defconfig
mips allnoconfig
mips fuloong2e_defconfig
mips txx9
sparc defconfig
sparc64 allnoconfig
sparc64 defconfig
i386 randconfig-x016-201720
i386 randconfig-x017-201720
i386 randconfig-x012-201720
i386 randconfig-x018-201720
i386 randconfig-x010-201720
i386 randconfig-x019-201720
i386 randconfig-x013-201720
i386 randconfig-x014-201720
i386 randconfig-x015-201720
i386 randconfig-x011-201720
sh titan_defconfig
sh rsk7269_defconfig
sh sh7785lcr_32bit_defconfig
sh allnoconfig
i386 randconfig-x074-05150639
i386 randconfig-x076-05150639
i386 randconfig-x070-05150639
i386 randconfig-x077-05150639
i386 randconfig-x078-05150639
i386 randconfig-x075-05150639
i386 randconfig-x072-05150639
i386 randconfig-x071-05150639
i386 randconfig-x079-05150639
i386 randconfig-x073-05150639
i386 allnoconfig
i386 defconfig
i386 alldefconfig
x86_64 randconfig-x006-201720
x86_64 randconfig-x007-201720
x86_64 randconfig-x001-201720
x86_64 randconfig-x004-201720
x86_64 randconfig-x005-201720
x86_64 randconfig-x000-201720
x86_64 randconfig-x008-201720
x86_64 randconfig-x002-201720
x86_64 randconfig-x003-201720
x86_64 randconfig-x009-201720
i386 allmodconfig
Thanks,
Fengguang
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH v1 00/11] mm/kasan: support per-page shadow memory to reduce memory consumption
From: Joonsoo Kim @ 2017-05-19 1:53 UTC (permalink / raw)
To: Andrey Ryabinin
Cc: Andrew Morton, Alexander Potapenko, Dmitry Vyukov, kasan-dev,
linux-mm, linux-kernel, Thomas Gleixner, Ingo Molnar,
H . Peter Anvin, kernel-team
In-Reply-To: <ebcc02d9-fa2b-30b1-2260-99cdf7434487@virtuozzo.com>
On Wed, May 17, 2017 at 03:17:13PM +0300, Andrey Ryabinin wrote:
> On 05/16/2017 04:16 AM, js1304@gmail.com wrote:
> > From: Joonsoo Kim <iamjoonsoo.kim@lge.com>
> >
> > Hello, all.
> >
> > This is an attempt to recude memory consumption of KASAN. Please see
> > following description to get the more information.
> >
> > 1. What is per-page shadow memory
> >
> > This patch introduces infrastructure to support per-page shadow memory.
> > Per-page shadow memory is the same with original shadow memory except
> > the granualarity. It's one byte shows the shadow value for the page.
> > The purpose of introducing this new shadow memory is to save memory
> > consumption.
> >
> > 2. Problem of current approach
> >
> > Until now, KASAN needs shadow memory for all the range of the memory
> > so the amount of statically allocated memory is so large. It causes
> > the problem that KASAN cannot run on the system with hard memory
> > constraint. Even if KASAN can run, large memory consumption due to
> > KASAN changes behaviour of the workload so we cannot validate
> > the moment that we want to check.
> >
> > 3. How does this patch fix the problem
> >
> > This patch tries to fix the problem by reducing memory consumption for
> > the shadow memory. There are two observations.
> >
>
>
> I think that the best way to deal with your problem is to increase shadow scale size.
>
> You'll need to add tunable to gcc to control shadow size. I expect that gcc has some
> places where 8-shadow scale size is hardcoded, but it should be fixable.
>
> The kernel also have some small amount of code written with KASAN_SHADOW_SCALE_SIZE == 8 in mind,
> which should be easy to fix.
>
> Note that bigger shadow scale size requires bigger alignment of allocated memory and variables.
> However, according to comments in gcc/asan.c gcc already aligns stack and global variables and at
> 32-bytes boundary.
> So we could bump shadow scale up to 32 without increasing current stack consumption.
>
> On a small machine (1Gb) 1/32 of shadow is just 32Mb which is comparable to yours 30Mb, but I expect it to be
> much faster. More importantly, this will require only small amount of simple changes in code, which will be
> a *lot* more easier to maintain.
I agree that it is also a good option to reduce memory consumption.
Nevertheless, there are two reasons that justifies this patchset.
1) With this patchset, memory consumption isn't increased in
proportional to total memory size. Please consider my 4Gb system
example on the below. With increasing shadow scale size to 32, memory
would be consumed by 128M. However, this patchset consumed 50MB. This
difference can be larger if we run KASAN with bigger machine.
2) These two optimization can be applied simulatenously. It is just an
orthogonal feature. If shadow scale size is increased to 32, memory
consumption will be decreased in case of my patchset, too.
Therefore, I think that this patchset is useful in any case.
Note that increasing shadow scale has it's own trade-off. It requires
that the size of slab object is aligned to shadow scale. It will
increase memory consumption due to slab.
Thanks.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox