* [PATCH man-pages 0/2] some more userfault pages updates
From: Mike Rapoport @ 2017-04-27 14:14 UTC (permalink / raw)
To: Michael Kerrisk (man-pages)
Cc: Andrea Arcangeli, linux-kernel, linux-mm, linux-man,
Mike Rapoport
Hi Michael,
Here are some more updates to {ioctl_}userfaultfd.2 pages.
Mike Rapoport (2):
userfaultfd.2: start documenting non-cooperative events
ioctl_userfaultfd.2: start adding details about userfaultfd features
man2/ioctl_userfaultfd.2 | 53 ++++++++++++++++++-
man2/userfaultfd.2 | 135 ++++++++++++++++++++++++++++++++++++++++++++---
2 files changed, 179 insertions(+), 9 deletions(-)
--
1.9.1
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* [PATCH man-pages 1/2] userfaultfd.2: start documenting non-cooperative events
From: Mike Rapoport @ 2017-04-27 14:14 UTC (permalink / raw)
To: Michael Kerrisk (man-pages)
Cc: Andrea Arcangeli, linux-kernel, linux-mm, linux-man,
Mike Rapoport
In-Reply-To: <1493302474-4701-1-git-send-email-rppt@linux.vnet.ibm.com>
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
---
man2/userfaultfd.2 | 135 ++++++++++++++++++++++++++++++++++++++++++++++++++---
1 file changed, 128 insertions(+), 7 deletions(-)
diff --git a/man2/userfaultfd.2 b/man2/userfaultfd.2
index cfea5cb..44af3e4 100644
--- a/man2/userfaultfd.2
+++ b/man2/userfaultfd.2
@@ -75,7 +75,7 @@ flag in
.PP
When the last file descriptor referring to a userfaultfd object is closed,
all memory ranges that were registered with the object are unregistered
-and unread page-fault events are flushed.
+and unread events are flushed.
.\"
.SS Usage
The userfaultfd mechanism is designed to allow a thread in a multithreaded
@@ -99,6 +99,20 @@ In such non-cooperative mode,
the process that monitors userfaultfd and handles page faults
needs to be aware of the changes in the virtual memory layout
of the faulting process to avoid memory corruption.
+
+Starting from Linux 4.11,
+userfaultfd may notify the fault-handling threads about changes
+in the virtual memory layout of the faulting process.
+In addition, if the faulting process invokes
+.BR fork (2)
+system call,
+the userfaultfd objects associated with the parent may be duplicated
+into the child process and the userfaultfd monitor will be notified
+about the file descriptor associated with the userfault objects
+created for the child process,
+which allows userfaultfd monitor to perform user-space paging
+for the child process.
+
.\" FIXME elaborate about non-cooperating mode, describe its limitations
.\" for kernels before 4.11, features added in 4.11
.\" and limitations remaining in 4.11
@@ -144,6 +158,10 @@ Details of the various
operations can be found in
.BR ioctl_userfaultfd (2).
+Since Linux 4.11, events other than page-fault may enabled during
+.B UFFDIO_API
+operation.
+
Up to Linux 4.11,
userfaultfd can be used only with anonymous private memory mappings.
@@ -156,7 +174,8 @@ Each
.BR read (2)
from the userfaultfd file descriptor returns one or more
.I uffd_msg
-structures, each of which describes a page-fault event:
+structures, each of which describes a page-fault event
+or an event required for the non-cooperative userfaultfd usage:
.nf
.in +4n
@@ -168,6 +187,23 @@ struct uffd_msg {
__u64 flags; /* Flags describing fault */
__u64 address; /* Faulting address */
} pagefault;
+ struct {
+ __u32 ufd; /* userfault file descriptor
+ of the child process */
+ } fork; /* since Linux 4.11 */
+ struct {
+ __u64 from; /* old address of the
+ remapped area */
+ __u64 to; /* new address of the
+ remapped area */
+ __u64 len; /* original mapping length */
+ } remap; /* since Linux 4.11 */
+ struct {
+ __u64 start; /* start address of the
+ removed area */
+ __u64 end; /* end address of the
+ removed area */
+ } remove; /* since Linux 4.11 */
...
} arg;
@@ -194,14 +230,73 @@ structure are as follows:
.TP
.I event
The type of event.
-Currently, only one value can appear in this field:
-.BR UFFD_EVENT_PAGEFAULT ,
-which indicates a page-fault event.
+Depending of the event type,
+different fields of the
+.I arg
+union represent details required for the event processing.
+The non-page-fault events are generated only when appropriate feature
+is enabled during API handshake with
+.B UFFDIO_API
+.BR ioctl (2).
+
+The following values can appear in the
+.I event
+field:
+.RS
+.TP
+.B UFFD_EVENT_PAGEFAULT
+A page-fault event.
+The page-fault details are available in the
+.I pagefault
+field.
.TP
-.I address
+.B UFFD_EVENT_FORK
+Generated when the faulting process invokes
+.BR fork (2)
+system call.
+The event details are available in the
+.I fork
+field.
+.\" FIXME descirbe duplication of userfault file descriptor during fork
+.TP
+.B UFFD_EVENT_REMAP
+Generated when the faulting process invokes
+.BR mremap (2)
+system call.
+The event details are available in the
+.I remap
+field.
+.TP
+.B UFFD_EVENT_REMOVE
+Generated when the faulting process invokes
+.BR madvise (2)
+system call with
+.BR MADV_DONTNEED
+or
+.BR MADV_REMOVE
+advice.
+The event details are available in the
+.I remove
+field.
+.TP
+.B UFFD_EVENT_UNMAP
+Generated when the faulting process unmaps a memory range,
+either explicitly using
+.BR munmap (2)
+system call or implicitly during
+.BR mmap (2)
+or
+.BR mremap (2)
+system calls.
+The event details are available in the
+.I remove
+field.
+.RE
+.TP
+.I pagefault.address
The address that triggered the page fault.
.TP
-.I flags
+.I pagefault.flags
A bit mask of flags that describe the event.
For
.BR UFFD_EVENT_PAGEFAULT ,
@@ -218,6 +313,32 @@ otherwise it is a read fault.
.\"
.\" UFFD_PAGEFAULT_FLAG_WP is not yet supported.
.RE
+.TP
+.I fork.ufd
+The file descriptor associated with the userfault object
+created for the child process
+.TP
+.I remap.from
+The original address of the memory range that was remapped using
+.BR mremap (2).
+.TP
+.I remap.to
+The new address of the memory range that was remapped using
+.BR mremap (2).
+.TP
+.I remap.len
+The original length of the the memory range that was remapped using
+.BR mremap (2).
+.TP
+.I remove.start
+The start address of the memory range that was freed using
+.BR madvise (2)
+or unmapped
+.TP
+.I remove.end
+The end address of the memory range that was freed using
+.BR madvise (2)
+or unmapped
.PP
A
.BR read (2)
--
1.9.1
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [PATCH man-pages 2/2] ioctl_userfaultfd.2: start adding details about userfaultfd features
From: Mike Rapoport @ 2017-04-27 14:14 UTC (permalink / raw)
To: Michael Kerrisk (man-pages)
Cc: Andrea Arcangeli, linux-kernel, linux-mm, linux-man,
Mike Rapoport
In-Reply-To: <1493302474-4701-1-git-send-email-rppt@linux.vnet.ibm.com>
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
---
man2/ioctl_userfaultfd.2 | 53 ++++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 51 insertions(+), 2 deletions(-)
diff --git a/man2/ioctl_userfaultfd.2 b/man2/ioctl_userfaultfd.2
index 42bf7a7..cdc07e0 100644
--- a/man2/ioctl_userfaultfd.2
+++ b/man2/ioctl_userfaultfd.2
@@ -121,22 +121,70 @@ and explicitly enable userfaultfd features that are disabled by default.
The kernel always reports all the available features in the
.I features
field.
+
+To enable userfaultfd features the application should set
+a bit corresponding to each feature it wants to enable in the
+.I features
+field.
+If the kernel supports all the requested features it will enable them.
+Otherwise it will zero out the returned
+.I uffdio_api
+structure and return
+.BR EINVAL .
.\" FIXME add more details about feature negotiation and enablement
Since Linux 4.11, the following feature bits may be set:
.TP
.B UFFD_FEATURE_EVENT_FORK
+When this feature is enabled,
+the userfaultfd objects associated with a parent process are duplicated
+into the child process during
+.BR fork (2)
+system call and the
+.I UFFD_EVENT_FORK
+is delivered to the userfaultfd monitor
.TP
.B UFFD_FEATURE_EVENT_REMAP
+If this feature is enabled,
+when the faulting process invokes
+.BR mremap (2)
+system call
+the userfaultfd monitor will receive an event of type
+.I UFFD_EVENT_REMAP.
.TP
.B UFFD_FEATURE_EVENT_REMOVE
+If this feature is enabled,
+when the faulting process calls
+.BR madvise(2)
+system call with
+.I MADV_DONTNEED
+or
+.I MADV_REMOVE
+advice to free a virtual memory area
+the userfaultfd monitor will receive an event of type
+.I UFFD_EVENT_REMOVE.
.TP
.B UFFD_FEATURE_EVENT_UNMAP
+If this feature is enabled,
+when the faulting process unmaps virtual memory either explicitly with
+.BR munmap (2)
+system call, or implicitly either during
+.BR mmap (2)
+or
+.BR mremap (2)
+system call,
+the userfaultfd monitor will receive an event of type
+.I UFFD_EVENT_UNMAP
.TP
.B UFFD_FEATURE_MISSING_HUGETLBFS
+If this feature bit is set,
+the kernel supports registering userfaultfd ranges on hugetlbfs
+virtual memory areas
.TP
.B UFFD_FEATURE_MISSING_SHMEM
-.\" FIXME add feature description
+If this feature bit is set,
+the kernel supports registering userfaultfd ranges on tmpfs
+virtual memory areas
The returned
.I ioctls
@@ -182,7 +230,8 @@ The API version requested in the
.I api
field is not supported by this kernel, or the
.I features
-field was not zero.
+field passed to the kernel includes feature bits that are not supported
+by the current kernel version.
.\" FIXME In the above error case, the returned 'uffdio_api' structure is
.\" zeroed out. Why is this done? This should be explained in the manual page.
.\"
--
1.9.1
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* Re: [PATCH v2 1/2] mm: Uncharge poisoned pages
From: Michal Hocko @ 2017-04-27 14:37 UTC (permalink / raw)
To: Laurent Dufour; +Cc: Naoya Horiguchi, linux-kernel, linux-mm, akpm
In-Reply-To: <1493130472-22843-2-git-send-email-ldufour@linux.vnet.ibm.com>
On Tue 25-04-17 16:27:51, Laurent Dufour wrote:
> When page are poisoned, they should be uncharged from the root memory
> cgroup.
>
> This is required to avoid a BUG raised when the page is onlined back:
> BUG: Bad page state in process mem-on-off-test pfn:7ae3b
> page:f000000001eb8ec0 count:0 mapcount:0 mapping: (null)
> index:0x1
> flags: 0x3ffff800200000(hwpoison)
My knowledge of memory poisoning is very rudimentary but aren't those
pages supposed to leak and never come back? In other words isn't the
hoplug code broken because it should leave them alone?
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH 1/3] mm: Silence vmap() allocation failures based on caller gfp_flags
From: Michal Hocko @ 2017-04-27 14:42 UTC (permalink / raw)
To: Florian Fainelli
Cc: linux-arm-kernel, Russell King, Catalin Marinas, Will Deacon,
Ard Biesheuvel, Andrew Morton, zijun_hu, Kirill A. Shutemov,
Andrey Ryabinin, Chris Wilson, open list,
open list:MEMORY MANAGEMENT, angus
In-Reply-To: <20170425223332.6999-4-f.fainelli@gmail.com>
On Tue 25-04-17 15:33:29, Florian Fainelli wrote:
> If the caller has set __GFP_NOWARN don't print the following message:
> vmap allocation for size 15736832 failed: use vmalloc=<size> to increase
> size.
>
> This can happen with the ARM/Linux module loader built with
> CONFIG_ARM_MODULE_PLTS=y which does a first attempt at loading a large
> module from module space, then falls back to vmalloc space.
>
> Signed-off-by: Florian Fainelli <f.fainelli@gmail.com>
> ---
> mm/vmalloc.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/mm/vmalloc.c b/mm/vmalloc.c
> index 0b057628a7ba..5a788eb58741 100644
> --- a/mm/vmalloc.c
> +++ b/mm/vmalloc.c
> @@ -521,7 +521,7 @@ static struct vmap_area *alloc_vmap_area(unsigned long size,
> }
> }
>
> - if (printk_ratelimit())
> + if (printk_ratelimit() && !(gfp_mask & __GFP_NOWARN))
Are you sure about this ordering? Should NOWARN requests alter the
ratelimit state?
> pr_warn("vmap allocation for size %lu failed: use vmalloc=<size> to increase size\n",
> size);
> kfree(va);
> --
> 2.9.3
>
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH v7 0/7] Introduce ZONE_CMA
From: Michal Hocko @ 2017-04-27 15:06 UTC (permalink / raw)
To: Joonsoo Kim
Cc: Andrew Morton, Rik van Riel, Johannes Weiner, mgorman,
Laura Abbott, Minchan Kim, Marek Szyprowski, Michal Nazarewicz,
Aneesh Kumar K . V, Vlastimil Babka, Russell King, Will Deacon,
linux-mm, linux-kernel, kernel-team
In-Reply-To: <20170425034255.GB32583@js1304-desktop>
On Tue 25-04-17 12:42:57, Joonsoo Kim wrote:
> On Mon, Apr 24, 2017 at 03:09:36PM +0200, Michal Hocko wrote:
> > On Mon 17-04-17 11:02:12, Joonsoo Kim wrote:
> > > On Thu, Apr 13, 2017 at 01:56:15PM +0200, Michal Hocko wrote:
> > > > On Wed 12-04-17 10:35:06, Joonsoo Kim wrote:
[...]
> > not for free. For most common configurations where we have ZONE_DMA,
> > ZONE_DMA32, ZONE_NORMAL and ZONE_MOVABLE all the 3 bits are already
> > consumed so a new zone will need a new one AFAICS.
>
> Yes, it requires one more bit for a new zone and it's handled by the patch.
I am pretty sure that you are aware that consuming new page flag bits
is usually a no-go and something we try to avoid as much as possible
because we are in a great shortage there. So there really have to be a
_strong_ reason if we go that way. My current understanding that the
whole zone concept is more about a more convenient implementation rather
than a fundamental change which will solve unsolvable problems with the
current approach. More on that below.
[...]
> MOVABLE allocation will fallback as following sequence.
>
> ZONE_CMA -> ZONE_MOVABLE -> ZONE_HIGHMEM -> ZONE_NORMAL -> ...
>
> I don't understand what you mean CMA allocation. In MM's context,
> there is no CMA allocation. That is just MOVABLE allocation.
>
> For device's context, there is CMA allocation. It is range specific
> allocation so it should be succeed for requested range. No fallback is
> allowed in this case.
OK. that answers my question. I guess... My main confusion comes from
__alloc_gigantic_page which shares alloc_contig_range with the cma
allocation. But from what you wrote above and my quick glance over the
code __alloc_gigantic_page simply changes the migrate type of the pfn
range and it doesn't move it to the zone CMA. Right?
[...]
> > > At a glance, special migratetype sound natural. I also did. However,
> > > it's not natural in implementation POV. Zone consists of the same type
> > > of memory (by definition ?) and MM subsystem is implemented with that
> > > assumption. If difference type of memory shares the same zone, it easily
> > > causes the problem and CMA problems are the such case.
> >
> > But this is not any different from the highmem vs. lowmem problems we
> > already have, no? I have looked at your example in the cover where you
> > mention utilization and the reclaim problems. With the node reclaim we
> > will have pages from all zones on the same LRU(s). isolate_lru_pages
> > will skip those from ZONE_CMA because their zone_idx is higher than
> > gfp_idx(GFP_KERNEL). The same could be achieved by an explicit check for
> > the pageblock migrate type. So the zone doesn't really help much. Or is
> > there some aspect that I am missing?
>
> Your understanding is correct. It can archieved by an explict check
> for migratetype. And, this is the main reason that we should avoid
> such approach.
>
> With ZONE approach, all these things are done naturally. We don't need
> any explicit check to anywhere. We already have a code to skip to
> reclaim such pages by checking zone_idx.
Yes, and as we have to filter pages anyway doing so for cma blocks
doesn't sound overly burdensome from the maintenance point of view.
> However, with MIGRATETYPE approach, all these things *cannot* be done
> naturally. We need extra checks to all the places (allocator fast
> path, reclaim path, compaction, etc...). It is really error-prone and
> it already causes many problems due to this aspect. For the
> performance wise, this approach is also bad since it requires to check
> migratetype for each pages.
>
> Moreover, even if we adds extra checks, things cannot be easily
> perfect.
I see this point and I agree that using a specific zone might be a
_nicer_ solution in the end but you have to consider another aspects as
well. The main one I am worried about is a long term maintainability.
We are really out of page flags and consuming one for a rather specific
usecase is not good. Look at ZONE_DMA. I am pretty sure that almost
no sane HW needs 16MB zone anymore, yet we have hard time to get rid
of it and so we have that memory laying around unused all the time
and blocking one page flag bit. CMA falls into a similar category
AFAIU. I wouldn't be all that surprised if a future HW will not need CMA
allocations in few years, yet we will have to fight to get rid of it
like we do with ZONE_DMA. And not only that. We will also have to fight
finding page flags for other more general usecases in the meantime.
> See 3) Atomic allocation failure problem. It's inherent
> problem if we have different types of memory in a single zone.
> We possibly can make things perfect even with MIGRATETYPE approach,
> however, it requires additional checks in hotpath than current. It's
> expensive and undesirable. It will make future maintenance of MM code
> much difficult.
I believe that the overhead in the hot path is not such a big deal. We
have means to make it 0 when CMA is not used by jumplabels. I assume
that the vast majority of systems will not use CMA. Those systems which
use CMA should be able to cope with some slight overhead IMHO.
I agree that the code maintenance cost is not free. And that is a valid
concern. CMA maintenance will not be for free in either case, though (if
for nothing else the page flags space mentioned above). Let's see what
what this means for mm/page_alloc.c
mm/page_alloc.c | 220 ++++++++++++++++++++++++++++----------------------------
1 file changed, 109 insertions(+), 111 deletions(-)
Not very convincing at first glance but this can be quite misleading as
you have already mentioned because you have moved a lot of code to to
init path. So let's just focus on the allocator hot paths
@@ -800,7 +805,7 @@ static inline void __free_one_page(struct page *page,
VM_BUG_ON(migratetype == -1);
if (likely(!is_migrate_isolate(migratetype)))
- __mod_zone_freepage_state(zone, 1 << order, migratetype);
+ __mod_zone_page_state(zone, NR_FREE_PAGES, 1 << order);
VM_BUG_ON_PAGE(pfn & ((1 << order) - 1), page);
VM_BUG_ON_PAGE(bad_range(zone, page), page);
@@ -1804,25 +1831,11 @@ static int fallbacks[MIGRATE_TYPES][4] = {
[MIGRATE_UNMOVABLE] = { MIGRATE_RECLAIMABLE, MIGRATE_MOVABLE, MIGRATE_TYPES },
[MIGRATE_RECLAIMABLE] = { MIGRATE_UNMOVABLE, MIGRATE_MOVABLE, MIGRATE_TYPES },
[MIGRATE_MOVABLE] = { MIGRATE_RECLAIMABLE, MIGRATE_UNMOVABLE, MIGRATE_TYPES },
-#ifdef CONFIG_CMA
- [MIGRATE_CMA] = { MIGRATE_TYPES }, /* Never used */
-#endif
#ifdef CONFIG_MEMORY_ISOLATION
[MIGRATE_ISOLATE] = { MIGRATE_TYPES }, /* Never used */
#endif
};
-#ifdef CONFIG_CMA
-static struct page *__rmqueue_cma_fallback(struct zone *zone,
- unsigned int order)
-{
- return __rmqueue_smallest(zone, order, MIGRATE_CMA);
-}
-#else
-static inline struct page *__rmqueue_cma_fallback(struct zone *zone,
- unsigned int order) { return NULL; }
-#endif
-
/*
* Move the free pages in a range to the free lists of the requested type.
* Note that start_page and end_pages are not aligned on a pageblock
@@ -2090,8 +2103,7 @@ static void reserve_highatomic_pageblock(struct page *page, struct zone *zone,
/* Yoink! */
mt = get_pageblock_migratetype(page);
- if (!is_migrate_highatomic(mt) && !is_migrate_isolate(mt)
- && !is_migrate_cma(mt)) {
+ if (!is_migrate_highatomic(mt) && !is_migrate_isolate(mt)) {
zone->nr_reserved_highatomic += pageblock_nr_pages;
set_pageblock_migratetype(page, MIGRATE_HIGHATOMIC);
move_freepages_block(zone, page, MIGRATE_HIGHATOMIC, NULL);
@@ -2235,13 +2247,8 @@ static struct page *__rmqueue(struct zone *zone, unsigned int order,
retry:
page = __rmqueue_smallest(zone, order, migratetype);
- if (unlikely(!page)) {
- if (migratetype == MIGRATE_MOVABLE)
- page = __rmqueue_cma_fallback(zone, order);
-
- if (!page && __rmqueue_fallback(zone, order, migratetype))
- goto retry;
- }
+ if (unlikely(!page) && __rmqueue_fallback(zone, order, migratetype))
+ goto retry;
trace_mm_page_alloc_zone_locked(page, order, migratetype);
return page;
@@ -2283,9 +2290,6 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order,
list_add_tail(&page->lru, list);
list = &page->lru;
alloced++;
- if (is_migrate_cma(get_pcppage_migratetype(page)))
- __mod_zone_page_state(zone, NR_FREE_CMA_PAGES,
- -(1 << order));
}
/*
@@ -2636,10 +2640,10 @@ int __isolate_free_page(struct page *page, unsigned int order)
* exists.
*/
watermark = min_wmark_pages(zone) + (1UL << order);
- if (!zone_watermark_ok(zone, 0, watermark, 0, ALLOC_CMA))
+ if (!zone_watermark_ok(zone, 0, watermark, 0, 0))
return 0;
- __mod_zone_freepage_state(zone, -(1UL << order), mt);
+ __mod_zone_page_state(zone, NR_FREE_PAGES, -(1UL << order));
}
/* Remove page from free list */
@@ -2655,8 +2659,8 @@ int __isolate_free_page(struct page *page, unsigned int order)
struct page *endpage = page + (1 << order) - 1;
for (; page < endpage; page += pageblock_nr_pages) {
int mt = get_pageblock_migratetype(page);
- if (!is_migrate_isolate(mt) && !is_migrate_cma(mt)
- && !is_migrate_highatomic(mt))
+ if (!is_migrate_isolate(mt) &&
+ !is_migrate_highatomic(mt))
set_pageblock_migratetype(page,
MIGRATE_MOVABLE);
}
@@ -2783,8 +2787,7 @@ struct page *rmqueue(struct zone *preferred_zone,
spin_unlock(&zone->lock);
if (!page)
goto failed;
- __mod_zone_freepage_state(zone, -(1 << order),
- get_pcppage_migratetype(page));
+ __mod_zone_page_state(zone, NR_FREE_PAGES, -(1 << order));
__count_zid_vm_events(PGALLOC, page_zonenum(page), 1 << order);
zone_statistics(preferred_zone, zone);
@@ -2907,12 +2910,6 @@ bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,
else
min -= min / 4;
-#ifdef CONFIG_CMA
- /* If allocation can't use CMA areas don't use free CMA pages */
- if (!(alloc_flags & ALLOC_CMA))
- free_pages -= zone_page_state(z, NR_FREE_CMA_PAGES);
-#endif
-
/*
* Check watermarks for an order-0 allocation request. If these
* are not met, then a high-order request also cannot go ahead
@@ -2940,13 +2937,6 @@ bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,
if (!list_empty(&area->free_list[mt]))
return true;
}
-
-#ifdef CONFIG_CMA
- if ((alloc_flags & ALLOC_CMA) &&
- !list_empty(&area->free_list[MIGRATE_CMA])) {
- return true;
- }
-#endif
}
return false;
}
@@ -2962,13 +2952,6 @@ static inline bool zone_watermark_fast(struct zone *z, unsigned int order,
unsigned long mark, int classzone_idx, unsigned int alloc_flags)
{
long free_pages = zone_page_state(z, NR_FREE_PAGES);
- long cma_pages = 0;
-
-#ifdef CONFIG_CMA
- /* If allocation can't use CMA areas don't use free CMA pages */
- if (!(alloc_flags & ALLOC_CMA))
- cma_pages = zone_page_state(z, NR_FREE_CMA_PAGES);
-#endif
/*
* Fast check for order-0 only. If this fails then the reserves
@@ -2977,7 +2960,7 @@ static inline bool zone_watermark_fast(struct zone *z, unsigned int order,
* the caller is !atomic then it'll uselessly search the free
* list. That corner case is then slower but it is harmless.
*/
- if (!order && (free_pages - cma_pages) > mark + z->lowmem_reserve[classzone_idx])
+ if (!order && free_pages > mark + z->lowmem_reserve[classzone_idx])
return true;
return __zone_watermark_ok(z, order, mark, classzone_idx, alloc_flags,
@@ -3547,10 +3530,6 @@ gfp_to_alloc_flags(gfp_t gfp_mask)
} else if (unlikely(rt_task(current)) && !in_interrupt())
alloc_flags |= ALLOC_HARDER;
-#ifdef CONFIG_CMA
- if (gfpflags_to_migratetype(gfp_mask) == MIGRATE_MOVABLE)
- alloc_flags |= ALLOC_CMA;
-#endif
return alloc_flags;
}
@@ -3972,9 +3951,6 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order,
if (should_fail_alloc_page(gfp_mask, order))
return false;
- if (IS_ENABLED(CONFIG_CMA) && ac->migratetype == MIGRATE_MOVABLE)
- *alloc_flags |= ALLOC_CMA;
-
return true;
}
This looks like a nice clean up. Those ifdefs are ugly as hell. One
could argue that some of that could be cleaned up by simply adding some
helpers (with a jump label to reduce the overhead), though. But is this
really strong enough reason to bring the whole zone in? I am not really
convinced to be honest.
[...]
> > Please do _not_ take this as a NAK from me. At least not at this time. I
> > am still trying to understand all the consequences but my intuition
> > tells me that building on top of highmem like approach will turn out to
> > be problematic in future (as we have already seen with the highmem and
> > movable zones) so this needs a very prudent consideration.
>
> I can understand that you are prudent to this issue. However, it takes more
> than two years and many people already expressed that ZONE approach is the
> way to go.
I can see a single Acked-by and one Reviewed-by. It would be much more
convincing to see much larger support. Do not take me wrong I am not
trying to undermine the feedback so far but we should be clear about one
thing. CMA is mostly motivated by the industry which tries to overcome
HW limitations which can change in future very easily. I would rather
see good enough solution for something like that than a nicer solution
which is pushing additional burden on more general usecases.
That being said, I would like to see a much larger consensus in the MM
community before a new zone is merged. I am staying very skeptical this
is the right direction though.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: your mail
From: Michal Hocko @ 2017-04-27 15:10 UTC (permalink / raw)
To: Joonsoo Kim
Cc: linux-mm, Andrew Morton, Mel Gorman, Vlastimil Babka,
Andrea Arcangeli, Jerome Glisse, Reza Arbab, Yasuaki Ishimatsu,
qiuxishi, Kani Toshimitsu, slaoub, Andi Kleen, David Rientjes,
Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML
In-Reply-To: <20170427020835.GA29169@js1304-desktop>
On Thu 27-04-17 11:08:38, Joonsoo Kim wrote:
> On Wed, Apr 26, 2017 at 11:19:06AM +0200, Michal Hocko wrote:
> > > > [...]
> > > >
> > > > > > You are trying to change a semantic of something that has a well defined
> > > > > > meaning. I disagree that we should change it. It might sound like a
> > > > > > simpler thing to do because pfn walkers will have to be checked but what
> > > > > > you are proposing is conflating two different things together.
> > > > >
> > > > > I don't think that *I* try to change the semantic of pfn_valid().
> > > > > It would be original semantic of pfn_valid().
> > > > >
> > > > > "If pfn_valid() returns true, we can get proper struct page and the
> > > > > zone information,"
> > > >
> > > > I do not see any guarantee about the zone information anywhere. In fact
> > > > this is not true with the original implementation as I've tried to
> > > > explain already. We do have new pages associated with a zone but that
> > > > association might change during the online phase. So you cannot really
> > > > rely on that information until the page is online. There is no real
> > > > change in that regards after my rework.
> > >
> > > I know that what you did doesn't change thing much. What I try to say
> > > is that previous implementation related to pfn_valid() in hotplug is
> > > wrong. Please do not assume that hotplug implementation is correct and
> > > other pfn_valid() users are incorrect. There is no design document so
> > > I'm not sure which one is correct but assumption that pfn_valid() user
> > > can access whole the struct page information makes much sense to me.
> >
> > Not really. E.g. ZONE_DEVICE pages are never online AFAIK. I believe we
> > still need pfn_valid to work for those pfns. Really, pfn_valid has a
>
> It's really contrary example to your insist. They requires not only
> struct page but also other information, especially, the zone index.
> They checks zone idx to know whether this page is for ZONE_DEVICE or not.
Yes and they guarantee this association is true. Without memory onlining
though. This memory is never online for anybody who is asking.
[...]
> I think that I did my best to explain my reasoning. It seems that we
> cannot agree with each other so it's better for some others to express
> their opinion to this problem. I will stop this discussion from now
> on.
I _do_ appreciate your feedback and if the general consensus is to
modify pfn_valid I can go that direction but my gut feeling tells me
that conflating "existing struct page" test and "fully online and
initialized" one is a wrong thing to do.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH -v3 0/13] mm: make movable onlining suck less
From: Jerome Glisse @ 2017-04-27 15:31 UTC (permalink / raw)
To: Michal Hocko
Cc: linux-mm, Andrew Morton, Mel Gorman, Vlastimil Babka,
Andrea Arcangeli, Reza Arbab, Yasuaki Ishimatsu, qiuxishi,
Kani Toshimitsu, slaoub, Joonsoo Kim, Andi Kleen, David Rientjes,
Daniel Kiper, Igor Mammedov, Vitaly Kuznetsov, LKML, Balbir Singh,
Dan Williams, Heiko Carstens, Martin Schwidefsky, Tobias Regnery,
Yasuaki Ishimatsu
In-Reply-To: <20170427075652.GA4706@dhcp22.suse.cz>
> Hi all,
> Andrew prefers to take this after the merge window so I will repost the
> full series then. Any feedback is still highly appreciated of course.
Andrew i will repost HMM too when Michal repost (unless there is no rebase
conflict but i doubt it).
Cheers,
Jérôme
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH v5 06/32] x86/mm: Add Secure Memory Encryption (SME) support
From: Borislav Petkov @ 2017-04-27 15:46 UTC (permalink / raw)
To: Tom Lendacky
Cc: linux-arch, linux-efi, kvm, linux-doc, x86, kexec, linux-kernel,
kasan-dev, linux-mm, iommu, Rik van Riel,
Radim Krčmář, Toshimitsu Kani, Arnd Bergmann,
Jonathan Corbet, Matt Fleming, Michael S. Tsirkin, Joerg Roedel,
Konrad Rzeszutek Wilk, Paolo Bonzini, Larry Woodman,
Brijesh Singh, Ingo Molnar, Andy Lutomirski, H. Peter Anvin,
Andrey Ryabinin, Alexander Potapenko, Dave Young, Thomas Gleixner,
Dmitry Vyukov
In-Reply-To: <20170418211727.10190.18774.stgit@tlendack-t1.amdoffice.net>
On Tue, Apr 18, 2017 at 04:17:27PM -0500, Tom Lendacky wrote:
> Add support for Secure Memory Encryption (SME). This initial support
> provides a Kconfig entry to build the SME support into the kernel and
> defines the memory encryption mask that will be used in subsequent
> patches to mark pages as encrypted.
...
> diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h
> new file mode 100644
> index 0000000..d5c4a2b
> --- /dev/null
> +++ b/arch/x86/include/asm/mem_encrypt.h
> @@ -0,0 +1,42 @@
> +/*
> + * AMD Memory Encryption Support
> + *
> + * Copyright (C) 2016 Advanced Micro Devices, Inc.
> + *
> + * Author: Tom Lendacky <thomas.lendacky@amd.com>
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + */
> +
These ifdeffery closing #endif markers look strange:
> +#ifndef __X86_MEM_ENCRYPT_H__
> +#define __X86_MEM_ENCRYPT_H__
> +
> +#ifndef __ASSEMBLY__
> +
> +#ifdef CONFIG_AMD_MEM_ENCRYPT
> +
> +extern unsigned long sme_me_mask;
> +
> +static inline bool sme_active(void)
> +{
> + return !!sme_me_mask;
> +}
> +
> +#else /* !CONFIG_AMD_MEM_ENCRYPT */
> +
> +#ifndef sme_me_mask
> +#define sme_me_mask 0UL
> +
> +static inline bool sme_active(void)
> +{
> + return false;
> +}
> +#endif
this endif is the sme_me_mask closing one and it has sme_active() in it.
Shouldn't it be:
#ifndef sme_me_mask
#define sme_me_mask 0UL
#endif
and have sme_active below it, in the !CONFIG_AMD_MEM_ENCRYPT branch?
The same thing is in include/linux/mem_encrypt.h
--
Regards/Gruss,
Boris.
Good mailing practices for 400: avoid top-posting and trim the reply.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH v5 31/32] x86: Add sysfs support for Secure Memory Encryption
From: Dave Hansen @ 2017-04-27 15:52 UTC (permalink / raw)
To: Dave Young
Cc: Tom Lendacky, linux-arch, linux-efi, kvm, linux-doc, x86, kexec,
linux-kernel, kasan-dev, linux-mm, iommu, Thomas Gleixner,
Rik van Riel, Brijesh Singh, Toshimitsu Kani, Arnd Bergmann,
Jonathan Corbet, Matt Fleming, Joerg Roedel,
Radim Krčmář, Konrad Rzeszutek Wilk,
Andrey Ryabinin, Ingo Molnar, Michael S. Tsirkin, Andy Lutomirski,
H. Peter Anvin, Borislav Petkov, Paolo Bonzini,
Alexander Potapenko, Larry Woodman, Dmitry Vyukov
In-Reply-To: <20170427072547.GB15297@dhcp-128-65.nay.redhat.com>
On 04/27/2017 12:25 AM, Dave Young wrote:
> On 04/21/17 at 02:55pm, Dave Hansen wrote:
>> On 04/18/2017 02:22 PM, Tom Lendacky wrote:
>>> Add sysfs support for SME so that user-space utilities (kdump, etc.) can
>>> determine if SME is active.
>>>
>>> A new directory will be created:
>>> /sys/kernel/mm/sme/
>>>
>>> And two entries within the new directory:
>>> /sys/kernel/mm/sme/active
>>> /sys/kernel/mm/sme/encryption_mask
>>
>> Why do they care, and what will they be doing with this information?
>
> Since kdump will copy old memory but need this to know if the old memory
> was encrypted or not. With this sysfs file we can know the previous SME
> status and pass to kdump kernel as like a kernel param.
>
> Tom, have you got chance to try if it works or not?
What will the kdump kernel do with it though? We kexec() into that
kernel so the SME keys will all be the same, right? So, will the kdump
kernel be just setting the encryption bit in the PTE so it can copy the
old plaintext out?
Why do we need both 'active' and 'encryption_mask'? How could it be
that the hardware-enumerated 'encryption_mask' changes across a kexec()?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* [RFC v3 00/17] Speculative page faults
From: Laurent Dufour @ 2017-04-27 15:52 UTC (permalink / raw)
To: paulmck, peterz, akpm, kirill, ak, mhocko, dave, jack
Cc: linux-kernel, linux-mm, haren, khandual, npiggin, bsingharora
This is a port on kernel 4.10 of the work done by Peter Zijlstra to
handle page fault without holding the mm semaphore.
http://linux-kernel.2935.n7.nabble.com/RFC-PATCH-0-6-Another-go-at-speculative-page-faults-tt965642.html#none
This series is functional on x86, but there may be some pending
issues. It's building on top of v4.10.
Compared to the Peter initial work, this series introduces a try spin
lock when dealing with speculative page fault. This is required to
avoid dead lock when handling a page fault while a TLB invalidate is
requested by an other CPU holding the PTE. Another change due to a
lock dependency issue with mapping->i_mmap_rwsem.
This series also protect changes to VMA's data which are read or
change by the page fault handler. The protections is done through the
VMA's sequence number.
Laurent Dufour (11):
mm: Introduce pte_spinlock
mm/spf: Try spin lock in speculative path
mm/spf: Fix fe.sequence init in __handle_mm_fault()
mm/spf: don't set fault entry's fields if locking failed
mm/spf; fix lock dependency against mapping->i_mmap_rwsem
mm/spf: Protect changes to vm_flags
mm/spf Protect vm_policy's changes against speculative pf
x86/mm: Update the handle_speculative_fault's path
mm/spf: Add check on the VMA's flags
mm: protect madvise vs speculative pf
mm/spf: protect mremap() against speculative pf
Peter Zijlstra (6):
mm: Dont assume page-table invariance during faults
mm: Prepare for FAULT_FLAG_SPECULATIVE
mm: VMA sequence count
RCU free VMAs
mm: Provide speculative fault infrastructure
mm,x86: Add speculative pagefault handling
arch/x86/mm/fault.c | 15 +++
fs/proc/task_mmu.c | 2 +
include/linux/mm.h | 4 +
include/linux/mm_types.h | 3 +
kernel/fork.c | 1 +
mm/init-mm.c | 1 +
mm/internal.h | 18 +++
mm/madvise.c | 5 +-
mm/memory.c | 284 +++++++++++++++++++++++++++++++++++++++--------
mm/mempolicy.c | 10 +-
mm/mlock.c | 9 +-
mm/mmap.c | 121 +++++++++++++++-----
mm/mprotect.c | 2 +
mm/mremap.c | 7 ++
14 files changed, 402 insertions(+), 80 deletions(-)
--
2.7.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* [RFC v3 02/17] mm: Prepare for FAULT_FLAG_SPECULATIVE
From: Laurent Dufour @ 2017-04-27 15:52 UTC (permalink / raw)
To: paulmck, peterz, akpm, kirill, ak, mhocko, dave, jack
Cc: linux-kernel, linux-mm, haren, khandual, npiggin, bsingharora
In-Reply-To: <1493308376-23851-1-git-send-email-ldufour@linux.vnet.ibm.com>
From: Peter Zijlstra <peterz@infradead.org>
When speculating faults (without holding mmap_sem) we need to validate
that the vma against which we loaded pages is still valid when we're
ready to install the new PTE.
Therefore, replace the pte_offset_map_lock() calls that (re)take the
PTL with pte_map_lock() which can fail in case we find the VMA changed
since we started the fault.
Instead of passing around the endless list of function arguments,
replace the lot with a single structure so we can change context
without endless function signature changes.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
[port to 4.10 kernel]
Signed-off-by: Laurent Dufour <ldufour@linux.vnet.ibm.com>
---
include/linux/mm.h | 1 +
mm/memory.c | 57 +++++++++++++++++++++++++++++++++++++++---------------
2 files changed, 42 insertions(+), 16 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index b84615b0f64c..555ac9ac7202 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -280,6 +280,7 @@ extern pgprot_t protection_map[16];
#define FAULT_FLAG_USER 0x40 /* The fault originated in userspace */
#define FAULT_FLAG_REMOTE 0x80 /* faulting for non current tsk/mm */
#define FAULT_FLAG_INSTRUCTION 0x100 /* The fault was during an instruction fetch */
+#define FAULT_FLAG_SPECULATIVE 0x200 /* Speculative fault, not holding mmap_sem */
/*
* vm_fault is filled by the the pagefault handler and passed to the vma's
diff --git a/mm/memory.c b/mm/memory.c
index 374b99de75a5..bce32c9d73c2 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2100,6 +2100,12 @@ static inline void wp_page_reuse(struct vm_fault *vmf)
pte_unmap_unlock(vmf->pte, vmf->ptl);
}
+static bool pte_map_lock(struct vm_fault *vmf)
+{
+ vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl);
+ return true;
+}
+
/*
* Handle the case of a page which we actually need to copy to a new page.
*
@@ -2127,6 +2133,7 @@ static int wp_page_copy(struct vm_fault *vmf)
const unsigned long mmun_start = vmf->address & PAGE_MASK;
const unsigned long mmun_end = mmun_start + PAGE_SIZE;
struct mem_cgroup *memcg;
+ int ret = VM_FAULT_OOM;
if (unlikely(anon_vma_prepare(vma)))
goto oom;
@@ -2154,7 +2161,11 @@ static int wp_page_copy(struct vm_fault *vmf)
/*
* Re-check the pte - we dropped the lock
*/
- vmf->pte = pte_offset_map_lock(mm, vmf->pmd, vmf->address, &vmf->ptl);
+ if (!pte_map_lock(vmf)) {
+ mem_cgroup_cancel_charge(new_page, memcg, false);
+ ret = VM_FAULT_RETRY;
+ goto oom_free_new;
+ }
if (likely(pte_same(*vmf->pte, vmf->orig_pte))) {
if (old_page) {
if (!PageAnon(old_page)) {
@@ -2242,7 +2253,7 @@ static int wp_page_copy(struct vm_fault *vmf)
oom:
if (old_page)
put_page(old_page);
- return VM_FAULT_OOM;
+ return ret;
}
/**
@@ -2263,8 +2274,8 @@ static int wp_page_copy(struct vm_fault *vmf)
int finish_mkwrite_fault(struct vm_fault *vmf)
{
WARN_ON_ONCE(!(vmf->vma->vm_flags & VM_SHARED));
- vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd, vmf->address,
- &vmf->ptl);
+ if (!pte_map_lock(vmf))
+ return VM_FAULT_RETRY;
/*
* We might have raced with another page fault while we released the
* pte_offset_map_lock.
@@ -2382,8 +2393,11 @@ static int do_wp_page(struct vm_fault *vmf)
get_page(vmf->page);
pte_unmap_unlock(vmf->pte, vmf->ptl);
lock_page(vmf->page);
- vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
- vmf->address, &vmf->ptl);
+ if (!pte_map_lock(vmf)) {
+ unlock_page(vmf->page);
+ put_page(vmf->page);
+ return VM_FAULT_RETRY;
+ }
if (!pte_same(*vmf->pte, vmf->orig_pte)) {
unlock_page(vmf->page);
pte_unmap_unlock(vmf->pte, vmf->ptl);
@@ -2541,8 +2555,10 @@ int do_swap_page(struct vm_fault *vmf)
* Back out if somebody else faulted in this pte
* while we released the pte lock.
*/
- vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
- vmf->address, &vmf->ptl);
+ if (!pte_map_lock(vmf)) {
+ delayacct_clear_flag(DELAYACCT_PF_SWAPIN);
+ return VM_FAULT_RETRY;
+ }
if (likely(pte_same(*vmf->pte, vmf->orig_pte)))
ret = VM_FAULT_OOM;
delayacct_clear_flag(DELAYACCT_PF_SWAPIN);
@@ -2598,8 +2614,11 @@ int do_swap_page(struct vm_fault *vmf)
/*
* Back out if somebody else already faulted in this pte.
*/
- vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address,
- &vmf->ptl);
+ if (!pte_map_lock(vmf)) {
+ ret = VM_FAULT_RETRY;
+ mem_cgroup_cancel_charge(page, memcg, false);
+ goto out_page;
+ }
if (unlikely(!pte_same(*vmf->pte, vmf->orig_pte)))
goto out_nomap;
@@ -2763,8 +2782,8 @@ static int do_anonymous_page(struct vm_fault *vmf)
!mm_forbids_zeropage(vma->vm_mm)) {
entry = pte_mkspecial(pfn_pte(my_zero_pfn(vmf->address),
vma->vm_page_prot));
- vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
- vmf->address, &vmf->ptl);
+ if (!pte_map_lock(vmf))
+ return VM_FAULT_RETRY;
if (!pte_none(*vmf->pte))
goto unlock;
/* Deliver the page fault to userland, check inside PT lock */
@@ -2796,8 +2815,12 @@ static int do_anonymous_page(struct vm_fault *vmf)
if (vma->vm_flags & VM_WRITE)
entry = pte_mkwrite(pte_mkdirty(entry));
- vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address,
- &vmf->ptl);
+ if (!pte_map_lock(vmf)) {
+ /* XXX: should be factorized */
+ mem_cgroup_cancel_charge(page, memcg, false);
+ put_page(page);
+ return VM_FAULT_RETRY;
+ }
if (!pte_none(*vmf->pte))
goto release;
@@ -2897,8 +2920,9 @@ static int pte_alloc_one_map(struct vm_fault *vmf)
if (pmd_trans_unstable(vmf->pmd) || pmd_devmap(*vmf->pmd))
return VM_FAULT_NOPAGE;
- vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address,
- &vmf->ptl);
+ if (!pte_map_lock(vmf))
+ return VM_FAULT_RETRY;
+
return 0;
}
@@ -3218,6 +3242,7 @@ static int do_read_fault(struct vm_fault *vmf)
* something).
*/
if (vma->vm_ops->map_pages && fault_around_bytes >> PAGE_SHIFT > 1) {
+ /* XXX: is a call to pte_map_lock(fe) required here ? */
ret = do_fault_around(vmf);
if (ret)
return ret;
--
2.7.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [RFC v3 01/17] mm: Dont assume page-table invariance during faults
From: Laurent Dufour @ 2017-04-27 15:52 UTC (permalink / raw)
To: paulmck, peterz, akpm, kirill, ak, mhocko, dave, jack
Cc: linux-kernel, linux-mm, haren, khandual, npiggin, bsingharora
In-Reply-To: <1493308376-23851-1-git-send-email-ldufour@linux.vnet.ibm.com>
From: Peter Zijlstra <peterz@infradead.org>
One of the side effects of speculating on faults (without holding
mmap_sem) is that we can race with free_pgtables() and therefore we
cannot assume the page-tables will stick around.
Remove the relyance on the pte pointer.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
mm/memory.c | 27 ---------------------------
1 file changed, 27 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index 6bf2b471e30c..374b99de75a5 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -1962,30 +1962,6 @@ int apply_to_page_range(struct mm_struct *mm, unsigned long addr,
}
EXPORT_SYMBOL_GPL(apply_to_page_range);
-/*
- * handle_pte_fault chooses page fault handler according to an entry which was
- * read non-atomically. Before making any commitment, on those architectures
- * or configurations (e.g. i386 with PAE) which might give a mix of unmatched
- * parts, do_swap_page must check under lock before unmapping the pte and
- * proceeding (but do_wp_page is only called after already making such a check;
- * and do_anonymous_page can safely check later on).
- */
-static inline int pte_unmap_same(struct mm_struct *mm, pmd_t *pmd,
- pte_t *page_table, pte_t orig_pte)
-{
- int same = 1;
-#if defined(CONFIG_SMP) || defined(CONFIG_PREEMPT)
- if (sizeof(pte_t) > sizeof(unsigned long)) {
- spinlock_t *ptl = pte_lockptr(mm, pmd);
- spin_lock(ptl);
- same = pte_same(*page_table, orig_pte);
- spin_unlock(ptl);
- }
-#endif
- pte_unmap(page_table);
- return same;
-}
-
static inline void cow_user_page(struct page *dst, struct page *src, unsigned long va, struct vm_area_struct *vma)
{
debug_dma_assert_idle(src);
@@ -2542,9 +2518,6 @@ int do_swap_page(struct vm_fault *vmf)
int exclusive = 0;
int ret = 0;
- if (!pte_unmap_same(vma->vm_mm, vmf->pmd, vmf->pte, vmf->orig_pte))
- goto out;
-
entry = pte_to_swp_entry(vmf->orig_pte);
if (unlikely(non_swap_entry(entry))) {
if (is_migration_entry(entry)) {
--
2.7.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [RFC v3 03/17] mm: Introduce pte_spinlock
From: Laurent Dufour @ 2017-04-27 15:52 UTC (permalink / raw)
To: paulmck, peterz, akpm, kirill, ak, mhocko, dave, jack
Cc: linux-kernel, linux-mm, haren, khandual, npiggin, bsingharora
In-Reply-To: <1493308376-23851-1-git-send-email-ldufour@linux.vnet.ibm.com>
This is needed because in handle_pte_fault() pte_offset_map() is
called and then fe->ptl is fetched and spin_locked.
This was previously embedded in the call to pte_offset_map_lock().
Signed-off-by: Laurent Dufour <ldufour@linux.vnet.ibm.com>
---
mm/memory.c | 15 +++++++++++----
1 file changed, 11 insertions(+), 4 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index bce32c9d73c2..441c0e3f3a0f 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2100,6 +2100,13 @@ static inline void wp_page_reuse(struct vm_fault *vmf)
pte_unmap_unlock(vmf->pte, vmf->ptl);
}
+static bool pte_spinlock(struct vm_fault *vmf)
+{
+ vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd);
+ spin_lock(vmf->ptl);
+ return true;
+}
+
static bool pte_map_lock(struct vm_fault *vmf)
{
vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl);
@@ -3398,8 +3405,8 @@ static int do_numa_page(struct vm_fault *vmf)
* page table entry is not accessible, so there would be no
* concurrent hardware modifications to the PTE.
*/
- vmf->ptl = pte_lockptr(vma->vm_mm, vmf->pmd);
- spin_lock(vmf->ptl);
+ if (!pte_spinlock(vmf))
+ return VM_FAULT_RETRY;
if (unlikely(!pte_same(*vmf->pte, pte))) {
pte_unmap_unlock(vmf->pte, vmf->ptl);
goto out;
@@ -3566,8 +3573,8 @@ static int handle_pte_fault(struct vm_fault *vmf)
if (pte_protnone(vmf->orig_pte) && vma_is_accessible(vmf->vma))
return do_numa_page(vmf);
- vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd);
- spin_lock(vmf->ptl);
+ if (!pte_spinlock(vmf))
+ return VM_FAULT_RETRY;
entry = vmf->orig_pte;
if (unlikely(!pte_same(*vmf->pte, entry)))
goto unlock;
--
2.7.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [RFC v3 04/17] mm: VMA sequence count
From: Laurent Dufour @ 2017-04-27 15:52 UTC (permalink / raw)
To: paulmck, peterz, akpm, kirill, ak, mhocko, dave, jack
Cc: linux-kernel, linux-mm, haren, khandual, npiggin, bsingharora
In-Reply-To: <1493308376-23851-1-git-send-email-ldufour@linux.vnet.ibm.com>
From: Peter Zijlstra <peterz@infradead.org>
Wrap the VMA modifications (vma_adjust/unmap_page_range) with sequence
counts such that we can easily test if a VMA is changed.
The unmap_page_range() one allows us to make assumptions about
page-tables; when we find the seqcount hasn't changed we can assume
page-tables are still valid.
The flip side is that we cannot distinguish between a vma_adjust() and
the unmap_page_range() -- where with the former we could have
re-checked the vma bounds against the address.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
[port to 4.10 kernel]
Signed-off-by: Laurent Dufour <ldufour@linux.vnet.ibm.com>
---
include/linux/mm_types.h | 1 +
mm/memory.c | 2 ++
mm/mmap.c | 13 +++++++++++++
3 files changed, 16 insertions(+)
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 808751d7b737..daa5fbba9349 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -358,6 +358,7 @@ struct vm_area_struct {
struct mempolicy *vm_policy; /* NUMA policy for the VMA */
#endif
struct vm_userfaultfd_ctx vm_userfaultfd_ctx;
+ seqcount_t vm_sequence;
};
struct core_thread {
diff --git a/mm/memory.c b/mm/memory.c
index 441c0e3f3a0f..0f7fbee554c4 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -1286,6 +1286,7 @@ void unmap_page_range(struct mmu_gather *tlb,
unsigned long next;
BUG_ON(addr >= end);
+ write_seqcount_begin(&vma->vm_sequence);
tlb_start_vma(tlb, vma);
pgd = pgd_offset(vma->vm_mm, addr);
do {
@@ -1295,6 +1296,7 @@ void unmap_page_range(struct mmu_gather *tlb,
next = zap_pud_range(tlb, vma, pgd, addr, next, details);
} while (pgd++, addr = next, addr != end);
tlb_end_vma(tlb, vma);
+ write_seqcount_end(&vma->vm_sequence);
}
diff --git a/mm/mmap.c b/mm/mmap.c
index dc4291dcc99b..cb41659bc9f9 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -541,6 +541,8 @@ void __vma_link_rb(struct mm_struct *mm, struct vm_area_struct *vma,
else
mm->highest_vm_end = vma->vm_end;
+ seqcount_init(&vma->vm_sequence);
+
/*
* vma->vm_prev wasn't known when we followed the rbtree to find the
* correct insertion point for that vma. As a result, we could not
@@ -675,6 +677,10 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start,
long adjust_next = 0;
int remove_next = 0;
+ write_seqcount_begin(&vma->vm_sequence);
+ if (next)
+ write_seqcount_begin_nested(&next->vm_sequence, SINGLE_DEPTH_NESTING);
+
if (next && !insert) {
struct vm_area_struct *exporter = NULL, *importer = NULL;
@@ -886,6 +892,7 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start,
mm->map_count--;
mpol_put(vma_policy(next));
kmem_cache_free(vm_area_cachep, next);
+ write_seqcount_end(&next->vm_sequence);
/*
* In mprotect's case 6 (see comments on vma_merge),
* we must remove another next too. It would clutter
@@ -899,6 +906,8 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start,
* "vma->vm_next" gap must be updated.
*/
next = vma->vm_next;
+ if (next)
+ write_seqcount_begin_nested(&next->vm_sequence, SINGLE_DEPTH_NESTING);
} else {
/*
* For the scope of the comment "next" and
@@ -945,6 +954,10 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start,
if (insert && file)
uprobe_mmap(insert);
+ if (next)
+ write_seqcount_end(&next->vm_sequence);
+ write_seqcount_end(&vma->vm_sequence);
+
validate_mm(mm);
return 0;
--
2.7.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [RFC v3 05/17] RCU free VMAs
From: Laurent Dufour @ 2017-04-27 15:52 UTC (permalink / raw)
To: paulmck, peterz, akpm, kirill, ak, mhocko, dave, jack
Cc: linux-kernel, linux-mm, haren, khandual, npiggin, bsingharora
In-Reply-To: <1493308376-23851-1-git-send-email-ldufour@linux.vnet.ibm.com>
From: Peter Zijlstra <peterz@infradead.org>
Manage the VMAs with SRCU such that we can do a lockless VMA lookup.
We put the fput(vma->vm_file) in the SRCU callback, this keeps files
valid during speculative faults, this is possible due to the delayed
fput work by Al Viro -- do we need srcu_barrier() in unmount
someplace?
We guard the mm_rb tree with a seqlock (XXX could be a seqcount but
we'd have to disable preemption around the write side in order to make
the retry loop in __read_seqcount_begin() work) such that we can know
if the rb tree walk was correct. We cannot trust the restult of a
lockless tree walk in the face of concurrent tree rotations; although
we can trust on the termination of such walks -- tree rotations
guarantee the end result is a tree again after all.
Furthermore, we rely on the WMB implied by the
write_seqlock/count_begin() to separate the VMA initialization and the
publishing stores, analogous to the RELEASE in rcu_assign_pointer().
We also rely on the RMB from read_seqretry() to separate the vma load
from further loads like the smp_read_barrier_depends() in regular
RCU.
We must not touch the vmacache while doing SRCU lookups as that is not
properly serialized against changes. We update gap information after
publishing the VMA, but A) we don't use that and B) the seqlock
read side would fix that anyhow.
We clear vma->vm_rb for nodes removed from the vma tree such that we
can easily detect such 'dead' nodes, we rely on the WMB from
write_sequnlock() to separate the tree removal and clearing the node.
Provide find_vma_srcu() which wraps the required magic.
XXX: mmap()/munmap() heavy workloads might suffer from the global lock
in call_srcu() -- this is fixable with a 'better' SRCU implementation.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
include/linux/mm_types.h | 2 +
kernel/fork.c | 1 +
mm/init-mm.c | 1 +
mm/internal.h | 18 +++++++++
mm/mmap.c | 100 +++++++++++++++++++++++++++++++++++------------
5 files changed, 96 insertions(+), 26 deletions(-)
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index daa5fbba9349..f276973b0f91 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -359,6 +359,7 @@ struct vm_area_struct {
#endif
struct vm_userfaultfd_ctx vm_userfaultfd_ctx;
seqcount_t vm_sequence;
+ struct rcu_head vm_rcu_head;
};
struct core_thread {
@@ -397,6 +398,7 @@ struct kioctx_table;
struct mm_struct {
struct vm_area_struct *mmap; /* list of VMAs */
struct rb_root mm_rb;
+ seqlock_t mm_seq;
u32 vmacache_seqnum; /* per-thread vmacache */
#ifdef CONFIG_MMU
unsigned long (*get_unmapped_area) (struct file *filp,
diff --git a/kernel/fork.c b/kernel/fork.c
index 11c5c8ab827c..352cf3fd6c19 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -753,6 +753,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p,
mm->mmap = NULL;
mm->mm_rb = RB_ROOT;
mm->vmacache_seqnum = 0;
+ seqlock_init(&mm->mm_seq);
atomic_set(&mm->mm_users, 1);
atomic_set(&mm->mm_count, 1);
init_rwsem(&mm->mmap_sem);
diff --git a/mm/init-mm.c b/mm/init-mm.c
index 975e49f00f34..2b1fa061684f 100644
--- a/mm/init-mm.c
+++ b/mm/init-mm.c
@@ -16,6 +16,7 @@
struct mm_struct init_mm = {
.mm_rb = RB_ROOT,
+ .mm_seq = __SEQLOCK_UNLOCKED(init_mm.mm_seq),
.pgd = swapper_pg_dir,
.mm_users = ATOMIC_INIT(2),
.mm_count = ATOMIC_INIT(1),
diff --git a/mm/internal.h b/mm/internal.h
index 7aa2ea0a8623..69df80ebc93d 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -40,6 +40,24 @@ void page_writeback_init(void);
int do_swap_page(struct vm_fault *vmf);
+extern struct srcu_struct vma_srcu;
+
+extern struct vm_area_struct *find_vma_srcu(struct mm_struct *mm, unsigned long addr);
+
+static inline bool vma_is_dead(struct vm_area_struct *vma, unsigned int sequence)
+{
+ int ret = RB_EMPTY_NODE(&vma->vm_rb);
+ unsigned seq = ACCESS_ONCE(vma->vm_sequence.sequence);
+
+ /*
+ * Matches both the wmb in write_seqlock_{begin,end}() and
+ * the wmb in vma_rb_erase().
+ */
+ smp_rmb();
+
+ return ret || seq != sequence;
+}
+
void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *start_vma,
unsigned long floor, unsigned long ceiling);
diff --git a/mm/mmap.c b/mm/mmap.c
index cb41659bc9f9..44e19aa31315 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -159,6 +159,23 @@ void unlink_file_vma(struct vm_area_struct *vma)
}
}
+DEFINE_SRCU(vma_srcu);
+
+static void __free_vma(struct rcu_head *head)
+{
+ struct vm_area_struct *vma =
+ container_of(head, struct vm_area_struct, vm_rcu_head);
+
+ if (vma->vm_file)
+ fput(vma->vm_file);
+ kmem_cache_free(vm_area_cachep, vma);
+}
+
+static void free_vma(struct vm_area_struct *vma)
+{
+ call_srcu(&vma_srcu, &vma->vm_rcu_head, __free_vma);
+}
+
/*
* Close a vm structure and free it, returning the next.
*/
@@ -169,10 +186,8 @@ static struct vm_area_struct *remove_vma(struct vm_area_struct *vma)
might_sleep();
if (vma->vm_ops && vma->vm_ops->close)
vma->vm_ops->close(vma);
- if (vma->vm_file)
- fput(vma->vm_file);
mpol_put(vma_policy(vma));
- kmem_cache_free(vm_area_cachep, vma);
+ free_vma(vma);
return next;
}
@@ -394,26 +409,37 @@ static void vma_gap_update(struct vm_area_struct *vma)
}
static inline void vma_rb_insert(struct vm_area_struct *vma,
- struct rb_root *root)
+ struct mm_struct *mm)
{
+ struct rb_root *root = &mm->mm_rb;
+
/* All rb_subtree_gap values must be consistent prior to insertion */
validate_mm_rb(root, NULL);
rb_insert_augmented(&vma->vm_rb, root, &vma_gap_callbacks);
}
-static void __vma_rb_erase(struct vm_area_struct *vma, struct rb_root *root)
+static void __vma_rb_erase(struct vm_area_struct *vma, struct mm_struct *mm)
{
+ struct rb_root *root = &mm->mm_rb;
/*
* Note rb_erase_augmented is a fairly large inline function,
* so make sure we instantiate it only once with our desired
* augmented rbtree callbacks.
*/
+ write_seqlock(&mm->mm_seq);
rb_erase_augmented(&vma->vm_rb, root, &vma_gap_callbacks);
+ write_sequnlock(&mm->mm_seq); /* wmb */
+
+ /*
+ * Ensure the removal is complete before clearing the node.
+ * Matched by vma_is_dead()/handle_speculative_fault().
+ */
+ RB_CLEAR_NODE(&vma->vm_rb);
}
static __always_inline void vma_rb_erase_ignore(struct vm_area_struct *vma,
- struct rb_root *root,
+ struct mm_struct *mm,
struct vm_area_struct *ignore)
{
/*
@@ -421,21 +447,21 @@ static __always_inline void vma_rb_erase_ignore(struct vm_area_struct *vma,
* with the possible exception of the "next" vma being erased if
* next->vm_start was reduced.
*/
- validate_mm_rb(root, ignore);
+ validate_mm_rb(&mm->mm_rb, ignore);
- __vma_rb_erase(vma, root);
+ __vma_rb_erase(vma, mm);
}
static __always_inline void vma_rb_erase(struct vm_area_struct *vma,
- struct rb_root *root)
+ struct mm_struct *mm)
{
/*
* All rb_subtree_gap values must be consistent prior to erase,
* with the possible exception of the vma being erased.
*/
- validate_mm_rb(root, vma);
+ validate_mm_rb(&mm->mm_rb, vma);
- __vma_rb_erase(vma, root);
+ __vma_rb_erase(vma, mm);
}
/*
@@ -552,10 +578,12 @@ void __vma_link_rb(struct mm_struct *mm, struct vm_area_struct *vma,
* immediately update the gap to the correct value. Finally we
* rebalance the rbtree after all augmented values have been set.
*/
+ write_seqlock(&mm->mm_seq);
rb_link_node(&vma->vm_rb, rb_parent, rb_link);
vma->rb_subtree_gap = 0;
vma_gap_update(vma);
- vma_rb_insert(vma, &mm->mm_rb);
+ vma_rb_insert(vma, mm);
+ write_sequnlock(&mm->mm_seq);
}
static void __vma_link_file(struct vm_area_struct *vma)
@@ -631,7 +659,7 @@ static __always_inline void __vma_unlink_common(struct mm_struct *mm,
{
struct vm_area_struct *next;
- vma_rb_erase_ignore(vma, &mm->mm_rb, ignore);
+ vma_rb_erase_ignore(vma, mm, ignore);
next = vma->vm_next;
if (has_prev)
prev->vm_next = next;
@@ -883,21 +911,19 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start,
}
if (remove_next) {
- if (file) {
+ if (file)
uprobe_munmap(next, next->vm_start, next->vm_end);
- fput(file);
- }
if (next->anon_vma)
anon_vma_merge(vma, next);
mm->map_count--;
mpol_put(vma_policy(next));
- kmem_cache_free(vm_area_cachep, next);
- write_seqcount_end(&next->vm_sequence);
+ free_vma(next);
/*
* In mprotect's case 6 (see comments on vma_merge),
* we must remove another next too. It would clutter
* up the code too much to do both in one go.
*/
+ write_seqcount_end(&next->vm_sequence);
if (remove_next != 3) {
/*
* If "next" was removed and vma->vm_end was
@@ -2103,16 +2129,11 @@ get_unmapped_area(struct file *file, unsigned long addr, unsigned long len,
EXPORT_SYMBOL(get_unmapped_area);
/* Look up the first VMA which satisfies addr < vm_end, NULL if none. */
-struct vm_area_struct *find_vma(struct mm_struct *mm, unsigned long addr)
+static struct vm_area_struct *__find_vma(struct mm_struct *mm, unsigned long addr)
{
struct rb_node *rb_node;
struct vm_area_struct *vma;
- /* Check the cache first. */
- vma = vmacache_find(mm, addr);
- if (likely(vma))
- return vma;
-
rb_node = mm->mm_rb.rb_node;
while (rb_node) {
@@ -2129,13 +2150,40 @@ struct vm_area_struct *find_vma(struct mm_struct *mm, unsigned long addr)
rb_node = rb_node->rb_right;
}
+ return vma;
+}
+
+struct vm_area_struct *find_vma(struct mm_struct *mm, unsigned long addr)
+{
+ struct vm_area_struct *vma;
+
+ /* Check the cache first. */
+ vma = vmacache_find(mm, addr);
+ if (likely(vma))
+ return vma;
+
+ vma = __find_vma(mm, addr);
if (vma)
vmacache_update(addr, vma);
return vma;
}
-
EXPORT_SYMBOL(find_vma);
+struct vm_area_struct *find_vma_srcu(struct mm_struct *mm, unsigned long addr)
+{
+ struct vm_area_struct *vma;
+ unsigned int seq;
+
+ WARN_ON_ONCE(!srcu_read_lock_held(&vma_srcu));
+
+ do {
+ seq = read_seqbegin(&mm->mm_seq);
+ vma = __find_vma(mm, addr);
+ } while (read_seqretry(&mm->mm_seq, seq));
+
+ return vma;
+}
+
/*
* Same as find_vma, but also return a pointer to the previous VMA in *pprev.
*/
@@ -2490,7 +2538,7 @@ detach_vmas_to_be_unmapped(struct mm_struct *mm, struct vm_area_struct *vma,
insertion_point = (prev ? &prev->vm_next : &mm->mmap);
vma->vm_prev = NULL;
do {
- vma_rb_erase(vma, &mm->mm_rb);
+ vma_rb_erase(vma, mm);
mm->map_count--;
tail_vma = vma;
vma = vma->vm_next;
--
2.7.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [RFC v3 07/17] mm,x86: Add speculative pagefault handling
From: Laurent Dufour @ 2017-04-27 15:52 UTC (permalink / raw)
To: paulmck, peterz, akpm, kirill, ak, mhocko, dave, jack
Cc: linux-kernel, linux-mm, haren, khandual, npiggin, bsingharora
In-Reply-To: <1493308376-23851-1-git-send-email-ldufour@linux.vnet.ibm.com>
From: Peter Zijlstra <peterz@infradead.org>
Try a speculative fault before acquiring mmap_sem, if it returns with
VM_FAULT_RETRY continue with the mmap_sem acquisition and do the
traditional fault.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
arch/x86/mm/fault.c | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)
diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
index e3254ca0eec4..ee6d8799d958 100644
--- a/arch/x86/mm/fault.c
+++ b/arch/x86/mm/fault.c
@@ -1316,6 +1316,16 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code,
if (error_code & PF_INSTR)
flags |= FAULT_FLAG_INSTRUCTION;
+ if (error_code & PF_USER) {
+ fault = handle_speculative_fault(mm, address,
+ flags & ~FAULT_FLAG_ALLOW_RETRY);
+
+ if (fault & VM_FAULT_RETRY)
+ goto retry;
+
+ goto done;
+ }
+
/*
* When running in the kernel we expect faults to occur only to
* addresses in user space. All other faults represent errors in
@@ -1419,7 +1429,15 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code,
return;
}
+ if (unlikely(fault & VM_FAULT_RETRY)) {
+ if (fatal_signal_pending(current))
+ return;
+
+ goto done;
+ }
+
up_read(&mm->mmap_sem);
+done:
if (unlikely(fault & VM_FAULT_ERROR)) {
mm_fault_error(regs, error_code, address, vma, fault);
return;
--
2.7.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [RFC v3 06/17] mm: Provide speculative fault infrastructure
From: Laurent Dufour @ 2017-04-27 15:52 UTC (permalink / raw)
To: paulmck, peterz, akpm, kirill, ak, mhocko, dave, jack
Cc: linux-kernel, linux-mm, haren, khandual, npiggin, bsingharora
In-Reply-To: <1493308376-23851-1-git-send-email-ldufour@linux.vnet.ibm.com>
From: Peter Zijlstra <peterz@infradead.org>
Provide infrastructure to do a speculative fault (not holding
mmap_sem).
The not holding of mmap_sem means we can race against VMA
change/removal and page-table destruction. We use the SRCU VMA freeing
to keep the VMA around. We use the VMA seqcount to detect change
(including umapping / page-table deletion) and we use gup_fast() style
page-table walking to deal with page-table races.
Once we've obtained the page and are ready to update the PTE, we
validate if the state we started the fault with is still valid, if
not, we'll fail the fault with VM_FAULT_RETRY, otherwise we update the
PTE and we're done.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
[Fix newly introduced pte_spinlock() for speculative page fault]
Signed-off-by: Laurent Dufour <ldufour@linux.vnet.ibm.com>
---
include/linux/mm.h | 3 ++
mm/memory.c | 149 +++++++++++++++++++++++++++++++++++++++++++++++++++--
2 files changed, 149 insertions(+), 3 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 555ac9ac7202..4667be54ba74 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -298,6 +298,7 @@ struct vm_fault {
gfp_t gfp_mask; /* gfp mask to be used for allocations */
pgoff_t pgoff; /* Logical page offset based on vma */
unsigned long address; /* Faulting virtual address */
+ unsigned int sequence;
pmd_t *pmd; /* Pointer to pmd entry matching
* the 'address' */
pte_t orig_pte; /* Value of PTE at the time of fault */
@@ -1237,6 +1238,8 @@ int invalidate_inode_page(struct page *page);
#ifdef CONFIG_MMU
extern int handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
unsigned int flags);
+extern int handle_speculative_fault(struct mm_struct *mm,
+ unsigned long address, unsigned int flags);
extern int fixup_user_fault(struct task_struct *tsk, struct mm_struct *mm,
unsigned long address, unsigned int fault_flags,
bool *unlocked);
diff --git a/mm/memory.c b/mm/memory.c
index 0f7fbee554c4..fd3a0dc122c5 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2104,15 +2104,66 @@ static inline void wp_page_reuse(struct vm_fault *vmf)
static bool pte_spinlock(struct vm_fault *vmf)
{
+ bool ret = false;
+
+ /* Check if vma is still valid */
+ if (!(vmf->flags & FAULT_FLAG_SPECULATIVE)) {
+ vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd);
+ spin_lock(vmf->ptl);
+ return true;
+ }
+
+ local_irq_disable();
+ if (vma_is_dead(vmf->vma, vmf->sequence))
+ goto out;
+
vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd);
spin_lock(vmf->ptl);
- return true;
+
+ if (vma_is_dead(vmf->vma, vmf->sequence)) {
+ spin_unlock(vmf->ptl);
+ goto out;
+ }
+
+ ret = true;
+out:
+ local_irq_enable();
+ return ret;
}
static bool pte_map_lock(struct vm_fault *vmf)
{
- vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl);
- return true;
+ bool ret = false;
+
+ if (!(vmf->flags & FAULT_FLAG_SPECULATIVE)) {
+ vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd,
+ vmf->address, &vmf->ptl);
+ return true;
+ }
+
+ /*
+ * The first vma_is_dead() guarantees the page-tables are still valid,
+ * having IRQs disabled ensures they stay around, hence the second
+ * vma_is_dead() to make sure they are still valid once we've got the
+ * lock. After that a concurrent zap_pte_range() will block on the PTL
+ * and thus we're safe.
+ */
+ local_irq_disable();
+ if (vma_is_dead(vmf->vma, vmf->sequence))
+ goto out;
+
+ vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd,
+ vmf->address, &vmf->ptl);
+
+ if (vma_is_dead(vmf->vma, vmf->sequence)) {
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
+ goto out;
+ }
+
+ ret = true;
+out:
+ local_irq_enable();
+ return ret;
}
/*
@@ -2544,6 +2595,7 @@ int do_swap_page(struct vm_fault *vmf)
entry = pte_to_swp_entry(vmf->orig_pte);
if (unlikely(non_swap_entry(entry))) {
if (is_migration_entry(entry)) {
+ /* XXX fe->pmd might be dead */
migration_entry_wait(vma->vm_mm, vmf->pmd,
vmf->address);
} else if (is_hwpoison_entry(entry)) {
@@ -3659,6 +3711,97 @@ static int __handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
return handle_pte_fault(&vmf);
}
+int handle_speculative_fault(struct mm_struct *mm, unsigned long address,
+ unsigned int flags)
+{
+ struct vm_fault vmf = {
+ .address = address,
+ .flags = flags | FAULT_FLAG_SPECULATIVE,
+ };
+ pgd_t *pgd;
+ pud_t *pud;
+ pmd_t *pmd;
+ int dead, seq, idx, ret = VM_FAULT_RETRY;
+ struct vm_area_struct *vma;
+
+ idx = srcu_read_lock(&vma_srcu);
+ vma = find_vma_srcu(mm, address);
+ if (!vma)
+ goto unlock;
+
+ /*
+ * Validate the VMA found by the lockless lookup.
+ */
+ dead = RB_EMPTY_NODE(&vma->vm_rb);
+ seq = raw_read_seqcount(&vma->vm_sequence); /* rmb <-> seqlock,vma_rb_erase() */
+ if ((seq & 1) || dead) /* XXX wait for !&1 instead? */
+ goto unlock;
+
+ if (address < vma->vm_start || vma->vm_end <= address)
+ goto unlock;
+
+ /*
+ * We need to re-validate the VMA after checking the bounds, otherwise
+ * we might have a false positive on the bounds.
+ */
+ if (read_seqcount_retry(&vma->vm_sequence, seq))
+ goto unlock;
+
+ /*
+ * Do a speculative lookup of the PTE entry.
+ */
+ local_irq_disable();
+ pgd = pgd_offset(mm, address);
+ if (pgd_none(*pgd) || unlikely(pgd_bad(*pgd)))
+ goto out_walk;
+
+ pud = pud_offset(pgd, address);
+ if (pud_none(*pud) || unlikely(pud_bad(*pud)))
+ goto out_walk;
+
+ pmd = pmd_offset(pud, address);
+ if (pmd_none(*pmd) || unlikely(pmd_bad(*pmd)))
+ goto out_walk;
+
+ /*
+ * The above does not allocate/instantiate page-tables because doing so
+ * would lead to the possibility of instantiating page-tables after
+ * free_pgtables() -- and consequently leaking them.
+ *
+ * The result is that we take at least one !speculative fault per PMD
+ * in order to instantiate it.
+ *
+ * XXX try and fix that.. should be possible somehow.
+ */
+
+ if (pmd_huge(*pmd)) /* XXX no huge support */
+ goto out_walk;
+
+ vmf.vma = vma;
+ vmf.pmd = pmd;
+ vmf.pgoff = linear_page_index(vma, address);
+ vmf.gfp_mask = __get_fault_gfp_mask(vma);
+ vmf.sequence = seq;
+
+#if 0
+#warning This is done in handle_pte_fault()...
+ pte = pte_offset_map(pmd, address);
+ fe.entry = ACCESS_ONCE(pte); /* XXX gup_get_pte() */
+ pte_unmap(pte);
+#endif
+ local_irq_enable();
+
+ ret = handle_pte_fault(&vmf);
+
+unlock:
+ srcu_read_unlock(&vma_srcu, idx);
+ return ret;
+
+out_walk:
+ local_irq_enable();
+ goto unlock;
+}
+
/*
* By the time we get here, we already hold the mm semaphore
*
--
2.7.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [RFC v3 08/17] mm/spf: Try spin lock in speculative path
From: Laurent Dufour @ 2017-04-27 15:52 UTC (permalink / raw)
To: paulmck, peterz, akpm, kirill, ak, mhocko, dave, jack
Cc: linux-kernel, linux-mm, haren, khandual, npiggin, bsingharora
In-Reply-To: <1493308376-23851-1-git-send-email-ldufour@linux.vnet.ibm.com>
There is a deadlock when a CPU is doing a speculative page fault and
another one is calling do_unmap().
The deadlock occurred because the speculative path try to spinlock the
pte while the interrupt are disabled. When the other CPU in the
unmap's path has locked the pte then is waiting for all the CPU to
invalidate the TLB. As the CPU doing the speculative fault have the
interrupt disable it can't invalidate the TLB, and can't get the lock.
Since we are in a speculative path, we can race with other mm action.
So let assume that the lock may not get acquired and fail the
speculative page fault.
Here are the stack captured during the deadlock:
CPU 0
native_flush_tlb_others+0x7c/0x260
flush_tlb_mm_range+0x6a/0x220
tlb_flush_mmu_tlbonly+0x63/0xc0
unmap_page_range+0x897/0x9d0
? unmap_single_vma+0x7d/0xe0
? release_pages+0x2b3/0x360
unmap_single_vma+0x7d/0xe0
unmap_vmas+0x51/0xa0
unmap_region+0xbd/0x130
do_munmap+0x279/0x460
SyS_munmap+0x53/0x70
CPU 1
do_raw_spin_lock+0x14e/0x160
_raw_spin_lock+0x5d/0x80
? pte_map_lock+0x169/0x1b0
pte_map_lock+0x169/0x1b0
handle_pte_fault+0xbf2/0xd80
? trace_hardirqs_on+0xd/0x10
handle_speculative_fault+0x272/0x280
handle_speculative_fault+0x5/0x280
__do_page_fault+0x187/0x580
trace_do_page_fault+0x52/0x260
do_async_page_fault+0x19/0x70
async_page_fault+0x28/0x30
Signed-off-by: Laurent Dufour <ldufour@linux.vnet.ibm.com>
---
mm/memory.c | 17 ++++++++++++++---
1 file changed, 14 insertions(+), 3 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index fd3a0dc122c5..458f579feb6f 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2118,7 +2118,8 @@ static bool pte_spinlock(struct vm_fault *vmf)
goto out;
vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd);
- spin_lock(vmf->ptl);
+ if (unlikely(!spin_trylock(vmf->ptl)))
+ goto out;
if (vma_is_dead(vmf->vma, vmf->sequence)) {
spin_unlock(vmf->ptl);
@@ -2152,8 +2153,18 @@ static bool pte_map_lock(struct vm_fault *vmf)
if (vma_is_dead(vmf->vma, vmf->sequence))
goto out;
- vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd,
- vmf->address, &vmf->ptl);
+ /* Same as pte_offset_map_lock() except that we call
+ * spin_trylock() in place of spin_lock() to avoid race with
+ * unmap path which may have the lock and wait for this CPU
+ * to invalidate TLB but this CPU has irq disabled.
+ * Since we are in a speculative patch, accept it could fail
+ */
+ vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd);
+ vmf->pte = pte_offset_map(vmf->pmd, vmf->address);
+ if (unlikely(!spin_trylock(vmf->ptl))) {
+ pte_unmap(vmf->pte);
+ goto out;
+ }
if (vma_is_dead(vmf->vma, vmf->sequence)) {
pte_unmap_unlock(vmf->pte, vmf->ptl);
--
2.7.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [RFC v3 09/17] mm/spf: Fix fe.sequence init in __handle_mm_fault()
From: Laurent Dufour @ 2017-04-27 15:52 UTC (permalink / raw)
To: paulmck, peterz, akpm, kirill, ak, mhocko, dave, jack
Cc: linux-kernel, linux-mm, haren, khandual, npiggin, bsingharora
In-Reply-To: <1493308376-23851-1-git-send-email-ldufour@linux.vnet.ibm.com>
__handle_mm_fault() calls handle_pte_fault which requires the sequence
field of the fault_env to be initialized.
Signed-off-by: Laurent Dufour <ldufour@linux.vnet.ibm.com>
---
mm/memory.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/mm/memory.c b/mm/memory.c
index 458f579feb6f..f8afd52f0d34 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -3694,6 +3694,7 @@ static int __handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
vmf.pmd = pmd_alloc(mm, pud, address);
if (!vmf.pmd)
return VM_FAULT_OOM;
+ vmf.sequence = raw_read_seqcount(&vma->vm_sequence);
if (pmd_none(*vmf.pmd) && transparent_hugepage_enabled(vma)) {
int ret = create_huge_pmd(&vmf);
if (!(ret & VM_FAULT_FALLBACK))
--
2.7.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [RFC v3 10/17] mm/spf: don't set fault entry's fields if locking failed
From: Laurent Dufour @ 2017-04-27 15:52 UTC (permalink / raw)
To: paulmck, peterz, akpm, kirill, ak, mhocko, dave, jack
Cc: linux-kernel, linux-mm, haren, khandual, npiggin, bsingharora
In-Reply-To: <1493308376-23851-1-git-send-email-ldufour@linux.vnet.ibm.com>
In the case pte_map_lock failed to lock the pte or if the VMA is no
more valid, the fault entry's fields should not be set so that caller
won't try to unlock it.
Signed-off-by: Laurent Dufour <ldufour@linux.vnet.ibm.com>
---
mm/memory.c | 14 +++++++++-----
1 file changed, 9 insertions(+), 5 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index f8afd52f0d34..3b28de5838c7 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2135,6 +2135,8 @@ static bool pte_spinlock(struct vm_fault *vmf)
static bool pte_map_lock(struct vm_fault *vmf)
{
bool ret = false;
+ pte_t *pte;
+ spinlock_t *ptl;
if (!(vmf->flags & FAULT_FLAG_SPECULATIVE)) {
vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd,
@@ -2159,18 +2161,20 @@ static bool pte_map_lock(struct vm_fault *vmf)
* to invalidate TLB but this CPU has irq disabled.
* Since we are in a speculative patch, accept it could fail
*/
- vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd);
- vmf->pte = pte_offset_map(vmf->pmd, vmf->address);
- if (unlikely(!spin_trylock(vmf->ptl))) {
- pte_unmap(vmf->pte);
+ ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd);
+ pte = pte_offset_map(vmf->pmd, vmf->address);
+ if (unlikely(!spin_trylock(ptl))) {
+ pte_unmap(pte);
goto out;
}
if (vma_is_dead(vmf->vma, vmf->sequence)) {
- pte_unmap_unlock(vmf->pte, vmf->ptl);
+ pte_unmap_unlock(pte, ptl);
goto out;
}
+ vmf->pte = pte;
+ vmf->ptl = ptl;
ret = true;
out:
local_irq_enable();
--
2.7.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [RFC v3 12/17] mm/spf: Protect changes to vm_flags
From: Laurent Dufour @ 2017-04-27 15:52 UTC (permalink / raw)
To: paulmck, peterz, akpm, kirill, ak, mhocko, dave, jack
Cc: linux-kernel, linux-mm, haren, khandual, npiggin, bsingharora
In-Reply-To: <1493308376-23851-1-git-send-email-ldufour@linux.vnet.ibm.com>
Protect VMA's flags change against the speculative page fault handler.
Signed-off-by: Laurent Dufour <ldufour@linux.vnet.ibm.com>
---
fs/proc/task_mmu.c | 2 ++
mm/mempolicy.c | 2 ++
mm/mlock.c | 9 ++++++---
mm/mmap.c | 2 ++
mm/mprotect.c | 2 ++
5 files changed, 14 insertions(+), 3 deletions(-)
diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index 8f96a49178d0..54c9a87530cb 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -1055,8 +1055,10 @@ static ssize_t clear_refs_write(struct file *file, const char __user *buf,
goto out_mm;
}
for (vma = mm->mmap; vma; vma = vma->vm_next) {
+ write_seqcount_begin(&vma->vm_sequence);
vma->vm_flags &= ~VM_SOFTDIRTY;
vma_set_page_prot(vma);
+ write_seqcount_end(&vma->vm_sequence);
}
downgrade_write(&mm->mmap_sem);
break;
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 1e7873e40c9a..1518b022927d 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -603,9 +603,11 @@ unsigned long change_prot_numa(struct vm_area_struct *vma,
{
int nr_updated;
+ write_seqcount_begin(&vma->vm_sequence);
nr_updated = change_protection(vma, addr, end, PAGE_NONE, 0, 1);
if (nr_updated)
count_vm_numa_events(NUMA_PTE_UPDATES, nr_updated);
+ write_seqcount_end(&vma->vm_sequence);
return nr_updated;
}
diff --git a/mm/mlock.c b/mm/mlock.c
index cdbed8aaa426..44cf70413530 100644
--- a/mm/mlock.c
+++ b/mm/mlock.c
@@ -437,7 +437,9 @@ static unsigned long __munlock_pagevec_fill(struct pagevec *pvec,
void munlock_vma_pages_range(struct vm_area_struct *vma,
unsigned long start, unsigned long end)
{
+ write_seqcount_begin(&vma->vm_sequence);
vma->vm_flags &= VM_LOCKED_CLEAR_MASK;
+ write_seqcount_end(&vma->vm_sequence);
while (start < end) {
struct page *page;
@@ -563,10 +565,11 @@ static int mlock_fixup(struct vm_area_struct *vma, struct vm_area_struct **prev,
* It's okay if try_to_unmap_one unmaps a page just after we
* set VM_LOCKED, populate_vma_page_range will bring it back.
*/
-
- if (lock)
+ if (lock) {
+ write_seqcount_begin(&vma->vm_sequence);
vma->vm_flags = newflags;
- else
+ write_seqcount_end(&vma->vm_sequence);
+ } else
munlock_vma_pages_range(vma, start, end);
out:
diff --git a/mm/mmap.c b/mm/mmap.c
index 27f407d8f7d7..815065d740c4 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -1742,6 +1742,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr,
out:
perf_event_mmap(vma);
+ write_seqcount_begin(&vma->vm_sequence);
vm_stat_account(mm, vm_flags, len >> PAGE_SHIFT);
if (vm_flags & VM_LOCKED) {
if (!((vm_flags & VM_SPECIAL) || is_vm_hugetlb_page(vma) ||
@@ -1764,6 +1765,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr,
vma->vm_flags |= VM_SOFTDIRTY;
vma_set_page_prot(vma);
+ write_seqcount_end(&vma->vm_sequence);
return addr;
diff --git a/mm/mprotect.c b/mm/mprotect.c
index f9c07f54dd62..646347faf4d5 100644
--- a/mm/mprotect.c
+++ b/mm/mprotect.c
@@ -341,6 +341,7 @@ mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,
* vm_flags and vm_page_prot are protected by the mmap_sem
* held in write mode.
*/
+ write_seqcount_begin(&vma->vm_sequence);
vma->vm_flags = newflags;
dirty_accountable = vma_wants_writenotify(vma, vma->vm_page_prot);
vma_set_page_prot(vma);
@@ -356,6 +357,7 @@ mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,
(newflags & VM_WRITE)) {
populate_vma_page_range(vma, start, end, NULL);
}
+ write_seqcount_end(&vma->vm_sequence);
vm_stat_account(mm, oldflags, -nrpages);
vm_stat_account(mm, newflags, nrpages);
--
2.7.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [RFC v3 11/17] mm/spf; fix lock dependency against mapping->i_mmap_rwsem
From: Laurent Dufour @ 2017-04-27 15:52 UTC (permalink / raw)
To: paulmck, peterz, akpm, kirill, ak, mhocko, dave, jack
Cc: linux-kernel, linux-mm, haren, khandual, npiggin, bsingharora
In-Reply-To: <1493308376-23851-1-git-send-email-ldufour@linux.vnet.ibm.com>
kworker/32:1/819 is trying to acquire lock:
(&vma->vm_sequence){+.+...}, at: [<c0000000002f20e0>]
zap_page_range_single+0xd0/0x1a0
but task is already holding lock:
(&mapping->i_mmap_rwsem){++++..}, at: [<c0000000002f229c>]
unmap_mapping_range+0x7c/0x160
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #2 (&mapping->i_mmap_rwsem){++++..}:
down_write+0x84/0x130
__vma_adjust+0x1f4/0xa80
__split_vma.isra.2+0x174/0x290
do_munmap+0x13c/0x4e0
vm_munmap+0x64/0xb0
elf_map+0x11c/0x130
load_elf_binary+0x6f0/0x15f0
search_binary_handler+0xe0/0x2a0
do_execveat_common.isra.14+0x7fc/0xbe0
call_usermodehelper_exec_async+0x14c/0x1d0
ret_from_kernel_thread+0x5c/0x68
-> #1 (&vma->vm_sequence/1){+.+...}:
__vma_adjust+0x124/0xa80
__split_vma.isra.2+0x174/0x290
do_munmap+0x13c/0x4e0
vm_munmap+0x64/0xb0
elf_map+0x11c/0x130
load_elf_binary+0x6f0/0x15f0
search_binary_handler+0xe0/0x2a0
do_execveat_common.isra.14+0x7fc/0xbe0
call_usermodehelper_exec_async+0x14c/0x1d0
ret_from_kernel_thread+0x5c/0x68
-> #0 (&vma->vm_sequence){+.+...}:
lock_acquire+0xf4/0x310
unmap_page_range+0xcc/0xfa0
zap_page_range_single+0xd0/0x1a0
unmap_mapping_range+0x138/0x160
truncate_pagecache+0x50/0xa0
put_aio_ring_file+0x48/0xb0
aio_free_ring+0x40/0x1b0
free_ioctx+0x38/0xc0
process_one_work+0x2cc/0x8a0
worker_thread+0xac/0x580
kthread+0x164/0x1b0
ret_from_kernel_thread+0x5c/0x68
other info that might help us debug this:
Chain exists of:
&vma->vm_sequence --> &vma->vm_sequence/1 --> &mapping->i_mmap_rwsem
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(&mapping->i_mmap_rwsem);
lock(&vma->vm_sequence/1);
lock(&mapping->i_mmap_rwsem);
lock(&vma->vm_sequence);
*** DEADLOCK ***
To fix that we must grab the vm_sequence lock after any mapping one in
__vma_adjust().
Signed-off-by: Laurent Dufour <ldufour@linux.vnet.ibm.com>
---
mm/mmap.c | 22 ++++++++++++----------
1 file changed, 12 insertions(+), 10 deletions(-)
diff --git a/mm/mmap.c b/mm/mmap.c
index 44e19aa31315..27f407d8f7d7 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -705,10 +705,6 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start,
long adjust_next = 0;
int remove_next = 0;
- write_seqcount_begin(&vma->vm_sequence);
- if (next)
- write_seqcount_begin_nested(&next->vm_sequence, SINGLE_DEPTH_NESTING);
-
if (next && !insert) {
struct vm_area_struct *exporter = NULL, *importer = NULL;
@@ -816,6 +812,11 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start,
}
}
+ write_seqcount_begin(&vma->vm_sequence);
+ if (next)
+ write_seqcount_begin_nested(&next->vm_sequence,
+ SINGLE_DEPTH_NESTING);
+
anon_vma = vma->anon_vma;
if (!anon_vma && adjust_next)
anon_vma = next->anon_vma;
@@ -932,8 +933,6 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start,
* "vma->vm_next" gap must be updated.
*/
next = vma->vm_next;
- if (next)
- write_seqcount_begin_nested(&next->vm_sequence, SINGLE_DEPTH_NESTING);
} else {
/*
* For the scope of the comment "next" and
@@ -950,11 +949,14 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start,
if (remove_next == 2) {
remove_next = 1;
end = next->vm_end;
+ write_seqcount_end(&vma->vm_sequence);
goto again;
- }
- else if (next)
+ } else if (next) {
+ if (next != vma)
+ write_seqcount_begin_nested(&next->vm_sequence,
+ SINGLE_DEPTH_NESTING);
vma_gap_update(next);
- else {
+ } else {
/*
* If remove_next == 2 we obviously can't
* reach this path.
@@ -980,7 +982,7 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start,
if (insert && file)
uprobe_mmap(insert);
- if (next)
+ if (next && next != vma)
write_seqcount_end(&next->vm_sequence);
write_seqcount_end(&vma->vm_sequence);
--
2.7.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [RFC v3 13/17] mm/spf Protect vm_policy's changes against speculative pf
From: Laurent Dufour @ 2017-04-27 15:52 UTC (permalink / raw)
To: paulmck, peterz, akpm, kirill, ak, mhocko, dave, jack
Cc: linux-kernel, linux-mm, haren, khandual, npiggin, bsingharora
In-Reply-To: <1493308376-23851-1-git-send-email-ldufour@linux.vnet.ibm.com>
Mark the VMA touched when policy changes are applied to it so that
speculative page fault will be aborted.
Signed-off-by: Laurent Dufour <ldufour@linux.vnet.ibm.com>
---
mm/mempolicy.c | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 1518b022927d..57ec8d0a9c95 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -444,8 +444,11 @@ void mpol_rebind_mm(struct mm_struct *mm, nodemask_t *new)
struct vm_area_struct *vma;
down_write(&mm->mmap_sem);
- for (vma = mm->mmap; vma; vma = vma->vm_next)
+ for (vma = mm->mmap; vma; vma = vma->vm_next) {
+ write_seqcount_begin(&vma->vm_sequence);
mpol_rebind_policy(vma->vm_policy, new, MPOL_REBIND_ONCE);
+ write_seqcount_end(&vma->vm_sequence);
+ }
up_write(&mm->mmap_sem);
}
@@ -708,6 +711,7 @@ static int vma_replace_policy(struct vm_area_struct *vma,
if (IS_ERR(new))
return PTR_ERR(new);
+ write_seqcount_begin(&vma->vm_sequence);
if (vma->vm_ops && vma->vm_ops->set_policy) {
err = vma->vm_ops->set_policy(vma, new);
if (err)
@@ -716,10 +720,12 @@ static int vma_replace_policy(struct vm_area_struct *vma,
old = vma->vm_policy;
vma->vm_policy = new; /* protected by mmap_sem */
+ write_seqcount_end(&vma->vm_sequence);
mpol_put(old);
return 0;
err_out:
+ write_seqcount_end(&vma->vm_sequence);
mpol_put(new);
return err;
}
--
2.7.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [RFC v3 14/17] x86/mm: Update the handle_speculative_fault's path
From: Laurent Dufour @ 2017-04-27 15:52 UTC (permalink / raw)
To: paulmck, peterz, akpm, kirill, ak, mhocko, dave, jack
Cc: linux-kernel, linux-mm, haren, khandual, npiggin, bsingharora
In-Reply-To: <1493308376-23851-1-git-send-email-ldufour@linux.vnet.ibm.com>
If handle_speculative_fault failed due to a VM ERROR, we try again the
slow path to allow the signal to be delivered.
Signed-off-by: Laurent Dufour <ldufour@linux.vnet.ibm.com>
---
arch/x86/mm/fault.c | 21 +++++++++------------
1 file changed, 9 insertions(+), 12 deletions(-)
diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
index ee6d8799d958..8f3bd8a53d66 100644
--- a/arch/x86/mm/fault.c
+++ b/arch/x86/mm/fault.c
@@ -1320,10 +1320,14 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code,
fault = handle_speculative_fault(mm, address,
flags & ~FAULT_FLAG_ALLOW_RETRY);
- if (fault & VM_FAULT_RETRY)
- goto retry;
-
- goto done;
+ /*
+ * We also check against VM_FAULT_ERROR because we have to
+ * raise a signal by calling later mm_fault_error() which
+ * requires the vma pointer to be set. So in that case,
+ * we fall through the normal path.
+ */
+ if (!(fault & VM_FAULT_RETRY || fault & VM_FAULT_ERROR))
+ goto done;
}
/*
@@ -1429,20 +1433,13 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code,
return;
}
- if (unlikely(fault & VM_FAULT_RETRY)) {
- if (fatal_signal_pending(current))
- return;
-
- goto done;
- }
-
up_read(&mm->mmap_sem);
-done:
if (unlikely(fault & VM_FAULT_ERROR)) {
mm_fault_error(regs, error_code, address, vma, fault);
return;
}
+done:
/*
* Major/minor page fault accounting. If any of the events
* returned VM_FAULT_MAJOR, we account it as a major fault.
--
2.7.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox