* Re: [PATCH man-pages 3/5] ioctl_userfaultfd.2: add BUGS section
From: Mike Rapoport @ 2017-05-02 9:32 UTC (permalink / raw)
To: Michael Kerrisk (man-pages)
Cc: Andrea Arcangeli, linux-kernel, linux-mm, linux-man
In-Reply-To: <345c064d-83fe-3e40-c5cb-5d4b6e5cdff4@gmail.com>
On Mon, May 01, 2017 at 08:33:50PM +0200, Michael Kerrisk (man-pages) wrote:
> Hi Mike,
>
> I've applied this, but have a question.
>
> On 05/01/2017 07:43 AM, Mike Rapoport wrote:
> > The features handshake is not quite convenient.
> > Elaborate about it in the BUGS section.
> >
> > Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
> > ---
> > man2/ioctl_userfaultfd.2 | 9 +++++++++
> > 1 file changed, 9 insertions(+)
> >
> > diff --git a/man2/ioctl_userfaultfd.2 b/man2/ioctl_userfaultfd.2
> > index e12b9de..50316de 100644
> > --- a/man2/ioctl_userfaultfd.2
> > +++ b/man2/ioctl_userfaultfd.2
> > @@ -650,6 +650,15 @@ operations are Linux-specific.
> > .SH EXAMPLE
> > See
> > .BR userfaultfd (2).
> > +.SH BUGS
> > +In order to detect available userfault features and
> > +enable certain subset of those features
>
> I changed "certain" to "some". ("certain subset" here also
> would sound like "some particular subset" of those features.)
> Okay?
Yes, sure.
> > +the usefault file descriptor must be closed after the first
> > +.BR UFFDIO_API
> > +operation that queries features availability and re-opened before
> > +the second
> > +.BR UFFDIO_API
> > +call that actually enables the desired features.
> > .SH SEE ALSO
> > .BR ioctl (2),
> > .BR mmap (2),
>
> Cheers,
>
> Michael
>
>
>
> --
> Michael Kerrisk
> Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
> Linux/UNIX System Programming Training: http://man7.org/training/
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH man-pages 1/5] ioctl_userfaultfd.2: update description of shared memory areas
From: Mike Rapoport @ 2017-05-02 9:31 UTC (permalink / raw)
To: Michael Kerrisk (man-pages)
Cc: Andrea Arcangeli, linux-kernel, linux-mm, linux-man
In-Reply-To: <7ec5dfc0-9d84-e142-bfaa-d96383acbee9@gmail.com>
On Mon, May 01, 2017 at 08:33:31PM +0200, Michael Kerrisk (man-pages) wrote:
> Hello Mike,
>
> I've applied this patch, but have a question.
>
> On 05/01/2017 07:43 AM, Mike Rapoport wrote:
> > Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
> > ---
> > man2/ioctl_userfaultfd.2 | 13 +++++++++++--
> > 1 file changed, 11 insertions(+), 2 deletions(-)
> >
> > diff --git a/man2/ioctl_userfaultfd.2 b/man2/ioctl_userfaultfd.2
> > index 889feb9..6edd396 100644
> > --- a/man2/ioctl_userfaultfd.2
> > +++ b/man2/ioctl_userfaultfd.2
> > @@ -181,8 +181,17 @@ virtual memory areas
> > .TP
> > .B UFFD_FEATURE_MISSING_SHMEM
> > If this feature bit is set,
> > -the kernel supports registering userfaultfd ranges on tmpfs
> > -virtual memory areas
> > +the kernel supports registering userfaultfd ranges on shared memory areas.
> > +This includes all kernel shared memory APIs:
> > +System V shared memory,
> > +tmpfs,
> > +/dev/zero,
> > +.BR mmap(2)
> > +with
> > +.I MAP_SHARED
> > +flag set,
> > +.BR memfd_create (2),
> > +etc.
> >
> > The returned
> > .I ioctls
>
> Does the change in this patch represent a change that occurred in
> Linux 4.11? If so, I think this needs to be said explicitly in the text.
The patch only extends the description of UFFD_FEATURE_MISSING_SHMEM. The
feature is indeed available from 4.11, but that is said a few lives above
(line 136 in ioctl_userfaultfd.2)
> Cheers,
>
> Michael
>
>
>
> --
> Michael Kerrisk
> Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
> Linux/UNIX System Programming Training: http://man7.org/training/
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH man-pages 1/2] userfaultfd.2: start documenting non-cooperative events
From: Mike Rapoport @ 2017-05-02 9:22 UTC (permalink / raw)
To: Michael Kerrisk (man-pages)
Cc: Andrea Arcangeli, linux-kernel, linux-mm, linux-man
In-Reply-To: <3cff5638-cb15-50e6-f5a4-d9a0fce643c5@gmail.com>
On Mon, May 01, 2017 at 08:34:16PM +0200, Michael Kerrisk (man-pages) wrote:
> Hi Mike,
>
> On 04/28/2017 11:45 AM, Mike Rapoprt wrote:
> >
> >
> > On April 27, 2017 8:26:16 PM GMT+03:00, "Michael Kerrisk (man-pages)" <mtk.manpages@gmail.com> wrote:
> >> Hi Mike,
> >>
> >> I've applied this, but have some questions/points I think
> >> further clarification.
> >>
> >> On 04/27/2017 04:14 PM, Mike Rapoport wrote:
> >>> Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
> >>> ---
> >>> man2/userfaultfd.2 | 135
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++---
> >>> 1 file changed, 128 insertions(+), 7 deletions(-)
> >>>
> >>> diff --git a/man2/userfaultfd.2 b/man2/userfaultfd.2
> >>> index cfea5cb..44af3e4 100644
> >>> --- a/man2/userfaultfd.2
> >>> +++ b/man2/userfaultfd.2
> >>> @@ -75,7 +75,7 @@ flag in
> >>> .PP
> >>> When the last file descriptor referring to a userfaultfd object is
> >> closed,
> >>> all memory ranges that were registered with the object are
> >> unregistered
> >>> -and unread page-fault events are flushed.
> >>> +and unread events are flushed.
> >>> .\"
> >>> .SS Usage
> >>> The userfaultfd mechanism is designed to allow a thread in a
> >> multithreaded
> >>> @@ -99,6 +99,20 @@ In such non-cooperative mode,
> >>> the process that monitors userfaultfd and handles page faults
> >>> needs to be aware of the changes in the virtual memory layout
> >>> of the faulting process to avoid memory corruption.
> >>> +
> >>> +Starting from Linux 4.11,
> >>> +userfaultfd may notify the fault-handling threads about changes
> >>> +in the virtual memory layout of the faulting process.
> >>> +In addition, if the faulting process invokes
> >>> +.BR fork (2)
> >>> +system call,
> >>> +the userfaultfd objects associated with the parent may be duplicated
> >>> +into the child process and the userfaultfd monitor will be notified
> >>> +about the file descriptor associated with the userfault objects
> >>
> >> What does "notified about the file descriptor" mean?
> >
> > Well, seems that I've made this one really awkward :)
> > When the monitored process forks, all the userfault objects
> > associateda?? with it are duplicated into the child process. For each
> > duplicated object, userfault generates event of type UFFD_EVENT_FORK
> > and the uffdio_msg for this event contains the file descriptor that
> > should be used to manipulate the duplicated userfault object.
> > Hope this clarifies.
>
> Yes, it's clearer now.
>
> Mostly what was needed here was a forward reference that mentions
> UFFD_EVENT_FORK explicitly. I added that, and also enhanced the
> text on UFFD_EVENT_FORK a little.
>
> Also, it's not just fork(2) for which UFFD_EVENT_FORK is generated,
> right? It can also be a clone(2) cal that does not specify
> CLONE_VM, right?
Yes.
> Could you review my changes in commit 522ab2ff6fc9010432a
> to make sure they are okay.
Yes, thats correct and with your updates the text is much clearer. Thanks.
> Cheers,
>
> Michael
>
> --
> Michael Kerrisk
> Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
> Linux/UNIX System Programming Training: http://man7.org/training/
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [RFC] dev/mem: "memtester -p 0x6c80000000000 10G" cause crash
From: Michal Hocko @ 2017-05-02 9:16 UTC (permalink / raw)
To: Xishi Qiu
Cc: Andrew Morton, Mel Gorman, Vlastimil Babka, Joonsoo Kim,
Johannes Weiner, Rik van Riel, Shakeel Butt, Linux MM, LKML,
zhong jiang
In-Reply-To: <590848B0.2000801@huawei.com>
On Tue 02-05-17 16:52:00, Xishi Qiu wrote:
> On 2017/5/2 16:43, Michal Hocko wrote:
>
> > On Tue 02-05-17 15:59:23, Xishi Qiu wrote:
> >> Hi, I use "memtester -p 0x6c80000000000 10G" to test physical address 0x6c80000000000
> >> Because this physical address is invalid, and valid_mmap_phys_addr_range()
> >> always return 1, so it causes crash.
> >>
> >> My question is that should the user assure the physical address is valid?
> >
> > We already seem to be checking range_is_allowed(). What is your
> > CONFIG_STRICT_DEVMEM setting? The code seems to be rather confusing but
> > my assumption is that you better know what you are doing when mapping
> > this file.
> >
>
> HI Michal,
>
> CONFIG_STRICT_DEVMEM=y, and range_is_allowed() will skip memory, but
> 0x6c80000000000 is not memory, it is just a invalid address, so it cause
> crash.
OK, I only now looked at the value. It is beyond addressable limit
(for 47b address space). None of the checks seems to stop this because
range_is_allowed() resp. its devmem_is_allowed() will allow it as a
non RAM (!page_is_ram check). I am not really sure how to fix this or
whether even we should try to fix this particular problem. As I've said
/dev/mem is dangerous and you should better know what you are doing when
accessing it.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [RFC] dev/mem: "memtester -p 0x6c80000000000 10G" cause crash
From: Xishi Qiu @ 2017-05-02 8:52 UTC (permalink / raw)
To: Michal Hocko
Cc: Andrew Morton, Mel Gorman, Vlastimil Babka, Joonsoo Kim,
Johannes Weiner, Rik van Riel, Shakeel Butt, Linux MM, LKML,
zhong jiang
In-Reply-To: <20170502084323.GG14593@dhcp22.suse.cz>
On 2017/5/2 16:43, Michal Hocko wrote:
> On Tue 02-05-17 15:59:23, Xishi Qiu wrote:
>> Hi, I use "memtester -p 0x6c80000000000 10G" to test physical address 0x6c80000000000
>> Because this physical address is invalid, and valid_mmap_phys_addr_range()
>> always return 1, so it causes crash.
>>
>> My question is that should the user assure the physical address is valid?
>
> We already seem to be checking range_is_allowed(). What is your
> CONFIG_STRICT_DEVMEM setting? The code seems to be rather confusing but
> my assumption is that you better know what you are doing when mapping
> this file.
>
HI Michal,
CONFIG_STRICT_DEVMEM=y, and range_is_allowed() will skip memory, but
0x6c80000000000 is not memory, it is just a invalid address, so it cause
crash.
You mean the user should assure the physical address is valid, right?
Thanks,
Xishi Qiu
>> ...
>> [ 169.147578] ? panic+0x1f1/0x239
>> [ 169.150789] oops_end+0xb8/0xd0
>> [ 169.153910] pgtable_bad+0x8a/0x95
>> [ 169.157294] __do_page_fault+0x3aa/0x4a0
>> [ 169.161194] do_page_fault+0x30/0x80
>> [ 169.164750] ? do_syscall_64+0x175/0x180
>> [ 169.168649] page_fault+0x28/0x30
>>
>> Thanks,
>> Xishi Qiu
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [RFC] dev/mem: "memtester -p 0x6c80000000000 10G" cause crash
From: Michal Hocko @ 2017-05-02 8:43 UTC (permalink / raw)
To: Xishi Qiu
Cc: Andrew Morton, Mel Gorman, Vlastimil Babka, Joonsoo Kim,
Johannes Weiner, Rik van Riel, Shakeel Butt, Linux MM, LKML,
zhong jiang
In-Reply-To: <59083C5B.5080204@huawei.com>
On Tue 02-05-17 15:59:23, Xishi Qiu wrote:
> Hi, I use "memtester -p 0x6c80000000000 10G" to test physical address 0x6c80000000000
> Because this physical address is invalid, and valid_mmap_phys_addr_range()
> always return 1, so it causes crash.
>
> My question is that should the user assure the physical address is valid?
We already seem to be checking range_is_allowed(). What is your
CONFIG_STRICT_DEVMEM setting? The code seems to be rather confusing but
my assumption is that you better know what you are doing when mapping
this file.
> ...
> [ 169.147578] ? panic+0x1f1/0x239
> [ 169.150789] oops_end+0xb8/0xd0
> [ 169.153910] pgtable_bad+0x8a/0x95
> [ 169.157294] __do_page_fault+0x3aa/0x4a0
> [ 169.161194] do_page_fault+0x30/0x80
> [ 169.164750] ? do_syscall_64+0x175/0x180
> [ 169.168649] page_fault+0x28/0x30
>
> Thanks,
> Xishi Qiu
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH v7 0/7] Introduce ZONE_CMA
From: Vlastimil Babka @ 2017-05-02 8:06 UTC (permalink / raw)
To: Michal Hocko, Joonsoo Kim
Cc: Andrew Morton, Rik van Riel, Johannes Weiner, mgorman,
Laura Abbott, Minchan Kim, Marek Szyprowski, Michal Nazarewicz,
Aneesh Kumar K . V, Russell King, Will Deacon, linux-mm,
linux-kernel, kernel-team
In-Reply-To: <20170427150636.GM4706@dhcp22.suse.cz>
On 04/27/2017 05:06 PM, Michal Hocko wrote:
> On Tue 25-04-17 12:42:57, Joonsoo Kim wrote:
>> On Mon, Apr 24, 2017 at 03:09:36PM +0200, Michal Hocko wrote:
>>> On Mon 17-04-17 11:02:12, Joonsoo Kim wrote:
>>>> On Thu, Apr 13, 2017 at 01:56:15PM +0200, Michal Hocko wrote:
>>>>> On Wed 12-04-17 10:35:06, Joonsoo Kim wrote:
> [...]
>>> not for free. For most common configurations where we have ZONE_DMA,
>>> ZONE_DMA32, ZONE_NORMAL and ZONE_MOVABLE all the 3 bits are already
>>> consumed so a new zone will need a new one AFAICS.
>>
>> Yes, it requires one more bit for a new zone and it's handled by the patch.
>
> I am pretty sure that you are aware that consuming new page flag bits
> is usually a no-go and something we try to avoid as much as possible
> because we are in a great shortage there. So there really have to be a
> _strong_ reason if we go that way. My current understanding that the
> whole zone concept is more about a more convenient implementation rather
> than a fundamental change which will solve unsolvable problems with the
> current approach. More on that below.
I don't see it as such a big issue. It's behind a CONFIG option (so we
also don't need the jump labels you suggest later) and enabling it
reduces the number of possible NUMA nodes (not page flags). So either
you are building a kernel for android phone that needs CMA but will have
a single NUMA node, or for a large server with many nodes that won't
have CMA. As long as there won't be large servers that need CMA, we
should be fine (yes, I know some HW vendors can be very creative, but
then it's their problem?).
> [...]
>> MOVABLE allocation will fallback as following sequence.
>>
>> ZONE_CMA -> ZONE_MOVABLE -> ZONE_HIGHMEM -> ZONE_NORMAL -> ...
Hmm, so this in effect resembles some of the aggressive CMA utilization
efforts that were never merged due to issues. Joonsoo, could you
summarize/expand the cover letter part on what were the issues with
aggressive CMA utilization, and why they no longer apply with ZONE_CMA,
especially given the current node-lru reclaim? Thanks.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH v3 4/4] mm: Adaptive hash table scaling
From: Michal Hocko @ 2017-05-02 8:04 UTC (permalink / raw)
To: Andrew Morton
Cc: Pavel Tatashin, linux-mm, sparclinux, linux-fsdevel, Al Viro
In-Reply-To: <20170426201126.GA32407@dhcp22.suse.cz>
Ping on this. Andrew, are you going to fold this or should I post a
separate patch?
[...]
> I cannot say I would be really happy about the chosen approach,
> though. Why HASH_ADAPT is not implicit? Which hash table would need
> gigabytes of memory and still benefit from it? Even if there is such an
> example then it should use the explicit high_limit. I do not like this
> opt-in because it is just too easy to miss that and hit the same issue
> again. And in fact only few users of alloc_large_system_hash are using
> the flag. E.g. why {dcache,inode}_init_early do not have the flag? I
> am pretty sure that having a physically contiguous hash table would be
> better over vmalloc from the TLB point of view.
>
> mount_hashtable resp. mountpoint_hashtable are another example. Other
> users just have a reasonable max value. So can we do the following
> on top of your commit? I think that we should rethink the scaling as
> well but I do not have a good answer for the maximum size so let's just
> start with a more reasonable API first.
> ---
> diff --git a/fs/dcache.c b/fs/dcache.c
> index 808ea99062c2..363502faa328 100644
> --- a/fs/dcache.c
> +++ b/fs/dcache.c
> @@ -3585,7 +3585,7 @@ static void __init dcache_init(void)
> sizeof(struct hlist_bl_head),
> dhash_entries,
> 13,
> - HASH_ZERO | HASH_ADAPT,
> + HASH_ZERO,
> &d_hash_shift,
> &d_hash_mask,
> 0,
> diff --git a/fs/inode.c b/fs/inode.c
> index a9caf53df446..b3c0731ec1fe 100644
> --- a/fs/inode.c
> +++ b/fs/inode.c
> @@ -1950,7 +1950,7 @@ void __init inode_init(void)
> sizeof(struct hlist_head),
> ihash_entries,
> 14,
> - HASH_ZERO | HASH_ADAPT,
> + HASH_ZERO,
> &i_hash_shift,
> &i_hash_mask,
> 0,
> diff --git a/include/linux/bootmem.h b/include/linux/bootmem.h
> index dbaf312b3317..e223d91b6439 100644
> --- a/include/linux/bootmem.h
> +++ b/include/linux/bootmem.h
> @@ -359,7 +359,6 @@ extern void *alloc_large_system_hash(const char *tablename,
> #define HASH_SMALL 0x00000002 /* sub-page allocation allowed, min
> * shift passed via *_hash_shift */
> #define HASH_ZERO 0x00000004 /* Zero allocated hash table */
> -#define HASH_ADAPT 0x00000008 /* Adaptive scale for large memory */
>
> /* Only NUMA needs hash distribution. 64bit NUMA architectures have
> * sufficient vmalloc space.
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index fa752de84eef..3bf60669d200 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -7226,7 +7226,7 @@ void *__init alloc_large_system_hash(const char *tablename,
> if (PAGE_SHIFT < 20)
> numentries = round_up(numentries, (1<<20)/PAGE_SIZE);
>
> - if (flags & HASH_ADAPT) {
> + if (!high_limit) {
> unsigned long adapt;
>
> for (adapt = ADAPT_SCALE_NPAGES; adapt < numentries;
>
> --
> Michal Hocko
> SUSE Labs
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [patch v2] mm, vmscan: avoid thrashing anon lru when free + file is low
From: Michal Hocko @ 2017-05-02 8:02 UTC (permalink / raw)
To: David Rientjes
Cc: Andrew Morton, Minchan Kim, Johannes Weiner, Mel Gorman,
linux-kernel, linux-mm
In-Reply-To: <alpine.DEB.2.10.1705011432220.137835@chino.kir.corp.google.com>
On Mon 01-05-17 14:34:21, David Rientjes wrote:
[...]
> @@ -2204,8 +2204,17 @@ static void get_scan_count(struct lruvec *lruvec, struct mem_cgroup *memcg,
> }
>
> if (unlikely(pgdatfile + pgdatfree <= total_high_wmark)) {
> - scan_balance = SCAN_ANON;
> - goto out;
> + /*
> + * Force SCAN_ANON if there are enough inactive
> + * anonymous pages on the LRU in eligible zones.
> + * Otherwise, the small LRU gets thrashed.
> + */
> + if (!inactive_list_is_low(lruvec, false, sc, false) &&
> + lruvec_lru_size(lruvec, LRU_INACTIVE_ANON, sc->reclaim_idx)
> + >> sc->priority) {
> + scan_balance = SCAN_ANON;
> + goto out;
> + }
I have already asked and my questions were ignored. So let me ask again
and hopefuly not get ignored this time. So Why do we need a different
criterion on anon pages than file pages? I do agree that blindly
scanning anon pages when file pages are low is very suboptimal but this
adds yet another heuristic without _any_ numbers. Why cannot we simply
treat anon and file pages equally? Something like the following
if (pgdatfile + pgdatanon + pgdatfree > 2*total_high_wmark) {
scan_balance = SCAN_FILE;
if (pgdatfile < pgdatanon)
scan_balance = SCAN_ANON;
goto out;
}
Also it would help to describe the workload which can trigger this
behavior so that we can compare numbers before and after this patch.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* [RFC] dev/mem: "memtester -p 0x6c80000000000 10G" cause crash
From: Xishi Qiu @ 2017-05-02 7:59 UTC (permalink / raw)
To: Andrew Morton, Mel Gorman, Vlastimil Babka, Joonsoo Kim,
Michal Hocko, Johannes Weiner, Rik van Riel, Shakeel Butt
Cc: Linux MM, LKML, zhong jiang
Hi, I use "memtester -p 0x6c80000000000 10G" to test physical address 0x6c80000000000
Because this physical address is invalid, and valid_mmap_phys_addr_range()
always return 1, so it causes crash.
My question is that should the user assure the physical address is valid?
...
[ 169.147578] ? panic+0x1f1/0x239
[ 169.150789] oops_end+0xb8/0xd0
[ 169.153910] pgtable_bad+0x8a/0x95
[ 169.157294] __do_page_fault+0x3aa/0x4a0
[ 169.161194] do_page_fault+0x30/0x80
[ 169.164750] ? do_syscall_64+0x175/0x180
[ 169.168649] page_fault+0x28/0x30
Thanks,
Xishi Qiu
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH] vmscan: scan pages until it founds eligible pages
From: Michal Hocko @ 2017-05-02 7:54 UTC (permalink / raw)
To: Minchan Kim
Cc: Andrew Morton, Johannes Weiner, Mel Gorman, kernel-team,
linux-kernel, linux-mm
In-Reply-To: <20170502051452.GA27264@bbox>
On Tue 02-05-17 14:14:52, Minchan Kim wrote:
> Oops, forgot to add lkml and linux-mm.
> Sorry for that.
> Send it again.
>
> >From 8ddf1c8aa15baf085bc6e8c62ce705459d57ea4c Mon Sep 17 00:00:00 2001
> From: Minchan Kim <minchan@kernel.org>
> Date: Tue, 2 May 2017 12:34:05 +0900
> Subject: [PATCH] vmscan: scan pages until it founds eligible pages
>
> On Tue, May 02, 2017 at 01:40:38PM +0900, Minchan Kim wrote:
> There are premature OOM happening. Although there are a ton of free
> swap and anonymous LRU list of elgible zones, OOM happened.
>
> With investigation, skipping page of isolate_lru_pages makes reclaim
> void because it returns zero nr_taken easily so LRU shrinking is
> effectively nothing and just increases priority aggressively.
> Finally, OOM happens.
I am not really sure I understand the problem you are facing. Could you
be more specific please? What is your configuration etc...
> balloon invoked oom-killer: gfp_mask=0x17080c0(GFP_KERNEL_ACCOUNT|__GFP_ZERO|__GFP_NOTRACK), nodemask=(null), order=0, oom_score_adj=0
[...]
> Node 0 active_anon:1698864kB inactive_anon:261256kB active_file:208kB inactive_file:184kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:532kB dirty:108kB writeback:0kB shmem:172kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
> DMA free:7316kB min:32kB low:44kB high:56kB active_anon:8064kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB slab_reclaimable:464kB slab_unreclaimable:40kB kernel_stack:0kB pagetables:24kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
> lowmem_reserve[]: 0 992 992 1952
> DMA32 free:9088kB min:2048kB low:3064kB high:4080kB active_anon:952176kB inactive_anon:0kB active_file:36kB inactive_file:0kB unevictable:0kB writepending:88kB present:1032192kB managed:1019388kB mlocked:0kB slab_reclaimable:13532kB slab_unreclaimable:16460kB kernel_stack:3552kB pagetables:6672kB bounce:0kB free_pcp:56kB local_pcp:24kB free_cma:0kB
> lowmem_reserve[]: 0 0 0 959
Hmm DMA32 has sufficient free memory to allow this order-0 request.
Inactive anon lru is basically empty. Why do not we rotate a really
large active anon list? Isn't this the primary problem?
I haven't really looked at the patch deeply yet. It looks quite scary at
first sight though. I would really like to understand what exactly is
going on here before we move to a patch to fix it.
Thanks!
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: 4.8.8 kernel trigger OOM killer repeatedly when I have lots of RAM that should be free
From: Michal Hocko @ 2017-05-02 7:44 UTC (permalink / raw)
To: Marc MERLIN
Cc: Linus Torvalds, Vlastimil Babka, linux-mm, LKML, Joonsoo Kim,
Tejun Heo, Greg Kroah-Hartman
In-Reply-To: <20170502041235.zqmywvj5tiiom3jk@merlins.org>
On Mon 01-05-17 21:12:35, Marc MERLIN wrote:
> Howdy,
>
> Well, sadly, the problem is more or less back is 4.11.0. The system doesn't really
> crash but it goes into an infinite loop with
> [34776.826800] BUG: workqueue lockup - pool cpus=6 node=0 flags=0x0 nice=0 stuck for 33s!
> More logs: https://pastebin.com/YqE4riw0
I am seeing a lot of traces where tasks is waiting for an IO. I do not
see any OOM report there. Why do you believe this is an OOM killer
issue?
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: 4.11.0-rc8+/x86_64 desktop lockup until applications closed
From: Michal Hocko @ 2017-05-02 7:31 UTC (permalink / raw)
To: Arthur Marsh; +Cc: Linux Kernel Mailing List, linux-mm
In-Reply-To: <99a78105-de58-a5e1-5191-d5f4de7ed5f4@internode.on.net>
On Sun 30-04-17 15:33:50, Arthur Marsh wrote:
>
>
> Michal Hocko wrote on 27/04/17 18:56:
> >On Thu 27-04-17 18:36:38, Arthur Marsh wrote:
> >[...]
> >>[55363.482931] QXcbEventReader: page allocation stalls for 10048ms, order:0,
> >>mode:0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null)
> >
> >Are there more of these stalls?
>
> I haven't seen the same kinds of logging in dmesg, but a few minutes ago I
> did see that the desktop had locked up and after remotely logging in and
> doing a kill -HUP of iceweasel/firefox, saw this:
>
> [92311.944443] swap_info_get: Bad swap offset entry 000ffffd
> [92311.944449] swap_info_get: Bad swap offset entry 000ffffe
> [92311.944451] swap_info_get: Bad swap offset entry 000fffff
Pte swap entry seem to be clobbered. That suggests a deeper problem and
a memory corruption.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [RFC 0/4] RFC - Coherent Device Memory (Not for inclusion)
From: Balbir Singh @ 2017-05-02 7:23 UTC (permalink / raw)
To: John Hubbard, linux-mm, akpm
Cc: khandual, benh, aneesh.kumar, paulmck, srikar, haren, jglisse,
mgorman, mhocko, arbab, vbabka, cl
In-Reply-To: <9e3b8b57-abd3-67cf-7c5c-a5cccc93f4b7@nvidia.com>
On Mon, 2017-05-01 at 22:47 -0700, John Hubbard wrote:
>
> On 05/01/2017 06:29 PM, Balbir Singh wrote:
> > On Mon, 2017-05-01 at 13:41 -0700, John Hubbard wrote:
> > > On 04/19/2017 12:52 AM, Balbir Singh wrote:
> > > > This is a request for comments on the discussed approaches
> > > > for coherent memory at mm-summit (some of the details are at
> > > > https://lwn.net/Articles/717601/). The latest posted patch
> > > > series is at https://lwn.net/Articles/713035/. I am reposting
> > > > this as RFC, Michal Hocko suggested using HMM for CDM, but
> > > > we believe there are stronger reasons to use the NUMA approach.
> > > > The earlier patches for Coherent Device memory were implemented
> > > > and designed by Anshuman Khandual.
> > > >
> > >
> > > Hi Balbir,
> > >
> > > Although I think everyone agrees that in the [very] long term, these
> > > hardware-coherent nodes probably want to be NUMA nodes, in order to decide what to
> > > code up over the next few years, we need to get a clear idea of what has to be done
> > > for each possible approach.
> > >
> > > Here, the CDM discussion is falling just a bit short, because it does not yet
> > > include the whole story of what we would need to do. Earlier threads pointed this
> > > out: the idea started as a large patchset RFC, but then, "for ease of review", it
> > > got turned into a smaller RFC, which loses too much context.
> >
> > Hi, John
> >
> > I thought I explained the context, but I'll try again. I see the whole solution
> > as a composite of the following primitives:
> >
> > 1. Enable hotplug of CDM nodes
> > 2. Isolation of CDM memory
> > 3. Migration to/from CDM memory
> > 4. Performance enhancements for migration
> >
>
> So, there is a little more than the above required, which is why I made that short
> list. I'm in particular concerned about the various system calls that userspace can
> make to control NUMA memory, and the device drivers will need notification (probably
> mmu_notifiers, I guess), and once they get notification, in many cases they'll need
> some way to deal with reverse mapping.
Are you suggesting that the system calls user space should be audited to
check if they should be used with a CDM device? I would
think a whole lot of this should be transparent to user space, unless it opts
in to using CDM and explictly wants to allocate and free memory -- the whole
isolation premise. w.r.t device drivers are you suggesting that the device
driver needs to know the state of each page -- free/in-use? Reverse mapping
for migration?
>
> HMM provides all of that support, so it needs to happen here, too.
>
>
>
> > The RFC here is for (2) above. (3) is handled by HMM and (4) is being discussed
> > in the community. I think the larger goals are same as HMM, except that we
> > don't need unaddressable memory, since the memory is cache coherent.
> >
> > >
> > > So, I'd suggest putting together something more complete, so that it can be fairly
> > > compared against the HMM-for-hardware-coherent-nodes approach.
> > >
> >
> > Since I intend to reuse bits of HMM, I am not sure if I want to repost those
> > patches as a part of my RFC. I hope my answers make sense, the goal is to
> > reuse as much of what is available. From a user perspective
>
> It's hard to keep track of what the plan is, so explaining exactly what you're doing
> helps.
>
Fair enough, I hope I answered the questions?
> >
> > 1. We see no new interface being added in either case, the programming model
> > would differ though
> > 2. We expect the programming model to be abstracted behind a user space
> > framework, potentially like CUDA or CXL
> >
> >
> > >
> > > > Jerome posted HMM-CDM at https://lwn.net/Articles/713035/.
> > > > The patches do a great deal to enable CDM with HMM, but we
> > > > still believe that HMM with CDM is not a natural way to
> > > > represent coherent device memory and the mm will need
> > > > to be audited and enhanced for it to even work.
> > >
> > > That is also true for the CDM approach. Specifically, in order for this to be of any
> > > use to device drivers, we'll need the following:
> > >
> >
> > Since Reza answered these questions, I'll skip them in this email
>
> Yes, but he skipped over the rmap question, which I think is an important one.
>
If it is for migration, then we are going to rely on changes from HMM-CDM.
How does HMM deal with the rmap case? I presume it is not required for
unaddressable memory?
Balbir Singh.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH -mm -v3] mm, swap: Sort swap entries before free
From: Huang, Ying @ 2017-05-02 6:08 UTC (permalink / raw)
To: Minchan Kim
Cc: Huang, Ying, Andrew Morton, linux-mm, linux-kernel, Hugh Dickins,
Shaohua Li, Rik van Riel
In-Reply-To: <20170502054858.GA27319@bbox>
Minchan Kim <minchan@kernel.org> writes:
> Hi Huang,
>
> On Tue, May 02, 2017 at 01:35:24PM +0800, Huang, Ying wrote:
>> Hi, Minchan,
>>
>> Minchan Kim <minchan@kernel.org> writes:
>>
>> > On Fri, Apr 28, 2017 at 09:35:37PM +0800, Huang, Ying wrote:
>> >> In fact, during the test, I found the overhead of sort() is comparable
>> >> with the performance difference of adding likely()/unlikely() to the
>> >> "if" in the function.
>> >
>> > Huang,
>> >
>> > This discussion is started from your optimization code:
>> >
>> > if (nr_swapfiles > 1)
>> > sort();
>> >
>> > I don't have such fast machine so cannot test it. However, you added
>> > such optimization code in there so I guess it's *worth* to review so
>> > with spending my time, I pointed out what you are missing and
>> > suggested a idea to find a compromise.
>>
>> Sorry for wasting your time and Thanks a lot for your review and
>> suggestion!
>>
>> When I started talking this with you, I found there is some measurable
>> overhead of sort(). But later when I done more tests, I found the
>> measurable overhead is at the same level of likely()/unlikely() compiler
>> notation. So you help me to find that, Thanks again!
>>
>> > Now you are saying sort is so fast so no worth to add more logics
>> > to avoid the overhead?
>> > Then, please just drop that if condition part and instead, sort
>> > it unconditionally.
>>
>> Now, because we found the overhead of sort() is low, I suggest to put
>> minimal effort to avoid it. Like the original implementation,
>>
>> if (nr_swapfiles > 1)
>> sort();
>
> It might confuse someone in future and would make him/her send a patch
> to fix like we discussed. If the logic is not clear and doesn't have
> measureable overhead, just leave it which is more simple/clear.
Because the added code is minimal and cheap, I tend to keep it and add
some comments to avoid confusion. For example,
/*
* Although nr_swapfiles isn't absolute correct, but the overhead of sort()
* is so low that it isn't necessary to optimize further.
*/
>>
>> Or, we can make nr_swapfiles more correct as Tim suggested (tracking
>> the number of the swap devices during swap on/off).
>
> It might be better option but it's still hard to justify the patch
> because you said it's hard to measure. Such optimiztion patch should
> be from numbers.
OK.
Best Regards,
Huang, Ying
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH -mm -v3] mm, swap: Sort swap entries before free
From: Minchan Kim @ 2017-05-02 5:48 UTC (permalink / raw)
To: Huang, Ying
Cc: Andrew Morton, linux-mm, linux-kernel, Hugh Dickins, Shaohua Li,
Rik van Riel
In-Reply-To: <87fugng6sj.fsf@yhuang-dev.intel.com>
Hi Huang,
On Tue, May 02, 2017 at 01:35:24PM +0800, Huang, Ying wrote:
> Hi, Minchan,
>
> Minchan Kim <minchan@kernel.org> writes:
>
> > On Fri, Apr 28, 2017 at 09:35:37PM +0800, Huang, Ying wrote:
> >> In fact, during the test, I found the overhead of sort() is comparable
> >> with the performance difference of adding likely()/unlikely() to the
> >> "if" in the function.
> >
> > Huang,
> >
> > This discussion is started from your optimization code:
> >
> > if (nr_swapfiles > 1)
> > sort();
> >
> > I don't have such fast machine so cannot test it. However, you added
> > such optimization code in there so I guess it's *worth* to review so
> > with spending my time, I pointed out what you are missing and
> > suggested a idea to find a compromise.
>
> Sorry for wasting your time and Thanks a lot for your review and
> suggestion!
>
> When I started talking this with you, I found there is some measurable
> overhead of sort(). But later when I done more tests, I found the
> measurable overhead is at the same level of likely()/unlikely() compiler
> notation. So you help me to find that, Thanks again!
>
> > Now you are saying sort is so fast so no worth to add more logics
> > to avoid the overhead?
> > Then, please just drop that if condition part and instead, sort
> > it unconditionally.
>
> Now, because we found the overhead of sort() is low, I suggest to put
> minimal effort to avoid it. Like the original implementation,
>
> if (nr_swapfiles > 1)
> sort();
It might confuse someone in future and would make him/her send a patch
to fix like we discussed. If the logic is not clear and doesn't have
measureable overhead, just leave it which is more simple/clear.
>
> Or, we can make nr_swapfiles more correct as Tim suggested (tracking
> the number of the swap devices during swap on/off).
It might be better option but it's still hard to justify the patch
because you said it's hard to measure. Such optimiztion patch should
be from numbers.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [RFC 0/4] RFC - Coherent Device Memory (Not for inclusion)
From: John Hubbard @ 2017-05-02 5:47 UTC (permalink / raw)
To: Balbir Singh, linux-mm, akpm
Cc: khandual, benh, aneesh.kumar, paulmck, srikar, haren, jglisse,
mgorman, mhocko, arbab, vbabka, cl
In-Reply-To: <1493688548.15044.1.camel@gmail.com>
On 05/01/2017 06:29 PM, Balbir Singh wrote:
> On Mon, 2017-05-01 at 13:41 -0700, John Hubbard wrote:
>> On 04/19/2017 12:52 AM, Balbir Singh wrote:
>>> This is a request for comments on the discussed approaches
>>> for coherent memory at mm-summit (some of the details are at
>>> https://lwn.net/Articles/717601/). The latest posted patch
>>> series is at https://lwn.net/Articles/713035/. I am reposting
>>> this as RFC, Michal Hocko suggested using HMM for CDM, but
>>> we believe there are stronger reasons to use the NUMA approach.
>>> The earlier patches for Coherent Device memory were implemented
>>> and designed by Anshuman Khandual.
>>>
>>
>> Hi Balbir,
>>
>> Although I think everyone agrees that in the [very] long term, these
>> hardware-coherent nodes probably want to be NUMA nodes, in order to decide what to
>> code up over the next few years, we need to get a clear idea of what has to be done
>> for each possible approach.
>>
>> Here, the CDM discussion is falling just a bit short, because it does not yet
>> include the whole story of what we would need to do. Earlier threads pointed this
>> out: the idea started as a large patchset RFC, but then, "for ease of review", it
>> got turned into a smaller RFC, which loses too much context.
>
> Hi, John
>
> I thought I explained the context, but I'll try again. I see the whole solution
> as a composite of the following primitives:
>
> 1. Enable hotplug of CDM nodes
> 2. Isolation of CDM memory
> 3. Migration to/from CDM memory
> 4. Performance enhancements for migration
>
So, there is a little more than the above required, which is why I made that short
list. I'm in particular concerned about the various system calls that userspace can
make to control NUMA memory, and the device drivers will need notification (probably
mmu_notifiers, I guess), and once they get notification, in many cases they'll need
some way to deal with reverse mapping.
HMM provides all of that support, so it needs to happen here, too.
> The RFC here is for (2) above. (3) is handled by HMM and (4) is being discussed
> in the community. I think the larger goals are same as HMM, except that we
> don't need unaddressable memory, since the memory is cache coherent.
>
>>
>> So, I'd suggest putting together something more complete, so that it can be fairly
>> compared against the HMM-for-hardware-coherent-nodes approach.
>>
>
> Since I intend to reuse bits of HMM, I am not sure if I want to repost those
> patches as a part of my RFC. I hope my answers make sense, the goal is to
> reuse as much of what is available. From a user perspective
It's hard to keep track of what the plan is, so explaining exactly what you're doing
helps.
>
> 1. We see no new interface being added in either case, the programming model
> would differ though
> 2. We expect the programming model to be abstracted behind a user space
> framework, potentially like CUDA or CXL
>
>
>>
>>> Jerome posted HMM-CDM at https://lwn.net/Articles/713035/.
>>> The patches do a great deal to enable CDM with HMM, but we
>>> still believe that HMM with CDM is not a natural way to
>>> represent coherent device memory and the mm will need
>>> to be audited and enhanced for it to even work.
>>
>> That is also true for the CDM approach. Specifically, in order for this to be of any
>> use to device drivers, we'll need the following:
>>
>
> Since Reza answered these questions, I'll skip them in this email
Yes, but he skipped over the rmap question, which I think is an important one.
thanks
john h
>
> Thanks for the review!
> Balbir Singh
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH -mm -v3] mm, swap: Sort swap entries before free
From: Huang, Ying @ 2017-05-02 5:35 UTC (permalink / raw)
To: Minchan Kim
Cc: Huang, Ying, Andrew Morton, linux-mm, linux-kernel, Hugh Dickins,
Shaohua Li, Rik van Riel
In-Reply-To: <20170502050228.GA27176@bbox>
Hi, Minchan,
Minchan Kim <minchan@kernel.org> writes:
> On Fri, Apr 28, 2017 at 09:35:37PM +0800, Huang, Ying wrote:
>> In fact, during the test, I found the overhead of sort() is comparable
>> with the performance difference of adding likely()/unlikely() to the
>> "if" in the function.
>
> Huang,
>
> This discussion is started from your optimization code:
>
> if (nr_swapfiles > 1)
> sort();
>
> I don't have such fast machine so cannot test it. However, you added
> such optimization code in there so I guess it's *worth* to review so
> with spending my time, I pointed out what you are missing and
> suggested a idea to find a compromise.
Sorry for wasting your time and Thanks a lot for your review and
suggestion!
When I started talking this with you, I found there is some measurable
overhead of sort(). But later when I done more tests, I found the
measurable overhead is at the same level of likely()/unlikely() compiler
notation. So you help me to find that, Thanks again!
> Now you are saying sort is so fast so no worth to add more logics
> to avoid the overhead?
> Then, please just drop that if condition part and instead, sort
> it unconditionally.
Now, because we found the overhead of sort() is low, I suggest to put
minimal effort to avoid it. Like the original implementation,
if (nr_swapfiles > 1)
sort();
Or, we can make nr_swapfiles more correct as Tim suggested (tracking
the number of the swap devices during swap on/off).
Best Regards,
Huang, Ying
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* [PATCH v3 3/3] powerpc/mm/hugetlb: Add support for page accounting
From: Balbir Singh @ 2017-05-02 5:17 UTC (permalink / raw)
To: vdavydov.dev, mpe, oss; +Cc: linuxppc-dev, linux-mm, Balbir Singh
In-Reply-To: <20170502051706.19043-1-bsingharora@gmail.com>
Add __GFP_ACCOUNT to __hugepte_alloc()
Signed-off-by: Balbir Singh <bsingharora@gmail.com>
---
arch/powerpc/mm/hugetlbpage.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/arch/powerpc/mm/hugetlbpage.c b/arch/powerpc/mm/hugetlbpage.c
index a4f33de..94e56b1 100644
--- a/arch/powerpc/mm/hugetlbpage.c
+++ b/arch/powerpc/mm/hugetlbpage.c
@@ -77,7 +77,7 @@ static int __hugepte_alloc(struct mm_struct *mm, hugepd_t *hpdp,
num_hugepd = 1;
}
- new = kmem_cache_zalloc(cachep, GFP_KERNEL);
+ new = kmem_cache_zalloc(cachep, pgtable_gfp_flags(mm, GFP_KERNEL));
BUG_ON(pshift > HUGEPD_SHIFT_MASK);
BUG_ON((unsigned long)new & HUGEPD_SHIFT_MASK);
--
2.9.3
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [PATCH v3 2/3] powerpc/mm/book(e)(3s)/32: Add page table accounting
From: Balbir Singh @ 2017-05-02 5:17 UTC (permalink / raw)
To: vdavydov.dev, mpe, oss; +Cc: linuxppc-dev, linux-mm, Balbir Singh
In-Reply-To: <20170502051706.19043-1-bsingharora@gmail.com>
Add support in pte_alloc_one() and pgd_alloc() by
passing __GFP_ACCOUNT in the flags
Signed-off-by: Balbir Singh <bsingharora@gmail.com>
---
arch/powerpc/include/asm/nohash/32/pgalloc.h | 3 ++-
arch/powerpc/mm/pgtable_32.c | 2 +-
2 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/arch/powerpc/include/asm/nohash/32/pgalloc.h b/arch/powerpc/include/asm/nohash/32/pgalloc.h
index 6331392..cc369a7 100644
--- a/arch/powerpc/include/asm/nohash/32/pgalloc.h
+++ b/arch/powerpc/include/asm/nohash/32/pgalloc.h
@@ -31,7 +31,8 @@ extern struct kmem_cache *pgtable_cache[];
static inline pgd_t *pgd_alloc(struct mm_struct *mm)
{
- return kmem_cache_alloc(PGT_CACHE(PGD_INDEX_SIZE), GFP_KERNEL);
+ return kmem_cache_alloc(PGT_CACHE(PGD_INDEX_SIZE),
+ pgtable_gfp_flags(mm, GFP_KERNEL));
}
static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)
diff --git a/arch/powerpc/mm/pgtable_32.c b/arch/powerpc/mm/pgtable_32.c
index a65c0b4..dc1e0c2 100644
--- a/arch/powerpc/mm/pgtable_32.c
+++ b/arch/powerpc/mm/pgtable_32.c
@@ -60,7 +60,7 @@ pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long address)
{
struct page *ptepage;
- gfp_t flags = GFP_KERNEL | __GFP_ZERO;
+ gfp_t flags = GFP_KERNEL | __GFP_ZERO | __GFP_ACCOUNT;
ptepage = alloc_pages(flags, 0);
if (!ptepage)
--
2.9.3
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [PATCH v3 1/3] powerpc/mm/book(e)(3s)/64: Add page table accounting
From: Balbir Singh @ 2017-05-02 5:17 UTC (permalink / raw)
To: vdavydov.dev, mpe, oss; +Cc: linuxppc-dev, linux-mm, Balbir Singh
In-Reply-To: <20170502051706.19043-1-bsingharora@gmail.com>
Introduce a helper pgtable_gfp_flags() which
just returns the current gfp flags and adds
__GFP_ACCOUNT to account for page table allocation.
The generic helper is added to include/asm/pgalloc.h
and has two variants - WARNING ugly bits ahead
1. If the header is included from a module, no check
for mm == &init_mm is done, since init_mm is not
exported
2. For kernel includes, the check is done and required
see (3e79ec7 arch: x86: charge page tables to kmemcg)
The fundamental assumption is that no module should be
doing pgd/pud/pmd and pte alloc's on behalf of init_mm
directly.
NOTE: This adds an overhead to pmd/pud/pgd allocations
similar to x86. The other alternative was to implement
pmd_alloc_kernel/pud_alloc_kernel and pgd_alloc_kernel
with their offset variants.
For 4k page size, pte_alloc_one no longer calls
pte_alloc_one_kernel.
Signed-off-by: Balbir Singh <bsingharora@gmail.com>
---
arch/powerpc/include/asm/book3s/32/pgalloc.h | 3 ++-
arch/powerpc/include/asm/book3s/64/pgalloc.h | 16 ++++++++++------
arch/powerpc/include/asm/nohash/64/pgalloc.h | 11 +++++++----
arch/powerpc/include/asm/pgalloc.h | 14 ++++++++++++++
arch/powerpc/mm/pgtable_64.c | 20 ++++++++++++++------
5 files changed, 47 insertions(+), 17 deletions(-)
diff --git a/arch/powerpc/include/asm/book3s/32/pgalloc.h b/arch/powerpc/include/asm/book3s/32/pgalloc.h
index d310546..a120e7f 100644
--- a/arch/powerpc/include/asm/book3s/32/pgalloc.h
+++ b/arch/powerpc/include/asm/book3s/32/pgalloc.h
@@ -31,7 +31,8 @@ extern struct kmem_cache *pgtable_cache[];
static inline pgd_t *pgd_alloc(struct mm_struct *mm)
{
- return kmem_cache_alloc(PGT_CACHE(PGD_INDEX_SIZE), GFP_KERNEL);
+ return kmem_cache_alloc(PGT_CACHE(PGD_INDEX_SIZE),
+ pgtable_gfp_flags(mm, GFP_KERNEL));
}
static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)
diff --git a/arch/powerpc/include/asm/book3s/64/pgalloc.h b/arch/powerpc/include/asm/book3s/64/pgalloc.h
index cd5e7aa..20b1485 100644
--- a/arch/powerpc/include/asm/book3s/64/pgalloc.h
+++ b/arch/powerpc/include/asm/book3s/64/pgalloc.h
@@ -53,10 +53,11 @@ extern void __tlb_remove_table(void *_table);
static inline pgd_t *radix__pgd_alloc(struct mm_struct *mm)
{
#ifdef CONFIG_PPC_64K_PAGES
- return (pgd_t *)__get_free_page(PGALLOC_GFP);
+ return (pgd_t *)__get_free_page(pgtable_gfp_flags(mm, PGALLOC_GFP));
#else
struct page *page;
- page = alloc_pages(PGALLOC_GFP | __GFP_REPEAT, 4);
+ page = alloc_pages(pgtable_gfp_flags(mm, PGALLOC_GFP | __GFP_REPEAT),
+ 4);
if (!page)
return NULL;
return (pgd_t *) page_address(page);
@@ -76,7 +77,8 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm)
{
if (radix_enabled())
return radix__pgd_alloc(mm);
- return kmem_cache_alloc(PGT_CACHE(PGD_INDEX_SIZE), GFP_KERNEL);
+ return kmem_cache_alloc(PGT_CACHE(PGD_INDEX_SIZE),
+ pgtable_gfp_flags(mm, GFP_KERNEL));
}
static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)
@@ -93,7 +95,8 @@ static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgd, pud_t *pud)
static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr)
{
- return kmem_cache_alloc(PGT_CACHE(PUD_INDEX_SIZE), GFP_KERNEL);
+ return kmem_cache_alloc(PGT_CACHE(PUD_INDEX_SIZE),
+ pgtable_gfp_flags(mm, GFP_KERNEL));
}
static inline void pud_free(struct mm_struct *mm, pud_t *pud)
@@ -119,7 +122,8 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud,
static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr)
{
- return kmem_cache_alloc(PGT_CACHE(PMD_CACHE_INDEX), GFP_KERNEL);
+ return kmem_cache_alloc(PGT_CACHE(PMD_CACHE_INDEX),
+ pgtable_gfp_flags(mm, GFP_KERNEL));
}
static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd)
@@ -168,7 +172,7 @@ static inline pgtable_t pte_alloc_one(struct mm_struct *mm,
struct page *page;
pte_t *pte;
- pte = pte_alloc_one_kernel(mm, address);
+ pte = (pte_t *)__get_free_page(GFP_KERNEL | __GFP_ZERO | __GFP_ACCOUNT);
if (!pte)
return NULL;
page = virt_to_page(pte);
diff --git a/arch/powerpc/include/asm/nohash/64/pgalloc.h b/arch/powerpc/include/asm/nohash/64/pgalloc.h
index 897d2e1..9721c78 100644
--- a/arch/powerpc/include/asm/nohash/64/pgalloc.h
+++ b/arch/powerpc/include/asm/nohash/64/pgalloc.h
@@ -43,7 +43,8 @@ extern struct kmem_cache *pgtable_cache[];
static inline pgd_t *pgd_alloc(struct mm_struct *mm)
{
- return kmem_cache_alloc(PGT_CACHE(PGD_INDEX_SIZE), GFP_KERNEL);
+ return kmem_cache_alloc(PGT_CACHE(PGD_INDEX_SIZE),
+ pgtable_gfp_flags(mm, GFP_KERNEL));
}
static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)
@@ -57,7 +58,8 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)
static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr)
{
- return kmem_cache_alloc(PGT_CACHE(PUD_INDEX_SIZE), GFP_KERNEL);
+ return kmem_cache_alloc(PGT_CACHE(PUD_INDEX_SIZE),
+ pgtable_gfp_flags(mm, GFP_KERNEL));
}
static inline void pud_free(struct mm_struct *mm, pud_t *pud)
@@ -96,7 +98,7 @@ static inline pgtable_t pte_alloc_one(struct mm_struct *mm,
struct page *page;
pte_t *pte;
- pte = pte_alloc_one_kernel(mm, address);
+ pte = (pte_t *)__get_free_page(GFP_KERNEL | __GFP_ZERO | __GFP_ACCOUNT);
if (!pte)
return NULL;
page = virt_to_page(pte);
@@ -189,7 +191,8 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t table,
static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr)
{
- return kmem_cache_alloc(PGT_CACHE(PMD_CACHE_INDEX), GFP_KERNEL);
+ return kmem_cache_alloc(PGT_CACHE(PMD_CACHE_INDEX),
+ pgtable_gfp_flags(mm, GFP_KERNEL));
}
static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd)
diff --git a/arch/powerpc/include/asm/pgalloc.h b/arch/powerpc/include/asm/pgalloc.h
index 0413457..d795c5d 100644
--- a/arch/powerpc/include/asm/pgalloc.h
+++ b/arch/powerpc/include/asm/pgalloc.h
@@ -3,6 +3,20 @@
#include <linux/mm.h>
+#ifndef MODULE
+static inline gfp_t pgtable_gfp_flags(struct mm_struct *mm, gfp_t gfp)
+{
+ if (unlikely(mm == &init_mm))
+ return gfp;
+ return gfp | __GFP_ACCOUNT;
+}
+#else /* !MODULE */
+static inline gfp_t pgtable_gfp_flags(struct mm_struct *mm, gfp_t gfp)
+{
+ return gfp | __GFP_ACCOUNT;
+}
+#endif /* MODULE */
+
#ifdef CONFIG_PPC_BOOK3S
#include <asm/book3s/pgalloc.h>
#else
diff --git a/arch/powerpc/mm/pgtable_64.c b/arch/powerpc/mm/pgtable_64.c
index db93cf7..8d2d674 100644
--- a/arch/powerpc/mm/pgtable_64.c
+++ b/arch/powerpc/mm/pgtable_64.c
@@ -351,12 +351,20 @@ static pte_t *get_from_cache(struct mm_struct *mm)
static pte_t *__alloc_for_cache(struct mm_struct *mm, int kernel)
{
void *ret = NULL;
- struct page *page = alloc_page(GFP_KERNEL | __GFP_NOTRACK | __GFP_ZERO);
- if (!page)
- return NULL;
- if (!kernel && !pgtable_page_ctor(page)) {
- __free_page(page);
- return NULL;
+ struct page *page;
+
+ if (!kernel) {
+ page = alloc_page(PGALLOC_GFP | __GFP_ACCOUNT);
+ if (!page)
+ return NULL;
+ if (!pgtable_page_ctor(page)) {
+ __free_page(page);
+ return NULL;
+ }
+ } else {
+ page = alloc_page(PGALLOC_GFP);
+ if (!page)
+ return NULL;
}
ret = page_address(page);
--
2.9.3
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [PATCH v3 0/3] Implement page table accounting for powerpc
From: Balbir Singh @ 2017-05-02 5:17 UTC (permalink / raw)
To: vdavydov.dev, mpe, oss; +Cc: linuxppc-dev, linux-mm, Balbir Singh
(3e79ec7 arch: x86: charge page tables to kmemcg) added support for page
table accounting). This patch is the second iteration to add
support, in the earlier iteration only book3s 64 bit was supported.
This iteration adds support for booke/3s/32 and 64 bit.
There is some ugliness in this patchset, pgalloc.h is included
from book3s_64_mmu_radix.c to reuse the pte/pmd/pud and pgd
management routines. We use #ifdef MODULE to provide a version
that provides full accounting. The alternatives are discussed
in patch 1 below
Changelog v3:
- Fixed a build failure with 32 bit powerpc
- Optimizations for pte_alloc_one()
Changelog v2:
- Added support for 32 bit and booke
- Added hugepte alloc accounting
Balbir Singh (3):
powerpc/mm/book(e)(3s)/64: Add page table accounting
powerpc/mm/book(e)(3s)/32: Add page table accounting
powerpc/mm/hugetlb: Add support for page accounting
arch/powerpc/include/asm/book3s/32/pgalloc.h | 3 ++-
arch/powerpc/include/asm/book3s/64/pgalloc.h | 16 ++++++++++------
arch/powerpc/include/asm/nohash/32/pgalloc.h | 3 ++-
arch/powerpc/include/asm/nohash/64/pgalloc.h | 12 ++++++++----
arch/powerpc/include/asm/pgalloc.h | 14 ++++++++++++++
arch/powerpc/mm/hugetlbpage.c | 2 +-
arch/powerpc/mm/pgtable_32.c | 2 +-
arch/powerpc/mm/pgtable_64.c | 20 ++++++++++++++------
8 files changed, 52 insertions(+), 20 deletions(-)
--
2.9.3
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH] vmscan: scan pages until it founds eligible pages
From: Minchan Kim @ 2017-05-02 5:14 UTC (permalink / raw)
To: Andrew Morton
Cc: Johannes Weiner, Mel Gorman, Michal Hocko, kernel-team,
linux-kernel, linux-mm
In-Reply-To: <1493700038-27091-1-git-send-email-minchan@kernel.org>
Oops, forgot to add lkml and linux-mm.
Sorry for that.
Send it again.
^ permalink raw reply
* Re: [PATCH -mm -v3] mm, swap: Sort swap entries before free
From: Minchan Kim @ 2017-05-02 5:02 UTC (permalink / raw)
To: Huang, Ying
Cc: Andrew Morton, linux-mm, linux-kernel, Hugh Dickins, Shaohua Li,
Rik van Riel
In-Reply-To: <878tmkvemu.fsf@yhuang-dev.intel.com>
On Fri, Apr 28, 2017 at 09:35:37PM +0800, Huang, Ying wrote:
> In fact, during the test, I found the overhead of sort() is comparable
> with the performance difference of adding likely()/unlikely() to the
> "if" in the function.
Huang,
This discussion is started from your optimization code:
if (nr_swapfiles > 1)
sort();
I don't have such fast machine so cannot test it. However, you added
such optimization code in there so I guess it's *worth* to review so
with spending my time, I pointed out what you are missing and
suggested a idea to find a compromise.
Now you are saying sort is so fast so no worth to add more logics
to avoid the overhead?
Then, please just drop that if condition part and instead, sort
it unconditionally.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: 4.8.8 kernel trigger OOM killer repeatedly when I have lots of RAM that should be free
From: Marc MERLIN @ 2017-05-02 4:12 UTC (permalink / raw)
To: Linus Torvalds
Cc: Michal Hocko, Vlastimil Babka, linux-mm, LKML, Joonsoo Kim,
Tejun Heo, Greg Kroah-Hartman
In-Reply-To: <20161129230135.GM7179@merlins.org>
Howdy,
Well, sadly, the problem is more or less back is 4.11.0. The system doesn't really
crash but it goes into an infinite loop with
[34776.826800] BUG: workqueue lockup - pool cpus=6 node=0 flags=0x0 nice=0 stuck for 33s!
More logs: https://pastebin.com/YqE4riw0
(I upgraded from 4.8 with custom patches you gave me, and went to 4.11.0
gargamel:~# cat /proc/sys/vm/dirty_ratio
2
gargamel:~# cat /proc/sys/vm/dirty_background_ratio
1
gargamel:~# free
total used free shared buffers cached
Mem: 24392600 16362660 8029940 0 8884 13739000
-/+ buffers/cache: 2614776 21777824
Swap: 15616764 0 15616764
And yet, I was doing a btrfs check repair on a busy filesystem, within 40mn or so,
it triggered the workqueue lockup.
gargamel:~# grep CONFIG_COMPACTION /boot/config-4.11.0-amd64-preempt-sysrq-20170406
CONFIG_COMPACTION=y
kernel config file: https://pastebin.com/7Tajse6L
To be fair, I didn't try to run btrfs check on 4.8 and now I'm busy
trying to recover a filesystem that apparently got corrupted by a bad
SAS driver in 4.8 which caused a lot of I/O errors and corruption.
This is just to say that btrfs on top of dmcrypt on top of bcache may
have been enough layers to hang on btrfs check on 4.8 too, but I can't
really go back to check right now due to the driver corruption issues.
Any idea what I should do next?
Thanks,
Marc
On Tue, Nov 29, 2016 at 03:01:35PM -0800, Marc MERLIN wrote:
> On Tue, Nov 29, 2016 at 09:40:19AM -0800, Marc MERLIN wrote:
> > Thanks for the reply and suggestions.
> >
> > On Tue, Nov 29, 2016 at 09:07:03AM -0800, Linus Torvalds wrote:
> > > On Tue, Nov 29, 2016 at 8:34 AM, Marc MERLIN <marc@merlins.org> wrote:
> > > > Now, to be fair, this is not a new problem, it's just varying degrees of
> > > > bad and usually only happens when I do a lot of I/O with btrfs.
> > >
> > > One situation where I've seen something like this happen is
> > >
> > > (a) lots and lots of dirty data queued up
> > > (b) horribly slow storage
> >
> > In my case, it is a 5x 4TB HDD with
> > software raid 5 < bcache < dmcrypt < btrfs
> > bcache is currently half disabled (as in I removed the actual cache) or
> > too many bcache requests pile up, and the kernel dies when too many
> > workqueues have piled up.
> > I'm just kind of worried that since I'm going through 4 subsystems
> > before my data can hit disk, that's a lot of memory allocations and
> > places where data can accumulate and cause bottlenecks if the next
> > subsystem isn't as fast.
> >
> > But this shouldn't be "horribly slow", should it? (it does copy a few
> > terabytes per day, not fast, but not horrible, about 30MB/s or so)
> >
> > > Sadly, our defaults for "how much dirty data do we allow" are somewhat
> > > buggered. The global defaults are in "percent of memory", and are
> > > generally _much_ too high for big-memory machines:
> > >
> > > [torvalds@i7 linux]$ cat /proc/sys/vm/dirty_ratio
> > > 20
> > > [torvalds@i7 linux]$ cat /proc/sys/vm/dirty_background_ratio
> > > 10
> >
> > I can confirm I have the same.
> >
> > > says that it only starts really throttling writes when you hit 20% of
> > > all memory used. You don't say how much memory you have in that
> > > machine, but if it's the same one you talked about earlier, it was
> > > 24GB. So you can have 4GB of dirty data waiting to be flushed out.
> >
> > Correct, 24GB and 4GB.
> >
> > > And we *try* to do this per-device backing-dev congestion thing to
> > > make things work better, but it generally seems to not work very well.
> > > Possibly because of inconsistent write speeds (ie _sometimes_ the SSD
> > > does really well, and we want to open up, and then it shuts down).
> > >
> > > One thing you can try is to just make the global limits much lower. As in
> > >
> > > echo 2 > /proc/sys/vm/dirty_ratio
> > > echo 1 > /proc/sys/vm/dirty_background_ratio
> >
> > I will give that a shot, thank you.
>
> And, after 5H of copying, not a single hang, or USB disconnect, or anything.
> Obviously this seems to point to other problems in the code, and I have no
> idea which layer is a culprit here, but reducing the buffers absolutely
> helped a lot.
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox