* Re: [RFC 4/8] lsm: smack: smack callbacks for kdbus security hooks
From: Paul Moore @ 2015-07-08 20:07 UTC (permalink / raw)
To: Casey Schaufler
Cc: Stephen Smalley, Paul Osmialowski, James Morris, Serge E. Hallyn,
Kees Cook, Tetsuo Handa, Neil Brown, Mark Rustad,
Greg Kroah-Hartman, Daniel Mack, David Herrmann, Djalal Harouni,
Shuah Khan, Al Viro, linux-security-module-u79uwXL29TY76Z2rM5mHXA,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
linux-api-u79uwXL29TY76Z2rM5mHXA, Karol Lewandowski,
Lukasz Skalski
In-Reply-To: <559D5201.6060400-iSGtlc1asvQWG2LlvL+J4A@public.gmane.org>
On Wednesday, July 08, 2015 09:38:25 AM Casey Schaufler wrote:
> On 7/8/2015 6:42 AM, Stephen Smalley wrote:
> > On 07/08/2015 06:25 AM, Paul Osmialowski wrote:
...
> > If Smack only truly needs 3 hooks, then it begs the question of why
> > there are so many other hooks defined. Are the other hooks just to
> > support finer-grained distinctions, or is Smack's coverage incomplete?
>
> I haven't been following kdbus closely for a while, but the original
> intent for Smack and kdbus was that it Smack controls would be on the
> objects involved, and that to accomplish that only a small number of
> hooks would be necessary. After all, Smack uses fewer hooks than SELinux
> on other things. I do agree that without a user there is no point in
> having hooks. If SELinux requires the other hooks we might want to
> hold off on asking for the hooks until the SELinux implementation is
> exposed. I also think that AppArmor should be examined as a potential
> user of the hooks, just to make sure the hooks aren't excessively
> oriented toward subject/object based security modules.
In Paul O.'s defense, we did have some discussion about the reasons for these
hooks, although that seems like ages ago and I would need to dig through the
archives (my inbox?) to find the reasoning for each.
However, I don't remember being very comfortable with the hooks back them
largely due to uncertainty about how we were treating kdbus with respect to
subjects/objects. I think it's worth restarting that discussion now before we
nit pick the patches themselves.
--
paul moore
security @ redhat
^ permalink raw reply
* [PATCH] fs, proc: add help for CONFIG_PROC_CHILDREN
From: Iago López Galeiras @ 2015-07-08 20:28 UTC (permalink / raw)
To: Jean Delvare
Cc: Iago López Galeiras, linux-fsdevel-u79uwXL29TY76Z2rM5mHXA,
linux-api-u79uwXL29TY76Z2rM5mHXA, Cyrill Gorcunov, Andrew Morton,
Djalal Harouni, Alban Crequy
The purpose of the option was documented in
Documentation/filesystems/proc.txt but the help text was missing.
Add small help text that also points to the documentation.
Signed-off-by: Iago López Galeiras <iago-973cpzSjLbNWk0Htik3J/w@public.gmane.org>
---
fs/proc/Kconfig | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/fs/proc/Kconfig b/fs/proc/Kconfig
index d751fcb..1ade120 100644
--- a/fs/proc/Kconfig
+++ b/fs/proc/Kconfig
@@ -75,3 +75,9 @@ config PROC_PAGE_MONITOR
config PROC_CHILDREN
bool "Include /proc/<pid>/task/<tid>/children file"
default n
+ help
+ Provides a fast way to retrieve first level children pids of a task. See
+ <file:Documentation/filesystems/proc.txt> for more information.
+
+ Say Y if you are running any user-space software which takes benefit from
+ this interface. For example, rkt is such a piece of software.
--
2.4.5
^ permalink raw reply related
* Re: [PATCH V3 3/5] mm: mlock: Introduce VM_LOCKONFAULT and add mlock flags to enable it
From: Eric B Munson @ 2015-07-08 20:34 UTC (permalink / raw)
To: Jonathan Corbet
Cc: Andrew Morton, Michal Hocko, Vlastimil Babka, linux-alpha,
linux-kernel, linux-mips, linux-parisc, linuxppc-dev, sparclinux,
linux-xtensa, linux-mm, linux-arch, linux-api
In-Reply-To: <20150708132351.61c13db6@lwn.net>
[-- Attachment #1: Type: text/plain, Size: 1853 bytes --]
On Wed, 08 Jul 2015, Jonathan Corbet wrote:
> On Tue, 7 Jul 2015 13:03:41 -0400
> Eric B Munson <emunson@akamai.com> wrote:
>
> > This patch introduces the ability to request that pages are not
> > pre-faulted, but are placed on the unevictable LRU when they are finally
> > faulted in. This can be done area at a time via the
> > mlock2(MLOCK_ONFAULT) or the mlockall(MCL_ONFAULT) system calls. These
> > calls can be undone via munlock2(MLOCK_ONFAULT) or
> > munlockall2(MCL_ONFAULT).
>
> Quick, possibly dumb question: I've been beating my head against these for
> a little bit, and I can't figure out what's supposed to happen in this
> case:
>
> mlock2(addr, len, MLOCK_ONFAULT);
> munlock2(addr, len, MLOCK_LOCKED);
>
> It looks to me like it will clear VM_LOCKED without actually unlocking any
> pages. Is that the intended result?
This is not quite right, what happens when you call munlock2(addr, len,
MLOCK_LOCKED); is we call apply_vma_flags(addr, len, VM_LOCKED, false).
The false argument means that we intend to clear the specified flags.
Here is the relevant snippet:
...
newflags = vma->vm_flags;
if (add_flags) {
newflags &= ~(VM_LOCKED | VM_LOCKONFAULT);
newflags |= flags;
} else {
newflags &= ~flags;
}
...
Note that when we are adding flags, we first clear both VM_LOCKED and
VM_LOCKONFAULT. This was done to match the behavior found in
mlockall(). When we are remove flags, we simply clear the specified
flag(s).
So in your example the state of the VMAs covered by addr and len would
remain unchanged.
It sounds like apply_vma_flags() needs a comment covering this topic, I
will include that in the set I am working on now.
Eric
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply
* Re: [PATCH V3 3/5] mm: mlock: Introduce VM_LOCKONFAULT and add mlock flags to enable it
From: Jonathan Corbet @ 2015-07-08 21:17 UTC (permalink / raw)
To: Eric B Munson
Cc: Andrew Morton, Michal Hocko, Vlastimil Babka, linux-alpha,
linux-kernel, linux-mips, linux-parisc, linuxppc-dev, sparclinux,
linux-xtensa, linux-mm, linux-arch, linux-api
In-Reply-To: <20150708203456.GC4669@akamai.com>
On Wed, 8 Jul 2015 16:34:56 -0400
Eric B Munson <emunson@akamai.com> wrote:
> > Quick, possibly dumb question: I've been beating my head against these for
> > a little bit, and I can't figure out what's supposed to happen in this
> > case:
> >
> > mlock2(addr, len, MLOCK_ONFAULT);
> > munlock2(addr, len, MLOCK_LOCKED);
> >
> > It looks to me like it will clear VM_LOCKED without actually unlocking any
> > pages. Is that the intended result?
>
> This is not quite right, what happens when you call munlock2(addr, len,
> MLOCK_LOCKED); is we call apply_vma_flags(addr, len, VM_LOCKED, false).
>From your explanation, it looks like what I said *was* right...what I was
missing was the fact that VM_LOCKED isn't set in the first place. So that
call would be a no-op, clearing a flag that's already cleared.
One other question...if I call mlock2(MLOCK_ONFAULT) on a range that
already has resident pages, I believe that those pages will not be locked
until they are reclaimed and faulted back in again, right? I suspect that
could be surprising to users.
Thanks,
jon
^ permalink raw reply
* Re: [PATCH v3] ipc: Modify message queue accounting to not take kernel data structures into account
From: Davidlohr Bueso @ 2015-07-08 21:49 UTC (permalink / raw)
To: Doug Ledford
Cc: mtk.manpages-Re5JQEeQqe8AvxtiuMwx3w, Marcus Gelderie, lkml,
David Howells, Alexander Viro, John Duffy, Arto Bendiken,
Linux API, Andrew Morton
In-Reply-To: <559D7760.1020909-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
On Wed, 2015-07-08 at 15:17 -0400, Doug Ledford wrote:
> I responded yesterday in the v2 patch thread I believe. In any case, I
> think this patch is fine, and can go to stable. This patch doesn't
> actually change the math related to the rlimit checks (which is the main
> thing I wanted to correct in my original patches), instead it corrects a
> mistake I made. At the time, I mistakenly thought that the qsize
> included the current message data total + the struct msg_msg size total.
> It didn't, it was just the current user data total. I added the rbtree
> nodes in order to keep the total accurate but I shouldn't have added the
> rbtree nodes to this total because none of the other kernel usage was
> previously included.
Exactly, this is what I was referring to when I suggested staying clear
from rlimit modifications. It just makes 0 sense to account for each
rbtree.
Thanks for taking a look!
>
> Acked-by: Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
Acked-by: Davidlohr Bueso <dbueso-l3A5Bk7waGM@public.gmane.org>
^ permalink raw reply
* Re: [RFCv3 0/5] enable migration of driver pages
From: Dave Airlie @ 2015-07-08 22:47 UTC (permalink / raw)
To: Gioh Kim, dri-devel
Cc: Andrew Morton, jlayton, bfields, vbabka, iamjoonsoo.kim, Al Viro,
Michael S. Tsirkin, koct9i, minchan, aquini, linux-fsdevel,
open list:VIRTIO CORE, NET..., LKML, open list:ABI/API,
Linux Memory Management List, gunho.lee, Gioh Kim
In-Reply-To: <559C6CA6.1050809@lge.com>
>>
>>
>> Can the various in-kernel GPU drivers benefit from this? If so, wiring
>> up one or more of those would be helpful?
>
>
> I'm sure that other in-kernel GPU drivers can have benefit.
> It must be helpful.
>
> If I was familiar with other in-kernel GPU drivers code, I tried to patch
> them.
> It's too bad.
I'll bring dri-devel into the loop here.
ARM GPU developers please take a look at this stuff, Laurent, Rob,
Eric I suppose.
Daniel Vetter you might have some opinions as well.
Dave.
^ permalink raw reply
* Re: [PATCH 0/24] kernel: add a netlink interface to get information about processes (v2)
From: Andrey Vagin @ 2015-07-08 22:49 UTC (permalink / raw)
To: Andy Lutomirski
Cc: Andrew Vagin, linux-kernel@vger.kernel.org, Linux API,
Oleg Nesterov, Andrew Morton, Cyrill Gorcunov, Pavel Emelyanov,
Roger Luethi, Arnd Bergmann, Arnaldo Carvalho de Melo,
David Ahern, Pavel Odintsov
In-Reply-To: <CALCETrW4LU3M2OAWjnckFR-rqenBjV+ROBi8B3eOo=Y_mCWfGQ@mail.gmail.com>
2015-07-08 20:39 GMT+03:00 Andy Lutomirski <luto@amacapital.net>:
> On Wed, Jul 8, 2015 at 9:10 AM, Andrew Vagin <avagin@odin.com> wrote:
>> On Tue, Jul 07, 2015 at 08:56:37AM -0700, Andy Lutomirski wrote:
>>> On Tue, Jul 7, 2015 at 8:43 AM, Andrew Vagin <avagin@odin.com> wrote:
>>> > On Mon, Jul 06, 2015 at 10:10:32AM -0700, Andy Lutomirski wrote:
>>> >> On Mon, Jul 6, 2015 at 1:47 AM, Andrey Vagin <avagin@openvz.org> wrote:
>>> >> > Currently we use the proc file system, where all information are
>>> >> > presented in text files, what is convenient for humans. But if we need
>>> >> > to get information about processes from code (e.g. in C), the procfs
>>> >> > doesn't look so cool.
>>> >> >
>>> >> > From code we would prefer to get information in binary format and to be
>>> >> > able to specify which information and for which tasks are required. Here
>>> >> > is a new interface with all these features, which is called task_diag.
>>> >> > In addition it's much faster than procfs.
>>> >> >
>>> >> > task_diag is based on netlink sockets and looks like socket-diag, which
>>> >> > is used to get information about sockets.
>>> >>
>>> >> I think I like this in principle, but I have can see a few potential
>>> >> problems with using netlink for this:
>>> >>
>>> >> 1. Netlink very naturally handles net namespaces, but it doesn't
>>> >> naturally handle any other kind of namespace. In fact, the taskstats
>>> >> code that you're building on has highly broken user and pid namespace
>>> >> support. (Look for some obviously useless init_user_ns and
>>> >> init_pid_ns references. But that's only the obvious problem. That
>>> >> code calls current_user_ns() and task_active_pid_ns(current) from
>>> >> .doit, which is, in turn, called from sys_write, and looking at
>>> >> current's security state from sys_write is a big no-no.)
>>> >>
>>> >> You could partially fix it by looking at f_cred's namespaces, but that
>>> >> would be a change of what it means to create a netlink socket, and I'm
>>> >> not sure that's a good idea.
>>> >
>>> > If I don't miss something, all problems around pidns and userns are
>>> > related with multicast functionality. task_diag is using
>>> > request/response scheme and doesn't send multicast packets.
>>>
>>> It has nothing to do with multicast. task_diag needs to know what
>>> pidns and userns to use for a request, but netlink isn't set up to
>>> give you any reasonably way to do that. A netlink socket is
>>> fundamentally tied to a *net* ns (it's a socket, after all). But you
>>> can send it requests using write(2), and calling current_user_ns()
>>> from write(2) is bad. There's a long history of bugs and
>>> vulnerabilities related to thinking that current_cred() and similar
>>> are acceptable things to use in write(2) implementations.
>>>
>>
>> As far as I understand, socket_diag doesn't have this problem, becaus
>> each socket has a link on a namespace where it was created.
>>
>> What if we will pin the current pidns and credentials to a task_diag
>> socket in a moment when it's created.
>
> That's certainly doable. OTOH, if anything does:
>
> socket(AF_NETLINK, ...);
> unshare(CLONE_PID);
> fork();
>
> then they now have a (minor) security problem.
What do you mean? Is it not the same when we open a file and change
uid and gid? Permissions are checked only in the "open" syscall.
[root@avagin-fc19-cr ~]# ls -l xxx
-rw-r--r-- 1 root root 5 Jul 9 01:42 xxx
open("xxx", O_WRONLY|O_APPEND) = 3
setgid(1000) = 0
setuid(1000) = 0
write(3, "a", 1) = 1
close(1) = 0
>
> --Andy
^ permalink raw reply
* Re: [PATCH -mm v6 5/6] proc: add kpageidle file
From: Andres Lagar-Cavilla @ 2015-07-08 23:01 UTC (permalink / raw)
To: Vladimir Davydov
Cc: Andrew Morton, Minchan Kim, Raghavendra K T, Johannes Weiner,
Michal Hocko, Greg Thelen, Michel Lespinasse, David Rientjes,
Pavel Emelyanov, Cyrill Gorcunov, Jonathan Corbet,
linux-api-u79uwXL29TY76Z2rM5mHXA,
linux-doc-u79uwXL29TY76Z2rM5mHXA, linux-mm-Bw31MaZKKs3YtjvyW6yDsg,
cgroups-u79uwXL29TY76Z2rM5mHXA,
linux-kernel-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <50b7cd0f35f651481ce32414fab5210de5dc1714.1434102076.git.vdavydov-bzQdu9zFT3WakBO8gow8eQ@public.gmane.org>
On Fri, Jun 12, 2015 at 2:52 AM, Vladimir Davydov
<vdavydov-bzQdu9zFT3WakBO8gow8eQ@public.gmane.org> wrote:
> Knowing the portion of memory that is not used by a certain application
> or memory cgroup (idle memory) can be useful for partitioning the system
> efficiently, e.g. by setting memory cgroup limits appropriately.
> Currently, the only means to estimate the amount of idle memory provided
> by the kernel is /proc/PID/{clear_refs,smaps}: the user can clear the
> access bit for all pages mapped to a particular process by writing 1 to
> clear_refs, wait for some time, and then count smaps:Referenced.
> However, this method has two serious shortcomings:
>
> - it does not count unmapped file pages
> - it affects the reclaimer logic
>
> To overcome these drawbacks, this patch introduces two new page flags,
> Idle and Young, and a new proc file, /proc/kpageidle. A page's Idle flag
> can only be set from userspace by setting bit in /proc/kpageidle at the
> offset corresponding to the page, and it is cleared whenever the page is
> accessed either through page tables (it is cleared in page_referenced()
> in this case) or using the read(2) system call (mark_page_accessed()).
> Thus by setting the Idle flag for pages of a particular workload, which
> can be found e.g. by reading /proc/PID/pagemap, waiting for some time to
> let the workload access its working set, and then reading the kpageidle
> file, one can estimate the amount of pages that are not used by the
> workload.
>
> The Young page flag is used to avoid interference with the memory
> reclaimer. A page's Young flag is set whenever the Access bit of a page
> table entry pointing to the page is cleared by writing to kpageidle. If
> page_referenced() is called on a Young page, it will add 1 to its return
> value, therefore concealing the fact that the Access bit was cleared.
>
> Note, since there is no room for extra page flags on 32 bit, this
> feature uses extended page flags when compiled on 32 bit.
>
> Signed-off-by: Vladimir Davydov <vdavydov-bzQdu9zFT3WakBO8gow8eQ@public.gmane.org>
Vladimir,
I've reviewed the other five patches on your series and they're
eminently reasonable, so I'll focus my comments here, inline below.
Comments apply to both this specific patch and more broadly to the
approach you present. If I think of more I will post again. Hope that
helps!
Andres
> ---
> Documentation/vm/pagemap.txt | 12 ++-
> fs/proc/page.c | 178 +++++++++++++++++++++++++++++++++++++++++++
> fs/proc/task_mmu.c | 4 +-
> include/linux/mm.h | 88 +++++++++++++++++++++
> include/linux/page-flags.h | 9 +++
> include/linux/page_ext.h | 4 +
> mm/Kconfig | 12 +++
> mm/debug.c | 4 +
> mm/page_ext.c | 3 +
> mm/rmap.c | 8 ++
> mm/swap.c | 2 +
> 11 files changed, 322 insertions(+), 2 deletions(-)
>
> diff --git a/Documentation/vm/pagemap.txt b/Documentation/vm/pagemap.txt
> index a9b7afc8fbc6..c9266340852c 100644
> --- a/Documentation/vm/pagemap.txt
> +++ b/Documentation/vm/pagemap.txt
> @@ -5,7 +5,7 @@ pagemap is a new (as of 2.6.25) set of interfaces in the kernel that allow
> userspace programs to examine the page tables and related information by
> reading files in /proc.
>
> -There are four components to pagemap:
> +There are five components to pagemap:
>
> * /proc/pid/pagemap. This file lets a userspace process find out which
> physical frame each virtual page is mapped to. It contains one 64-bit
> @@ -69,6 +69,16 @@ There are four components to pagemap:
> memory cgroup each page is charged to, indexed by PFN. Only available when
> CONFIG_MEMCG is set.
>
> + * /proc/kpageidle. This file implements a bitmap where each bit corresponds
> + to a page, indexed by PFN. When the bit is set, the corresponding page is
> + idle. A page is considered idle if it has not been accessed since it was
> + marked idle. To mark a page idle one should set the bit corresponding to the
> + page by writing to the file. A value written to the file is OR-ed with the
> + current bitmap value. Only user memory pages can be marked idle, for other
> + page types input is silently ignored. Writing to this file beyond max PFN
> + results in the ENXIO error. Only available when CONFIG_IDLE_PAGE_TRACKING is
> + set.
> +
> Short descriptions to the page flags:
>
> 0. LOCKED
> diff --git a/fs/proc/page.c b/fs/proc/page.c
> index 70d23245dd43..1e342270b9c0 100644
> --- a/fs/proc/page.c
> +++ b/fs/proc/page.c
> @@ -16,6 +16,7 @@
>
> #define KPMSIZE sizeof(u64)
> #define KPMMASK (KPMSIZE - 1)
> +#define KPMBITS (KPMSIZE * BITS_PER_BYTE)
>
> /* /proc/kpagecount - an array exposing page counts
> *
> @@ -275,6 +276,179 @@ static const struct file_operations proc_kpagecgroup_operations = {
> };
> #endif /* CONFIG_MEMCG */
>
> +#ifdef CONFIG_IDLE_PAGE_TRACKING
> +/*
> + * Idle page tracking only considers user memory pages, for other types of
> + * pages the idle flag is always unset and an attempt to set it is silently
> + * ignored.
> + *
> + * We treat a page as a user memory page if it is on an LRU list, because it is
> + * always safe to pass such a page to page_referenced(), which is essential for
> + * idle page tracking. With such an indicator of user pages we can skip
> + * isolated pages, but since there are not usually many of them, it will hardly
> + * affect the overall result.
> + *
> + * This function tries to get a user memory page by pfn as described above.
> + */
> +static struct page *kpageidle_get_page(unsigned long pfn)
> +{
> + struct page *page;
> + struct zone *zone;
> +
> + if (!pfn_valid(pfn))
> + return NULL;
> +
> + page = pfn_to_page(pfn);
> + if (!page || !PageLRU(page))
Isolation can race in while you're processing the page, after these
checks. This is ok, but worth a small comment.
> + return NULL;
> + if (!get_page_unless_zero(page))
> + return NULL;
> +
> + zone = page_zone(page);
> + spin_lock_irq(&zone->lru_lock);
> + if (unlikely(!PageLRU(page))) {
> + put_page(page);
> + page = NULL;
> + }
> + spin_unlock_irq(&zone->lru_lock);
> + return page;
> +}
> +
> +/*
> + * This function calls page_referenced() to clear the referenced bit for all
> + * mappings to a page. Since the latter also clears the page idle flag if the
> + * page was referenced, it can be used to update the idle flag of a page.
> + */
> +static void kpageidle_clear_pte_refs(struct page *page)
> +{
> + unsigned long dummy;
> +
> + if (page_referenced(page, 0, NULL, &dummy, NULL))
Because of pte/pmd_clear_flush_young* called in the guts of
page_referenced_one, an N byte write or read to /proc/kpageidle will
cause N * 64 TLB flushes.
Additionally, because of the _notify connection to mmu notifiers, this
will also cause N * 64 EPT TLB flushes (in the KVM Intel case, similar
for other notifier flavors, you get the point).
The solution is relatively straightforward: augment
page_referenced_one with a mode marker or boolean that determines
whether tlb flushing is required.
For an access pattern tracker such as the one you propose, flushing is
not strictly necessary: the next context switch will take care. Too
bad if you missed a few accesses because the pte/pmd was loaded in the
TLB. Not so easy for MMU notifiers, because each secondary MMU has its
own semantics. You could arguably throw the towel in there, or try to
provide a framework (i.e. propagate the flushing flag) and let each
implementation fill the gaps.
> + /*
> + * We cleared the referenced bit in a mapping to this page. To
> + * avoid interference with the reclaimer, mark it young so that
> + * the next call to page_referenced() will also return > 0 (see
> + * page_referenced_one())
> + */
> + set_page_young(page);
> +}
> +
> +static ssize_t kpageidle_read(struct file *file, char __user *buf,
> + size_t count, loff_t *ppos)
> +{
> + u64 __user *out = (u64 __user *)buf;
> + struct page *page;
> + unsigned long pfn, end_pfn;
> + ssize_t ret = 0;
> + u64 idle_bitmap = 0;
> + int bit;
> +
> + if (*ppos & KPMMASK || count & KPMMASK)
> + return -EINVAL;
> +
> + pfn = *ppos * BITS_PER_BYTE;
> + if (pfn >= max_pfn)
> + return 0;
> +
> + end_pfn = pfn + count * BITS_PER_BYTE;
> + if (end_pfn > max_pfn)
> + end_pfn = ALIGN(max_pfn, KPMBITS);
> +
> + for (; pfn < end_pfn; pfn++) {
> + bit = pfn % KPMBITS;
> + page = kpageidle_get_page(pfn);
> + if (page) {
> + if (page_is_idle(page)) {
> + /*
> + * The page might have been referenced via a
> + * pte, in which case it is not idle. Clear
> + * refs and recheck.
> + */
> + kpageidle_clear_pte_refs(page);
> + if (page_is_idle(page))
> + idle_bitmap |= 1ULL << bit;
> + }
> + put_page(page);
> + }
> + if (bit == KPMBITS - 1) {
> + if (put_user(idle_bitmap, out)) {
> + ret = -EFAULT;
> + break;
> + }
> + idle_bitmap = 0;
> + out++;
> + }
> + }
> +
> + *ppos += (char __user *)out - buf;
> + if (!ret)
> + ret = (char __user *)out - buf;
> + return ret;
> +}
> +
> +static ssize_t kpageidle_write(struct file *file, const char __user *buf,
Your reasoning for a host wide /proc/kpageidle is well argued, but I'm
still hesitant.
mincore() shows how to (relatively simply) resolve unmapped file pages
to their backing page cache destination. You could recycle that code
and then you'd have per process idle/idling interfaces. With the
advantage of a clear TLB flush demarcation.
> + size_t count, loff_t *ppos)
> +{
> + const u64 __user *in = (const u64 __user *)buf;
> + struct page *page;
> + unsigned long pfn, end_pfn;
> + ssize_t ret = 0;
> + u64 idle_bitmap = 0;
> + int bit;
> +
> + if (*ppos & KPMMASK || count & KPMMASK)
> + return -EINVAL;
> +
> + pfn = *ppos * BITS_PER_BYTE;
> + if (pfn >= max_pfn)
> + return -ENXIO;
> +
> + end_pfn = pfn + count * BITS_PER_BYTE;
> + if (end_pfn > max_pfn)
> + end_pfn = ALIGN(max_pfn, KPMBITS);
> +
> + for (; pfn < end_pfn; pfn++) {
Relatively straight forward to teleport forward 512 (or more
correctly: 1 << compound_order(page)) pages for THP pages, once done
with a THP head, and avoid 511 fruitless trips down rmap.c for each
tail.
> + bit = pfn % KPMBITS;
> + if (bit == 0) {
> + if (get_user(idle_bitmap, in)) {
> + ret = -EFAULT;
> + break;
> + }
> + in++;
> + }
> + if (idle_bitmap >> bit & 1) {
> + page = kpageidle_get_page(pfn);
> + if (page) {
> + kpageidle_clear_pte_refs(page);
> + set_page_idle(page);
In the common case this will make a page both young and idle. This is
fine. We will come back to it below.
> + put_page(page);
> + }
> + }
> + }
> +
> + *ppos += (const char __user *)in - buf;
> + if (!ret)
> + ret = (const char __user *)in - buf;
> + return ret;
> +}
> +
> +static const struct file_operations proc_kpageidle_operations = {
> + .llseek = mem_lseek,
> + .read = kpageidle_read,
> + .write = kpageidle_write,
> +};
> +
> +#ifndef CONFIG_64BIT
> +static bool need_page_idle(void)
> +{
> + return true;
> +}
> +struct page_ext_operations page_idle_ops = {
> + .need = need_page_idle,
> +};
> +#endif
> +#endif /* CONFIG_IDLE_PAGE_TRACKING */
> +
> static int __init proc_page_init(void)
> {
> proc_create("kpagecount", S_IRUSR, NULL, &proc_kpagecount_operations);
> @@ -282,6 +456,10 @@ static int __init proc_page_init(void)
> #ifdef CONFIG_MEMCG
> proc_create("kpagecgroup", S_IRUSR, NULL, &proc_kpagecgroup_operations);
> #endif
> +#ifdef CONFIG_IDLE_PAGE_TRACKING
> + proc_create("kpageidle", S_IRUSR | S_IWUSR, NULL,
> + &proc_kpageidle_operations);
> +#endif
> return 0;
> }
> fs_initcall(proc_page_init);
> diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> index 58be92e11939..fcec9ccb8f7e 100644
> --- a/fs/proc/task_mmu.c
> +++ b/fs/proc/task_mmu.c
> @@ -458,7 +458,7 @@ static void smaps_account(struct mem_size_stats *mss, struct page *page,
>
> mss->resident += size;
> /* Accumulate the size in pages that have been accessed. */
> - if (young || PageReferenced(page))
> + if (young || page_is_young(page) || PageReferenced(page))
> mss->referenced += size;
> mapcount = page_mapcount(page);
> if (mapcount >= 2) {
> @@ -810,6 +810,7 @@ static int clear_refs_pte_range(pmd_t *pmd, unsigned long addr,
>
> /* Clear accessed and referenced bits. */
> pmdp_test_and_clear_young(vma, addr, pmd);
> + clear_page_young(page);
> ClearPageReferenced(page);
> out:
> spin_unlock(ptl);
> @@ -837,6 +838,7 @@ out:
>
> /* Clear accessed and referenced bits. */
> ptep_test_and_clear_young(vma, addr, pte);
> + clear_page_young(page);
> ClearPageReferenced(page);
> }
> pte_unmap_unlock(pte - 1, ptl);
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index 7f471789781a..4545ac6e27eb 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -2205,5 +2205,93 @@ void __init setup_nr_node_ids(void);
> static inline void setup_nr_node_ids(void) {}
> #endif
>
> +#ifdef CONFIG_IDLE_PAGE_TRACKING
> +#ifdef CONFIG_64BIT
> +static inline bool page_is_young(struct page *page)
> +{
> + return PageYoung(page);
> +}
> +
> +static inline void set_page_young(struct page *page)
> +{
> + SetPageYoung(page);
> +}
> +
> +static inline void clear_page_young(struct page *page)
> +{
> + ClearPageYoung(page);
> +}
Below I will comment more on the value of test_and_clear_page_young. I
think you should strive to support that, and it's trivial in the
common case of 64 bits (and requires some syntactic sugar and relaxed
guarantees for the page_ext case. Fine)
> +
> +static inline bool page_is_idle(struct page *page)
> +{
> + return PageIdle(page);
> +}
> +
> +static inline void set_page_idle(struct page *page)
> +{
> + SetPageIdle(page);
> +}
> +
> +static inline void clear_page_idle(struct page *page)
> +{
> + ClearPageIdle(page);
> +}
> +#else /* !CONFIG_64BIT */
> +/*
> + * If there is not enough space to store Idle and Young bits in page flags, use
> + * page ext flags instead.
> + */
> +extern struct page_ext_operations page_idle_ops;
> +
> +static inline bool page_is_young(struct page *page)
> +{
> + return test_bit(PAGE_EXT_YOUNG, &lookup_page_ext(page)->flags);
> +}
> +
> +static inline void set_page_young(struct page *page)
> +{
> + set_bit(PAGE_EXT_YOUNG, &lookup_page_ext(page)->flags);
> +}
> +
> +static inline void clear_page_young(struct page *page)
> +{
> + clear_bit(PAGE_EXT_YOUNG, &lookup_page_ext(page)->flags);
> +}
> +
> +static inline bool page_is_idle(struct page *page)
> +{
> + return test_bit(PAGE_EXT_IDLE, &lookup_page_ext(page)->flags);
> +}
> +
> +static inline void set_page_idle(struct page *page)
> +{
> + set_bit(PAGE_EXT_IDLE, &lookup_page_ext(page)->flags);
> +}
> +
> +static inline void clear_page_idle(struct page *page)
> +{
> + clear_bit(PAGE_EXT_IDLE, &lookup_page_ext(page)->flags);
> +}
> +#endif /* CONFIG_64BIT */
> +#else /* !CONFIG_IDLE_PAGE_TRACKING */
> +static inline bool page_is_young(struct page *page)
> +{
> + return false;
> +}
> +
> +static inline void clear_page_young(struct page *page)
> +{
> +}
> +
> +static inline bool page_is_idle(struct page *page)
> +{
> + return false;
> +}
> +
> +static inline void clear_page_idle(struct page *page)
> +{
> +}
> +#endif /* CONFIG_IDLE_PAGE_TRACKING */
> +
> #endif /* __KERNEL__ */
> #endif /* _LINUX_MM_H */
> diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
> index 91b7f9b2b774..14c5d774ad70 100644
> --- a/include/linux/page-flags.h
> +++ b/include/linux/page-flags.h
> @@ -109,6 +109,10 @@ enum pageflags {
> #ifdef CONFIG_TRANSPARENT_HUGEPAGE
> PG_compound_lock,
> #endif
> +#if defined(CONFIG_IDLE_PAGE_TRACKING) && defined(CONFIG_64BIT)
> + PG_young,
> + PG_idle,
> +#endif
> __NR_PAGEFLAGS,
>
> /* Filesystems */
> @@ -363,6 +367,11 @@ PAGEFLAG_FALSE(HWPoison)
> #define __PG_HWPOISON 0
> #endif
>
> +#if defined(CONFIG_IDLE_PAGE_TRACKING) && defined(CONFIG_64BIT)
> +PAGEFLAG(Young, young, PF_ANY)
> +PAGEFLAG(Idle, idle, PF_ANY)
> +#endif
> +
> /*
> * On an anonymous page mapped into a user virtual memory area,
> * page->mapping points to its anon_vma, not to a struct address_space;
> diff --git a/include/linux/page_ext.h b/include/linux/page_ext.h
> index c42981cd99aa..17f118a82854 100644
> --- a/include/linux/page_ext.h
> +++ b/include/linux/page_ext.h
> @@ -26,6 +26,10 @@ enum page_ext_flags {
> PAGE_EXT_DEBUG_POISON, /* Page is poisoned */
> PAGE_EXT_DEBUG_GUARD,
> PAGE_EXT_OWNER,
> +#if defined(CONFIG_IDLE_PAGE_TRACKING) && !defined(CONFIG_64BIT)
> + PAGE_EXT_YOUNG,
> + PAGE_EXT_IDLE,
> +#endif
> };
>
> /*
> diff --git a/mm/Kconfig b/mm/Kconfig
> index e79de2bd12cd..db817e2c2ec8 100644
> --- a/mm/Kconfig
> +++ b/mm/Kconfig
> @@ -654,3 +654,15 @@ config DEFERRED_STRUCT_PAGE_INIT
> when kswapd starts. This has a potential performance impact on
> processes running early in the lifetime of the systemm until kswapd
> finishes the initialisation.
> +
> +config IDLE_PAGE_TRACKING
> + bool "Enable idle page tracking"
> + select PROC_PAGE_MONITOR
> + select PAGE_EXTENSION if !64BIT
> + help
> + This feature allows to estimate the amount of user pages that have
> + not been touched during a given period of time. This information can
> + be useful to tune memory cgroup limits and/or for job placement
> + within a compute cluster.
> +
> + See Documentation/vm/pagemap.txt for more details.
> diff --git a/mm/debug.c b/mm/debug.c
> index 76089ddf99ea..6c1b3ea61bfd 100644
> --- a/mm/debug.c
> +++ b/mm/debug.c
> @@ -48,6 +48,10 @@ static const struct trace_print_flags pageflag_names[] = {
> #ifdef CONFIG_TRANSPARENT_HUGEPAGE
> {1UL << PG_compound_lock, "compound_lock" },
> #endif
> +#if defined(CONFIG_IDLE_PAGE_TRACKING) && defined(CONFIG_64BIT)
> + {1UL << PG_young, "young" },
> + {1UL << PG_idle, "idle" },
> +#endif
> };
>
> static void dump_flags(unsigned long flags,
> diff --git a/mm/page_ext.c b/mm/page_ext.c
> index d86fd2f5353f..e4b3af054bf2 100644
> --- a/mm/page_ext.c
> +++ b/mm/page_ext.c
> @@ -59,6 +59,9 @@ static struct page_ext_operations *page_ext_ops[] = {
> #ifdef CONFIG_PAGE_OWNER
> &page_owner_ops,
> #endif
> +#if defined(CONFIG_IDLE_PAGE_TRACKING) && !defined(CONFIG_64BIT)
> + &page_idle_ops,
> +#endif
> };
>
> static unsigned long total_usage;
> diff --git a/mm/rmap.c b/mm/rmap.c
> index 49b244b1f18c..8db3a6fc0c91 100644
> --- a/mm/rmap.c
> +++ b/mm/rmap.c
> @@ -798,6 +798,14 @@ static int page_referenced_one(struct page *page, struct vm_area_struct *vma,
> pte_unmap_unlock(pte, ptl);
This is not in your patch, but further up in page_referenced_one there
is the pmd case.
So what happens on THP split? That was a leading question: you should
propagate the young and idle flags to the split-up tail pages.
> }
>
> + if (referenced && page_is_idle(page))
> + clear_page_idle(page);
Is it so expensive to just call clear without the test .. ?
> +
> + if (page_is_young(page)) {
> + clear_page_young(page);
referenced += test_and_clear_page_young(page) .. ?
> + referenced++;
> + }
> +
Invert the order. A page can be both young and idle -- we noted that
closer to the top of the patch.
So young bumps referenced up, and then the final referenced value is
used to clear idle.
> if (referenced) {
At this point, if you follow my suggestion of augmenting
page_referenced_one with a mode indicator (for TLB flushing), you can
set page young here. There is the added benefit of holding the
mmap_mutex lock or vma_lock, which prevents reclaim, try_to_unmap,
migration, from exploiting a small window where page young is not set
but should.
> pra->referenced++;
> pra->vm_flags |= vma->vm_flags;
> diff --git a/mm/swap.c b/mm/swap.c
> index ab7c338eda87..db43c9b4891d 100644
> --- a/mm/swap.c
> +++ b/mm/swap.c
> @@ -623,6 +623,8 @@ void mark_page_accessed(struct page *page)
> } else if (!PageReferenced(page)) {
> SetPageReferenced(page);
> }
> + if (page_is_idle(page))
> + clear_page_idle(page);
> }
> EXPORT_SYMBOL(mark_page_accessed);
>
> --
> 2.1.4
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
--
Andres Lagar-Cavilla | Google Kernel Team | andreslc-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org
^ permalink raw reply
* Re: [PATCH 0/24] kernel: add a netlink interface to get information about processes (v2)
From: Andy Lutomirski @ 2015-07-08 23:48 UTC (permalink / raw)
To: Andrey Vagin
Cc: Andrew Vagin,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Linux API,
Oleg Nesterov, Andrew Morton, Cyrill Gorcunov, Pavel Emelyanov,
Roger Luethi, Arnd Bergmann, Arnaldo Carvalho de Melo,
David Ahern, Pavel Odintsov
In-Reply-To: <CANaxB-yMKGWJ1r0GMR9VfAq_xHn6bTjYmkDXST4suNNqu4GVjA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
On Wed, Jul 8, 2015 at 3:49 PM, Andrey Vagin <avagin-GEFAQzZX7r8dnm+yROfE0A@public.gmane.org> wrote:
> 2015-07-08 20:39 GMT+03:00 Andy Lutomirski <luto-kltTT9wpgjJwATOyAt5JVQ@public.gmane.org>:
>> On Wed, Jul 8, 2015 at 9:10 AM, Andrew Vagin <avagin-wo1vFcy6AUs@public.gmane.org> wrote:
>>>
>>> As far as I understand, socket_diag doesn't have this problem, becaus
>>> each socket has a link on a namespace where it was created.
>>>
>>> What if we will pin the current pidns and credentials to a task_diag
>>> socket in a moment when it's created.
>>
>> That's certainly doable. OTOH, if anything does:
>>
>> socket(AF_NETLINK, ...);
>> unshare(CLONE_PID);
>> fork();
>>
>> then they now have a (minor) security problem.
>
> What do you mean? Is it not the same when we open a file and change
> uid and gid? Permissions are checked only in the "open" syscall.
>
> [root@avagin-fc19-cr ~]# ls -l xxx
> -rw-r--r-- 1 root root 5 Jul 9 01:42 xxx
>
> open("xxx", O_WRONLY|O_APPEND) = 3
> setgid(1000) = 0
> setuid(1000) = 0
> write(3, "a", 1) = 1
> close(1) = 0
Yes and no.
open(2) is supposed to return an fd that retains the access to the
file that existed when open(2) was called. socket(2) is supposed* to
capture the access to the netns that existed at the time it was
called, but capturing access to a userns and/or pidns is new.
If you added socket(AF_NETLINK, SOCK_DGRAM, NETLINK_PIDNS), then maybe
that would work, but the userns interaction is a bit odd. OTOH every
pidns has an associated userns, so you could just use that. I don't
know whether that would annoy someone.
* There's some question as to whether socket(2) or connect(2) should
do this, but connect handling in netlink is quite broken and iproute2
relies on the broken handling. The historical behavior was different,
too, but the old behavior was exploitable. I have a cute little
program that does 'ip set dev lo down' but doesn't need to be run as
root :)
--Andy
^ permalink raw reply
* Re: [RFCv3 0/5] enable migration of driver pages
From: Gioh Kim @ 2015-07-08 23:55 UTC (permalink / raw)
To: Dave Airlie, dri-devel
Cc: Andrew Morton, jlayton, bfields, vbabka, iamjoonsoo.kim, Al Viro,
Michael S. Tsirkin, koct9i, minchan, aquini, linux-fsdevel,
open list:VIRTIO CORE, NET..., LKML, open, list, ABI/API,
Linux Memory Management List, gunho.lee, Gioh Kim
In-Reply-To: <CAPM=9txmUJ58=CAxDhf12Y3Y8wz7CGBy-Bd4pQ8YAAKDsCxU8w@mail.gmail.com>
2015-07-09 오전 7:47에 Dave Airlie 이(가) 쓴 글:
>>>
>>>
>>> Can the various in-kernel GPU drivers benefit from this? If so, wiring
>>> up one or more of those would be helpful?
>>
>>
>> I'm sure that other in-kernel GPU drivers can have benefit.
>> It must be helpful.
>>
>> If I was familiar with other in-kernel GPU drivers code, I tried to patch
>> them.
>> It's too bad.
>
> I'll bring dri-devel into the loop here.
>
> ARM GPU developers please take a look at this stuff, Laurent, Rob,
> Eric I suppose.
I sent a patch, https://lkml.org/lkml/2015/3/24/1182, and my opinion about compaction
to ARM GPU developers via Korea ARM branch.
I got a reply that they had no time to review it.
I hope they're interested to this patch.
>
> Daniel Vetter you might have some opinions as well.
>
> Dave.
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* [PATCH v2 00/10] hugetlbfs: add fallocate support
From: Mike Kravetz @ 2015-07-09 0:21 UTC (permalink / raw)
To: linux-mm, linux-kernel, linux-api
Cc: Dave Hansen, Naoya Horiguchi, David Rientjes, Hugh Dickins,
Davidlohr Bueso, Aneesh Kumar, Hillf Danton, Christoph Hellwig,
Andrew Morton, Michal Hocko, Mike Kravetz
Primary change in this revision is to fix the leak found by Hillf
Danton. Also, changed region_del to use LONG_MAX as an indicator
of "truncate" functionality. Considered using LLONG_MAX for same
type of indicator in remove_inode_hugepages instead of -1, but kept
-1 as LLONG_MAX could potentially be a valid offset some day. All
changes are in the log below.
As suggested during the RFC process, tests have been proposed to
libhugetlbfs as described at:
http://librelist.com/browser//libhugetlbfs/2015/6/25/patch-tests-add-tests-for-fallocate-system-call/
fallocate(2) man page modifications are also necessary to specify
that fallocate for hugetlbfs only operates on whole pages. This
change will be submitted once the code has stabilized and been
proposed for merging.
hugetlbfs is used today by applications that want a high degree of
control over huge page usage. Often, large hugetlbfs files are used
to map a large number huge pages into the application processes.
The applications know when page ranges within these large files will
no longer be used, and ideally would like to release them back to
the subpool or global pools for other uses. The fallocate() system
call provides an interface for preallocation and hole punching within
files. This patch set adds fallocate functionality to hugetlbfs.
v2:
Fixed leak in resv_map_release discovered by Hillf Danton.
Used LONG_MAX as indicator of truncate function for region_del.
v1:
Add a cache of region descriptors to the resv_map for use by
region_add in case hole punch deletes entries necessary for
a successful operation.
RFC v4:
Removed alloc_huge_page/hugetlb_reserve_pages race patches as already
in mmotm
Moved hugetlb_fix_reserve_counts in series as suggested by Naoya Horiguchi
Inline'ed hugetlb_fault_mutex routines as suggested by Davidlohr Bueso and
existing code changed to use new interfaces as suggested by Naoya
fallocate preallocation code cleaned up and made simpler
Modified alloc_huge_page to handle special case where allocation is
for a hole punched area with spool reserves
RFC v3:
Folded in patch for alloc_huge_page/hugetlb_reserve_pages race
in current code
fallocate allocation and hole punch is synchronized with page
faults via existing mutex table
hole punch uses existing hugetlb_vmtruncate_list instead of more
generic unmap_mapping_range for unmapping
Error handling for the case when region_del() fauils
RFC v2:
Addressed alignment and error handling issues noticed by Hillf Danton
New region_del() routine for region tracking/resv_map of ranges
Fixed several issues found during more extensive testing
Error handling in region_del() when kmalloc() fails stills needs
to be addressed
madvise remove support remains
Mike Kravetz (10):
mm/hugetlb: add cache of descriptors to resv_map for region_add
mm/hugetlb: add region_del() to delete a specific range of entries
mm/hugetlb: expose hugetlb fault mutex for use by fallocate
hugetlbfs: hugetlb_vmtruncate_list() needs to take a range to delete
hugetlbfs: truncate_hugepages() takes a range of pages
mm/hugetlb: vma_has_reserves() needs to handle fallocate hole punch
mm/hugetlb: alloc_huge_page handle areas hole punched by fallocate
hugetlbfs: New huge_add_to_page_cache helper routine
hugetlbfs: add hugetlbfs_fallocate()
mm: madvise allow remove operation for hugetlbfs
fs/hugetlbfs/inode.c | 281 +++++++++++++++++++++++++++++---
include/linux/hugetlb.h | 17 +-
mm/hugetlb.c | 422 ++++++++++++++++++++++++++++++++++++++----------
mm/madvise.c | 2 +-
4 files changed, 618 insertions(+), 104 deletions(-)
--
2.1.0
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* [PATCH v2 01/10] mm/hugetlb: add cache of descriptors to resv_map for region_add
From: Mike Kravetz @ 2015-07-09 0:21 UTC (permalink / raw)
To: linux-mm, linux-kernel, linux-api
Cc: Dave Hansen, Naoya Horiguchi, David Rientjes, Hugh Dickins,
Davidlohr Bueso, Aneesh Kumar, Hillf Danton, Christoph Hellwig,
Andrew Morton, Michal Hocko, Mike Kravetz
In-Reply-To: <1436401301-18839-1-git-send-email-mike.kravetz@oracle.com>
fallocate hole punch will want to remove a specific range of
pages. When pages are removed, their associated entries in
the region/reserve map will also be removed. This will break
an assumption in the region_chg/region_add calling sequence.
If a new region descriptor must be allocated, it is done as
part of the region_chg processing. In this way, region_add
can not fail because it does not need to attempt an allocation.
To prepare for fallocate hole punch, create a "cache" of
descriptors that can be used by region_add if necessary.
region_chg will ensure there are sufficient entries in the
cache. It will be necessary to track the number of in progress
add operations to know a sufficient number of descriptors
reside in the cache. A new routine region_abort is added to
adjust this in progress count when add operations are aborted.
vma_abort_reservation is also added for callers creating
reservations with vma_needs_reservation/vma_commit_reservation.
Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com>
---
include/linux/hugetlb.h | 3 +
mm/hugetlb.c | 168 ++++++++++++++++++++++++++++++++++++++++++------
2 files changed, 152 insertions(+), 19 deletions(-)
diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index d891f94..667cf44 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -35,6 +35,9 @@ struct resv_map {
struct kref refs;
spinlock_t lock;
struct list_head regions;
+ long adds_in_progress;
+ struct list_head rgn_cache;
+ long rgn_cache_count;
};
extern struct resv_map *resv_map_alloc(void);
void resv_map_release(struct kref *ref);
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index a8c3087..de03374 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -240,11 +240,14 @@ struct file_region {
/*
* Add the huge page range represented by [f, t) to the reserve
- * map. Existing regions will be expanded to accommodate the
- * specified range. We know only existing regions need to be
- * expanded, because region_add is only called after region_chg
- * with the same range. If a new file_region structure must
- * be allocated, it is done in region_chg.
+ * map. In the normal case, existing regions will be expanded
+ * to accommodate the specified range. Sufficient regions should
+ * exist for expansion due to the previous call to region_chg
+ * with the same range. However, it is possible that region_del
+ * could have been called after region_chg and modifed the map
+ * in such a way that no region exists to be expanded. In this
+ * case, pull a region descriptor from the cache associated with
+ * the map and use that for the new range.
*
* Return the number of new huge pages added to the map. This
* number is greater than or equal to zero.
@@ -261,6 +264,27 @@ static long region_add(struct resv_map *resv, long f, long t)
if (f <= rg->to)
break;
+ if (&rg->link == head || t < rg->from) {
+ /*
+ * No region exists which can be expanded to include the
+ * specified range. Pull a region descriptor from the
+ * cache, and use it for this range.
+ */
+ VM_BUG_ON(!resv->rgn_cache_count);
+
+ resv->rgn_cache_count--;
+ nrg = list_first_entry(&resv->rgn_cache, struct file_region,
+ link);
+ list_del(&nrg->link);
+
+ nrg->from = f;
+ nrg->to = t;
+ list_add(&nrg->link, rg->link.prev);
+
+ add += t - f;
+ goto out_locked;
+ }
+
/* Round our left edge to the current segment if it encloses us. */
if (f > rg->from)
f = rg->from;
@@ -294,6 +318,8 @@ static long region_add(struct resv_map *resv, long f, long t)
add += t - nrg->to; /* Added to end of region */
nrg->to = t;
+out_locked:
+ resv->adds_in_progress--;
spin_unlock(&resv->lock);
VM_BUG_ON(add < 0);
return add;
@@ -312,11 +338,16 @@ static long region_add(struct resv_map *resv, long f, long t)
* so that the subsequent region_add call will have all the
* regions it needs and will not fail.
*
+ * Upon entry, region_chg will also examine the cache of
+ * region descriptors associated with the map. If there
+ * not enough descriptors cached, one will be allocated
+ * for the in progress add operation.
+ *
* Returns the number of huge pages that need to be added
* to the existing reservation map for the range [f, t).
* This number is greater or equal to zero. -ENOMEM is
- * returned if a new file_region structure is needed and can
- * not be allocated.
+ * returned if a new file_region structure or cache entry
+ * is needed and can not be allocated.
*/
static long region_chg(struct resv_map *resv, long f, long t)
{
@@ -326,6 +357,30 @@ static long region_chg(struct resv_map *resv, long f, long t)
retry:
spin_lock(&resv->lock);
+ resv->adds_in_progress++;
+
+ /*
+ * Check for sufficient descriptors in the cache to accommodate
+ * the number of in progress add operations.
+ */
+ if (resv->adds_in_progress > resv->rgn_cache_count) {
+ struct file_region *trg;
+
+ VM_BUG_ON(resv->adds_in_progress - resv->rgn_cache_count > 1);
+ /* Must drop lock to allocate a new descriptor. */
+ resv->adds_in_progress--;
+ spin_unlock(&resv->lock);
+
+ trg = kmalloc(sizeof(*trg), GFP_KERNEL);
+ if (!trg)
+ return -ENOMEM;
+
+ spin_lock(&resv->lock);
+ resv->adds_in_progress++;
+ list_add(&trg->link, &resv->rgn_cache);
+ resv->rgn_cache_count++;
+ }
+
/* Locate the region we are before or in. */
list_for_each_entry(rg, head, link)
if (f <= rg->to)
@@ -336,6 +391,7 @@ retry:
* size such that we can guarantee to record the reservation. */
if (&rg->link == head || t < rg->from) {
if (!nrg) {
+ resv->adds_in_progress--;
spin_unlock(&resv->lock);
nrg = kmalloc(sizeof(*nrg), GFP_KERNEL);
if (!nrg)
@@ -385,6 +441,25 @@ out_nrg:
}
/*
+ * Abort the in progress add operation. The adds_in_progress field
+ * of the resv_map keeps track of the operations in progress between
+ * calls to region_chg and region_add. Operations are sometimes
+ * aborted after the call to region_chg. In such cases, region_abort
+ * is called to decrement the adds_in_progress counter.
+ *
+ * NOTE: The range arguments [f, t) are not needed or used in this
+ * routine. They are kept to make reading the calling code easier as
+ * arguments will match the associated region_chg call.
+ */
+static void region_abort(struct resv_map *resv, long f, long t)
+{
+ spin_lock(&resv->lock);
+ VM_BUG_ON(!resv->rgn_cache_count);
+ resv->adds_in_progress--;
+ spin_unlock(&resv->lock);
+}
+
+/*
* Truncate the reserve map at index 'end'. Modify/truncate any
* region which contains end. Delete any regions past end.
* Return the number of huge pages removed from the map.
@@ -544,22 +619,44 @@ static void set_vma_private_data(struct vm_area_struct *vma,
struct resv_map *resv_map_alloc(void)
{
struct resv_map *resv_map = kmalloc(sizeof(*resv_map), GFP_KERNEL);
- if (!resv_map)
+ struct file_region *rg = kmalloc(sizeof(*rg), GFP_KERNEL);
+
+ if (!resv_map || !rg) {
+ kfree(resv_map);
+ kfree(rg);
return NULL;
+ }
kref_init(&resv_map->refs);
spin_lock_init(&resv_map->lock);
INIT_LIST_HEAD(&resv_map->regions);
+ resv_map->adds_in_progress = 0;
+
+ INIT_LIST_HEAD(&resv_map->rgn_cache);
+ list_add(&rg->link, &resv_map->rgn_cache);
+ resv_map->rgn_cache_count = 1;
+
return resv_map;
}
void resv_map_release(struct kref *ref)
{
struct resv_map *resv_map = container_of(ref, struct resv_map, refs);
+ struct list_head *head = &resv_map->rgn_cache;
+ struct file_region *rg, *trg;
/* Clear out any active regions before we release the map. */
region_truncate(resv_map, 0);
+
+ /* ... and any entries left in the cache */
+ list_for_each_entry_safe(rg, trg, head, link) {
+ list_del(&rg->link);
+ kfree(rg);
+ }
+
+ VM_BUG_ON(resv_map->adds_in_progress);
+
kfree(resv_map);
}
@@ -1473,16 +1570,18 @@ static void return_unused_surplus_pages(struct hstate *h,
}
}
+
/*
- * vma_needs_reservation and vma_commit_reservation are used by the huge
- * page allocation routines to manage reservations.
+ * vma_needs_reservation, vma_commit_reservation and vma_abort_reservation
+ * are used by the huge page allocation routines to manage reservations.
*
* vma_needs_reservation is called to determine if the huge page at addr
* within the vma has an associated reservation. If a reservation is
* needed, the value 1 is returned. The caller is then responsible for
* managing the global reservation and subpool usage counts. After
* the huge page has been allocated, vma_commit_reservation is called
- * to add the page to the reservation map.
+ * to add the page to the reservation map. If the reservation must be
+ * aborted instead of committed, vma_abort_reservation is called.
*
* In the normal case, vma_commit_reservation returns the same value
* as the preceding vma_needs_reservation call. The only time this
@@ -1490,9 +1589,14 @@ static void return_unused_surplus_pages(struct hstate *h,
* is the responsibility of the caller to notice the difference and
* take appropriate action.
*/
+enum vma_resv_mode {
+ VMA_NEEDS_RESV,
+ VMA_COMMIT_RESV,
+ VMA_ABORT_RESV,
+};
static long __vma_reservation_common(struct hstate *h,
struct vm_area_struct *vma, unsigned long addr,
- bool commit)
+ enum vma_resv_mode mode)
{
struct resv_map *resv;
pgoff_t idx;
@@ -1503,10 +1607,20 @@ static long __vma_reservation_common(struct hstate *h,
return 1;
idx = vma_hugecache_offset(h, vma, addr);
- if (commit)
- ret = region_add(resv, idx, idx + 1);
- else
+ switch (mode) {
+ case VMA_NEEDS_RESV:
ret = region_chg(resv, idx, idx + 1);
+ break;
+ case VMA_COMMIT_RESV:
+ ret = region_add(resv, idx, idx + 1);
+ break;
+ case VMA_ABORT_RESV:
+ region_abort(resv, idx, idx + 1);
+ ret = 0;
+ break;
+ default:
+ BUG();
+ }
if (vma->vm_flags & VM_MAYSHARE)
return ret;
@@ -1517,13 +1631,19 @@ static long __vma_reservation_common(struct hstate *h,
static long vma_needs_reservation(struct hstate *h,
struct vm_area_struct *vma, unsigned long addr)
{
- return __vma_reservation_common(h, vma, addr, false);
+ return __vma_reservation_common(h, vma, addr, VMA_NEEDS_RESV);
}
static long vma_commit_reservation(struct hstate *h,
struct vm_area_struct *vma, unsigned long addr)
{
- return __vma_reservation_common(h, vma, addr, true);
+ return __vma_reservation_common(h, vma, addr, VMA_COMMIT_RESV);
+}
+
+static void vma_abort_reservation(struct hstate *h,
+ struct vm_area_struct *vma, unsigned long addr)
+{
+ (void)__vma_reservation_common(h, vma, addr, VMA_ABORT_RESV);
}
static struct page *alloc_huge_page(struct vm_area_struct *vma,
@@ -1549,8 +1669,10 @@ static struct page *alloc_huge_page(struct vm_area_struct *vma,
if (chg < 0)
return ERR_PTR(-ENOMEM);
if (chg || avoid_reserve)
- if (hugepage_subpool_get_pages(spool, 1) < 0)
+ if (hugepage_subpool_get_pages(spool, 1) < 0) {
+ vma_abort_reservation(h, vma, addr);
return ERR_PTR(-ENOSPC);
+ }
ret = hugetlb_cgroup_charge_cgroup(idx, pages_per_huge_page(h), &h_cg);
if (ret)
@@ -1596,6 +1718,7 @@ out_uncharge_cgroup:
out_subpool_put:
if (chg || avoid_reserve)
hugepage_subpool_put_pages(spool, 1);
+ vma_abort_reservation(h, vma, addr);
return ERR_PTR(-ENOSPC);
}
@@ -3236,11 +3359,14 @@ retry:
* any allocations necessary to record that reservation occur outside
* the spinlock.
*/
- if ((flags & FAULT_FLAG_WRITE) && !(vma->vm_flags & VM_SHARED))
+ if ((flags & FAULT_FLAG_WRITE) && !(vma->vm_flags & VM_SHARED)) {
if (vma_needs_reservation(h, vma, address) < 0) {
ret = VM_FAULT_OOM;
goto backout_unlocked;
}
+ /* Just decrements count, does not deallocate */
+ vma_abort_reservation(h, vma, address);
+ }
ptl = huge_pte_lockptr(h, mm, ptep);
spin_lock(ptl);
@@ -3387,6 +3513,8 @@ int hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,
ret = VM_FAULT_OOM;
goto out_mutex;
}
+ /* Just decrements count, does not deallocate */
+ vma_abort_reservation(h, vma, address);
if (!(vma->vm_flags & VM_MAYSHARE))
pagecache_page = hugetlbfs_pagecache_page(h,
@@ -3726,6 +3854,8 @@ int hugetlb_reserve_pages(struct inode *inode,
}
return 0;
out_err:
+ if (!vma || vma->vm_flags & VM_MAYSHARE)
+ region_abort(resv_map, from, to);
if (vma && is_vma_resv_set(vma, HPAGE_RESV_OWNER))
kref_put(&resv_map->refs, resv_map_release);
return ret;
--
2.1.0
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [PATCH v2 02/10] mm/hugetlb: add region_del() to delete a specific range of entries
From: Mike Kravetz @ 2015-07-09 0:21 UTC (permalink / raw)
To: linux-mm, linux-kernel, linux-api
Cc: Dave Hansen, Naoya Horiguchi, David Rientjes, Hugh Dickins,
Davidlohr Bueso, Aneesh Kumar, Hillf Danton, Christoph Hellwig,
Andrew Morton, Michal Hocko, Mike Kravetz
In-Reply-To: <1436401301-18839-1-git-send-email-mike.kravetz@oracle.com>
fallocate hole punch will want to remove a specific range of pages.
The existing region_truncate() routine deletes all region/reserve
map entries after a specified offset. region_del() will provide
this same functionality if the end of region is specified as LONG_MAX.
Hence, region_del() can replace region_truncate().
Unlike region_truncate(), region_del() can return an error in the
rare case where it can not allocate memory for a region descriptor.
This ONLY happens in the case where an existing region must be split.
Current callers passing LONG_MAX as end of range will never experience
this error and do not need to deal with error handling. Future
callers of region_del() (such as fallocate hole punch) will need to
handle this error.
Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com>
---
mm/hugetlb.c | 99 ++++++++++++++++++++++++++++++++++++++++++++----------------
1 file changed, 73 insertions(+), 26 deletions(-)
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index de03374..1b21fb7 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -460,43 +460,90 @@ static void region_abort(struct resv_map *resv, long f, long t)
}
/*
- * Truncate the reserve map at index 'end'. Modify/truncate any
- * region which contains end. Delete any regions past end.
- * Return the number of huge pages removed from the map.
+ * Delete the specified range [f, t) from the reserve map. If the
+ * t parameter is LONG_MAX, this indicates that ALL regions after f
+ * should be deleted. Locate the regions which intersect [f, t)
+ * and either trim, delete or split the existing regions.
+ *
+ * Returns the number of huge pages deleted from the reserve map.
+ * In the normal case, the return value is zero or more. In the
+ * case where a region must be split, a new region descriptor must
+ * be allocated. If the allocation fails, -ENOMEM will be returned.
+ * NOTE: If the parameter t == LONG_MAX, then we will never split
+ * a region and possibly return -ENOMEM. Callers specifying
+ * t == LONG_MAX do not need to check for -ENOMEM error.
*/
-static long region_truncate(struct resv_map *resv, long end)
+static long region_del(struct resv_map *resv, long f, long t)
{
struct list_head *head = &resv->regions;
struct file_region *rg, *trg;
- long chg = 0;
+ struct file_region *nrg = NULL;
+ long del = 0;
+retry:
spin_lock(&resv->lock);
- /* Locate the region we are either in or before. */
- list_for_each_entry(rg, head, link)
- if (end <= rg->to)
+ list_for_each_entry_safe(rg, trg, head, link) {
+ if (rg->to <= f)
+ continue;
+ if (rg->from >= t)
break;
- if (&rg->link == head)
- goto out;
- /* If we are in the middle of a region then adjust it. */
- if (end > rg->from) {
- chg = rg->to - end;
- rg->to = end;
- rg = list_entry(rg->link.next, typeof(*rg), link);
- }
+ if (f > rg->from && t < rg->to) { /* Must split region */
+ /*
+ * Check for an entry in the cache before dropping
+ * lock and attempting allocation.
+ */
+ if (!nrg &&
+ resv->rgn_cache_count > resv->adds_in_progress) {
+ nrg = list_first_entry(&resv->rgn_cache,
+ struct file_region,
+ link);
+ list_del(&nrg->link);
+ resv->rgn_cache_count--;
+ }
- /* Drop any remaining regions. */
- list_for_each_entry_safe(rg, trg, rg->link.prev, link) {
- if (&rg->link == head)
+ if (!nrg) {
+ spin_unlock(&resv->lock);
+ nrg = kmalloc(sizeof(*nrg), GFP_KERNEL);
+ if (!nrg)
+ return -ENOMEM;
+ goto retry;
+ }
+
+ del += t - f;
+
+ /* New entry for end of split region */
+ nrg->from = t;
+ nrg->to = rg->to;
+ INIT_LIST_HEAD(&nrg->link);
+
+ /* Original entry is trimmed */
+ rg->to = f;
+
+ list_add(&nrg->link, &rg->link);
+ nrg = NULL;
break;
- chg += rg->to - rg->from;
- list_del(&rg->link);
- kfree(rg);
+ }
+
+ if (f <= rg->from && t >= rg->to) { /* Remove entire region */
+ del += rg->to - rg->from;
+ list_del(&rg->link);
+ kfree(rg);
+ continue;
+ }
+
+ if (f <= rg->from) { /* Trim beginning of region */
+ del += t - rg->from;
+ rg->from = t;
+ } else { /* Trim end of region */
+ del += rg->to - f;
+ rg->to = f;
+ }
}
-out:
spin_unlock(&resv->lock);
- return chg;
+ kfree(nrg);
+ return del;
}
/*
@@ -647,7 +694,7 @@ void resv_map_release(struct kref *ref)
struct file_region *rg, *trg;
/* Clear out any active regions before we release the map. */
- region_truncate(resv_map, 0);
+ region_del(resv_map, 0, LONG_MAX);
/* ... and any entries left in the cache */
list_for_each_entry_safe(rg, trg, head, link) {
@@ -3870,7 +3917,7 @@ void hugetlb_unreserve_pages(struct inode *inode, long offset, long freed)
long gbl_reserve;
if (resv_map)
- chg = region_truncate(resv_map, offset);
+ chg = region_del(resv_map, offset, LONG_MAX);
spin_lock(&inode->i_lock);
inode->i_blocks -= (blocks_per_huge_page(h) * freed);
spin_unlock(&inode->i_lock);
--
2.1.0
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [PATCH v2 03/10] mm/hugetlb: expose hugetlb fault mutex for use by fallocate
From: Mike Kravetz @ 2015-07-09 0:21 UTC (permalink / raw)
To: linux-mm, linux-kernel, linux-api
Cc: Dave Hansen, Naoya Horiguchi, David Rientjes, Hugh Dickins,
Davidlohr Bueso, Aneesh Kumar, Hillf Danton, Christoph Hellwig,
Andrew Morton, Michal Hocko, Mike Kravetz
In-Reply-To: <1436401301-18839-1-git-send-email-mike.kravetz@oracle.com>
hugetlb page faults are currently synchronized by the table of
mutexes (htlb_fault_mutex_table). fallocate code will need to
synchronize with the page fault code when it allocates or
deletes pages. Expose interfaces so that fallocate operations
can be synchronized with page faults. Minor name changes to
be more consistent with other global hugetlb symbols.
Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com>
---
include/linux/hugetlb.h | 5 +++++
mm/hugetlb.c | 20 ++++++++++----------
2 files changed, 15 insertions(+), 10 deletions(-)
diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index 667cf44..933da39 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -88,6 +88,11 @@ int dequeue_hwpoisoned_huge_page(struct page *page);
bool isolate_huge_page(struct page *page, struct list_head *list);
void putback_active_hugepage(struct page *page);
void free_huge_page(struct page *page);
+extern struct mutex *hugetlb_fault_mutex_table;
+u32 hugetlb_fault_mutex_hash(struct hstate *h, struct mm_struct *mm,
+ struct vm_area_struct *vma,
+ struct address_space *mapping,
+ pgoff_t idx, unsigned long address);
#ifdef CONFIG_ARCH_WANT_HUGE_PMD_SHARE
pte_t *huge_pmd_share(struct mm_struct *mm, unsigned long addr, pud_t *pud);
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 1b21fb7..58113d1 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -64,7 +64,7 @@ DEFINE_SPINLOCK(hugetlb_lock);
* prevent spurious OOMs when the hugepage pool is fully utilized.
*/
static int num_fault_mutexes;
-static struct mutex *htlb_fault_mutex_table ____cacheline_aligned_in_smp;
+struct mutex *hugetlb_fault_mutex_table ____cacheline_aligned_in_smp;
/* Forward declaration */
static int hugetlb_acct_memory(struct hstate *h, long delta);
@@ -2481,7 +2481,7 @@ static void __exit hugetlb_exit(void)
}
kobject_put(hugepages_kobj);
- kfree(htlb_fault_mutex_table);
+ kfree(hugetlb_fault_mutex_table);
}
module_exit(hugetlb_exit);
@@ -2514,12 +2514,12 @@ static int __init hugetlb_init(void)
#else
num_fault_mutexes = 1;
#endif
- htlb_fault_mutex_table =
+ hugetlb_fault_mutex_table =
kmalloc(sizeof(struct mutex) * num_fault_mutexes, GFP_KERNEL);
- BUG_ON(!htlb_fault_mutex_table);
+ BUG_ON(!hugetlb_fault_mutex_table);
for (i = 0; i < num_fault_mutexes; i++)
- mutex_init(&htlb_fault_mutex_table[i]);
+ mutex_init(&hugetlb_fault_mutex_table[i]);
return 0;
}
module_init(hugetlb_init);
@@ -3453,7 +3453,7 @@ backout_unlocked:
}
#ifdef CONFIG_SMP
-static u32 fault_mutex_hash(struct hstate *h, struct mm_struct *mm,
+u32 hugetlb_fault_mutex_hash(struct hstate *h, struct mm_struct *mm,
struct vm_area_struct *vma,
struct address_space *mapping,
pgoff_t idx, unsigned long address)
@@ -3478,7 +3478,7 @@ static u32 fault_mutex_hash(struct hstate *h, struct mm_struct *mm,
* For uniprocesor systems we always use a single mutex, so just
* return 0 and avoid the hashing overhead.
*/
-static u32 fault_mutex_hash(struct hstate *h, struct mm_struct *mm,
+u32 hugetlb_fault_mutex_hash(struct hstate *h, struct mm_struct *mm,
struct vm_area_struct *vma,
struct address_space *mapping,
pgoff_t idx, unsigned long address)
@@ -3526,8 +3526,8 @@ int hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,
* get spurious allocation failures if two CPUs race to instantiate
* the same page in the page cache.
*/
- hash = fault_mutex_hash(h, mm, vma, mapping, idx, address);
- mutex_lock(&htlb_fault_mutex_table[hash]);
+ hash = hugetlb_fault_mutex_hash(h, mm, vma, mapping, idx, address);
+ mutex_lock(&hugetlb_fault_mutex_table[hash]);
entry = huge_ptep_get(ptep);
if (huge_pte_none(entry)) {
@@ -3612,7 +3612,7 @@ out_ptl:
put_page(pagecache_page);
}
out_mutex:
- mutex_unlock(&htlb_fault_mutex_table[hash]);
+ mutex_unlock(&hugetlb_fault_mutex_table[hash]);
/*
* Generally it's safe to hold refcount during waiting page lock. But
* here we just wait to defer the next page fault to avoid busy loop and
--
2.1.0
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [PATCH v2 04/10] hugetlbfs: hugetlb_vmtruncate_list() needs to take a range to delete
From: Mike Kravetz @ 2015-07-09 0:21 UTC (permalink / raw)
To: linux-mm, linux-kernel, linux-api
Cc: Dave Hansen, Naoya Horiguchi, David Rientjes, Hugh Dickins,
Davidlohr Bueso, Aneesh Kumar, Hillf Danton, Christoph Hellwig,
Andrew Morton, Michal Hocko, Mike Kravetz
In-Reply-To: <1436401301-18839-1-git-send-email-mike.kravetz@oracle.com>
fallocate hole punch will want to unmap a specific range of pages.
Modify the existing hugetlb_vmtruncate_list() routine to take a
start/end range. If end is 0, this indicates all pages after start
should be unmapped. This is the same as the existing truncate
functionality. Modify existing callers to add 0 as end of range.
Since the routine will be used in hole punch as well as truncate
operations, it is more appropriately renamed to hugetlb_vmdelete_list().
Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com>
---
fs/hugetlbfs/inode.c | 25 ++++++++++++++++++-------
1 file changed, 18 insertions(+), 7 deletions(-)
diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c
index 0cf74df..ed40f56 100644
--- a/fs/hugetlbfs/inode.c
+++ b/fs/hugetlbfs/inode.c
@@ -349,11 +349,15 @@ static void hugetlbfs_evict_inode(struct inode *inode)
}
static inline void
-hugetlb_vmtruncate_list(struct rb_root *root, pgoff_t pgoff)
+hugetlb_vmdelete_list(struct rb_root *root, pgoff_t start, pgoff_t end)
{
struct vm_area_struct *vma;
- vma_interval_tree_foreach(vma, root, pgoff, ULONG_MAX) {
+ /*
+ * end == 0 indicates that the entire range after
+ * start should be unmapped.
+ */
+ vma_interval_tree_foreach(vma, root, start, end ? end : ULONG_MAX) {
unsigned long v_offset;
/*
@@ -362,13 +366,20 @@ hugetlb_vmtruncate_list(struct rb_root *root, pgoff_t pgoff)
* which overlap the truncated area starting at pgoff,
* and no vma on a 32-bit arch can span beyond the 4GB.
*/
- if (vma->vm_pgoff < pgoff)
- v_offset = (pgoff - vma->vm_pgoff) << PAGE_SHIFT;
+ if (vma->vm_pgoff < start)
+ v_offset = (start - vma->vm_pgoff) << PAGE_SHIFT;
else
v_offset = 0;
- unmap_hugepage_range(vma, vma->vm_start + v_offset,
- vma->vm_end, NULL);
+ if (end) {
+ end = ((end - start) << PAGE_SHIFT) +
+ vma->vm_start + v_offset;
+ if (end > vma->vm_end)
+ end = vma->vm_end;
+ } else
+ end = vma->vm_end;
+
+ unmap_hugepage_range(vma, vma->vm_start + v_offset, end, NULL);
}
}
@@ -384,7 +395,7 @@ static int hugetlb_vmtruncate(struct inode *inode, loff_t offset)
i_size_write(inode, offset);
i_mmap_lock_write(mapping);
if (!RB_EMPTY_ROOT(&mapping->i_mmap))
- hugetlb_vmtruncate_list(&mapping->i_mmap, pgoff);
+ hugetlb_vmdelete_list(&mapping->i_mmap, pgoff, 0);
i_mmap_unlock_write(mapping);
truncate_hugepages(inode, offset);
return 0;
--
2.1.0
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [PATCH v2 05/10] hugetlbfs: truncate_hugepages() takes a range of pages
From: Mike Kravetz @ 2015-07-09 0:21 UTC (permalink / raw)
To: linux-mm, linux-kernel, linux-api
Cc: Dave Hansen, Naoya Horiguchi, David Rientjes, Hugh Dickins,
Davidlohr Bueso, Aneesh Kumar, Hillf Danton, Christoph Hellwig,
Andrew Morton, Michal Hocko, Mike Kravetz
In-Reply-To: <1436401301-18839-1-git-send-email-mike.kravetz@oracle.com>
Modify truncate_hugepages() to take a range of pages (start, end)
instead of simply start. If an end value of -1 is passed, the
current "truncate" functionality is maintained. Existing callers
are modified to pass -1 as end of range. By keying off end == -1,
the routine behaves differently for truncate and hole punch.
Page removal is now synchronized with page allocation via faults
by using the fault mutex table. The hole punch case can experience
the rare region_del error and must handle accordingly.
Add the routine hugetlb_fix_reserve_counts to fix up reserve counts
in the case where region_del returns an error.
Since the routine handles more than just the truncate case, it is
renamed to remove_inode_hugepages(). To be consistent, the routine
truncate_huge_page() is renamed remove_huge_page().
Downstream of remove_inode_hugepages(), the routine
hugetlb_unreserve_pages() is also modified to take a range of pages.
hugetlb_unreserve_pages is modified to detect an error from
region_del and pass it back to the caller.
Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com>
---
fs/hugetlbfs/inode.c | 98 ++++++++++++++++++++++++++++++++++++++++++++-----
include/linux/hugetlb.h | 4 +-
mm/hugetlb.c | 40 ++++++++++++++++++--
3 files changed, 128 insertions(+), 14 deletions(-)
diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c
index ed40f56..f869993 100644
--- a/fs/hugetlbfs/inode.c
+++ b/fs/hugetlbfs/inode.c
@@ -293,26 +293,61 @@ static int hugetlbfs_write_end(struct file *file, struct address_space *mapping,
return -EINVAL;
}
-static void truncate_huge_page(struct page *page)
+static void remove_huge_page(struct page *page)
{
ClearPageDirty(page);
ClearPageUptodate(page);
delete_from_page_cache(page);
}
-static void truncate_hugepages(struct inode *inode, loff_t lstart)
+
+/*
+ * remove_inode_hugepages handles two distinct cases: truncation and hole
+ * punch. There are subtle differences in operation for each case.
+
+ * truncation is indicated by end of range being -1
+ * In this case, we first scan the range and release found pages.
+ * After releasing pages, hugetlb_unreserve_pages cleans up region/reserv
+ * maps and global counts.
+ * hole punch is indicated if end is not -1
+ * In the hole punch case we scan the range and release found pages.
+ * Only when releasing a page is the associated region/reserv map
+ * deleted. The region/reserv map for ranges without associated
+ * pages are not modified.
+ * Note: If the passed end of range value is beyond the end of file, but
+ * not -1 this routine still performs a hole punch operation.
+ */
+static void remove_inode_hugepages(struct inode *inode, loff_t lstart,
+ loff_t lend)
{
struct hstate *h = hstate_inode(inode);
struct address_space *mapping = &inode->i_data;
const pgoff_t start = lstart >> huge_page_shift(h);
+ const pgoff_t end = lend >> huge_page_shift(h);
+ struct vm_area_struct pseudo_vma;
struct pagevec pvec;
pgoff_t next;
int i, freed = 0;
+ long lookup_nr = PAGEVEC_SIZE;
+ bool truncate_op = (lend == -1);
+ memset(&pseudo_vma, 0, sizeof(struct vm_area_struct));
+ pseudo_vma.vm_flags = (VM_HUGETLB | VM_MAYSHARE | VM_SHARED);
pagevec_init(&pvec, 0);
next = start;
- while (1) {
- if (!pagevec_lookup(&pvec, mapping, next, PAGEVEC_SIZE)) {
+ while (next < end) {
+ /*
+ * Make sure to never grab more pages that we
+ * might possibly need.
+ */
+ if (end - next < lookup_nr)
+ lookup_nr = end - next;
+
+ /*
+ * This pagevec_lookup() may return pages past 'end',
+ * so we must check for page->index > end.
+ */
+ if (!pagevec_lookup(&pvec, mapping, next, lookup_nr)) {
if (next == start)
break;
next = start;
@@ -321,26 +356,69 @@ static void truncate_hugepages(struct inode *inode, loff_t lstart)
for (i = 0; i < pagevec_count(&pvec); ++i) {
struct page *page = pvec.pages[i];
+ u32 hash;
+
+ hash = hugetlb_fault_mutex_hash(h, current->mm,
+ &pseudo_vma,
+ mapping, next, 0);
+ mutex_lock(&hugetlb_fault_mutex_table[hash]);
lock_page(page);
+ if (page->index >= end) {
+ unlock_page(page);
+ mutex_unlock(&hugetlb_fault_mutex_table[hash]);
+ next = end; /* we are done */
+ break;
+ }
+
+ /*
+ * If page is mapped, it was faulted in after being
+ * unmapped. Do nothing in this race case. In the
+ * normal case page is not mapped.
+ */
+ if (!page_mapped(page)) {
+ bool rsv_on_error = !PagePrivate(page);
+ /*
+ * We must free the huge page and remove
+ * from page cache (remove_huge_page) BEFORE
+ * removing the region/reserve map
+ * (hugetlb_unreserve_pages). In rare out
+ * of memory conditions, removal of the
+ * region/reserve map could fail. Before
+ * free'ing the page, note PagePrivate which
+ * is used in case of error.
+ */
+ remove_huge_page(page);
+ freed++;
+ if (!truncate_op) {
+ if (unlikely(hugetlb_unreserve_pages(
+ inode, next,
+ next + 1, 1)))
+ hugetlb_fix_reserve_counts(
+ inode, rsv_on_error);
+ }
+ }
+
if (page->index > next)
next = page->index;
+
++next;
- truncate_huge_page(page);
unlock_page(page);
- freed++;
+
+ mutex_unlock(&hugetlb_fault_mutex_table[hash]);
}
huge_pagevec_release(&pvec);
}
- BUG_ON(!lstart && mapping->nrpages);
- hugetlb_unreserve_pages(inode, start, freed);
+
+ if (truncate_op)
+ (void)hugetlb_unreserve_pages(inode, start, LONG_MAX, freed);
}
static void hugetlbfs_evict_inode(struct inode *inode)
{
struct resv_map *resv_map;
- truncate_hugepages(inode, 0);
+ remove_inode_hugepages(inode, 0, -1);
resv_map = (struct resv_map *)inode->i_mapping->private_data;
/* root inode doesn't have the resv_map, so we should check it */
if (resv_map)
@@ -397,7 +475,7 @@ static int hugetlb_vmtruncate(struct inode *inode, loff_t offset)
if (!RB_EMPTY_ROOT(&mapping->i_mmap))
hugetlb_vmdelete_list(&mapping->i_mmap, pgoff, 0);
i_mmap_unlock_write(mapping);
- truncate_hugepages(inode, offset);
+ remove_inode_hugepages(inode, offset, -1);
return 0;
}
diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index 933da39..e7825c9 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -83,11 +83,13 @@ int hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,
int hugetlb_reserve_pages(struct inode *inode, long from, long to,
struct vm_area_struct *vma,
vm_flags_t vm_flags);
-void hugetlb_unreserve_pages(struct inode *inode, long offset, long freed);
+long hugetlb_unreserve_pages(struct inode *inode, long start, long end,
+ long freed);
int dequeue_hwpoisoned_huge_page(struct page *page);
bool isolate_huge_page(struct page *page, struct list_head *list);
void putback_active_hugepage(struct page *page);
void free_huge_page(struct page *page);
+void hugetlb_fix_reserve_counts(struct inode *inode, bool restore_reserve);
extern struct mutex *hugetlb_fault_mutex_table;
u32 hugetlb_fault_mutex_hash(struct hstate *h, struct mm_struct *mm,
struct vm_area_struct *vma,
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 58113d1..d39a6ad 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -547,6 +547,28 @@ retry:
}
/*
+ * A rare out of memory error was encountered which prevented removal of
+ * the reserve map region for a page. The huge page itself was free'ed
+ * and removed from the page cache. This routine will adjust the subpool
+ * usage count, and the global reserve count if needed. By incrementing
+ * these counts, the reserve map entry which could not be deleted will
+ * appear as a "reserved" entry instead of simply dangling with incorrect
+ * counts.
+ */
+void hugetlb_fix_reserve_counts(struct inode *inode, bool restore_reserve)
+{
+ struct hugepage_subpool *spool = subpool_inode(inode);
+ long rsv_adjust;
+
+ rsv_adjust = hugepage_subpool_get_pages(spool, 1);
+ if (restore_reserve && rsv_adjust) {
+ struct hstate *h = hstate_inode(inode);
+
+ hugetlb_acct_memory(h, 1);
+ }
+}
+
+/*
* Count and return the number of huge pages in the reserve map
* that intersect with the range [f, t).
*/
@@ -3908,7 +3930,8 @@ out_err:
return ret;
}
-void hugetlb_unreserve_pages(struct inode *inode, long offset, long freed)
+long hugetlb_unreserve_pages(struct inode *inode, long start, long end,
+ long freed)
{
struct hstate *h = hstate_inode(inode);
struct resv_map *resv_map = inode_resv_map(inode);
@@ -3916,8 +3939,17 @@ void hugetlb_unreserve_pages(struct inode *inode, long offset, long freed)
struct hugepage_subpool *spool = subpool_inode(inode);
long gbl_reserve;
- if (resv_map)
- chg = region_del(resv_map, offset, LONG_MAX);
+ if (resv_map) {
+ chg = region_del(resv_map, start, end);
+ /*
+ * region_del() can fail in the rare case where a region
+ * must be split and another region descriptor can not be
+ * allocated. If end == LONG_MAX, it will not fail.
+ */
+ if (chg < 0)
+ return chg;
+ }
+
spin_lock(&inode->i_lock);
inode->i_blocks -= (blocks_per_huge_page(h) * freed);
spin_unlock(&inode->i_lock);
@@ -3928,6 +3960,8 @@ void hugetlb_unreserve_pages(struct inode *inode, long offset, long freed)
*/
gbl_reserve = hugepage_subpool_put_pages(spool, (chg - freed));
hugetlb_acct_memory(h, -gbl_reserve);
+
+ return 0;
}
#ifdef CONFIG_ARCH_WANT_HUGE_PMD_SHARE
--
2.1.0
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [PATCH v2 06/10] mm/hugetlb: vma_has_reserves() needs to handle fallocate hole punch
From: Mike Kravetz @ 2015-07-09 0:21 UTC (permalink / raw)
To: linux-mm, linux-kernel, linux-api
Cc: Dave Hansen, Naoya Horiguchi, David Rientjes, Hugh Dickins,
Davidlohr Bueso, Aneesh Kumar, Hillf Danton, Christoph Hellwig,
Andrew Morton, Michal Hocko, Mike Kravetz
In-Reply-To: <1436401301-18839-1-git-send-email-mike.kravetz@oracle.com>
In vma_has_reserves(), the current assumption is that reserves are
always present for shared mappings. However, this will not be the
case with fallocate hole punch. When punching a hole, the present
page will be deleted as well as the region/reserve map entry (and
hence any reservation). vma_has_reserves is passed "chg" which
indicates whether or not a region/reserve map is present. Use
this to determine if reserves are actually present or were removed
via hole punch.
Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com>
---
mm/hugetlb.c | 16 +++++++++++++---
1 file changed, 13 insertions(+), 3 deletions(-)
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index d39a6ad..93c7089 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -800,9 +800,19 @@ static int vma_has_reserves(struct vm_area_struct *vma, long chg)
return 0;
}
- /* Shared mappings always use reserves */
- if (vma->vm_flags & VM_MAYSHARE)
- return 1;
+ if (vma->vm_flags & VM_MAYSHARE) {
+ /*
+ * We know VM_NORESERVE is not set. Therefore, there SHOULD
+ * be a region map for all pages. The only situation where
+ * there is no region map is if a hole was punched via
+ * fallocate. In this case, there really are no reverves to
+ * use. This situation is indicated if chg != 0.
+ */
+ if (chg)
+ return 0;
+ else
+ return 1;
+ }
/*
* Only the process that called mmap() has reserves for
--
2.1.0
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [PATCH v2 07/10] mm/hugetlb: alloc_huge_page handle areas hole punched by fallocate
From: Mike Kravetz @ 2015-07-09 0:21 UTC (permalink / raw)
To: linux-mm, linux-kernel, linux-api
Cc: Dave Hansen, Naoya Horiguchi, David Rientjes, Hugh Dickins,
Davidlohr Bueso, Aneesh Kumar, Hillf Danton, Christoph Hellwig,
Andrew Morton, Michal Hocko, Mike Kravetz
In-Reply-To: <1436401301-18839-1-git-send-email-mike.kravetz@oracle.com>
Areas hole punched by fallocate will not have entries in the
region/reserve map. However, shared mappings with min_size subpool
reservations may still have reserved pages. alloc_huge_page needs
to handle this special case and do the proper accounting.
Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com>
---
mm/hugetlb.c | 54 +++++++++++++++++++++++++++++++++++++++---------------
1 file changed, 39 insertions(+), 15 deletions(-)
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 93c7089..66f69d7 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -1731,34 +1731,58 @@ static struct page *alloc_huge_page(struct vm_area_struct *vma,
struct hugepage_subpool *spool = subpool_vma(vma);
struct hstate *h = hstate_vma(vma);
struct page *page;
- long chg, commit;
+ long map_chg, map_commit;
+ long gbl_chg;
int ret, idx;
struct hugetlb_cgroup *h_cg;
idx = hstate_index(h);
/*
- * Processes that did not create the mapping will have no
- * reserves and will not have accounted against subpool
- * limit. Check that the subpool limit can be made before
- * satisfying the allocation MAP_NORESERVE mappings may also
- * need pages and subpool limit allocated allocated if no reserve
- * mapping overlaps.
+ * Examine the region/reserve map to determine if the process
+ * has a reservation for the page to be allocated. A return
+ * code of zero indicates a reservation exists (no change).
*/
- chg = vma_needs_reservation(h, vma, addr);
- if (chg < 0)
+ map_chg = gbl_chg = vma_needs_reservation(h, vma, addr);
+ if (map_chg < 0)
return ERR_PTR(-ENOMEM);
- if (chg || avoid_reserve)
- if (hugepage_subpool_get_pages(spool, 1) < 0) {
+
+ /*
+ * Processes that did not create the mapping will have no
+ * reserves as indicated by the region/reserve map. Check
+ * that the allocation will not exceed the subpool limit.
+ * Allocations for MAP_NORESERVE mappings also need to be
+ * checked against any subpool limit.
+ */
+ if (map_chg || avoid_reserve) {
+ gbl_chg = hugepage_subpool_get_pages(spool, 1);
+ if (gbl_chg < 0) {
vma_abort_reservation(h, vma, addr);
return ERR_PTR(-ENOSPC);
}
+ /*
+ * Even though there was no reservation in the region/reserve
+ * map, there could be reservations associated with the
+ * subpool that can be used. This would be indicated if the
+ * return value of hugepage_subpool_get_pages() is zero.
+ * However, if avoid_reserve is specified we still avoid even
+ * the subpool reservations.
+ */
+ if (avoid_reserve)
+ gbl_chg = 1;
+ }
+
ret = hugetlb_cgroup_charge_cgroup(idx, pages_per_huge_page(h), &h_cg);
if (ret)
goto out_subpool_put;
spin_lock(&hugetlb_lock);
- page = dequeue_huge_page_vma(h, vma, addr, avoid_reserve, chg);
+ /*
+ * glb_chg is passed to indicate whether or not a page must be taken
+ * from the global free pool (global change). gbl_chg == 0 indicates
+ * a reservation exists for the allocation.
+ */
+ page = dequeue_huge_page_vma(h, vma, addr, avoid_reserve, gbl_chg);
if (!page) {
spin_unlock(&hugetlb_lock);
page = alloc_buddy_huge_page(h, NUMA_NO_NODE);
@@ -1774,8 +1798,8 @@ static struct page *alloc_huge_page(struct vm_area_struct *vma,
set_page_private(page, (unsigned long)spool);
- commit = vma_commit_reservation(h, vma, addr);
- if (unlikely(chg > commit)) {
+ map_commit = vma_commit_reservation(h, vma, addr);
+ if (unlikely(map_chg > map_commit)) {
/*
* The page was added to the reservation map between
* vma_needs_reservation and vma_commit_reservation.
@@ -1795,7 +1819,7 @@ static struct page *alloc_huge_page(struct vm_area_struct *vma,
out_uncharge_cgroup:
hugetlb_cgroup_uncharge_cgroup(idx, pages_per_huge_page(h), h_cg);
out_subpool_put:
- if (chg || avoid_reserve)
+ if (map_chg || avoid_reserve)
hugepage_subpool_put_pages(spool, 1);
vma_abort_reservation(h, vma, addr);
return ERR_PTR(-ENOSPC);
--
2.1.0
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [PATCH v2 08/10] hugetlbfs: New huge_add_to_page_cache helper routine
From: Mike Kravetz @ 2015-07-09 0:21 UTC (permalink / raw)
To: linux-mm, linux-kernel, linux-api
Cc: Dave Hansen, Naoya Horiguchi, David Rientjes, Hugh Dickins,
Davidlohr Bueso, Aneesh Kumar, Hillf Danton, Christoph Hellwig,
Andrew Morton, Michal Hocko, Mike Kravetz
In-Reply-To: <1436401301-18839-1-git-send-email-mike.kravetz@oracle.com>
Currently, there is only a single place where hugetlbfs pages are
added to the page cache. The new fallocate code be adding a second
one, so break the functionality out into its own helper.
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com>
---
include/linux/hugetlb.h | 2 ++
mm/hugetlb.c | 27 ++++++++++++++++++---------
2 files changed, 20 insertions(+), 9 deletions(-)
diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index e7825c9..657ef26 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -333,6 +333,8 @@ struct huge_bootmem_page {
struct page *alloc_huge_page_node(struct hstate *h, int nid);
struct page *alloc_huge_page_noerr(struct vm_area_struct *vma,
unsigned long addr, int avoid_reserve);
+int huge_add_to_page_cache(struct page *page, struct address_space *mapping,
+ pgoff_t idx);
/* arch callback */
int __init alloc_bootmem_huge_page(struct hstate *h);
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 66f69d7..5d48436 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3373,6 +3373,23 @@ static bool hugetlbfs_pagecache_present(struct hstate *h,
return page != NULL;
}
+int huge_add_to_page_cache(struct page *page, struct address_space *mapping,
+ pgoff_t idx)
+{
+ struct inode *inode = mapping->host;
+ struct hstate *h = hstate_inode(inode);
+ int err = add_to_page_cache(page, mapping, idx, GFP_KERNEL);
+
+ if (err)
+ return err;
+ ClearPagePrivate(page);
+
+ spin_lock(&inode->i_lock);
+ inode->i_blocks += blocks_per_huge_page(h);
+ spin_unlock(&inode->i_lock);
+ return 0;
+}
+
static int hugetlb_no_page(struct mm_struct *mm, struct vm_area_struct *vma,
struct address_space *mapping, pgoff_t idx,
unsigned long address, pte_t *ptep, unsigned int flags)
@@ -3420,21 +3437,13 @@ retry:
set_page_huge_active(page);
if (vma->vm_flags & VM_MAYSHARE) {
- int err;
- struct inode *inode = mapping->host;
-
- err = add_to_page_cache(page, mapping, idx, GFP_KERNEL);
+ int err = huge_add_to_page_cache(page, mapping, idx);
if (err) {
put_page(page);
if (err == -EEXIST)
goto retry;
goto out;
}
- ClearPagePrivate(page);
-
- spin_lock(&inode->i_lock);
- inode->i_blocks += blocks_per_huge_page(h);
- spin_unlock(&inode->i_lock);
} else {
lock_page(page);
if (unlikely(anon_vma_prepare(vma))) {
--
2.1.0
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [PATCH v2 09/10] hugetlbfs: add hugetlbfs_fallocate()
From: Mike Kravetz @ 2015-07-09 0:21 UTC (permalink / raw)
To: linux-mm, linux-kernel, linux-api
Cc: Dave Hansen, Naoya Horiguchi, David Rientjes, Hugh Dickins,
Davidlohr Bueso, Aneesh Kumar, Hillf Danton, Christoph Hellwig,
Andrew Morton, Michal Hocko, Mike Kravetz
In-Reply-To: <1436401301-18839-1-git-send-email-mike.kravetz@oracle.com>
This is based on the shmem version, but it has diverged quite
a bit. We have no swap to worry about, nor the new file sealing.
Add synchronication via the fault mutex table to coordinate
page faults, fallocate allocation and fallocate hole punch.
What this allows us to do is move physical memory in and out of
a hugetlbfs file without having it mapped. This also gives us
the ability to support MADV_REMOVE since it is currently
implemented using fallocate(). MADV_REMOVE lets madvise() remove
pages from the middle of a hugetlbfs file, which wasn't possible
before.
hugetlbfs fallocate only operates on whole huge pages.
Based-on code-by: Dave Hansen <dave.hansen@linux.intel.com>
Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com>
---
fs/hugetlbfs/inode.c | 158 +++++++++++++++++++++++++++++++++++++++++++++++-
include/linux/hugetlb.h | 3 +
mm/hugetlb.c | 2 +-
3 files changed, 161 insertions(+), 2 deletions(-)
diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c
index f869993..002bafa 100644
--- a/fs/hugetlbfs/inode.c
+++ b/fs/hugetlbfs/inode.c
@@ -12,6 +12,7 @@
#include <linux/thread_info.h>
#include <asm/current.h>
#include <linux/sched.h> /* remove ASAP */
+#include <linux/falloc.h>
#include <linux/fs.h>
#include <linux/mount.h>
#include <linux/file.h>
@@ -479,6 +480,160 @@ static int hugetlb_vmtruncate(struct inode *inode, loff_t offset)
return 0;
}
+static long hugetlbfs_punch_hole(struct inode *inode, loff_t offset, loff_t len)
+{
+ struct hstate *h = hstate_inode(inode);
+ loff_t hpage_size = huge_page_size(h);
+ loff_t hole_start, hole_end;
+
+ /*
+ * For hole punch round up the beginning offset of the hole and
+ * round down the end.
+ */
+ hole_start = round_up(offset, hpage_size);
+ hole_end = round_down(offset + len, hpage_size);
+
+ if (hole_end > hole_start) {
+ struct address_space *mapping = inode->i_mapping;
+
+ mutex_lock(&inode->i_mutex);
+ i_mmap_lock_write(mapping);
+ if (!RB_EMPTY_ROOT(&mapping->i_mmap))
+ hugetlb_vmdelete_list(&mapping->i_mmap,
+ hole_start >> PAGE_SHIFT,
+ hole_end >> PAGE_SHIFT);
+ i_mmap_unlock_write(mapping);
+ remove_inode_hugepages(inode, hole_start, hole_end);
+ mutex_unlock(&inode->i_mutex);
+ }
+
+ return 0;
+}
+
+static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset,
+ loff_t len)
+{
+ struct inode *inode = file_inode(file);
+ struct address_space *mapping = inode->i_mapping;
+ struct hstate *h = hstate_inode(inode);
+ struct vm_area_struct pseudo_vma;
+ struct mm_struct *mm = current->mm;
+ loff_t hpage_size = huge_page_size(h);
+ unsigned long hpage_shift = huge_page_shift(h);
+ pgoff_t start, index, end;
+ int error;
+ u32 hash;
+
+ if (mode & ~(FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE))
+ return -EOPNOTSUPP;
+
+ if (mode & FALLOC_FL_PUNCH_HOLE)
+ return hugetlbfs_punch_hole(inode, offset, len);
+
+ /*
+ * Default preallocate case.
+ * For this range, start is rounded down and end is rounded up
+ * as well as being converted to page offsets.
+ */
+ start = offset >> hpage_shift;
+ end = (offset + len + hpage_size - 1) >> hpage_shift;
+
+ mutex_lock(&inode->i_mutex);
+
+ /* We need to check rlimit even when FALLOC_FL_KEEP_SIZE */
+ error = inode_newsize_ok(inode, offset + len);
+ if (error)
+ goto out;
+
+ /*
+ * Initialize a pseudo vma that just contains the policy used
+ * when allocating the huge pages. The actual policy field
+ * (vm_policy) is determined based on the index in the loop below.
+ */
+ memset(&pseudo_vma, 0, sizeof(struct vm_area_struct));
+ pseudo_vma.vm_flags = (VM_HUGETLB | VM_MAYSHARE | VM_SHARED);
+ pseudo_vma.vm_file = file;
+
+ for (index = start; index < end; index++) {
+ /*
+ * This is supposed to be the vaddr where the page is being
+ * faulted in, but we have no vaddr here.
+ */
+ struct page *page;
+ unsigned long addr;
+ int avoid_reserve = 0;
+
+ cond_resched();
+
+ /*
+ * fallocate(2) manpage permits EINTR; we may have been
+ * interrupted because we are using up too much memory.
+ */
+ if (signal_pending(current)) {
+ error = -EINTR;
+ break;
+ }
+
+ /* Get policy based on index */
+ pseudo_vma.vm_policy =
+ mpol_shared_policy_lookup(&HUGETLBFS_I(inode)->policy,
+ index);
+
+ /* addr is the offset within the file (zero based) */
+ addr = index * hpage_size;
+
+ /* mutex taken here, fault path and hole punch */
+ hash = hugetlb_fault_mutex_hash(h, mm, &pseudo_vma, mapping,
+ index, addr);
+ mutex_lock(&hugetlb_fault_mutex_table[hash]);
+
+ /* See if already present in mapping to avoid alloc/free */
+ page = find_get_page(mapping, index);
+ if (page) {
+ put_page(page);
+ mutex_unlock(&hugetlb_fault_mutex_table[hash]);
+ mpol_cond_put(pseudo_vma.vm_policy);
+ continue;
+ }
+
+ /* Allocate page and add to page cache */
+ page = alloc_huge_page(&pseudo_vma, addr, avoid_reserve);
+ mpol_cond_put(pseudo_vma.vm_policy);
+ if (IS_ERR(page)) {
+ mutex_unlock(&hugetlb_fault_mutex_table[hash]);
+ error = PTR_ERR(page);
+ goto out;
+ }
+ clear_huge_page(page, addr, pages_per_huge_page(h));
+ __SetPageUptodate(page);
+ error = huge_add_to_page_cache(page, mapping, index);
+ if (unlikely(error)) {
+ put_page(page);
+ mutex_unlock(&hugetlb_fault_mutex_table[hash]);
+ goto out;
+ }
+
+ mutex_unlock(&hugetlb_fault_mutex_table[hash]);
+
+ /*
+ * page_put due to reference from alloc_huge_page()
+ * unlock_page because locked by add_to_page_cache()
+ */
+ put_page(page);
+ unlock_page(page);
+ }
+
+ if (!(mode & FALLOC_FL_KEEP_SIZE) && offset + len > inode->i_size)
+ i_size_write(inode, offset + len);
+ inode->i_ctime = CURRENT_TIME;
+ spin_lock(&inode->i_lock);
+ inode->i_private = NULL;
+ spin_unlock(&inode->i_lock);
+out:
+ mutex_unlock(&inode->i_mutex);
+ return error;
+}
+
static int hugetlbfs_setattr(struct dentry *dentry, struct iattr *attr)
{
struct inode *inode = d_inode(dentry);
@@ -790,7 +945,8 @@ const struct file_operations hugetlbfs_file_operations = {
.mmap = hugetlbfs_file_mmap,
.fsync = noop_fsync,
.get_unmapped_area = hugetlb_get_unmapped_area,
- .llseek = default_llseek,
+ .llseek = default_llseek,
+ .fallocate = hugetlbfs_fallocate,
};
static const struct inode_operations hugetlbfs_dir_inode_operations = {
diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index 657ef26..386dcbf 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -330,6 +330,8 @@ struct huge_bootmem_page {
#endif
};
+struct page *alloc_huge_page(struct vm_area_struct *vma,
+ unsigned long addr, int avoid_reserve);
struct page *alloc_huge_page_node(struct hstate *h, int nid);
struct page *alloc_huge_page_noerr(struct vm_area_struct *vma,
unsigned long addr, int avoid_reserve);
@@ -483,6 +485,7 @@ static inline spinlock_t *huge_pte_lockptr(struct hstate *h,
#else /* CONFIG_HUGETLB_PAGE */
struct hstate {};
+#define alloc_huge_page(v, a, r) NULL
#define alloc_huge_page_node(h, nid) NULL
#define alloc_huge_page_noerr(v, a, r) NULL
#define alloc_bootmem_huge_page(h) NULL
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 5d48436..aded2c3 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -1725,7 +1725,7 @@ static void vma_abort_reservation(struct hstate *h,
(void)__vma_reservation_common(h, vma, addr, VMA_ABORT_RESV);
}
-static struct page *alloc_huge_page(struct vm_area_struct *vma,
+struct page *alloc_huge_page(struct vm_area_struct *vma,
unsigned long addr, int avoid_reserve)
{
struct hugepage_subpool *spool = subpool_vma(vma);
--
2.1.0
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [PATCH v2 10/10] mm: madvise allow remove operation for hugetlbfs
From: Mike Kravetz @ 2015-07-09 0:21 UTC (permalink / raw)
To: linux-mm, linux-kernel, linux-api
Cc: Dave Hansen, Naoya Horiguchi, David Rientjes, Hugh Dickins,
Davidlohr Bueso, Aneesh Kumar, Hillf Danton, Christoph Hellwig,
Andrew Morton, Michal Hocko, Mike Kravetz
In-Reply-To: <1436401301-18839-1-git-send-email-mike.kravetz@oracle.com>
Now that we have hole punching support for hugetlbfs, we can
also support the MADV_REMOVE interface to it.
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com>
---
mm/madvise.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/mm/madvise.c b/mm/madvise.c
index 70ce0d4..a235367 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -468,7 +468,7 @@ static long madvise_remove(struct vm_area_struct *vma,
*prev = NULL; /* tell sys_madvise we drop mmap_sem */
- if (vma->vm_flags & (VM_LOCKED | VM_HUGETLB))
+ if (vma->vm_flags & VM_LOCKED)
return -EINVAL;
f = vma->vm_file;
--
2.1.0
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* Re: [PATCH v2 01/10] mm/hugetlb: add cache of descriptors to resv_map for region_add
From: Mike Kravetz @ 2015-07-09 0:43 UTC (permalink / raw)
To: linux-mm, linux-kernel, linux-api
Cc: Dave Hansen, Naoya Horiguchi, David Rientjes, Hugh Dickins,
Davidlohr Bueso, Aneesh Kumar, Hillf Danton, Christoph Hellwig,
Andrew Morton, Michal Hocko
In-Reply-To: <1436401301-18839-2-git-send-email-mike.kravetz@oracle.com>
On 07/08/2015 05:21 PM, Mike Kravetz wrote:
> fallocate hole punch will want to remove a specific range of
> pages. When pages are removed, their associated entries in
> the region/reserve map will also be removed. This will break
> an assumption in the region_chg/region_add calling sequence.
> If a new region descriptor must be allocated, it is done as
> part of the region_chg processing. In this way, region_add
> can not fail because it does not need to attempt an allocation.
>
> To prepare for fallocate hole punch, create a "cache" of
> descriptors that can be used by region_add if necessary.
> region_chg will ensure there are sufficient entries in the
> cache. It will be necessary to track the number of in progress
> add operations to know a sufficient number of descriptors
> reside in the cache. A new routine region_abort is added to
> adjust this in progress count when add operations are aborted.
> vma_abort_reservation is also added for callers creating
> reservations with vma_needs_reservation/vma_commit_reservation.
>
> Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com>
> ---
> include/linux/hugetlb.h | 3 +
> mm/hugetlb.c | 168 ++++++++++++++++++++++++++++++++++++++++++------
> 2 files changed, 152 insertions(+), 19 deletions(-)
>
> diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
> index d891f94..667cf44 100644
> --- a/include/linux/hugetlb.h
> +++ b/include/linux/hugetlb.h
> @@ -35,6 +35,9 @@ struct resv_map {
> struct kref refs;
> spinlock_t lock;
> struct list_head regions;
> + long adds_in_progress;
> + struct list_head rgn_cache;
> + long rgn_cache_count;
> };
> extern struct resv_map *resv_map_alloc(void);
> void resv_map_release(struct kref *ref);
> diff --git a/mm/hugetlb.c b/mm/hugetlb.c
> index a8c3087..de03374 100644
> --- a/mm/hugetlb.c
> +++ b/mm/hugetlb.c
> @@ -240,11 +240,14 @@ struct file_region {
>
> /*
> * Add the huge page range represented by [f, t) to the reserve
> - * map. Existing regions will be expanded to accommodate the
> - * specified range. We know only existing regions need to be
> - * expanded, because region_add is only called after region_chg
> - * with the same range. If a new file_region structure must
> - * be allocated, it is done in region_chg.
> + * map. In the normal case, existing regions will be expanded
> + * to accommodate the specified range. Sufficient regions should
> + * exist for expansion due to the previous call to region_chg
> + * with the same range. However, it is possible that region_del
> + * could have been called after region_chg and modifed the map
> + * in such a way that no region exists to be expanded. In this
> + * case, pull a region descriptor from the cache associated with
> + * the map and use that for the new range.
> *
> * Return the number of new huge pages added to the map. This
> * number is greater than or equal to zero.
> @@ -261,6 +264,27 @@ static long region_add(struct resv_map *resv, long f, long t)
> if (f <= rg->to)
> break;
>
> + if (&rg->link == head || t < rg->from) {
> + /*
> + * No region exists which can be expanded to include the
> + * specified range. Pull a region descriptor from the
> + * cache, and use it for this range.
> + */
> + VM_BUG_ON(!resv->rgn_cache_count);
> +
> + resv->rgn_cache_count--;
> + nrg = list_first_entry(&resv->rgn_cache, struct file_region,
> + link);
> + list_del(&nrg->link);
> +
> + nrg->from = f;
> + nrg->to = t;
> + list_add(&nrg->link, rg->link.prev);
> +
> + add += t - f;
> + goto out_locked;
> + }
> +
> /* Round our left edge to the current segment if it encloses us. */
> if (f > rg->from)
> f = rg->from;
> @@ -294,6 +318,8 @@ static long region_add(struct resv_map *resv, long f, long t)
> add += t - nrg->to; /* Added to end of region */
> nrg->to = t;
>
> +out_locked:
> + resv->adds_in_progress--;
> spin_unlock(&resv->lock);
> VM_BUG_ON(add < 0);
> return add;
> @@ -312,11 +338,16 @@ static long region_add(struct resv_map *resv, long f, long t)
> * so that the subsequent region_add call will have all the
> * regions it needs and will not fail.
> *
> + * Upon entry, region_chg will also examine the cache of
> + * region descriptors associated with the map. If there
> + * not enough descriptors cached, one will be allocated
> + * for the in progress add operation.
> + *
> * Returns the number of huge pages that need to be added
> * to the existing reservation map for the range [f, t).
> * This number is greater or equal to zero. -ENOMEM is
> - * returned if a new file_region structure is needed and can
> - * not be allocated.
> + * returned if a new file_region structure or cache entry
> + * is needed and can not be allocated.
> */
> static long region_chg(struct resv_map *resv, long f, long t)
> {
> @@ -326,6 +357,30 @@ static long region_chg(struct resv_map *resv, long f, long t)
>
> retry:
> spin_lock(&resv->lock);
> + resv->adds_in_progress++;
> +
> + /*
> + * Check for sufficient descriptors in the cache to accommodate
> + * the number of in progress add operations.
> + */
> + if (resv->adds_in_progress > resv->rgn_cache_count) {
> + struct file_region *trg;
> +
> + VM_BUG_ON(resv->adds_in_progress - resv->rgn_cache_count > 1);
> + /* Must drop lock to allocate a new descriptor. */
> + resv->adds_in_progress--;
> + spin_unlock(&resv->lock);
> +
> + trg = kmalloc(sizeof(*trg), GFP_KERNEL);
> + if (!trg)
> + return -ENOMEM;
> +
> + spin_lock(&resv->lock);
> + resv->adds_in_progress++;
> + list_add(&trg->link, &resv->rgn_cache);
> + resv->rgn_cache_count++;
Doh! I noticed shortly after sending that this needs to go back and
check the condition (sufficient descriptors) after acquiring the lock.
Apologies for the obvious bug. More comments are welcome.
--
Mike Kravetz
> + }
> +
> /* Locate the region we are before or in. */
> list_for_each_entry(rg, head, link)
> if (f <= rg->to)
> @@ -336,6 +391,7 @@ retry:
> * size such that we can guarantee to record the reservation. */
> if (&rg->link == head || t < rg->from) {
> if (!nrg) {
> + resv->adds_in_progress--;
> spin_unlock(&resv->lock);
> nrg = kmalloc(sizeof(*nrg), GFP_KERNEL);
> if (!nrg)
> @@ -385,6 +441,25 @@ out_nrg:
> }
>
> /*
> + * Abort the in progress add operation. The adds_in_progress field
> + * of the resv_map keeps track of the operations in progress between
> + * calls to region_chg and region_add. Operations are sometimes
> + * aborted after the call to region_chg. In such cases, region_abort
> + * is called to decrement the adds_in_progress counter.
> + *
> + * NOTE: The range arguments [f, t) are not needed or used in this
> + * routine. They are kept to make reading the calling code easier as
> + * arguments will match the associated region_chg call.
> + */
> +static void region_abort(struct resv_map *resv, long f, long t)
> +{
> + spin_lock(&resv->lock);
> + VM_BUG_ON(!resv->rgn_cache_count);
> + resv->adds_in_progress--;
> + spin_unlock(&resv->lock);
> +}
> +
> +/*
> * Truncate the reserve map at index 'end'. Modify/truncate any
> * region which contains end. Delete any regions past end.
> * Return the number of huge pages removed from the map.
> @@ -544,22 +619,44 @@ static void set_vma_private_data(struct vm_area_struct *vma,
> struct resv_map *resv_map_alloc(void)
> {
> struct resv_map *resv_map = kmalloc(sizeof(*resv_map), GFP_KERNEL);
> - if (!resv_map)
> + struct file_region *rg = kmalloc(sizeof(*rg), GFP_KERNEL);
> +
> + if (!resv_map || !rg) {
> + kfree(resv_map);
> + kfree(rg);
> return NULL;
> + }
>
> kref_init(&resv_map->refs);
> spin_lock_init(&resv_map->lock);
> INIT_LIST_HEAD(&resv_map->regions);
>
> + resv_map->adds_in_progress = 0;
> +
> + INIT_LIST_HEAD(&resv_map->rgn_cache);
> + list_add(&rg->link, &resv_map->rgn_cache);
> + resv_map->rgn_cache_count = 1;
> +
> return resv_map;
> }
>
> void resv_map_release(struct kref *ref)
> {
> struct resv_map *resv_map = container_of(ref, struct resv_map, refs);
> + struct list_head *head = &resv_map->rgn_cache;
> + struct file_region *rg, *trg;
>
> /* Clear out any active regions before we release the map. */
> region_truncate(resv_map, 0);
> +
> + /* ... and any entries left in the cache */
> + list_for_each_entry_safe(rg, trg, head, link) {
> + list_del(&rg->link);
> + kfree(rg);
> + }
> +
> + VM_BUG_ON(resv_map->adds_in_progress);
> +
> kfree(resv_map);
> }
>
> @@ -1473,16 +1570,18 @@ static void return_unused_surplus_pages(struct hstate *h,
> }
> }
>
> +
> /*
> - * vma_needs_reservation and vma_commit_reservation are used by the huge
> - * page allocation routines to manage reservations.
> + * vma_needs_reservation, vma_commit_reservation and vma_abort_reservation
> + * are used by the huge page allocation routines to manage reservations.
> *
> * vma_needs_reservation is called to determine if the huge page at addr
> * within the vma has an associated reservation. If a reservation is
> * needed, the value 1 is returned. The caller is then responsible for
> * managing the global reservation and subpool usage counts. After
> * the huge page has been allocated, vma_commit_reservation is called
> - * to add the page to the reservation map.
> + * to add the page to the reservation map. If the reservation must be
> + * aborted instead of committed, vma_abort_reservation is called.
> *
> * In the normal case, vma_commit_reservation returns the same value
> * as the preceding vma_needs_reservation call. The only time this
> @@ -1490,9 +1589,14 @@ static void return_unused_surplus_pages(struct hstate *h,
> * is the responsibility of the caller to notice the difference and
> * take appropriate action.
> */
> +enum vma_resv_mode {
> + VMA_NEEDS_RESV,
> + VMA_COMMIT_RESV,
> + VMA_ABORT_RESV,
> +};
> static long __vma_reservation_common(struct hstate *h,
> struct vm_area_struct *vma, unsigned long addr,
> - bool commit)
> + enum vma_resv_mode mode)
> {
> struct resv_map *resv;
> pgoff_t idx;
> @@ -1503,10 +1607,20 @@ static long __vma_reservation_common(struct hstate *h,
> return 1;
>
> idx = vma_hugecache_offset(h, vma, addr);
> - if (commit)
> - ret = region_add(resv, idx, idx + 1);
> - else
> + switch (mode) {
> + case VMA_NEEDS_RESV:
> ret = region_chg(resv, idx, idx + 1);
> + break;
> + case VMA_COMMIT_RESV:
> + ret = region_add(resv, idx, idx + 1);
> + break;
> + case VMA_ABORT_RESV:
> + region_abort(resv, idx, idx + 1);
> + ret = 0;
> + break;
> + default:
> + BUG();
> + }
>
> if (vma->vm_flags & VM_MAYSHARE)
> return ret;
> @@ -1517,13 +1631,19 @@ static long __vma_reservation_common(struct hstate *h,
> static long vma_needs_reservation(struct hstate *h,
> struct vm_area_struct *vma, unsigned long addr)
> {
> - return __vma_reservation_common(h, vma, addr, false);
> + return __vma_reservation_common(h, vma, addr, VMA_NEEDS_RESV);
> }
>
> static long vma_commit_reservation(struct hstate *h,
> struct vm_area_struct *vma, unsigned long addr)
> {
> - return __vma_reservation_common(h, vma, addr, true);
> + return __vma_reservation_common(h, vma, addr, VMA_COMMIT_RESV);
> +}
> +
> +static void vma_abort_reservation(struct hstate *h,
> + struct vm_area_struct *vma, unsigned long addr)
> +{
> + (void)__vma_reservation_common(h, vma, addr, VMA_ABORT_RESV);
> }
>
> static struct page *alloc_huge_page(struct vm_area_struct *vma,
> @@ -1549,8 +1669,10 @@ static struct page *alloc_huge_page(struct vm_area_struct *vma,
> if (chg < 0)
> return ERR_PTR(-ENOMEM);
> if (chg || avoid_reserve)
> - if (hugepage_subpool_get_pages(spool, 1) < 0)
> + if (hugepage_subpool_get_pages(spool, 1) < 0) {
> + vma_abort_reservation(h, vma, addr);
> return ERR_PTR(-ENOSPC);
> + }
>
> ret = hugetlb_cgroup_charge_cgroup(idx, pages_per_huge_page(h), &h_cg);
> if (ret)
> @@ -1596,6 +1718,7 @@ out_uncharge_cgroup:
> out_subpool_put:
> if (chg || avoid_reserve)
> hugepage_subpool_put_pages(spool, 1);
> + vma_abort_reservation(h, vma, addr);
> return ERR_PTR(-ENOSPC);
> }
>
> @@ -3236,11 +3359,14 @@ retry:
> * any allocations necessary to record that reservation occur outside
> * the spinlock.
> */
> - if ((flags & FAULT_FLAG_WRITE) && !(vma->vm_flags & VM_SHARED))
> + if ((flags & FAULT_FLAG_WRITE) && !(vma->vm_flags & VM_SHARED)) {
> if (vma_needs_reservation(h, vma, address) < 0) {
> ret = VM_FAULT_OOM;
> goto backout_unlocked;
> }
> + /* Just decrements count, does not deallocate */
> + vma_abort_reservation(h, vma, address);
> + }
>
> ptl = huge_pte_lockptr(h, mm, ptep);
> spin_lock(ptl);
> @@ -3387,6 +3513,8 @@ int hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,
> ret = VM_FAULT_OOM;
> goto out_mutex;
> }
> + /* Just decrements count, does not deallocate */
> + vma_abort_reservation(h, vma, address);
>
> if (!(vma->vm_flags & VM_MAYSHARE))
> pagecache_page = hugetlbfs_pagecache_page(h,
> @@ -3726,6 +3854,8 @@ int hugetlb_reserve_pages(struct inode *inode,
> }
> return 0;
> out_err:
> + if (!vma || vma->vm_flags & VM_MAYSHARE)
> + region_abort(resv_map, from, to);
> if (vma && is_vma_resv_set(vma, HPAGE_RESV_OWNER))
> kref_put(&resv_map->refs, resv_map_release);
> return ret;
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [RFCv3 0/5] enable migration of driver pages
From: Rafael Aquini @ 2015-07-09 1:00 UTC (permalink / raw)
To: Gioh Kim
Cc: jlayton-vpEMnDpepFuMZCB2o+C8xQ, bfields-uC3wQj2KruNg9hUCZPvPmw,
vbabka-AlSwsSmVLrQ, iamjoonsoo.kim-Hm3cg6mZ9cc,
viro-RmSDqhL/yNMiFSDQTTA3OLVCufUGDwFn, mst-H+wXaHxf7aLQT0dZR+AlfA,
koct9i-Re5JQEeQqe8AvxtiuMwx3w, minchan-DgEjT+Ai2ygdnm+yROfE0A,
linux-fsdevel-u79uwXL29TY76Z2rM5mHXA,
virtualization-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
linux-api-u79uwXL29TY76Z2rM5mHXA, linux-mm-Bw31MaZKKs3YtjvyW6yDsg,
gunho.lee-Hm3cg6mZ9cc, akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b,
Gioh Kim
In-Reply-To: <1436243785-24105-1-git-send-email-gioh.kim-Hm3cg6mZ9cc@public.gmane.org>
On Tue, Jul 07, 2015 at 01:36:20PM +0900, Gioh Kim wrote:
> From: Gioh Kim <gurugio-A7HHaXTFBl7R7s880joybQ@public.gmane.org>
>
> Hello,
>
> This series try to enable migration of non-LRU pages, such as driver's page.
>
> My ARM-based platform occured severe fragmentation problem after long-term
> (several days) test. Sometimes even order-3 page allocation failed. It has
> memory size 512MB ~ 1024MB. 30% ~ 40% memory is consumed for graphic processing
> and 20~30 memory is reserved for zram.
>
> I found that many pages of GPU driver and zram are non-movable pages. So I
> reported Minchan Kim, the maintainer of zram, and he made the internal
> compaction logic of zram. And I made the internal compaction of GPU driver.
>
> They reduced some fragmentation but they are not enough effective.
> They are activated by its own interface, /sys, so they are not cooperative
> with kernel compaction. If there is too much fragmentation and kernel starts
> to compaction, zram and GPU driver cannot work with the kernel compaction.
>
> This patch set combines 5 patches.
>
> 1. patch 1/5: get inode from anon_inodes
> This patch adds new interface to create inode from anon_inodes.
>
> 2. patch 2/5: framework to isolate/migrate/putback page
> Add isolatepage, putbackpage into address_space_operations
> and wrapper function to call them
>
> 3. patch 3/5: apply the framework into balloon driver
> The balloon driver is applied into the framework. It gets a inode
> from anon_inodes and register operations in the inode.
> Any other drivers can register operations via inode like this
> to migrate it's pages.
>
> 4. patch 4/5: compaction/migration call the generic interfaces
> Compaction and migration pages call the generic interfaces of the framework,
> instead of calling balloon migration directly.
>
> 5. patch 5/5: remove direct calling of migration of driver pages
> Non-lru pages are migrated with lru pages by move_to_new_page().
>
> This patch set is tested:
> - turn on Ubuntu 14.04 with 1G memory on qemu.
> - do kernel building
> - after several seconds check more than 512MB is used with free command
> - command "balloon 512" in qemu monitor
> - check hundreds MB of pages are migrated
>
> My thanks to Konstantin Khlebnikov for his reviews of the v2 patch set.
> Most of the changes were based on his feedback.
>
> Changes since v2:
> - change the name of page type from migratable page into mobile page
> - get and lock page to isolate page
> - add wrapper interfaces for page->mapping->a_ops->isolate/putback
> - leave balloon pages marked as balloon
>
> This patch-set is based on v4.1
>
> Gioh Kim (5):
> fs/anon_inodes: new interface to create new inode
> mm/compaction: enable mobile-page migration
> mm/balloon: apply mobile page migratable into balloon
> mm/compaction: call generic migration callbacks
> mm: remove direct calling of migration
>
> drivers/virtio/virtio_balloon.c | 3 ++
> fs/anon_inodes.c | 6 +++
> fs/proc/page.c | 3 ++
> include/linux/anon_inodes.h | 1 +
> include/linux/balloon_compaction.h | 15 +++++--
> include/linux/compaction.h | 76 ++++++++++++++++++++++++++++++++++
> include/linux/fs.h | 2 +
> include/linux/page-flags.h | 19 +++++++++
> include/uapi/linux/kernel-page-flags.h | 1 +
> mm/balloon_compaction.c | 71 ++++++++++---------------------
> mm/compaction.c | 8 ++--
> mm/migrate.c | 24 +++--------
> 12 files changed, 154 insertions(+), 75 deletions(-)
>
> --
> 2.1.4
>
Acked-by: Rafael Aquini <aquini-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
^ permalink raw reply
* Re: [RFCv3 3/5] mm/balloon: apply mobile page migratable into balloon
From: Michael S. Tsirkin @ 2015-07-09 8:07 UTC (permalink / raw)
To: Gioh Kim
Cc: jlayton-vpEMnDpepFuMZCB2o+C8xQ, bfields-uC3wQj2KruNg9hUCZPvPmw,
vbabka-AlSwsSmVLrQ, iamjoonsoo.kim-Hm3cg6mZ9cc,
viro-RmSDqhL/yNMiFSDQTTA3OLVCufUGDwFn,
koct9i-Re5JQEeQqe8AvxtiuMwx3w, minchan-DgEjT+Ai2ygdnm+yROfE0A,
aquini-H+wXaHxf7aLQT0dZR+AlfA,
linux-fsdevel-u79uwXL29TY76Z2rM5mHXA,
virtualization-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
linux-api-u79uwXL29TY76Z2rM5mHXA, linux-mm-Bw31MaZKKs3YtjvyW6yDsg,
gunho.lee-Hm3cg6mZ9cc, akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b,
Gioh Kim
In-Reply-To: <1436243785-24105-4-git-send-email-gioh.kim-Hm3cg6mZ9cc@public.gmane.org>
On Tue, Jul 07, 2015 at 01:36:23PM +0900, Gioh Kim wrote:
> From: Gioh Kim <gurugio-A7HHaXTFBl7R7s880joybQ@public.gmane.org>
>
> Apply mobile page migration into balloon driver.
> The balloong driver has an anonymous inode that manages
> address_space_operation for page migration.
>
> Signed-off-by: Gioh Kim <gioh.kim-Hm3cg6mZ9cc@public.gmane.org>
> ---
> drivers/virtio/virtio_balloon.c | 3 ++
> include/linux/balloon_compaction.h | 15 +++++++--
> mm/balloon_compaction.c | 65 +++++++++++++-------------------------
> mm/compaction.c | 2 +-
> mm/migrate.c | 2 +-
> 5 files changed, 39 insertions(+), 48 deletions(-)
>
> diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c
> index 82e80e0..ef5b9b5 100644
> --- a/drivers/virtio/virtio_balloon.c
> +++ b/drivers/virtio/virtio_balloon.c
> @@ -30,6 +30,7 @@
> #include <linux/balloon_compaction.h>
> #include <linux/oom.h>
> #include <linux/wait.h>
> +#include <linux/anon_inodes.h>
>
> /*
> * Balloon device works in 4K page units. So each page is pointed to by
> @@ -505,6 +506,8 @@ static int virtballoon_probe(struct virtio_device *vdev)
> balloon_devinfo_init(&vb->vb_dev_info);
> #ifdef CONFIG_BALLOON_COMPACTION
> vb->vb_dev_info.migratepage = virtballoon_migratepage;
> + vb->vb_dev_info.inode = anon_inode_new();
> + vb->vb_dev_info.inode->i_mapping->a_ops = &balloon_aops;
> #endif
>
> err = init_vqs(vb);
> diff --git a/include/linux/balloon_compaction.h b/include/linux/balloon_compaction.h
> index 9b0a15d..a9e0bde 100644
> --- a/include/linux/balloon_compaction.h
> +++ b/include/linux/balloon_compaction.h
> @@ -48,6 +48,7 @@
> #include <linux/migrate.h>
> #include <linux/gfp.h>
> #include <linux/err.h>
> +#include <linux/fs.h>
>
> /*
> * Balloon device information descriptor.
> @@ -62,6 +63,7 @@ struct balloon_dev_info {
> struct list_head pages; /* Pages enqueued & handled to Host */
> int (*migratepage)(struct balloon_dev_info *, struct page *newpage,
> struct page *page, enum migrate_mode mode);
> + struct inode *inode;
> };
>
> extern struct page *balloon_page_enqueue(struct balloon_dev_info *b_dev_info);
> @@ -73,12 +75,16 @@ static inline void balloon_devinfo_init(struct balloon_dev_info *balloon)
> spin_lock_init(&balloon->pages_lock);
> INIT_LIST_HEAD(&balloon->pages);
> balloon->migratepage = NULL;
> + balloon->inode = NULL;
> }
>
> #ifdef CONFIG_BALLOON_COMPACTION
> -extern bool balloon_page_isolate(struct page *page);
> +extern const struct address_space_operations balloon_aops;
> +extern bool balloon_page_isolate(struct page *page,
> + isolate_mode_t mode);
> extern void balloon_page_putback(struct page *page);
> -extern int balloon_page_migrate(struct page *newpage,
> +extern int balloon_page_migrate(struct address_space *mapping,
> + struct page *newpage,
> struct page *page, enum migrate_mode mode);
>
> /*
> @@ -124,6 +130,7 @@ static inline void balloon_page_insert(struct balloon_dev_info *balloon,
> struct page *page)
> {
> __SetPageBalloon(page);
> + page->mapping = balloon->inode->i_mapping;
> SetPagePrivate(page);
> set_page_private(page, (unsigned long)balloon);
> list_add(&page->lru, &balloon->pages);
> @@ -140,6 +147,7 @@ static inline void balloon_page_insert(struct balloon_dev_info *balloon,
> static inline void balloon_page_delete(struct page *page)
> {
> __ClearPageBalloon(page);
> + page->mapping = NULL;
> set_page_private(page, 0);
> if (PagePrivate(page)) {
> ClearPagePrivate(page);
Order of cleanup here is not the reverse of the order of initialization.
Better make it exactly the reverse.
Also, I have a question: is it enough to lock the page to make changing
the mapping safe? Do all users lock the page too?
> @@ -191,7 +199,8 @@ static inline bool isolated_balloon_page(struct page *page)
> return false;
> }
>
> -static inline bool balloon_page_isolate(struct page *page)
> +static inline bool balloon_page_isolate(struct page *page,
> + isolate_mode_t mode)
> {
> return false;
> }
> diff --git a/mm/balloon_compaction.c b/mm/balloon_compaction.c
> index fcad832..0dd0b0d 100644
> --- a/mm/balloon_compaction.c
> +++ b/mm/balloon_compaction.c
> @@ -131,43 +131,16 @@ static inline void __putback_balloon_page(struct page *page)
> }
>
> /* __isolate_lru_page() counterpart for a ballooned page */
> -bool balloon_page_isolate(struct page *page)
> +bool balloon_page_isolate(struct page *page, isolate_mode_t mode)
> {
> /*
> - * Avoid burning cycles with pages that are yet under __free_pages(),
> - * or just got freed under us.
> - *
> - * In case we 'win' a race for a balloon page being freed under us and
> - * raise its refcount preventing __free_pages() from doing its job
> - * the put_page() at the end of this block will take care of
> - * release this page, thus avoiding a nasty leakage.
> + * A ballooned page, by default, has PagePrivate set.
> + * Prevent concurrent compaction threads from isolating
> + * an already isolated balloon page by clearing it.
> */
> - if (likely(get_page_unless_zero(page))) {
> - /*
> - * As balloon pages are not isolated from LRU lists, concurrent
> - * compaction threads can race against page migration functions
> - * as well as race against the balloon driver releasing a page.
> - *
> - * In order to avoid having an already isolated balloon page
> - * being (wrongly) re-isolated while it is under migration,
> - * or to avoid attempting to isolate pages being released by
> - * the balloon driver, lets be sure we have the page lock
> - * before proceeding with the balloon page isolation steps.
> - */
> - if (likely(trylock_page(page))) {
> - /*
> - * A ballooned page, by default, has PagePrivate set.
> - * Prevent concurrent compaction threads from isolating
> - * an already isolated balloon page by clearing it.
> - */
> - if (balloon_page_movable(page)) {
> - __isolate_balloon_page(page);
> - unlock_page(page);
> - return true;
> - }
> - unlock_page(page);
> - }
> - put_page(page);
> + if (balloon_page_movable(page)) {
> + __isolate_balloon_page(page);
> + return true;
> }
> return false;
> }
> @@ -175,30 +148,28 @@ bool balloon_page_isolate(struct page *page)
> /* putback_lru_page() counterpart for a ballooned page */
> void balloon_page_putback(struct page *page)
> {
> - /*
> - * 'lock_page()' stabilizes the page and prevents races against
> - * concurrent isolation threads attempting to re-isolate it.
> - */
> - lock_page(page);
> + if (!isolated_balloon_page(page))
> + return;
>
> if (__is_movable_balloon_page(page)) {
> __putback_balloon_page(page);
> - /* drop the extra ref count taken for page isolation */
> - put_page(page);
> } else {
> WARN_ON(1);
> dump_page(page, "not movable balloon page");
> }
> - unlock_page(page);
> }
>
> /* move_to_new_page() counterpart for a ballooned page */
> -int balloon_page_migrate(struct page *newpage,
> +int balloon_page_migrate(struct address_space *mapping,
> + struct page *newpage,
> struct page *page, enum migrate_mode mode)
> {
> struct balloon_dev_info *balloon = balloon_page_device(page);
> int rc = -EAGAIN;
>
> + if (!isolated_balloon_page(page))
> + return rc;
> +
> /*
> * Block others from accessing the 'newpage' when we get around to
> * establishing additional references. We should be the only one
> @@ -218,4 +189,12 @@ int balloon_page_migrate(struct page *newpage,
> unlock_page(newpage);
> return rc;
> }
> +
> +/* define the balloon_mapping->a_ops callback to allow balloon page migration */
> +const struct address_space_operations balloon_aops = {
> + .migratepage = balloon_page_migrate,
> + .isolatepage = balloon_page_isolate,
> + .putbackpage = balloon_page_putback,
> +};
> +EXPORT_SYMBOL_GPL(balloon_aops);
> #endif /* CONFIG_BALLOON_COMPACTION */
> diff --git a/mm/compaction.c b/mm/compaction.c
> index 018f08d..81bafaf 100644
> --- a/mm/compaction.c
> +++ b/mm/compaction.c
> @@ -719,7 +719,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
> */
> if (!PageLRU(page)) {
> if (unlikely(balloon_page_movable(page))) {
> - if (balloon_page_isolate(page)) {
> + if (balloon_page_isolate(page, isolate_mode)) {
> /* Successfully isolated */
> goto isolate_success;
> }
> diff --git a/mm/migrate.c b/mm/migrate.c
> index f53838f..c94038e 100644
> --- a/mm/migrate.c
> +++ b/mm/migrate.c
> @@ -852,7 +852,7 @@ static int __unmap_and_move(struct page *page, struct page *newpage,
> * in order to avoid burning cycles at rmap level, and perform
> * the page migration right away (proteced by page lock).
> */
> - rc = balloon_page_migrate(newpage, page, mode);
> + rc = balloon_page_migrate(page->mapping, newpage, page, mode);
> goto out_unlock;
> }
>
> --
> 2.1.4
^ permalink raw reply
* Re: [PATCH] fs, proc: add help for CONFIG_PROC_CHILDREN
From: Jean Delvare @ 2015-07-09 9:04 UTC (permalink / raw)
To: Iago López Galeiras
Cc: linux-fsdevel-u79uwXL29TY76Z2rM5mHXA,
linux-api-u79uwXL29TY76Z2rM5mHXA, Cyrill Gorcunov, Andrew Morton,
Djalal Harouni, Alban Crequy
In-Reply-To: <1436387284-3998-1-git-send-email-iago-973cpzSjLbNWk0Htik3J/w@public.gmane.org>
Le Wednesday 08 July 2015 à 22:28 +0200, Iago López Galeiras a écrit :
> The purpose of the option was documented in
> Documentation/filesystems/proc.txt but the help text was missing.
>
> Add small help text that also points to the documentation.
>
> Signed-off-by: Iago López Galeiras <iago-973cpzSjLbNWk0Htik3J/w@public.gmane.org>
> ---
> fs/proc/Kconfig | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> diff --git a/fs/proc/Kconfig b/fs/proc/Kconfig
> index d751fcb..1ade120 100644
> --- a/fs/proc/Kconfig
> +++ b/fs/proc/Kconfig
> @@ -75,3 +75,9 @@ config PROC_PAGE_MONITOR
> config PROC_CHILDREN
> bool "Include /proc/<pid>/task/<tid>/children file"
> default n
> + help
> + Provides a fast way to retrieve first level children pids of a task. See
> + <file:Documentation/filesystems/proc.txt> for more information.
> +
> + Say Y if you are running any user-space software which takes benefit from
> + this interface. For example, rkt is such a piece of software.
Thanks for doing that.
Reviewed-by: Jean Delvare <jdelvare-l3A5Bk7waGM@public.gmane.org>
--
Jean Delvare
SUSE L3 Support
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox