* Re: [PATCH] pci_regs: reintroduce PCI_MSIX_FLAGS_BIRMASK
From: Michael S. Tsirkin @ 2015-07-13 16:32 UTC (permalink / raw)
To: Bjorn Helgaas
Cc: linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Rajat Jain,
Chen, Gong, Guenter Roeck, Rafał Miłecki,
linux-api-u79uwXL29TY76Z2rM5mHXA,
linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
In-Reply-To: <CAErSpo4cbj58VheRbQOJ3XnyBn_SvmUi9ZyFhRnPWhf8ciJ7jg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
On Mon, Jul 13, 2015 at 10:49:02AM -0500, Bjorn Helgaas wrote:
> On Mon, Jul 13, 2015 at 10:11 AM, Michael S. Tsirkin <mst-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> > On Wed, Jul 01, 2015 at 12:14:10PM +0200, Michael S. Tsirkin wrote:
> >> This partially reverts commit 09a2c73ddfc7f173237fc7209a65b34dd5bcb5ed.
> >> PCI: Remove unused PCI_MSIX_FLAGS_BIRMASK definition
> >>
> >> That commit dropped a symbol from an exported header claiming "no one
> >> uses it". This isn't how Linux normally approaches userspace API though,
> >> and in fact QEMU build fails if trying to use updated headers from linux
> >> 3.12 and up.
> >>
> >> Sure, userspace can be fixed to use the new symbol, but the cost
> >> of keeping the old one around is fairly low, too.
> >>
> >> Signed-off-by: Michael S. Tsirkin <mst-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> >
> > Ping. Can this be included upstream please?
>
> Yes, I plan to merge it, and I'll mark it for stable. This looks like
> a complete revert of 09a2c73ddfc7, not a partial revert. Am I missing
> something?
Only that I redefine PCI_MSIX_FLAGS_BIRMASK in terms of PCI_MSIX_PBA_BIR
so that there's no duplication.
> 09a2c73ddfc7 appeared in v3.13 (not v3.12), so it wouldn't ordinarily
> be post-merge window material. But it seems pretty safe, so I might
> put it in for v4.2 instead of waiting for v4.3.
>
> >> ---
> >> include/uapi/linux/pci_regs.h | 1 +
> >> 1 file changed, 1 insertion(+)
> >>
> >> diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h
> >> index efe3443..66644ac 100644
> >> --- a/include/uapi/linux/pci_regs.h
> >> +++ b/include/uapi/linux/pci_regs.h
> >> @@ -319,6 +319,7 @@
> >> #define PCI_MSIX_PBA 8 /* Pending Bit Array offset */
> >> #define PCI_MSIX_PBA_BIR 0x00000007 /* BAR index */
> >> #define PCI_MSIX_PBA_OFFSET 0xfffffff8 /* Offset into specified BAR */
> >> +#define PCI_MSIX_FLAGS_BIRMASK PCI_MSIX_PBA_BIR /* deprecated */
> >> #define PCI_CAP_MSIX_SIZEOF 12 /* size of MSIX registers */
> >>
> >> /* MSI-X Table entry format */
> >> --
> >> MST
^ permalink raw reply
* Re: [RFC PATCH] getcpu_cache system call: caching current CPU number (x86)
From: Mathieu Desnoyers @ 2015-07-13 16:07 UTC (permalink / raw)
To: Andrew Hunter
Cc: Andy Lutomirski, Paul E. McKenney, Ben Maurer, Ingo Molnar,
Andrew Morton, Josh Triplett, Lai Jiangshan, Paul Turner, rostedt,
linux-api, Linus Torvalds, Peter Zijlstra
In-Reply-To: <CADroS=7MnUULrjDeQtmscxjkpjCtti9V-HfFXU0sjKhi6PsaAg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
----- On Jul 13, 2015, at 11:30 AM, Andrew Hunter ahh-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org wrote:
> On Mon, Jul 13, 2015 at 8:27 AM, Mathieu Desnoyers
> <mathieu.desnoyers-vg+e7yoeK/dWk0Htik3J/w@public.gmane.org> wrote:
>> percpu segments will likely not solve everything. I have a use-case
>> with dynamically allocated per-cpu ring buffer in user-space (lttng-ust)
>> which can be a challenge for percpu segments. Having a fast getcpu()
>> is a win in those cases.
>>
>
> Note that percpu segments allow userspace to trivially implement fast getcpu.
>
> For the record, Paul and I currently think the best solution is percpu
> segments + some variant of a restart-sequence API (we'll have a patch
> soon.)
Although useful in many situations, percpu segments still have some
limitations AFAIU:
- They are not available on all architectures (very x86-specific),
- Some user-space applications already use those segments.
So as long as we only target user-space code that does not use GS,
and which runs on x86, the percpu segments seems to be a good idea.
However, implementing a more general approach for a fast getcpu cache
still appears somewhat useful for the general case.
Thanks,
Mathieu
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
^ permalink raw reply
* Re: [PATCH] pci_regs: reintroduce PCI_MSIX_FLAGS_BIRMASK
From: Bjorn Helgaas @ 2015-07-13 15:49 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: linux-kernel@vger.kernel.org, Rajat Jain, Chen, Gong,
Guenter Roeck, Rafał Miłecki, linux-api,
linux-pci@vger.kernel.org
In-Reply-To: <20150713181025-mutt-send-email-mst@redhat.com>
On Mon, Jul 13, 2015 at 10:11 AM, Michael S. Tsirkin <mst@redhat.com> wrote:
> On Wed, Jul 01, 2015 at 12:14:10PM +0200, Michael S. Tsirkin wrote:
>> This partially reverts commit 09a2c73ddfc7f173237fc7209a65b34dd5bcb5ed.
>> PCI: Remove unused PCI_MSIX_FLAGS_BIRMASK definition
>>
>> That commit dropped a symbol from an exported header claiming "no one
>> uses it". This isn't how Linux normally approaches userspace API though,
>> and in fact QEMU build fails if trying to use updated headers from linux
>> 3.12 and up.
>>
>> Sure, userspace can be fixed to use the new symbol, but the cost
>> of keeping the old one around is fairly low, too.
>>
>> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>
> Ping. Can this be included upstream please?
Yes, I plan to merge it, and I'll mark it for stable. This looks like
a complete revert of 09a2c73ddfc7, not a partial revert. Am I missing
something?
09a2c73ddfc7 appeared in v3.13 (not v3.12), so it wouldn't ordinarily
be post-merge window material. But it seems pretty safe, so I might
put it in for v4.2 instead of waiting for v4.3.
>> ---
>> include/uapi/linux/pci_regs.h | 1 +
>> 1 file changed, 1 insertion(+)
>>
>> diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h
>> index efe3443..66644ac 100644
>> --- a/include/uapi/linux/pci_regs.h
>> +++ b/include/uapi/linux/pci_regs.h
>> @@ -319,6 +319,7 @@
>> #define PCI_MSIX_PBA 8 /* Pending Bit Array offset */
>> #define PCI_MSIX_PBA_BIR 0x00000007 /* BAR index */
>> #define PCI_MSIX_PBA_OFFSET 0xfffffff8 /* Offset into specified BAR */
>> +#define PCI_MSIX_FLAGS_BIRMASK PCI_MSIX_PBA_BIR /* deprecated */
>> #define PCI_CAP_MSIX_SIZEOF 12 /* size of MSIX registers */
>>
>> /* MSI-X Table entry format */
>> --
>> MST
^ permalink raw reply
* Re: [RFC PATCH] getcpu_cache system call: caching current CPU number (x86)
From: Andrew Hunter @ 2015-07-13 15:30 UTC (permalink / raw)
To: Mathieu Desnoyers
Cc: Andy Lutomirski, Paul E. McKenney, Ben Maurer, Ingo Molnar,
Andrew Morton, Josh Triplett, Lai Jiangshan, Paul Turner, rostedt,
linux-api, Linus Torvalds, Peter Zijlstra
In-Reply-To: <1050138282.1065.1436801252018.JavaMail.zimbra-vg+e7yoeK/dWk0Htik3J/w@public.gmane.org>
On Mon, Jul 13, 2015 at 8:27 AM, Mathieu Desnoyers
<mathieu.desnoyers-vg+e7yoeK/dWk0Htik3J/w@public.gmane.org> wrote:
> percpu segments will likely not solve everything. I have a use-case
> with dynamically allocated per-cpu ring buffer in user-space (lttng-ust)
> which can be a challenge for percpu segments. Having a fast getcpu()
> is a win in those cases.
>
Note that percpu segments allow userspace to trivially implement fast getcpu.
For the record, Paul and I currently think the best solution is percpu
segments + some variant of a restart-sequence API (we'll have a patch
soon.)
^ permalink raw reply
* Re: [RFC PATCH] getcpu_cache system call: caching current CPU number (x86)
From: Mathieu Desnoyers @ 2015-07-13 15:27 UTC (permalink / raw)
To: Andy Lutomirski
Cc: Paul E. McKenney, Ben Maurer, Ingo Molnar, Andrew Morton,
Josh Triplett, Lai Jiangshan, Paul Turner, rostedt, Andrew Hunter,
linux-api, Linus Torvalds, Peter Zijlstra
In-Reply-To: <CALCETrV1suAbvMgD1jOEFyn3JcDE_hhi6X7+sGs9e3Oqw_6jUw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
----- On Jul 12, 2015, at 11:38 PM, Andy Lutomirski luto-kltTT9wpgjJwATOyAt5JVQ@public.gmane.org wrote:
> On Jul 12, 2015 12:06 PM, "Mathieu Desnoyers"
> <mathieu.desnoyers-vg+e7yoeK/dWk0Htik3J/w@public.gmane.org> wrote:
>>
>> Expose a new system call allowing threads to register a userspace memory
>> area where to store the current CPU number. Scheduler migration sets the
>> TIF_NOTIFY_RESUME flag on the current thread. Upon return to user-space,
>> a notify-resume handler updates the current CPU value within that
>> user-space memory area.
>>
>> This getcpu cache is an alternative to the sched_getcpu() vdso which has
>> a few benefits:
>> - It is faster to do a memory read that to call a vDSO,
>> - This cache value can be read from within an inline assembly, which
>> makes it a useful building block for restartable sequences.
>>
>
> Let's wait and see what the final percpu atomic solution is. If it
> involves percpu segments, then this is unnecessary.
percpu segments will likely not solve everything. I have a use-case
with dynamically allocated per-cpu ring buffer in user-space (lttng-ust)
which can be a challenge for percpu segments. Having a fast getcpu()
is a win in those cases.
>
> Also, this will need to be rebased onto -tip, and that should wait
> until the big exit rewrite is farther along.
I don't really care which thread flag it ends up using, and this is
more or less an internal implementation detail. The important part is
the ABI exposed to user-space, and it's good to start the discussion
on this aspect early.
>
>> This approach is inspired by Paul Turner and Andrew Hunter's work
>> on percpu atomics, which lets the kernel handle restart of critical
>> sections:
>> Ref.:
>> * https://lkml.org/lkml/2015/6/24/665
>> * https://lwn.net/Articles/650333/
>> *
>> http://www.linuxplumbersconf.org/2013/ocw/system/presentations/1695/original/LPC%20-%20PerCpu%20Atomics.pdf
>>
>> Benchmarking sched_getcpu() vs tls cache approach. Getting the
>> current CPU number:
>>
>> - With Linux vdso: 12.7 ns
>
> This is a bit unfair, because the glibc wrapper sucks and the
> __vdso_getcpu interface is overcomplicated. We can fix it with a
> better API. It won't make it *that* much faster, though.
Even if we improve the vDSO function, we are at a point where just
the function call is not that cheap.
>
>>
>> diff --git a/arch/x86/kernel/signal.c b/arch/x86/kernel/signal.c
>> index e504246..157cec0 100644
>> --- a/arch/x86/kernel/signal.c
>> +++ b/arch/x86/kernel/signal.c
>> @@ -750,6 +750,8 @@ do_notify_resume(struct pt_regs *regs, void *unused, __u32
>> thread_info_flags)
>> if (thread_info_flags & _TIF_NOTIFY_RESUME) {
>> clear_thread_flag(TIF_NOTIFY_RESUME);
>> tracehook_notify_resume(regs);
>> + if (getcpu_cache_active(current))
>> + getcpu_cache_handle_notify_resume(current);
>
> We need to disentangle this stuff. This is buried way too deeply here.
>
> Fortunately, do_notify_resume is going away. It's already unused on
> 64-bit kernels in -next.
Cool! Of course, I'm willing to rebase this on whichever thread flag
and notification upon resume to userspace makes more sense.
>
>> +/*
>> + * This resume handler should always be executed between a migration
>> + * triggered by preemption and return to user-space.
>> + */
>> +void getcpu_cache_handle_notify_resume(struct task_struct *t)
>> +{
>> + int32_t __user *gcp = t->getcpu_cache;
>> +
>> + if (gcp == NULL)
>> + return;
>> + if (unlikely(t->flags & PF_EXITING))
>> + return;
>> + /*
>> + * access_ok() of gcp_user has already been checked by
>> + * sys_getcpu_cache().
>> + */
>> + if (__put_user(raw_smp_processor_id(), gcp))
>> + force_sig(SIGSEGV, current);
>
> We're preemptible here, although I think it's okay. But I'd at least
> clear the getcpu_cache state if __put_user fails, because otherwise
> it's not entirely obvious to me that we can't infinite loop.
Good point. For safety's sake, I'll set t->getcpu_cache to NULL.
>
>> +/*
>> + * sys_getcpu_cache - setup getcpu cache for caller thread
>> + */
>> +SYSCALL_DEFINE2(getcpu_cache, int32_t __user *, gcp, int, flags)
>> +{
>> + if (flags)
>> + return -EINVAL;
>> + if (gcp != NULL && !access_ok(VERIFY_WRITE, gcp, sizeof(int32_t)))
>> + return -EFAULT;
>> + current->getcpu_cache = gcp;
>> + /* Will update *gcp on resume */
>> + if (gcp)
>> + set_thread_flag(TIF_NOTIFY_RESUME);
>> + return 0;
>> +}
>
> IMO this is impolite. If the pointer is bad, we should return -EFAULT
> rather than sending SIGSEGV.
OK, so I guess you mean we should do the __put_user() in getcpu_cache
too, rather than relying on the one in notify_resume, so we can handle
faults there and return -EFAULT rather than sending SIGSEGV. Yep, it
makes sense, will fix.
Thanks!
Mathieu
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
^ permalink raw reply
* Re: [PATCH] pci_regs: reintroduce PCI_MSIX_FLAGS_BIRMASK
From: Michael S. Tsirkin @ 2015-07-13 15:11 UTC (permalink / raw)
To: linux-kernel-u79uwXL29TY76Z2rM5mHXA
Cc: Bjorn Helgaas, Rajat Jain, Chen, Gong, Guenter Roeck,
=?UTF-8?q?Rafa=C5=82=20Mi=C5=82ecki?=,
linux-api-u79uwXL29TY76Z2rM5mHXA,
linux-pci-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <1435745608-21217-1-git-send-email-mst-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
On Wed, Jul 01, 2015 at 12:14:10PM +0200, Michael S. Tsirkin wrote:
> This partially reverts commit 09a2c73ddfc7f173237fc7209a65b34dd5bcb5ed.
> PCI: Remove unused PCI_MSIX_FLAGS_BIRMASK definition
>
> That commit dropped a symbol from an exported header claiming "no one
> uses it". This isn't how Linux normally approaches userspace API though,
> and in fact QEMU build fails if trying to use updated headers from linux
> 3.12 and up.
>
> Sure, userspace can be fixed to use the new symbol, but the cost
> of keeping the old one around is fairly low, too.
>
> Signed-off-by: Michael S. Tsirkin <mst-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
Ping. Can this be included upstream please?
> ---
> include/uapi/linux/pci_regs.h | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h
> index efe3443..66644ac 100644
> --- a/include/uapi/linux/pci_regs.h
> +++ b/include/uapi/linux/pci_regs.h
> @@ -319,6 +319,7 @@
> #define PCI_MSIX_PBA 8 /* Pending Bit Array offset */
> #define PCI_MSIX_PBA_BIR 0x00000007 /* BAR index */
> #define PCI_MSIX_PBA_OFFSET 0xfffffff8 /* Offset into specified BAR */
> +#define PCI_MSIX_FLAGS_BIRMASK PCI_MSIX_PBA_BIR /* deprecated */
> #define PCI_CAP_MSIX_SIZEOF 12 /* size of MSIX registers */
>
> /* MSI-X Table entry format */
> --
> MST
^ permalink raw reply
* Re: [RFC PATCH] getcpu_cache system call: caching current CPU number (x86)
From: Mathieu Desnoyers @ 2015-07-13 15:09 UTC (permalink / raw)
To: Josh Triplett
Cc: Paul Turner, Andrew Hunter, Peter Zijlstra, Ingo Molnar,
Ben Maurer, rostedt, Paul E. McKenney, Lai Jiangshan,
Linus Torvalds, Andrew Morton, linux-api
In-Reply-To: <20150712184730.GD18191@x>
----- On Jul 12, 2015, at 2:47 PM, Josh Triplett josh-iaAMLnmF4UmaiuxdJuQwMA@public.gmane.org wrote:
> On Sun, Jul 12, 2015 at 02:06:26PM -0400, Mathieu Desnoyers wrote:
>> Expose a new system call allowing threads to register a userspace memory
>> area where to store the current CPU number. Scheduler migration sets the
>> TIF_NOTIFY_RESUME flag on the current thread. Upon return to user-space,
>> a notify-resume handler updates the current CPU value within that
>> user-space memory area.
>>
>> This getcpu cache is an alternative to the sched_getcpu() vdso which has
>> a few benefits:
>> - It is faster to do a memory read that to call a vDSO,
>> - This cache value can be read from within an inline assembly, which
>> makes it a useful building block for restartable sequences.
>>
>> This approach is inspired by Paul Turner and Andrew Hunter's work
>> on percpu atomics, which lets the kernel handle restart of critical
>> sections:
>> Ref.:
>> * https://lkml.org/lkml/2015/6/24/665
>> * https://lwn.net/Articles/650333/
>> *
>> http://www.linuxplumbersconf.org/2013/ocw/system/presentations/1695/original/LPC%20-%20PerCpu%20Atomics.pdf
>>
>> Benchmarking sched_getcpu() vs tls cache approach. Getting the
>> current CPU number:
>>
>> - With Linux vdso: 12.7 ns
>> - With TLS-cached cpu number: 0.3 ns
>
> Nice. One comment below about an interesting assumption that needs
> confirmation.
>
>> --- /dev/null
>> +++ b/kernel/getcpu-cache.c
> [...]
>> +void getcpu_cache_handle_notify_resume(struct task_struct *t)
>> +{
>> + int32_t __user *gcp = t->getcpu_cache;
>> +
>> + if (gcp == NULL)
>> + return;
>> + if (unlikely(t->flags & PF_EXITING))
>> + return;
>> + /*
>> + * access_ok() of gcp_user has already been checked by
>> + * sys_getcpu_cache().
>> + */
>> + if (__put_user(raw_smp_processor_id(), gcp))
>> + force_sig(SIGSEGV, current);
>> +}
>> +
>> +/*
>> + * If parent process has a getcpu_cache, the child inherits. Only
>> + * applies when forking a process, not a thread.
>> + */
>> +void getcpu_cache_fork(struct task_struct *t)
>> +{
>> + t->getcpu_cache = current->getcpu_cache;
>> +}
>> +
>> +void getcpu_cache_execve(struct task_struct *t)
>> +{
>> + t->getcpu_cache = NULL;
>> +}
>> +
>> +/*
>> + * sys_getcpu_cache - setup getcpu cache for caller thread
>> + */
>> +SYSCALL_DEFINE2(getcpu_cache, int32_t __user *, gcp, int, flags)
>> +{
>> + if (flags)
>> + return -EINVAL;
>> + if (gcp != NULL && !access_ok(VERIFY_WRITE, gcp, sizeof(int32_t)))
>> + return -EFAULT;
>> + current->getcpu_cache = gcp;
>
> So, you store a userspace address, and intentionally only validate it
> when initially set, not when used. You clear it on exec, though not on
> fork. Could any cases other than exec could make this problematic? In
> particular, what about unusual personality flags, such as
> ADDR_LIMIT_32BIT or ADDR_LIMIT_3GB?
That's an interesting point. Looking at those personalities, I don't think
it should be an issue, but just the fact that you raise the question makes
me think we should user put_user() rather than __put_user() in the
notify_resume handler, just to be on the safe side. It should not be a
frequent code path anyway.
>
>> + /* Will update *gcp on resume */
>> + if (gcp)
>
> Minor nit: you're using the pointer as a boolean here, but comparing it
> to NULL elsewhere; you should be consistent. I'd suggest consistently
> using gcp and !gcp, without the comparison to NULL.
Good point, fixing,
Thanks for the feedback!
Mathieu
>
> - Josh Triplett
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
^ permalink raw reply
* Re: [PATCH 0/4] enable migration of driver pages
From: Rafael Aquini @ 2015-07-13 14:20 UTC (permalink / raw)
To: Gioh Kim
Cc: jlayton-vpEMnDpepFuMZCB2o+C8xQ, bfields-uC3wQj2KruNg9hUCZPvPmw,
vbabka-AlSwsSmVLrQ, iamjoonsoo.kim-Hm3cg6mZ9cc,
viro-RmSDqhL/yNMiFSDQTTA3OLVCufUGDwFn, mst-H+wXaHxf7aLQT0dZR+AlfA,
koct9i-Re5JQEeQqe8AvxtiuMwx3w, minchan-DgEjT+Ai2ygdnm+yROfE0A,
linux-fsdevel-u79uwXL29TY76Z2rM5mHXA,
virtualization-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
linux-api-u79uwXL29TY76Z2rM5mHXA, linux-mm-Bw31MaZKKs3YtjvyW6yDsg,
dri-devel-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW,
akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b, Gioh Kim
In-Reply-To: <1436776519-17337-1-git-send-email-gioh.kim-Hm3cg6mZ9cc@public.gmane.org>
On Mon, Jul 13, 2015 at 05:35:15PM +0900, Gioh Kim wrote:
> From: Gioh Kim <gurugio-A7HHaXTFBl7R7s880joybQ@public.gmane.org>
>
> Hello,
>
> This series try to enable migration of non-LRU pages, such as driver's page.
>
> My ARM-based platform occured severe fragmentation problem after long-term
> (several days) test. Sometimes even order-3 page allocation failed. It has
> memory size 512MB ~ 1024MB. 30% ~ 40% memory is consumed for graphic processing
> and 20~30 memory is reserved for zram.
>
> I found that many pages of GPU driver and zram are non-movable pages. So I
> reported Minchan Kim, the maintainer of zram, and he made the internal
> compaction logic of zram. And I made the internal compaction of GPU driver.
>
> They reduced some fragmentation but they are not enough effective.
> They are activated by its own interface, /sys, so they are not cooperative
> with kernel compaction. If there is too much fragmentation and kernel starts
> to compaction, zram and GPU driver cannot work with the kernel compaction.
>
> So I thought there needs a interface to combine driver and kernel compaction.
> This patch adds a generic isolate/migrate/putback callbacks for page
> address-space and a new interface to create anon-inode to manage
> address_space_operation. The zram and GPU, and any other modules can create
> anon_inode and register its own migration method. The kernel compaction can
> call the registered migration when it does compaction.
>
> My GPU driver source is not in-kernel driver so that I apply the interface
> into balloon driver. The balloon driver is already merged
> into the kernel compaction as a corner-case. This patch have the balloon
> driver migration be called by the generic interface.
>
>
> This patch set combines 4 patches.
>
> 1. patch 1/4: get inode from anon_inodes
> This patch adds new interface to create inode from anon_inodes.
>
> 2. patch 2/4: framework to isolate/migrate/putback page
> Add isolatepage, putbackpage into address_space_operations
> and wrapper function to call them.
>
> 3. patch 3/4: apply the framework into balloon driver
> The balloon driver is applied into the framework. It gets a inode
> from anon_inodes and register operations in the inode.
> The kernel compaction calls generic interfaces, not balloon
> driver interfaces.
> Any other drivers can register operations via inode like this
> to migrate it's pages.
>
> 4. patch 4/4: remove direct calling of migration of driver pages
> Non-lru pages are also migrated with lru pages by move_to_new_page().
>
> This patch set is tested:
> - turn on Ubuntu 14.04 with 1G memory on qemu.
> - do kernel building
> - after several seconds check more than 512MB is used with free command
> - command "balloon 512" in qemu monitor
> - check hundreds MB of pages are migrated
>
> My thanks to Konstantin Khlebnikov for his reviews of the RFC patch set.
> Most of the changes were based on his feedback.
>
> This patch-set is based on v4.1
>
>
> Gioh Kim (4):
> fs/anon_inodes: new interface to create new inode
> mm/compaction: enable mobile-page migration
> mm/balloon: apply mobile page migratable into balloon
> mm: remove direct calling of migration
>
> drivers/virtio/virtio_balloon.c | 3 ++
> fs/anon_inodes.c | 6 +++
> fs/proc/page.c | 3 ++
> include/linux/anon_inodes.h | 1 +
> include/linux/balloon_compaction.h | 15 +++++--
> include/linux/compaction.h | 80 ++++++++++++++++++++++++++++++++++
> include/linux/fs.h | 2 +
> include/linux/page-flags.h | 19 ++++++++
> include/uapi/linux/kernel-page-flags.h | 1 +
> mm/balloon_compaction.c | 72 ++++++++++--------------------
> mm/compaction.c | 8 ++--
> mm/migrate.c | 24 +++-------
> 12 files changed, 160 insertions(+), 74 deletions(-)
>
> --
> 2.1.4
>
Acked-by: Rafael Aquini <aquini-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
^ permalink raw reply
* RE: [RFC PATCH] getcpu_cache system call: caching current CPU number (x86)
From: Ben Maurer @ 2015-07-13 11:17 UTC (permalink / raw)
To: Mathieu Desnoyers, Paul Turner
Cc: Andrew Hunter, Peter Zijlstra, Ingo Molnar, Steven Rostedt,
Paul E. McKenney, Josh Triplett, Lai Jiangshan, Linus Torvalds,
Andrew Morton, linux-api-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
In-Reply-To: <1436724386-30909-1-git-send-email-mathieu.desnoyers-vg+e7yoeK/dWk0Htik3J/w@public.gmane.org>
At Facebook we already use getcpu in folly, our base C++ library, to provide high performance concurrency algorithms. Folly includes an abstraction called AccessSpreader which helps engineers write abstractions which shard themselves across different cores to prevent cache contention (https://github.com/facebook/folly/blob/master/folly/detail/CacheLocality.cpp). We have used this primative to create faster reader writer locks (https://github.com/facebook/folly/blob/master/folly/SharedMutex.h), as well as in an abstraction that powers workqueues (https://github.com/facebook/folly/blob/master/folly/IndexedMemPool.h). This would be a great perf improvement for these types of abstractions and probably encourage us to use the idea more widely.
One quick comment on the approach -- it'd be really great if we had a method that didn't require users to register each thread. This can often lead to requiring an additional branch in critical code to check if the appropriate caches have been initialized. Also, one of the most interesting potential applications of the restartable sequences concept is in malloc. having a brief period at the beginning of the life of a thread where malloc didn't work would be pretty tricky to program around.
-b
________________________________________
From: Mathieu Desnoyers [mathieu.desnoyers-vg+e7yoeK/dWk0Htik3J/w@public.gmane.org]
Sent: Sunday, July 12, 2015 11:06 AM
To: Paul Turner
Cc: Mathieu Desnoyers; Andrew Hunter; Peter Zijlstra; Ingo Molnar; Ben Maurer; Steven Rostedt; Paul E. McKenney; Josh Triplett; Lai Jiangshan; Linus Torvalds; Andrew Morton; linux-api-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
Subject: [RFC PATCH] getcpu_cache system call: caching current CPU number (x86)
Expose a new system call allowing threads to register a userspace memory
area where to store the current CPU number. Scheduler migration sets the
TIF_NOTIFY_RESUME flag on the current thread. Upon return to user-space,
a notify-resume handler updates the current CPU value within that
user-space memory area.
This getcpu cache is an alternative to the sched_getcpu() vdso which has
a few benefits:
- It is faster to do a memory read that to call a vDSO,
- This cache value can be read from within an inline assembly, which
makes it a useful building block for restartable sequences.
This approach is inspired by Paul Turner and Andrew Hunter's work
on percpu atomics, which lets the kernel handle restart of critical
sections:
Ref.:
* https://lkml.org/lkml/2015/6/24/665
* https://urldefense.proofpoint.com/v1/url?u=https://lwn.net/Articles/650333/&k=ZVNjlDMF0FElm4dQtryO4A%3D%3D%0A&r=ykFqYBj5kD4j7jlBP4F60A%3D%3D%0A&m=rgSZ590AsXQ7Hp6Q6Oyt6xFrzk0JFmcxItletGiGhNE%3D%0A&s=6446da0c7a6cc256b68c865d26b333a37d4b40548ebeebc39114a782a6ce6ea3
* https://urldefense.proofpoint.com/v1/url?u=http://www.linuxplumbersconf.org/2013/ocw/system/presentations/1695/original/LPC%2520-%2520PerCpu%2520Atomics.pdf&k=ZVNjlDMF0FElm4dQtryO4A%3D%3D%0A&r=ykFqYBj5kD4j7jlBP4F60A%3D%3D%0A&m=rgSZ590AsXQ7Hp6Q6Oyt6xFrzk0JFmcxItletGiGhNE%3D%0A&s=71e097fe57910235e8b23b0bb1b20aa8c8c4b80106663980b62ccaa7e13fba9a
Benchmarking sched_getcpu() vs tls cache approach. Getting the
current CPU number:
- With Linux vdso: 12.7 ns
- With TLS-cached cpu number: 0.3 ns
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers-vg+e7yoeK/dWk0Htik3J/w@public.gmane.org>
CC: Paul Turner <pjt-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org>
CC: Andrew Hunter <ahh-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org>
CC: Peter Zijlstra <peterz-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
CC: Ingo Molnar <mingo-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
CC: Ben Maurer <bmaurer-b10kYP2dOMg@public.gmane.org>
CC: Steven Rostedt <rostedt-nx8X9YLhiw1AfugRpC6u6w@public.gmane.org>
CC: "Paul E. McKenney" <paulmck-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org>
CC: Josh Triplett <josh-iaAMLnmF4UmaiuxdJuQwMA@public.gmane.org>
CC: Lai Jiangshan <laijs-BthXqXjhjHXQFUHtdCDX3A@public.gmane.org>
CC: Linus Torvalds <torvalds-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b@public.gmane.org>
CC: Andrew Morton <akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b@public.gmane.org>
CC: linux-api-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
---
arch/x86/kernel/signal.c | 2 ++
arch/x86/syscalls/syscall_64.tbl | 1 +
fs/exec.c | 1 +
include/linux/sched.h | 27 +++++++++++++++
include/uapi/asm-generic/unistd.h | 4 ++-
init/Kconfig | 9 +++++
kernel/Makefile | 1 +
kernel/fork.c | 2 ++
kernel/getcpu-cache.c | 70 +++++++++++++++++++++++++++++++++++++++
kernel/sched/core.c | 3 ++
kernel/sched/sched.h | 2 ++
kernel/sys_ni.c | 3 ++
12 files changed, 124 insertions(+), 1 deletion(-)
create mode 100644 kernel/getcpu-cache.c
diff --git a/arch/x86/kernel/signal.c b/arch/x86/kernel/signal.c
index e504246..157cec0 100644
--- a/arch/x86/kernel/signal.c
+++ b/arch/x86/kernel/signal.c
@@ -750,6 +750,8 @@ do_notify_resume(struct pt_regs *regs, void *unused, __u32 thread_info_flags)
if (thread_info_flags & _TIF_NOTIFY_RESUME) {
clear_thread_flag(TIF_NOTIFY_RESUME);
tracehook_notify_resume(regs);
+ if (getcpu_cache_active(current))
+ getcpu_cache_handle_notify_resume(current);
}
if (thread_info_flags & _TIF_USER_RETURN_NOTIFY)
fire_user_return_notifiers();
diff --git a/arch/x86/syscalls/syscall_64.tbl b/arch/x86/syscalls/syscall_64.tbl
index 8d656fb..cfcf8e7 100644
--- a/arch/x86/syscalls/syscall_64.tbl
+++ b/arch/x86/syscalls/syscall_64.tbl
@@ -329,6 +329,7 @@
320 common kexec_file_load sys_kexec_file_load
321 common bpf sys_bpf
322 64 execveat stub_execveat
+323 common getcpu_cache sys_getcpu_cache
#
# x32-specific system call numbers start at 512 to avoid cache impact
diff --git a/fs/exec.c b/fs/exec.c
index c7f9b73..20ef2e6 100644
--- a/fs/exec.c
+++ b/fs/exec.c
@@ -1555,6 +1555,7 @@ static int do_execveat_common(int fd, struct filename *filename,
/* execve succeeded */
current->fs->in_exec = 0;
current->in_execve = 0;
+ getcpu_cache_execve(current);
acct_update_integrals(current);
task_numa_free(current);
free_bprm(bprm);
diff --git a/include/linux/sched.h b/include/linux/sched.h
index a419b65..0654cc2 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1710,6 +1710,9 @@ struct task_struct {
#ifdef CONFIG_DEBUG_ATOMIC_SLEEP
unsigned long task_state_change;
#endif
+#ifdef CONFIG_GETCPU_CACHE
+ int32_t __user *getcpu_cache;
+#endif
};
/* Future-safe accessor for struct task_struct's cpus_allowed. */
@@ -3090,4 +3093,28 @@ static inline unsigned long rlimit_max(unsigned int limit)
return task_rlimit_max(current, limit);
}
+#ifdef CONFIG_GETCPU_CACHE
+void getcpu_cache_fork(struct task_struct *t);
+void getcpu_cache_execve(struct task_struct *t);
+void getcpu_cache_handle_notify_resume(struct task_struct *t);
+static inline bool getcpu_cache_active(struct task_struct *t)
+{
+ return t->getcpu_cache;
+}
+#else
+static inline void getcpu_cache_fork(struct task_struct *t)
+{
+}
+static inline void getcpu_cache_execve(struct task_struct *t)
+{
+}
+static inline void getcpu_cache_handle_notify_resume(struct task_struct *t)
+{
+}
+static inline bool getcpu_cache_active(struct task_struct *t)
+{
+ return false;
+}
+#endif
+
#endif
diff --git a/include/uapi/asm-generic/unistd.h b/include/uapi/asm-generic/unistd.h
index e016bd9..f82b70d 100644
--- a/include/uapi/asm-generic/unistd.h
+++ b/include/uapi/asm-generic/unistd.h
@@ -709,9 +709,11 @@ __SYSCALL(__NR_memfd_create, sys_memfd_create)
__SYSCALL(__NR_bpf, sys_bpf)
#define __NR_execveat 281
__SC_COMP(__NR_execveat, sys_execveat, compat_sys_execveat)
+#define __NR_getcpu_cache 282
+__SYSCALL(__NR_getcpu_cache, sys_getcpu_cache)
#undef __NR_syscalls
-#define __NR_syscalls 282
+#define __NR_syscalls 283
/*
* All syscalls below here should go away really,
diff --git a/init/Kconfig b/init/Kconfig
index f5dbc6d..fac919b 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -1559,6 +1559,15 @@ config PCI_QUIRKS
bugs/quirks. Disable this only if your target machine is
unaffected by PCI quirks.
+config GETCPU_CACHE
+ bool "Enable getcpu_cache() system call" if EXPERT
+ default y
+ help
+ Enable the getcpu_cache() system call which provides a
+ user-space cache for the current CPU number value.
+
+ If unsure, say Y.
+
config EMBEDDED
bool "Embedded system"
option allnoconfig_y
diff --git a/kernel/Makefile b/kernel/Makefile
index 1408b33..3350ba1 100644
--- a/kernel/Makefile
+++ b/kernel/Makefile
@@ -96,6 +96,7 @@ obj-$(CONFIG_CRASH_DUMP) += crash_dump.o
obj-$(CONFIG_JUMP_LABEL) += jump_label.o
obj-$(CONFIG_CONTEXT_TRACKING) += context_tracking.o
obj-$(CONFIG_TORTURE_TEST) += torture.o
+obj-$(CONFIG_GETCPU_CACHE) += getcpu-cache.o
$(obj)/configs.o: $(obj)/config_data.h
diff --git a/kernel/fork.c b/kernel/fork.c
index cf65139..334e62d 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -1549,6 +1549,8 @@ static struct task_struct *copy_process(unsigned long clone_flags,
cgroup_post_fork(p);
if (clone_flags & CLONE_THREAD)
threadgroup_change_end(current);
+ if (!(clone_flags & CLONE_THREAD))
+ getcpu_cache_fork(p);
perf_event_fork(p);
trace_task_newtask(p, clone_flags);
diff --git a/kernel/getcpu-cache.c b/kernel/getcpu-cache.c
new file mode 100644
index 0000000..b4e5c77
--- /dev/null
+++ b/kernel/getcpu-cache.c
@@ -0,0 +1,70 @@
+/*
+ * Copyright (C) 2015 Mathieu Desnoyers <mathieu.desnoyers-vg+e7yoeK/dWk0Htik3J/w@public.gmane.org>
+ *
+ * getcpu_cache system call
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/init.h>
+#include <linux/sched.h>
+#include <linux/uaccess.h>
+#include <linux/syscalls.h>
+
+/*
+ * This resume handler should always be executed between a migration
+ * triggered by preemption and return to user-space.
+ */
+void getcpu_cache_handle_notify_resume(struct task_struct *t)
+{
+ int32_t __user *gcp = t->getcpu_cache;
+
+ if (gcp == NULL)
+ return;
+ if (unlikely(t->flags & PF_EXITING))
+ return;
+ /*
+ * access_ok() of gcp_user has already been checked by
+ * sys_getcpu_cache().
+ */
+ if (__put_user(raw_smp_processor_id(), gcp))
+ force_sig(SIGSEGV, current);
+}
+
+/*
+ * If parent process has a getcpu_cache, the child inherits. Only
+ * applies when forking a process, not a thread.
+ */
+void getcpu_cache_fork(struct task_struct *t)
+{
+ t->getcpu_cache = current->getcpu_cache;
+}
+
+void getcpu_cache_execve(struct task_struct *t)
+{
+ t->getcpu_cache = NULL;
+}
+
+/*
+ * sys_getcpu_cache - setup getcpu cache for caller thread
+ */
+SYSCALL_DEFINE2(getcpu_cache, int32_t __user *, gcp, int, flags)
+{
+ if (flags)
+ return -EINVAL;
+ if (gcp != NULL && !access_ok(VERIFY_WRITE, gcp, sizeof(int32_t)))
+ return -EFAULT;
+ current->getcpu_cache = gcp;
+ /* Will update *gcp on resume */
+ if (gcp)
+ set_thread_flag(TIF_NOTIFY_RESUME);
+ return 0;
+}
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 62671f5..a9009d4 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -1823,6 +1823,9 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p)
p->numa_group = NULL;
#endif /* CONFIG_NUMA_BALANCING */
+#ifdef CONFIG_GETCPU_CACHE
+ p->getcpu_cache = NULL;
+#endif
}
#ifdef CONFIG_NUMA_BALANCING
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index dc0f435..bf3e346 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -921,6 +921,8 @@ static inline void __set_task_cpu(struct task_struct *p, unsigned int cpu)
{
set_task_rq(p, cpu);
#ifdef CONFIG_SMP
+ if (getcpu_cache_active(p))
+ set_tsk_thread_flag(p, TIF_NOTIFY_RESUME);
/*
* After ->cpu is set up to a new value, task_rq_lock(p, ...) can be
* successfuly executed on another CPU. We must ensure that updates of
diff --git a/kernel/sys_ni.c b/kernel/sys_ni.c
index 5adcb0a..3691dc8 100644
--- a/kernel/sys_ni.c
+++ b/kernel/sys_ni.c
@@ -229,3 +229,6 @@ cond_syscall(sys_bpf);
/* execveat */
cond_syscall(sys_execveat);
+
+/* current CPU number cache */
+cond_syscall(sys_getcpu_cache);
--
2.1.4
^ permalink raw reply related
* Re: [PATCH v7 4/9] nvmem: Add bindings for simple nvmem framework
From: Srinivas Kandagatla @ 2015-07-13 10:21 UTC (permalink / raw)
To: Rob Herring
Cc: linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org,
Greg Kroah-Hartman, Rob Herring, Kumar Gala, Mark Brown,
Sascha Hauer, linux-api-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
devicetree-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-arm-msm,
Arnd Bergmann, Stephen Boyd, Pantelis Antoniou, Matt Porter,
Stefan Wahren, wxt-TNX95d0MmH7DzftRWevZcw, Maxime Ripard
In-Reply-To: <CAL_JsqKYT-+T=D=TKCb1TcydOvBTE8_yc-EEe1cwdmhr1B0dHQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
Thanks Rob for quick review,
On 10/07/15 20:04, Rob Herring wrote:
> On Fri, Jul 10, 2015 at 4:45 AM, Srinivas Kandagatla
> <srinivas.kandagatla-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org> wrote:
>> This patch adds bindings for simple nvmem framework which allows nvmem
>> consumers to talk to nvmem providers to get access to nvmem cell data.
>>
>> Signed-off-by: Maxime Ripard <maxime.ripard-wi1+55ScJUtKEb57/3fJTNBPR1lH4CV8@public.gmane.org>
>> [Maxime Ripard: intial version of eeprom framework]
>> Signed-off-by: Srinivas Kandagatla <srinivas.kandagatla-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org>
>> ---
>> Documentation/devicetree/bindings/nvmem/nvmem.txt | 85 +++++++++++++++++++++++
>> 1 file changed, 85 insertions(+)
>> create mode 100644 Documentation/devicetree/bindings/nvmem/nvmem.txt
>>
>> diff --git a/Documentation/devicetree/bindings/nvmem/nvmem.txt b/Documentation/devicetree/bindings/nvmem/nvmem.txt
>> new file mode 100644
>> index 0000000..849f1e1
>> --- /dev/null
>> +++ b/Documentation/devicetree/bindings/nvmem/nvmem.txt
>> @@ -0,0 +1,85 @@
>> += NVMEM(Non Volatile Memory) Data Device Tree Bindings =
>> +
>> +This binding is intended to represent the location of hardware
>> +configuration data stored in NVMEMs like eeprom, efuses and so on.
>> +
>> +On a significant proportion of boards, the manufacturer has stored
>> +some data on NVMEM, for the OS to be able to retrieve these information
>> +and act upon it. Obviously, the OS has to know about where to retrieve
>> +these data from, and where they are stored on the storage device.
>> +
>> +This document is here to document this.
>> +
>> += Data providers =
>> +Contains bindings specific to provider drivers and data cells as children
>> +of this node.
>
> #address-cells and #size-cells are required here.
>
>> +
>> +Optional properties:
>> + read-only: Mark the provider as read only.
>
> Couldn't this be per field rather than global?
>
Not ATM, The reason for making this property global is to mark the
complete nvmem to be readonly/read-write. Which most of the use-cases
will have. Currently this property is used for setting permissions on
the sysfs binary file, also it would be impossible to apply per field
read-only property to such file.
Am also planning to send few patches on top of these to expose fields in
sysfs which would then allow us to use per field read-only property.
Again not sure how the direct access to nvmem would fit in with such
requirements. Need to evaluate this option in more detail though. :-)
>> +
>> += Data cells =
>> +These are the child nodes of the provider which contain data cell
>> +information like offset and size in nvmem provider.
>> +
>> +Required properties:
>> +reg: specifies the offset in byte within that storage device, start bit
>> + in the byte and the length in bytes of the data we care about.
>> + There could be more than one offset-length pairs in this property.
>> +
>> +Optional properties:
>> +
>> +bit-offset: specifies the offset in bit within the address range specified
>> + by reg property. Can take values from 0-7.
>> +nbits: specifies number of bits this cell occupies starting from bit-offset.
>
> How about just: "bits = <<offset> <size>>"
>
Thats another possible way to specify the same info, Only reason I came
up with bit-offset and nbits is due to the fact that similar properties
were seen in other device DT bindings.
I will try your suggestion and see how it looks before I send new version.
> Then the bit specification is more aligned with the byte location
> (i.e. reg property).
>
> You could also do this all in the reg property with 2 address cells
> for byte and bit position and then size can be in bits. reg doesn't
> have to match a memory mapped bus addressing meanings. If you wanted
> to handle ranges and address translation, then you would need custom
> functions like PCI does. I'm not sure you would need that.
I wanted to keep things simple for this first version.
>
--srini
^ permalink raw reply
* Re: [PATCH 0/4] enable migration of driver pages
From: Gioh Kim @ 2015-07-13 10:02 UTC (permalink / raw)
To: Konstantin Khlebnikov
Cc: Rafael Aquini, Michael S. Tsirkin, Linux API,
Linux Kernel Mailing List, dri-devel, virtualization,
Bruce Fields, Minchan Kim, Gioh Kim, linux-mm@kvack.org, Al Viro,
Andrew Morton, linux-fsdevel, Jeff Layton, Joonsoo Kim,
Vlastimil Babka
In-Reply-To: <CALYGNiPZtJqcYW5Ob6TbRMGrJHP6zV7cKfbesBxprVQjqVmUSw@mail.gmail.com>
2015-07-13 오후 6:24에 Konstantin Khlebnikov 이(가) 쓴 글:
> On Mon, Jul 13, 2015 at 11:35 AM, Gioh Kim <gioh.kim@lge.com> wrote:
>> From: Gioh Kim <gurugio@hanmail.net>
>>
>> Hello,
>>
>> This series try to enable migration of non-LRU pages, such as driver's page.
>>
>> My ARM-based platform occured severe fragmentation problem after long-term
>> (several days) test. Sometimes even order-3 page allocation failed. It has
>> memory size 512MB ~ 1024MB. 30% ~ 40% memory is consumed for graphic processing
>> and 20~30 memory is reserved for zram.
>>
>> I found that many pages of GPU driver and zram are non-movable pages. So I
>> reported Minchan Kim, the maintainer of zram, and he made the internal
>> compaction logic of zram. And I made the internal compaction of GPU driver.
>>
>> They reduced some fragmentation but they are not enough effective.
>> They are activated by its own interface, /sys, so they are not cooperative
>> with kernel compaction. If there is too much fragmentation and kernel starts
>> to compaction, zram and GPU driver cannot work with the kernel compaction.
>>
>> So I thought there needs a interface to combine driver and kernel compaction.
>> This patch adds a generic isolate/migrate/putback callbacks for page
>> address-space and a new interface to create anon-inode to manage
>> address_space_operation. The zram and GPU, and any other modules can create
>> anon_inode and register its own migration method. The kernel compaction can
>> call the registered migration when it does compaction.
>>
>> My GPU driver source is not in-kernel driver so that I apply the interface
>> into balloon driver. The balloon driver is already merged
>> into the kernel compaction as a corner-case. This patch have the balloon
>> driver migration be called by the generic interface.
>>
>>
>> This patch set combines 4 patches.
>>
>> 1. patch 1/4: get inode from anon_inodes
>> This patch adds new interface to create inode from anon_inodes.
>>
>> 2. patch 2/4: framework to isolate/migrate/putback page
>> Add isolatepage, putbackpage into address_space_operations
>> and wrapper function to call them.
>>
>> 3. patch 3/4: apply the framework into balloon driver
>> The balloon driver is applied into the framework. It gets a inode
>> from anon_inodes and register operations in the inode.
>> The kernel compaction calls generic interfaces, not balloon
>> driver interfaces.
>> Any other drivers can register operations via inode like this
>> to migrate it's pages.
>>
>> 4. patch 4/4: remove direct calling of migration of driver pages
>> Non-lru pages are also migrated with lru pages by move_to_new_page().
>
> The whole patchset looks good.
>
> Reviewed-by: Konstantin Khlebnikov <koct9i@gmail.com>
>
>>
>> This patch set is tested:
>> - turn on Ubuntu 14.04 with 1G memory on qemu.
>> - do kernel building
>> - after several seconds check more than 512MB is used with free command
>> - command "balloon 512" in qemu monitor
>> - check hundreds MB of pages are migrated
>
> Another simple test is several instances of
> tools/testing/selftests/vm/transhuge-stress.c
> runnng in parallel with balloon inflating/deflating.
> (transparent huge pages must be enabled of course)
> That catched a lot of races in ballooning code.
>
Great!
I'll do it and inform you the result in this week.
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* Re: [PATCH 0/4] enable migration of driver pages
From: Konstantin Khlebnikov @ 2015-07-13 9:24 UTC (permalink / raw)
To: Gioh Kim
Cc: Jeff Layton, Bruce Fields, Vlastimil Babka, Joonsoo Kim, Al Viro,
Michael S. Tsirkin, Minchan Kim, Rafael Aquini, linux-fsdevel,
virtualization, Linux Kernel Mailing List, Linux API,
linux-mm@kvack.org, dri-devel, Andrew Morton, Gioh Kim
In-Reply-To: <1436776519-17337-1-git-send-email-gioh.kim@lge.com>
On Mon, Jul 13, 2015 at 11:35 AM, Gioh Kim <gioh.kim@lge.com> wrote:
> From: Gioh Kim <gurugio@hanmail.net>
>
> Hello,
>
> This series try to enable migration of non-LRU pages, such as driver's page.
>
> My ARM-based platform occured severe fragmentation problem after long-term
> (several days) test. Sometimes even order-3 page allocation failed. It has
> memory size 512MB ~ 1024MB. 30% ~ 40% memory is consumed for graphic processing
> and 20~30 memory is reserved for zram.
>
> I found that many pages of GPU driver and zram are non-movable pages. So I
> reported Minchan Kim, the maintainer of zram, and he made the internal
> compaction logic of zram. And I made the internal compaction of GPU driver.
>
> They reduced some fragmentation but they are not enough effective.
> They are activated by its own interface, /sys, so they are not cooperative
> with kernel compaction. If there is too much fragmentation and kernel starts
> to compaction, zram and GPU driver cannot work with the kernel compaction.
>
> So I thought there needs a interface to combine driver and kernel compaction.
> This patch adds a generic isolate/migrate/putback callbacks for page
> address-space and a new interface to create anon-inode to manage
> address_space_operation. The zram and GPU, and any other modules can create
> anon_inode and register its own migration method. The kernel compaction can
> call the registered migration when it does compaction.
>
> My GPU driver source is not in-kernel driver so that I apply the interface
> into balloon driver. The balloon driver is already merged
> into the kernel compaction as a corner-case. This patch have the balloon
> driver migration be called by the generic interface.
>
>
> This patch set combines 4 patches.
>
> 1. patch 1/4: get inode from anon_inodes
> This patch adds new interface to create inode from anon_inodes.
>
> 2. patch 2/4: framework to isolate/migrate/putback page
> Add isolatepage, putbackpage into address_space_operations
> and wrapper function to call them.
>
> 3. patch 3/4: apply the framework into balloon driver
> The balloon driver is applied into the framework. It gets a inode
> from anon_inodes and register operations in the inode.
> The kernel compaction calls generic interfaces, not balloon
> driver interfaces.
> Any other drivers can register operations via inode like this
> to migrate it's pages.
>
> 4. patch 4/4: remove direct calling of migration of driver pages
> Non-lru pages are also migrated with lru pages by move_to_new_page().
The whole patchset looks good.
Reviewed-by: Konstantin Khlebnikov <koct9i@gmail.com>
>
> This patch set is tested:
> - turn on Ubuntu 14.04 with 1G memory on qemu.
> - do kernel building
> - after several seconds check more than 512MB is used with free command
> - command "balloon 512" in qemu monitor
> - check hundreds MB of pages are migrated
Another simple test is several instances of
tools/testing/selftests/vm/transhuge-stress.c
runnng in parallel with balloon inflating/deflating.
(transparent huge pages must be enabled of course)
That catched a lot of races in ballooning code.
>
> My thanks to Konstantin Khlebnikov for his reviews of the RFC patch set.
> Most of the changes were based on his feedback.
>
> This patch-set is based on v4.1
>
>
> Gioh Kim (4):
> fs/anon_inodes: new interface to create new inode
> mm/compaction: enable mobile-page migration
> mm/balloon: apply mobile page migratable into balloon
> mm: remove direct calling of migration
>
> drivers/virtio/virtio_balloon.c | 3 ++
> fs/anon_inodes.c | 6 +++
> fs/proc/page.c | 3 ++
> include/linux/anon_inodes.h | 1 +
> include/linux/balloon_compaction.h | 15 +++++--
> include/linux/compaction.h | 80 ++++++++++++++++++++++++++++++++++
> include/linux/fs.h | 2 +
> include/linux/page-flags.h | 19 ++++++++
> include/uapi/linux/kernel-page-flags.h | 1 +
> mm/balloon_compaction.c | 72 ++++++++++--------------------
> mm/compaction.c | 8 ++--
> mm/migrate.c | 24 +++-------
> 12 files changed, 160 insertions(+), 74 deletions(-)
>
> --
> 2.1.4
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [RFCv3 2/5] mm/compaction: enable mobile-page migration
From: Gioh Kim @ 2015-07-13 8:45 UTC (permalink / raw)
To: Konstantin Khlebnikov
Cc: Jeff Layton, Bruce Fields, Vlastimil Babka, Joonsoo Kim, Al Viro,
Michael S. Tsirkin, Minchan Kim, Rafael Aquini, linux-fsdevel,
virtualization, Linux Kernel Mailing List, Linux API,
linux-mm@kvack.org, gunho.lee, Andrew Morton, Gioh Kim
In-Reply-To: <CALYGNiPBPzA0QCXZKXKye++xVSeO_nBW4gV+ukk2jPiBOM+n=A@mail.gmail.com>
>> @@ -51,6 +54,66 @@ extern void compaction_defer_reset(struct zone *zone, int order,
>> bool alloc_success);
>> extern bool compaction_restarting(struct zone *zone, int order);
>>
>> +static inline bool mobile_page(struct page *page)
>> +{
>> + return page->mapping && page->mapping->a_ops &&
>
> Dereferncing mapping->a_ops isn't safe without page-lock and isn't required:
> all mappings always have ->a_ops.
>
I got it.
>> +static inline void putback_mobilepage(struct page *page)
>> +{
>> + /*
>> + * 'lock_page()' stabilizes the page and prevents races against
>> + * concurrent isolation threads attempting to re-isolate it.
>> + */
>> + lock_page(page);
>> + if (mobile_page(page) && page->mapping->a_ops->putbackpage) {
>
> It seems "if (page->mapping && page->mapping->a_ops->putbackpage)"
> should be enough: we already seen that page as mobile.
Ditto.
>
>> + page->mapping->a_ops->putbackpage(page);
>> + /* drop the extra ref count taken for mobile page isolation */
>> + put_page(page);
>> + }
>> + unlock_page(page);
>
> call put_page() after unlock and do that always -- putback must drop
> page reference from caller.
>
> lock_page(page);
> if (page->mapping && page->mapping->a_ops->putbackpage)
> page->mapping->a_ops->putbackpage(page);
> unlock_page();
> put_page(page);
>
Ditto.
>> +}
>> #else
>> static inline unsigned long try_to_compact_pages(gfp_t gfp_mask,
>> unsigned int order, int alloc_flags,
>> @@ -83,6 +146,19 @@ static inline bool compaction_deferred(struct zone *zone, int order)
>> return true;
>> }
>>
>> +static inline bool mobile_page(struct page *page)
>> +{
>> + return false;
>> +}
>> +
>> +static inline bool isolate_mobilepage(struct page *page, isolate_mode_t mode)
>> +{
>> + return false;
>> +}
>> +
>> +static inline void putback_mobilepage(struct page *page)
>> +{
>> +}
>> #endif /* CONFIG_COMPACTION */
>>
>> #if defined(CONFIG_COMPACTION) && defined(CONFIG_SYSFS) && defined(CONFIG_NUMA)
>> diff --git a/include/linux/fs.h b/include/linux/fs.h
>> index 35ec87e..33c9aa5 100644
>> --- a/include/linux/fs.h
>> +++ b/include/linux/fs.h
>> @@ -395,6 +395,8 @@ struct address_space_operations {
>> */
>> int (*migratepage) (struct address_space *,
>> struct page *, struct page *, enum migrate_mode);
>> + bool (*isolatepage) (struct page *, isolate_mode_t);
>> + void (*putbackpage) (struct page *);
>> int (*launder_page) (struct page *);
>> int (*is_partially_uptodate) (struct page *, unsigned long,
>> unsigned long);
>> diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
>> index f34e040..abef145 100644
>> --- a/include/linux/page-flags.h
>> +++ b/include/linux/page-flags.h
>> @@ -582,6 +582,25 @@ static inline void __ClearPageBalloon(struct page *page)
>> atomic_set(&page->_mapcount, -1);
>> }
>>
>> +#define PAGE_MOBILE_MAPCOUNT_VALUE (-255)
>> +
>> +static inline int PageMobile(struct page *page)
>> +{
>> + return atomic_read(&page->_mapcount) == PAGE_MOBILE_MAPCOUNT_VALUE;
>> +}
>> +
>> +static inline void __SetPageMobile(struct page *page)
>> +{
>> + VM_BUG_ON_PAGE(atomic_read(&page->_mapcount) != -1, page);
>> + atomic_set(&page->_mapcount, PAGE_MOBILE_MAPCOUNT_VALUE);
>> +}
>> +
>> +static inline void __ClearPageMobile(struct page *page)
>> +{
>> + VM_BUG_ON_PAGE(!PageMobile(page), page);
>> + atomic_set(&page->_mapcount, -1);
>> +}
>> +
>> /*
>> * If network-based swap is enabled, sl*b must keep track of whether pages
>> * were allocated from pfmemalloc reserves.
>> diff --git a/include/uapi/linux/kernel-page-flags.h b/include/uapi/linux/kernel-page-flags.h
>> index a6c4962..d50d9e8 100644
>> --- a/include/uapi/linux/kernel-page-flags.h
>> +++ b/include/uapi/linux/kernel-page-flags.h
>> @@ -33,6 +33,7 @@
>> #define KPF_THP 22
>> #define KPF_BALLOON 23
>> #define KPF_ZERO_PAGE 24
>> +#define KPF_MOBILE 25
>>
>>
>> #endif /* _UAPILINUX_KERNEL_PAGE_FLAGS_H */
>> --
>> 2.1.4
>>
>
I fixed the code as your comments and I found patch 3/5 and 4/5 could not be applied separately.
So I merge them and report new [PATCH].
I appreciate your reviews.
^ permalink raw reply
* [PATCH 4/4] mm: remove direct calling of migration
From: Gioh Kim @ 2015-07-13 8:35 UTC (permalink / raw)
To: jlayton-vpEMnDpepFuMZCB2o+C8xQ, bfields-uC3wQj2KruNg9hUCZPvPmw,
vbabka-AlSwsSmVLrQ, iamjoonsoo.kim-Hm3cg6mZ9cc,
viro-RmSDqhL/yNMiFSDQTTA3OLVCufUGDwFn, mst-H+wXaHxf7aLQT0dZR+AlfA,
koct9i-Re5JQEeQqe8AvxtiuMwx3w, minchan-DgEjT+Ai2ygdnm+yROfE0A,
aquini-H+wXaHxf7aLQT0dZR+AlfA,
linux-fsdevel-u79uwXL29TY76Z2rM5mHXA,
virtualization-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
linux-api-u79uwXL29TY76Z2rM5mHXA, linux-mm-Bw31MaZKKs3YtjvyW6yDsg
Cc: dri-devel-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW,
akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b, Gioh Kim, Gioh Kim
In-Reply-To: <1436776519-17337-1-git-send-email-gioh.kim-Hm3cg6mZ9cc@public.gmane.org>
From: Gioh Kim <gurugio-A7HHaXTFBl7R7s880joybQ@public.gmane.org>
Migration is completely generalized so that migrating mobile page
is processed with lru-pages in move_to_new_page.
Signed-off-by: Gioh Kim <gioh.kim-Hm3cg6mZ9cc@public.gmane.org>
Acked-by: Rafael Aquini <aquini-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
---
mm/migrate.c | 15 ---------------
1 file changed, 15 deletions(-)
diff --git a/mm/migrate.c b/mm/migrate.c
index 53f0081d..e6644ac 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -844,21 +844,6 @@ static int __unmap_and_move(struct page *page, struct page *newpage,
}
}
- if (unlikely(mobile_page(page))) {
- /*
- * A mobile page does not need any special attention from
- * physical to virtual reverse mapping procedures.
- * Skip any attempt to unmap PTEs or to remap swap cache,
- * in order to avoid burning cycles at rmap level, and perform
- * the page migration right away (proteced by page lock).
- */
- lock_page(newpage);
- rc = page->mapping->a_ops->migratepage(page->mapping,
- newpage, page, mode);
- unlock_page(newpage);
- goto out_unlock;
- }
-
/*
* Corner case handling:
* 1. When a new swap-cache page is read into, it is added to the LRU
--
2.1.4
^ permalink raw reply related
* [PATCH 3/4] mm/balloon: apply mobile page migratable into balloon
From: Gioh Kim @ 2015-07-13 8:35 UTC (permalink / raw)
To: jlayton, bfields, vbabka, iamjoonsoo.kim, viro, mst, koct9i,
minchan, aquini, linux-fsdevel, virtualization, linux-kernel,
linux-api, linux-mm
Cc: dri-devel, akpm, Gioh Kim, Gioh Kim
In-Reply-To: <1436776519-17337-1-git-send-email-gioh.kim@lge.com>
From: Gioh Kim <gurugio@hanmail.net>
Apply mobile page migration into balloon driver.
The balloong driver has an anonymous inode that manages
address_space_operation for page migration.
Compaction calls interfaces of mobile page migration
instead of calling balloon migration directly.
Signed-off-by: Gioh Kim <gioh.kim@lge.com>
Acked-by: Rafael Aquini <aquini@redhat.com>
---
drivers/virtio/virtio_balloon.c | 3 ++
include/linux/balloon_compaction.h | 15 ++++++--
mm/balloon_compaction.c | 72 ++++++++++++--------------------------
mm/compaction.c | 8 ++---
mm/migrate.c | 21 ++++++-----
5 files changed, 54 insertions(+), 65 deletions(-)
diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c
index 82e80e0..ef5b9b5 100644
--- a/drivers/virtio/virtio_balloon.c
+++ b/drivers/virtio/virtio_balloon.c
@@ -30,6 +30,7 @@
#include <linux/balloon_compaction.h>
#include <linux/oom.h>
#include <linux/wait.h>
+#include <linux/anon_inodes.h>
/*
* Balloon device works in 4K page units. So each page is pointed to by
@@ -505,6 +506,8 @@ static int virtballoon_probe(struct virtio_device *vdev)
balloon_devinfo_init(&vb->vb_dev_info);
#ifdef CONFIG_BALLOON_COMPACTION
vb->vb_dev_info.migratepage = virtballoon_migratepage;
+ vb->vb_dev_info.inode = anon_inode_new();
+ vb->vb_dev_info.inode->i_mapping->a_ops = &balloon_aops;
#endif
err = init_vqs(vb);
diff --git a/include/linux/balloon_compaction.h b/include/linux/balloon_compaction.h
index 9b0a15d..a9e0bde 100644
--- a/include/linux/balloon_compaction.h
+++ b/include/linux/balloon_compaction.h
@@ -48,6 +48,7 @@
#include <linux/migrate.h>
#include <linux/gfp.h>
#include <linux/err.h>
+#include <linux/fs.h>
/*
* Balloon device information descriptor.
@@ -62,6 +63,7 @@ struct balloon_dev_info {
struct list_head pages; /* Pages enqueued & handled to Host */
int (*migratepage)(struct balloon_dev_info *, struct page *newpage,
struct page *page, enum migrate_mode mode);
+ struct inode *inode;
};
extern struct page *balloon_page_enqueue(struct balloon_dev_info *b_dev_info);
@@ -73,12 +75,16 @@ static inline void balloon_devinfo_init(struct balloon_dev_info *balloon)
spin_lock_init(&balloon->pages_lock);
INIT_LIST_HEAD(&balloon->pages);
balloon->migratepage = NULL;
+ balloon->inode = NULL;
}
#ifdef CONFIG_BALLOON_COMPACTION
-extern bool balloon_page_isolate(struct page *page);
+extern const struct address_space_operations balloon_aops;
+extern bool balloon_page_isolate(struct page *page,
+ isolate_mode_t mode);
extern void balloon_page_putback(struct page *page);
-extern int balloon_page_migrate(struct page *newpage,
+extern int balloon_page_migrate(struct address_space *mapping,
+ struct page *newpage,
struct page *page, enum migrate_mode mode);
/*
@@ -124,6 +130,7 @@ static inline void balloon_page_insert(struct balloon_dev_info *balloon,
struct page *page)
{
__SetPageBalloon(page);
+ page->mapping = balloon->inode->i_mapping;
SetPagePrivate(page);
set_page_private(page, (unsigned long)balloon);
list_add(&page->lru, &balloon->pages);
@@ -140,6 +147,7 @@ static inline void balloon_page_insert(struct balloon_dev_info *balloon,
static inline void balloon_page_delete(struct page *page)
{
__ClearPageBalloon(page);
+ page->mapping = NULL;
set_page_private(page, 0);
if (PagePrivate(page)) {
ClearPagePrivate(page);
@@ -191,7 +199,8 @@ static inline bool isolated_balloon_page(struct page *page)
return false;
}
-static inline bool balloon_page_isolate(struct page *page)
+static inline bool balloon_page_isolate(struct page *page,
+ isolate_mode_t mode)
{
return false;
}
diff --git a/mm/balloon_compaction.c b/mm/balloon_compaction.c
index fcad832..8fbcf9c 100644
--- a/mm/balloon_compaction.c
+++ b/mm/balloon_compaction.c
@@ -131,43 +131,16 @@ static inline void __putback_balloon_page(struct page *page)
}
/* __isolate_lru_page() counterpart for a ballooned page */
-bool balloon_page_isolate(struct page *page)
+bool balloon_page_isolate(struct page *page, isolate_mode_t mode)
{
/*
- * Avoid burning cycles with pages that are yet under __free_pages(),
- * or just got freed under us.
- *
- * In case we 'win' a race for a balloon page being freed under us and
- * raise its refcount preventing __free_pages() from doing its job
- * the put_page() at the end of this block will take care of
- * release this page, thus avoiding a nasty leakage.
+ * A ballooned page, by default, has PagePrivate set.
+ * Prevent concurrent compaction threads from isolating
+ * an already isolated balloon page by clearing it.
*/
- if (likely(get_page_unless_zero(page))) {
- /*
- * As balloon pages are not isolated from LRU lists, concurrent
- * compaction threads can race against page migration functions
- * as well as race against the balloon driver releasing a page.
- *
- * In order to avoid having an already isolated balloon page
- * being (wrongly) re-isolated while it is under migration,
- * or to avoid attempting to isolate pages being released by
- * the balloon driver, lets be sure we have the page lock
- * before proceeding with the balloon page isolation steps.
- */
- if (likely(trylock_page(page))) {
- /*
- * A ballooned page, by default, has PagePrivate set.
- * Prevent concurrent compaction threads from isolating
- * an already isolated balloon page by clearing it.
- */
- if (balloon_page_movable(page)) {
- __isolate_balloon_page(page);
- unlock_page(page);
- return true;
- }
- unlock_page(page);
- }
- put_page(page);
+ if (balloon_page_movable(page)) {
+ __isolate_balloon_page(page);
+ return true;
}
return false;
}
@@ -175,37 +148,31 @@ bool balloon_page_isolate(struct page *page)
/* putback_lru_page() counterpart for a ballooned page */
void balloon_page_putback(struct page *page)
{
- /*
- * 'lock_page()' stabilizes the page and prevents races against
- * concurrent isolation threads attempting to re-isolate it.
- */
- lock_page(page);
+ if (!isolated_balloon_page(page))
+ return;
if (__is_movable_balloon_page(page)) {
__putback_balloon_page(page);
- /* drop the extra ref count taken for page isolation */
- put_page(page);
} else {
WARN_ON(1);
dump_page(page, "not movable balloon page");
}
- unlock_page(page);
}
/* move_to_new_page() counterpart for a ballooned page */
-int balloon_page_migrate(struct page *newpage,
+int balloon_page_migrate(struct address_space *mapping,
+ struct page *newpage,
struct page *page, enum migrate_mode mode)
{
struct balloon_dev_info *balloon = balloon_page_device(page);
int rc = -EAGAIN;
+ if (!isolated_balloon_page(page))
+ return rc;
+
/*
- * Block others from accessing the 'newpage' when we get around to
- * establishing additional references. We should be the only one
- * holding a reference to the 'newpage' at this point.
+ * newpage and page should be already locked.
*/
- BUG_ON(!trylock_page(newpage));
-
if (WARN_ON(!__is_movable_balloon_page(page))) {
dump_page(page, "not movable balloon page");
unlock_page(newpage);
@@ -215,7 +182,14 @@ int balloon_page_migrate(struct page *newpage,
if (balloon && balloon->migratepage)
rc = balloon->migratepage(balloon, newpage, page, mode);
- unlock_page(newpage);
return rc;
}
+
+/* define the balloon_mapping->a_ops callback to allow balloon page migration */
+const struct address_space_operations balloon_aops = {
+ .migratepage = balloon_page_migrate,
+ .isolatepage = balloon_page_isolate,
+ .putbackpage = balloon_page_putback,
+};
+EXPORT_SYMBOL_GPL(balloon_aops);
#endif /* CONFIG_BALLOON_COMPACTION */
diff --git a/mm/compaction.c b/mm/compaction.c
index 018f08d..60e4cbb 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -14,7 +14,7 @@
#include <linux/backing-dev.h>
#include <linux/sysctl.h>
#include <linux/sysfs.h>
-#include <linux/balloon_compaction.h>
+#include <linux/compaction.h>
#include <linux/page-isolation.h>
#include <linux/kasan.h>
#include "internal.h"
@@ -714,12 +714,12 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
/*
* Check may be lockless but that's ok as we recheck later.
- * It's possible to migrate LRU pages and balloon pages
+ * It's possible to migrate LRU pages and mobile pages
* Skip any other type of page
*/
if (!PageLRU(page)) {
- if (unlikely(balloon_page_movable(page))) {
- if (balloon_page_isolate(page)) {
+ if (unlikely(mobile_page(page))) {
+ if (isolate_mobilepage(page, isolate_mode)) {
/* Successfully isolated */
goto isolate_success;
}
diff --git a/mm/migrate.c b/mm/migrate.c
index ee401e4..53f0081d 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -35,7 +35,7 @@
#include <linux/hugetlb.h>
#include <linux/hugetlb_cgroup.h>
#include <linux/gfp.h>
-#include <linux/balloon_compaction.h>
+#include <linux/compaction.h>
#include <linux/mmu_notifier.h>
#include <asm/tlbflush.h>
@@ -76,7 +76,7 @@ int migrate_prep_local(void)
* from where they were once taken off for compaction/migration.
*
* This function shall be used whenever the isolated pageset has been
- * built from lru, balloon, hugetlbfs page. See isolate_migratepages_range()
+ * built from lru, mobile, hugetlbfs page. See isolate_migratepages_range()
* and isolate_huge_page().
*/
void putback_movable_pages(struct list_head *l)
@@ -92,8 +92,8 @@ void putback_movable_pages(struct list_head *l)
list_del(&page->lru);
dec_zone_page_state(page, NR_ISOLATED_ANON +
page_is_file_cache(page));
- if (unlikely(isolated_balloon_page(page)))
- balloon_page_putback(page);
+ if (unlikely(mobile_page(page)))
+ putback_mobilepage(page);
else
putback_lru_page(page);
}
@@ -844,15 +844,18 @@ static int __unmap_and_move(struct page *page, struct page *newpage,
}
}
- if (unlikely(isolated_balloon_page(page))) {
+ if (unlikely(mobile_page(page))) {
/*
- * A ballooned page does not need any special attention from
+ * A mobile page does not need any special attention from
* physical to virtual reverse mapping procedures.
* Skip any attempt to unmap PTEs or to remap swap cache,
* in order to avoid burning cycles at rmap level, and perform
* the page migration right away (proteced by page lock).
*/
- rc = balloon_page_migrate(newpage, page, mode);
+ lock_page(newpage);
+ rc = page->mapping->a_ops->migratepage(page->mapping,
+ newpage, page, mode);
+ unlock_page(newpage);
goto out_unlock;
}
@@ -962,8 +965,8 @@ out:
if (rc != MIGRATEPAGE_SUCCESS && put_new_page) {
ClearPageSwapBacked(newpage);
put_new_page(newpage, private);
- } else if (unlikely(__is_movable_balloon_page(newpage))) {
- /* drop our reference, page already in the balloon */
+ } else if (unlikely(mobile_page(newpage))) {
+ /* drop our reference */
put_page(newpage);
} else
putback_lru_page(newpage);
--
2.1.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [PATCH 2/4] mm/compaction: enable mobile-page migration
From: Gioh Kim @ 2015-07-13 8:35 UTC (permalink / raw)
To: jlayton, bfields, vbabka, iamjoonsoo.kim, viro, mst, koct9i,
minchan, aquini, linux-fsdevel, virtualization, linux-kernel,
linux-api, linux-mm
Cc: akpm, Gioh Kim, dri-devel, Gioh Kim
In-Reply-To: <1436776519-17337-1-git-send-email-gioh.kim@lge.com>
From: Gioh Kim <gurugio@hanmail.net>
Add framework to register callback functions and check page mobility.
There are some modes for page isolation so that isolate interface
has arguments of page address and isolation mode while putback
interface has only page address as argument.
Signed-off-by: Gioh Kim <gioh.kim@lge.com>
Acked-by: Rafael Aquini <aquini@redhat.com>
---
fs/proc/page.c | 3 ++
include/linux/compaction.h | 80 ++++++++++++++++++++++++++++++++++
include/linux/fs.h | 2 +
include/linux/page-flags.h | 19 ++++++++
include/uapi/linux/kernel-page-flags.h | 1 +
5 files changed, 105 insertions(+)
diff --git a/fs/proc/page.c b/fs/proc/page.c
index 7eee2d8..a4f5a00 100644
--- a/fs/proc/page.c
+++ b/fs/proc/page.c
@@ -146,6 +146,9 @@ u64 stable_page_flags(struct page *page)
if (PageBalloon(page))
u |= 1 << KPF_BALLOON;
+ if (PageMobile(page))
+ u |= 1 << KPF_MOBILE;
+
u |= kpf_copy_bit(k, KPF_LOCKED, PG_locked);
u |= kpf_copy_bit(k, KPF_SLAB, PG_slab);
diff --git a/include/linux/compaction.h b/include/linux/compaction.h
index aa8f61c..f693072 100644
--- a/include/linux/compaction.h
+++ b/include/linux/compaction.h
@@ -1,6 +1,9 @@
#ifndef _LINUX_COMPACTION_H
#define _LINUX_COMPACTION_H
+#include <linux/page-flags.h>
+#include <linux/pagemap.h>
+
/* Return values for compact_zone() and try_to_compact_pages() */
/* compaction didn't start as it was deferred due to past failures */
#define COMPACT_DEFERRED 0
@@ -51,6 +54,70 @@ extern void compaction_defer_reset(struct zone *zone, int order,
bool alloc_success);
extern bool compaction_restarting(struct zone *zone, int order);
+static inline bool mobile_page(struct page *page)
+{
+ return page->mapping && (PageMobile(page) || PageBalloon(page));
+}
+
+static inline bool isolate_mobilepage(struct page *page, isolate_mode_t mode)
+{
+ bool ret = false;
+
+ /*
+ * Avoid burning cycles with pages that are yet under __free_pages(),
+ * or just got freed under us.
+ *
+ * In case we 'win' a race for a mobile page being freed under us and
+ * raise its refcount preventing __free_pages() from doing its job
+ * the put_page() at the end of this block will take care of
+ * release this page, thus avoiding a nasty leakage.
+ */
+ if (unlikely(!get_page_unless_zero(page)))
+ goto out;
+
+ /*
+ * As mobile pages are not isolated from LRU lists, concurrent
+ * compaction threads can race against page migration functions
+ * as well as race against the releasing a page.
+ *
+ * In order to avoid having an already isolated mobile page
+ * being (wrongly) re-isolated while it is under migration,
+ * or to avoid attempting to isolate pages being released,
+ * lets be sure we have the page lock
+ * before proceeding with the mobile page isolation steps.
+ */
+ if (unlikely(!trylock_page(page)))
+ goto out_putpage;
+
+ if (!(mobile_page(page) && page->mapping->a_ops->isolatepage))
+ goto out_not_isolated;
+ ret = page->mapping->a_ops->isolatepage(page, mode);
+ if (!ret)
+ goto out_not_isolated;
+ unlock_page(page);
+ return ret;
+
+out_not_isolated:
+ unlock_page(page);
+out_putpage:
+ put_page(page);
+out:
+ return ret;
+}
+
+static inline void putback_mobilepage(struct page *page)
+{
+ /*
+ * 'lock_page()' stabilizes the page and prevents races against
+ * concurrent isolation threads attempting to re-isolate it.
+ */
+ lock_page(page);
+ if (page->mapping && page->mapping->a_ops->putbackpage)
+ page->mapping->a_ops->putbackpage(page);
+ unlock_page(page);
+ /* drop the extra ref count taken for mobile page isolation */
+ put_page(page);
+}
#else
static inline unsigned long try_to_compact_pages(gfp_t gfp_mask,
unsigned int order, int alloc_flags,
@@ -83,6 +150,19 @@ static inline bool compaction_deferred(struct zone *zone, int order)
return true;
}
+static inline bool mobile_page(struct page *page)
+{
+ return false;
+}
+
+static inline bool isolate_mobilepage(struct page *page, isolate_mode_t mode)
+{
+ return false;
+}
+
+static inline void putback_mobilepage(struct page *page)
+{
+}
#endif /* CONFIG_COMPACTION */
#if defined(CONFIG_COMPACTION) && defined(CONFIG_SYSFS) && defined(CONFIG_NUMA)
diff --git a/include/linux/fs.h b/include/linux/fs.h
index a0653e5..2cc4b24 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -396,6 +396,8 @@ struct address_space_operations {
*/
int (*migratepage) (struct address_space *,
struct page *, struct page *, enum migrate_mode);
+ bool (*isolatepage) (struct page *, isolate_mode_t);
+ void (*putbackpage) (struct page *);
int (*launder_page) (struct page *);
int (*is_partially_uptodate) (struct page *, unsigned long,
unsigned long);
diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
index f34e040..abef145 100644
--- a/include/linux/page-flags.h
+++ b/include/linux/page-flags.h
@@ -582,6 +582,25 @@ static inline void __ClearPageBalloon(struct page *page)
atomic_set(&page->_mapcount, -1);
}
+#define PAGE_MOBILE_MAPCOUNT_VALUE (-255)
+
+static inline int PageMobile(struct page *page)
+{
+ return atomic_read(&page->_mapcount) == PAGE_MOBILE_MAPCOUNT_VALUE;
+}
+
+static inline void __SetPageMobile(struct page *page)
+{
+ VM_BUG_ON_PAGE(atomic_read(&page->_mapcount) != -1, page);
+ atomic_set(&page->_mapcount, PAGE_MOBILE_MAPCOUNT_VALUE);
+}
+
+static inline void __ClearPageMobile(struct page *page)
+{
+ VM_BUG_ON_PAGE(!PageMobile(page), page);
+ atomic_set(&page->_mapcount, -1);
+}
+
/*
* If network-based swap is enabled, sl*b must keep track of whether pages
* were allocated from pfmemalloc reserves.
diff --git a/include/uapi/linux/kernel-page-flags.h b/include/uapi/linux/kernel-page-flags.h
index a6c4962..d50d9e8 100644
--- a/include/uapi/linux/kernel-page-flags.h
+++ b/include/uapi/linux/kernel-page-flags.h
@@ -33,6 +33,7 @@
#define KPF_THP 22
#define KPF_BALLOON 23
#define KPF_ZERO_PAGE 24
+#define KPF_MOBILE 25
#endif /* _UAPILINUX_KERNEL_PAGE_FLAGS_H */
--
2.1.4
^ permalink raw reply related
* [PATCH 1/4] fs/anon_inodes: new interface to create new inode
From: Gioh Kim @ 2015-07-13 8:35 UTC (permalink / raw)
To: jlayton, bfields, vbabka, iamjoonsoo.kim, viro, mst, koct9i,
minchan, aquini, linux-fsdevel, virtualization, linux-kernel,
linux-api, linux-mm
Cc: akpm, Gioh Kim, dri-devel, Gioh Kim
In-Reply-To: <1436776519-17337-1-git-send-email-gioh.kim@lge.com>
From: Gioh Kim <gurugio@hanmail.net>
The anon_inodes has already complete interfaces to create manage
many anonymous inodes but don't have interface to get
new inode. Other sub-modules can create anonymous inode
without creating and mounting it's own pseudo filesystem.
Signed-off-by: Gioh Kim <gioh.kim@lge.com>
Acked-by: Rafael Aquini <aquini@redhat.com>
---
fs/anon_inodes.c | 6 ++++++
include/linux/anon_inodes.h | 1 +
2 files changed, 7 insertions(+)
diff --git a/fs/anon_inodes.c b/fs/anon_inodes.c
index 80ef38c..1d51f96 100644
--- a/fs/anon_inodes.c
+++ b/fs/anon_inodes.c
@@ -162,6 +162,12 @@ err_put_unused_fd:
}
EXPORT_SYMBOL_GPL(anon_inode_getfd);
+struct inode *anon_inode_new(void)
+{
+ return alloc_anon_inode(anon_inode_mnt->mnt_sb);
+}
+EXPORT_SYMBOL_GPL(anon_inode_new);
+
static int __init anon_inode_init(void)
{
anon_inode_mnt = kern_mount(&anon_inode_fs_type);
diff --git a/include/linux/anon_inodes.h b/include/linux/anon_inodes.h
index 8013a45..ddbd67f 100644
--- a/include/linux/anon_inodes.h
+++ b/include/linux/anon_inodes.h
@@ -15,6 +15,7 @@ struct file *anon_inode_getfile(const char *name,
void *priv, int flags);
int anon_inode_getfd(const char *name, const struct file_operations *fops,
void *priv, int flags);
+struct inode *anon_inode_new(void);
#endif /* _LINUX_ANON_INODES_H */
--
2.1.4
^ permalink raw reply related
* [PATCH 0/4] enable migration of driver pages
From: Gioh Kim @ 2015-07-13 8:35 UTC (permalink / raw)
To: jlayton, bfields, vbabka, iamjoonsoo.kim, viro, mst, koct9i,
minchan, aquini, linux-fsdevel, virtualization, linux-kernel,
linux-api, linux-mm
Cc: dri-devel, akpm, Gioh Kim
From: Gioh Kim <gurugio@hanmail.net>
Hello,
This series try to enable migration of non-LRU pages, such as driver's page.
My ARM-based platform occured severe fragmentation problem after long-term
(several days) test. Sometimes even order-3 page allocation failed. It has
memory size 512MB ~ 1024MB. 30% ~ 40% memory is consumed for graphic processing
and 20~30 memory is reserved for zram.
I found that many pages of GPU driver and zram are non-movable pages. So I
reported Minchan Kim, the maintainer of zram, and he made the internal
compaction logic of zram. And I made the internal compaction of GPU driver.
They reduced some fragmentation but they are not enough effective.
They are activated by its own interface, /sys, so they are not cooperative
with kernel compaction. If there is too much fragmentation and kernel starts
to compaction, zram and GPU driver cannot work with the kernel compaction.
So I thought there needs a interface to combine driver and kernel compaction.
This patch adds a generic isolate/migrate/putback callbacks for page
address-space and a new interface to create anon-inode to manage
address_space_operation. The zram and GPU, and any other modules can create
anon_inode and register its own migration method. The kernel compaction can
call the registered migration when it does compaction.
My GPU driver source is not in-kernel driver so that I apply the interface
into balloon driver. The balloon driver is already merged
into the kernel compaction as a corner-case. This patch have the balloon
driver migration be called by the generic interface.
This patch set combines 4 patches.
1. patch 1/4: get inode from anon_inodes
This patch adds new interface to create inode from anon_inodes.
2. patch 2/4: framework to isolate/migrate/putback page
Add isolatepage, putbackpage into address_space_operations
and wrapper function to call them.
3. patch 3/4: apply the framework into balloon driver
The balloon driver is applied into the framework. It gets a inode
from anon_inodes and register operations in the inode.
The kernel compaction calls generic interfaces, not balloon
driver interfaces.
Any other drivers can register operations via inode like this
to migrate it's pages.
4. patch 4/4: remove direct calling of migration of driver pages
Non-lru pages are also migrated with lru pages by move_to_new_page().
This patch set is tested:
- turn on Ubuntu 14.04 with 1G memory on qemu.
- do kernel building
- after several seconds check more than 512MB is used with free command
- command "balloon 512" in qemu monitor
- check hundreds MB of pages are migrated
My thanks to Konstantin Khlebnikov for his reviews of the RFC patch set.
Most of the changes were based on his feedback.
This patch-set is based on v4.1
Gioh Kim (4):
fs/anon_inodes: new interface to create new inode
mm/compaction: enable mobile-page migration
mm/balloon: apply mobile page migratable into balloon
mm: remove direct calling of migration
drivers/virtio/virtio_balloon.c | 3 ++
fs/anon_inodes.c | 6 +++
fs/proc/page.c | 3 ++
include/linux/anon_inodes.h | 1 +
include/linux/balloon_compaction.h | 15 +++++--
include/linux/compaction.h | 80 ++++++++++++++++++++++++++++++++++
include/linux/fs.h | 2 +
include/linux/page-flags.h | 19 ++++++++
include/uapi/linux/kernel-page-flags.h | 1 +
mm/balloon_compaction.c | 72 ++++++++++--------------------
mm/compaction.c | 8 ++--
mm/migrate.c | 24 +++-------
12 files changed, 160 insertions(+), 74 deletions(-)
--
2.1.4
^ permalink raw reply
* [PATCH v3 10/10] mm: madvise allow remove operation for hugetlbfs
From: Mike Kravetz @ 2015-07-13 4:21 UTC (permalink / raw)
To: linux-mm, linux-kernel, linux-api
Cc: Dave Hansen, Naoya Horiguchi, David Rientjes, Hugh Dickins,
Davidlohr Bueso, Aneesh Kumar, Hillf Danton, Christoph Hellwig,
Andrew Morton, Michal Hocko, Mike Kravetz
In-Reply-To: <1436761268-6397-1-git-send-email-mike.kravetz@oracle.com>
Now that we have hole punching support for hugetlbfs, we can
also support the MADV_REMOVE interface to it.
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com>
---
mm/madvise.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/mm/madvise.c b/mm/madvise.c
index 70ce0d4..a235367 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -468,7 +468,7 @@ static long madvise_remove(struct vm_area_struct *vma,
*prev = NULL; /* tell sys_madvise we drop mmap_sem */
- if (vma->vm_flags & (VM_LOCKED | VM_HUGETLB))
+ if (vma->vm_flags & VM_LOCKED)
return -EINVAL;
f = vma->vm_file;
--
2.1.0
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [PATCH v3 09/10] hugetlbfs: add hugetlbfs_fallocate()
From: Mike Kravetz @ 2015-07-13 4:21 UTC (permalink / raw)
To: linux-mm, linux-kernel, linux-api
Cc: Dave Hansen, Naoya Horiguchi, David Rientjes, Hugh Dickins,
Davidlohr Bueso, Aneesh Kumar, Hillf Danton, Christoph Hellwig,
Andrew Morton, Michal Hocko, Mike Kravetz
In-Reply-To: <1436761268-6397-1-git-send-email-mike.kravetz@oracle.com>
This is based on the shmem version, but it has diverged quite
a bit. We have no swap to worry about, nor the new file sealing.
Add synchronication via the fault mutex table to coordinate
page faults, fallocate allocation and fallocate hole punch.
What this allows us to do is move physical memory in and out of
a hugetlbfs file without having it mapped. This also gives us
the ability to support MADV_REMOVE since it is currently
implemented using fallocate(). MADV_REMOVE lets madvise() remove
pages from the middle of a hugetlbfs file, which wasn't possible
before.
hugetlbfs fallocate only operates on whole huge pages.
Based-on code-by: Dave Hansen <dave.hansen@linux.intel.com>
Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com>
---
fs/hugetlbfs/inode.c | 158 +++++++++++++++++++++++++++++++++++++++++++++++-
include/linux/hugetlb.h | 3 +
mm/hugetlb.c | 2 +-
3 files changed, 161 insertions(+), 2 deletions(-)
diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c
index a974e4b..6e565a4 100644
--- a/fs/hugetlbfs/inode.c
+++ b/fs/hugetlbfs/inode.c
@@ -12,6 +12,7 @@
#include <linux/thread_info.h>
#include <asm/current.h>
#include <linux/sched.h> /* remove ASAP */
+#include <linux/falloc.h>
#include <linux/fs.h>
#include <linux/mount.h>
#include <linux/file.h>
@@ -479,6 +480,160 @@ static int hugetlb_vmtruncate(struct inode *inode, loff_t offset)
return 0;
}
+static long hugetlbfs_punch_hole(struct inode *inode, loff_t offset, loff_t len)
+{
+ struct hstate *h = hstate_inode(inode);
+ loff_t hpage_size = huge_page_size(h);
+ loff_t hole_start, hole_end;
+
+ /*
+ * For hole punch round up the beginning offset of the hole and
+ * round down the end.
+ */
+ hole_start = round_up(offset, hpage_size);
+ hole_end = round_down(offset + len, hpage_size);
+
+ if (hole_end > hole_start) {
+ struct address_space *mapping = inode->i_mapping;
+
+ mutex_lock(&inode->i_mutex);
+ i_mmap_lock_write(mapping);
+ if (!RB_EMPTY_ROOT(&mapping->i_mmap))
+ hugetlb_vmdelete_list(&mapping->i_mmap,
+ hole_start >> PAGE_SHIFT,
+ hole_end >> PAGE_SHIFT);
+ i_mmap_unlock_write(mapping);
+ remove_inode_hugepages(inode, hole_start, hole_end);
+ mutex_unlock(&inode->i_mutex);
+ }
+
+ return 0;
+}
+
+static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset,
+ loff_t len)
+{
+ struct inode *inode = file_inode(file);
+ struct address_space *mapping = inode->i_mapping;
+ struct hstate *h = hstate_inode(inode);
+ struct vm_area_struct pseudo_vma;
+ struct mm_struct *mm = current->mm;
+ loff_t hpage_size = huge_page_size(h);
+ unsigned long hpage_shift = huge_page_shift(h);
+ pgoff_t start, index, end;
+ int error;
+ u32 hash;
+
+ if (mode & ~(FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE))
+ return -EOPNOTSUPP;
+
+ if (mode & FALLOC_FL_PUNCH_HOLE)
+ return hugetlbfs_punch_hole(inode, offset, len);
+
+ /*
+ * Default preallocate case.
+ * For this range, start is rounded down and end is rounded up
+ * as well as being converted to page offsets.
+ */
+ start = offset >> hpage_shift;
+ end = (offset + len + hpage_size - 1) >> hpage_shift;
+
+ mutex_lock(&inode->i_mutex);
+
+ /* We need to check rlimit even when FALLOC_FL_KEEP_SIZE */
+ error = inode_newsize_ok(inode, offset + len);
+ if (error)
+ goto out;
+
+ /*
+ * Initialize a pseudo vma that just contains the policy used
+ * when allocating the huge pages. The actual policy field
+ * (vm_policy) is determined based on the index in the loop below.
+ */
+ memset(&pseudo_vma, 0, sizeof(struct vm_area_struct));
+ pseudo_vma.vm_flags = (VM_HUGETLB | VM_MAYSHARE | VM_SHARED);
+ pseudo_vma.vm_file = file;
+
+ for (index = start; index < end; index++) {
+ /*
+ * This is supposed to be the vaddr where the page is being
+ * faulted in, but we have no vaddr here.
+ */
+ struct page *page;
+ unsigned long addr;
+ int avoid_reserve = 0;
+
+ cond_resched();
+
+ /*
+ * fallocate(2) manpage permits EINTR; we may have been
+ * interrupted because we are using up too much memory.
+ */
+ if (signal_pending(current)) {
+ error = -EINTR;
+ break;
+ }
+
+ /* Get policy based on index */
+ pseudo_vma.vm_policy =
+ mpol_shared_policy_lookup(&HUGETLBFS_I(inode)->policy,
+ index);
+
+ /* addr is the offset within the file (zero based) */
+ addr = index * hpage_size;
+
+ /* mutex taken here, fault path and hole punch */
+ hash = hugetlb_fault_mutex_hash(h, mm, &pseudo_vma, mapping,
+ index, addr);
+ mutex_lock(&hugetlb_fault_mutex_table[hash]);
+
+ /* See if already present in mapping to avoid alloc/free */
+ page = find_get_page(mapping, index);
+ if (page) {
+ put_page(page);
+ mutex_unlock(&hugetlb_fault_mutex_table[hash]);
+ mpol_cond_put(pseudo_vma.vm_policy);
+ continue;
+ }
+
+ /* Allocate page and add to page cache */
+ page = alloc_huge_page(&pseudo_vma, addr, avoid_reserve);
+ mpol_cond_put(pseudo_vma.vm_policy);
+ if (IS_ERR(page)) {
+ mutex_unlock(&hugetlb_fault_mutex_table[hash]);
+ error = PTR_ERR(page);
+ goto out;
+ }
+ clear_huge_page(page, addr, pages_per_huge_page(h));
+ __SetPageUptodate(page);
+ error = huge_add_to_page_cache(page, mapping, index);
+ if (unlikely(error)) {
+ put_page(page);
+ mutex_unlock(&hugetlb_fault_mutex_table[hash]);
+ goto out;
+ }
+
+ mutex_unlock(&hugetlb_fault_mutex_table[hash]);
+
+ /*
+ * page_put due to reference from alloc_huge_page()
+ * unlock_page because locked by add_to_page_cache()
+ */
+ put_page(page);
+ unlock_page(page);
+ }
+
+ if (!(mode & FALLOC_FL_KEEP_SIZE) && offset + len > inode->i_size)
+ i_size_write(inode, offset + len);
+ inode->i_ctime = CURRENT_TIME;
+ spin_lock(&inode->i_lock);
+ inode->i_private = NULL;
+ spin_unlock(&inode->i_lock);
+out:
+ mutex_unlock(&inode->i_mutex);
+ return error;
+}
+
static int hugetlbfs_setattr(struct dentry *dentry, struct iattr *attr)
{
struct inode *inode = d_inode(dentry);
@@ -790,7 +945,8 @@ const struct file_operations hugetlbfs_file_operations = {
.mmap = hugetlbfs_file_mmap,
.fsync = noop_fsync,
.get_unmapped_area = hugetlb_get_unmapped_area,
- .llseek = default_llseek,
+ .llseek = default_llseek,
+ .fallocate = hugetlbfs_fallocate,
};
static const struct inode_operations hugetlbfs_dir_inode_operations = {
diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index 657ef26..386dcbf 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -330,6 +330,8 @@ struct huge_bootmem_page {
#endif
};
+struct page *alloc_huge_page(struct vm_area_struct *vma,
+ unsigned long addr, int avoid_reserve);
struct page *alloc_huge_page_node(struct hstate *h, int nid);
struct page *alloc_huge_page_noerr(struct vm_area_struct *vma,
unsigned long addr, int avoid_reserve);
@@ -483,6 +485,7 @@ static inline spinlock_t *huge_pte_lockptr(struct hstate *h,
#else /* CONFIG_HUGETLB_PAGE */
struct hstate {};
+#define alloc_huge_page(v, a, r) NULL
#define alloc_huge_page_node(h, nid) NULL
#define alloc_huge_page_noerr(v, a, r) NULL
#define alloc_bootmem_huge_page(h) NULL
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index aedc5e7..a775a65 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -1726,7 +1726,7 @@ static void vma_abort_reservation(struct hstate *h,
(void)__vma_reservation_common(h, vma, addr, VMA_ABORT_RESV);
}
-static struct page *alloc_huge_page(struct vm_area_struct *vma,
+struct page *alloc_huge_page(struct vm_area_struct *vma,
unsigned long addr, int avoid_reserve)
{
struct hugepage_subpool *spool = subpool_vma(vma);
--
2.1.0
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [PATCH v3 08/10] hugetlbfs: New huge_add_to_page_cache helper routine
From: Mike Kravetz @ 2015-07-13 4:21 UTC (permalink / raw)
To: linux-mm, linux-kernel, linux-api
Cc: Dave Hansen, Naoya Horiguchi, David Rientjes, Hugh Dickins,
Davidlohr Bueso, Aneesh Kumar, Hillf Danton, Christoph Hellwig,
Andrew Morton, Michal Hocko, Mike Kravetz
In-Reply-To: <1436761268-6397-1-git-send-email-mike.kravetz@oracle.com>
Currently, there is only a single place where hugetlbfs pages are
added to the page cache. The new fallocate code be adding a second
one, so break the functionality out into its own helper.
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com>
---
include/linux/hugetlb.h | 2 ++
mm/hugetlb.c | 27 ++++++++++++++++++---------
2 files changed, 20 insertions(+), 9 deletions(-)
diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index e7825c9..657ef26 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -333,6 +333,8 @@ struct huge_bootmem_page {
struct page *alloc_huge_page_node(struct hstate *h, int nid);
struct page *alloc_huge_page_noerr(struct vm_area_struct *vma,
unsigned long addr, int avoid_reserve);
+int huge_add_to_page_cache(struct page *page, struct address_space *mapping,
+ pgoff_t idx);
/* arch callback */
int __init alloc_bootmem_huge_page(struct hstate *h);
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 5a2ee06..aedc5e7 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3374,6 +3374,23 @@ static bool hugetlbfs_pagecache_present(struct hstate *h,
return page != NULL;
}
+int huge_add_to_page_cache(struct page *page, struct address_space *mapping,
+ pgoff_t idx)
+{
+ struct inode *inode = mapping->host;
+ struct hstate *h = hstate_inode(inode);
+ int err = add_to_page_cache(page, mapping, idx, GFP_KERNEL);
+
+ if (err)
+ return err;
+ ClearPagePrivate(page);
+
+ spin_lock(&inode->i_lock);
+ inode->i_blocks += blocks_per_huge_page(h);
+ spin_unlock(&inode->i_lock);
+ return 0;
+}
+
static int hugetlb_no_page(struct mm_struct *mm, struct vm_area_struct *vma,
struct address_space *mapping, pgoff_t idx,
unsigned long address, pte_t *ptep, unsigned int flags)
@@ -3421,21 +3438,13 @@ retry:
set_page_huge_active(page);
if (vma->vm_flags & VM_MAYSHARE) {
- int err;
- struct inode *inode = mapping->host;
-
- err = add_to_page_cache(page, mapping, idx, GFP_KERNEL);
+ int err = huge_add_to_page_cache(page, mapping, idx);
if (err) {
put_page(page);
if (err == -EEXIST)
goto retry;
goto out;
}
- ClearPagePrivate(page);
-
- spin_lock(&inode->i_lock);
- inode->i_blocks += blocks_per_huge_page(h);
- spin_unlock(&inode->i_lock);
} else {
lock_page(page);
if (unlikely(anon_vma_prepare(vma))) {
--
2.1.0
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [PATCH v3 07/10] mm/hugetlb: alloc_huge_page handle areas hole punched by fallocate
From: Mike Kravetz @ 2015-07-13 4:21 UTC (permalink / raw)
To: linux-mm, linux-kernel, linux-api
Cc: Dave Hansen, Naoya Horiguchi, David Rientjes, Hugh Dickins,
Davidlohr Bueso, Aneesh Kumar, Hillf Danton, Christoph Hellwig,
Andrew Morton, Michal Hocko, Mike Kravetz
In-Reply-To: <1436761268-6397-1-git-send-email-mike.kravetz@oracle.com>
Areas hole punched by fallocate will not have entries in the
region/reserve map. However, shared mappings with min_size subpool
reservations may still have reserved pages. alloc_huge_page needs
to handle this special case and do the proper accounting.
Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com>
---
mm/hugetlb.c | 54 +++++++++++++++++++++++++++++++++++++++---------------
1 file changed, 39 insertions(+), 15 deletions(-)
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index f72cb96..5a2ee06 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -1732,34 +1732,58 @@ static struct page *alloc_huge_page(struct vm_area_struct *vma,
struct hugepage_subpool *spool = subpool_vma(vma);
struct hstate *h = hstate_vma(vma);
struct page *page;
- long chg, commit;
+ long map_chg, map_commit;
+ long gbl_chg;
int ret, idx;
struct hugetlb_cgroup *h_cg;
idx = hstate_index(h);
/*
- * Processes that did not create the mapping will have no
- * reserves and will not have accounted against subpool
- * limit. Check that the subpool limit can be made before
- * satisfying the allocation MAP_NORESERVE mappings may also
- * need pages and subpool limit allocated allocated if no reserve
- * mapping overlaps.
+ * Examine the region/reserve map to determine if the process
+ * has a reservation for the page to be allocated. A return
+ * code of zero indicates a reservation exists (no change).
*/
- chg = vma_needs_reservation(h, vma, addr);
- if (chg < 0)
+ map_chg = gbl_chg = vma_needs_reservation(h, vma, addr);
+ if (map_chg < 0)
return ERR_PTR(-ENOMEM);
- if (chg || avoid_reserve)
- if (hugepage_subpool_get_pages(spool, 1) < 0) {
+
+ /*
+ * Processes that did not create the mapping will have no
+ * reserves as indicated by the region/reserve map. Check
+ * that the allocation will not exceed the subpool limit.
+ * Allocations for MAP_NORESERVE mappings also need to be
+ * checked against any subpool limit.
+ */
+ if (map_chg || avoid_reserve) {
+ gbl_chg = hugepage_subpool_get_pages(spool, 1);
+ if (gbl_chg < 0) {
vma_abort_reservation(h, vma, addr);
return ERR_PTR(-ENOSPC);
}
+ /*
+ * Even though there was no reservation in the region/reserve
+ * map, there could be reservations associated with the
+ * subpool that can be used. This would be indicated if the
+ * return value of hugepage_subpool_get_pages() is zero.
+ * However, if avoid_reserve is specified we still avoid even
+ * the subpool reservations.
+ */
+ if (avoid_reserve)
+ gbl_chg = 1;
+ }
+
ret = hugetlb_cgroup_charge_cgroup(idx, pages_per_huge_page(h), &h_cg);
if (ret)
goto out_subpool_put;
spin_lock(&hugetlb_lock);
- page = dequeue_huge_page_vma(h, vma, addr, avoid_reserve, chg);
+ /*
+ * glb_chg is passed to indicate whether or not a page must be taken
+ * from the global free pool (global change). gbl_chg == 0 indicates
+ * a reservation exists for the allocation.
+ */
+ page = dequeue_huge_page_vma(h, vma, addr, avoid_reserve, gbl_chg);
if (!page) {
spin_unlock(&hugetlb_lock);
page = alloc_buddy_huge_page(h, NUMA_NO_NODE);
@@ -1775,8 +1799,8 @@ static struct page *alloc_huge_page(struct vm_area_struct *vma,
set_page_private(page, (unsigned long)spool);
- commit = vma_commit_reservation(h, vma, addr);
- if (unlikely(chg > commit)) {
+ map_commit = vma_commit_reservation(h, vma, addr);
+ if (unlikely(map_chg > map_commit)) {
/*
* The page was added to the reservation map between
* vma_needs_reservation and vma_commit_reservation.
@@ -1796,7 +1820,7 @@ static struct page *alloc_huge_page(struct vm_area_struct *vma,
out_uncharge_cgroup:
hugetlb_cgroup_uncharge_cgroup(idx, pages_per_huge_page(h), h_cg);
out_subpool_put:
- if (chg || avoid_reserve)
+ if (map_chg || avoid_reserve)
hugepage_subpool_put_pages(spool, 1);
vma_abort_reservation(h, vma, addr);
return ERR_PTR(-ENOSPC);
--
2.1.0
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [PATCH v3 06/10] mm/hugetlb: vma_has_reserves() needs to handle fallocate hole punch
From: Mike Kravetz @ 2015-07-13 4:21 UTC (permalink / raw)
To: linux-mm, linux-kernel, linux-api
Cc: Dave Hansen, Naoya Horiguchi, David Rientjes, Hugh Dickins,
Davidlohr Bueso, Aneesh Kumar, Hillf Danton, Christoph Hellwig,
Andrew Morton, Michal Hocko, Mike Kravetz
In-Reply-To: <1436761268-6397-1-git-send-email-mike.kravetz@oracle.com>
In vma_has_reserves(), the current assumption is that reserves are
always present for shared mappings. However, this will not be the
case with fallocate hole punch. When punching a hole, the present
page will be deleted as well as the region/reserve map entry (and
hence any reservation). vma_has_reserves is passed "chg" which
indicates whether or not a region/reserve map is present. Use
this to determine if reserves are actually present or were removed
via hole punch.
Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com>
---
mm/hugetlb.c | 16 +++++++++++++---
1 file changed, 13 insertions(+), 3 deletions(-)
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index def39e3..f72cb96 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -801,9 +801,19 @@ static int vma_has_reserves(struct vm_area_struct *vma, long chg)
return 0;
}
- /* Shared mappings always use reserves */
- if (vma->vm_flags & VM_MAYSHARE)
- return 1;
+ if (vma->vm_flags & VM_MAYSHARE) {
+ /*
+ * We know VM_NORESERVE is not set. Therefore, there SHOULD
+ * be a region map for all pages. The only situation where
+ * there is no region map is if a hole was punched via
+ * fallocate. In this case, there really are no reverves to
+ * use. This situation is indicated if chg != 0.
+ */
+ if (chg)
+ return 0;
+ else
+ return 1;
+ }
/*
* Only the process that called mmap() has reserves for
--
2.1.0
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [PATCH v3 05/10] hugetlbfs: truncate_hugepages() takes a range of pages
From: Mike Kravetz @ 2015-07-13 4:21 UTC (permalink / raw)
To: linux-mm, linux-kernel, linux-api
Cc: Dave Hansen, Naoya Horiguchi, David Rientjes, Hugh Dickins,
Davidlohr Bueso, Aneesh Kumar, Hillf Danton, Christoph Hellwig,
Andrew Morton, Michal Hocko, Mike Kravetz
In-Reply-To: <1436761268-6397-1-git-send-email-mike.kravetz@oracle.com>
Modify truncate_hugepages() to take a range of pages (start, end)
instead of simply start. If an end value of LLONG_MAX is passed,
the current "truncate" functionality is maintained. Existing
callers are modified to pass LLONG_MAX as end of range. By keying
off end == LLONG_MAX, the routine behaves differently for truncate
and hole punch. Page removal is now synchronized with page
allocation via faults by using the fault mutex table. The hole
punch case can experience the rare region_del error and must
handle accordingly.
Add the routine hugetlb_fix_reserve_counts to fix up reserve counts
in the case where region_del returns an error.
Since the routine handles more than just the truncate case, it is
renamed to remove_inode_hugepages(). To be consistent, the routine
truncate_huge_page() is renamed remove_huge_page().
Downstream of remove_inode_hugepages(), the routine
hugetlb_unreserve_pages() is also modified to take a range of pages.
hugetlb_unreserve_pages is modified to detect an error from
region_del and pass it back to the caller.
Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com>
---
fs/hugetlbfs/inode.c | 98 ++++++++++++++++++++++++++++++++++++++++++++-----
include/linux/hugetlb.h | 4 +-
mm/hugetlb.c | 40 ++++++++++++++++++--
3 files changed, 128 insertions(+), 14 deletions(-)
diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c
index ed40f56..a974e4b 100644
--- a/fs/hugetlbfs/inode.c
+++ b/fs/hugetlbfs/inode.c
@@ -293,26 +293,61 @@ static int hugetlbfs_write_end(struct file *file, struct address_space *mapping,
return -EINVAL;
}
-static void truncate_huge_page(struct page *page)
+static void remove_huge_page(struct page *page)
{
ClearPageDirty(page);
ClearPageUptodate(page);
delete_from_page_cache(page);
}
-static void truncate_hugepages(struct inode *inode, loff_t lstart)
+
+/*
+ * remove_inode_hugepages handles two distinct cases: truncation and hole
+ * punch. There are subtle differences in operation for each case.
+
+ * truncation is indicated by end of range being LLONG_MAX
+ * In this case, we first scan the range and release found pages.
+ * After releasing pages, hugetlb_unreserve_pages cleans up region/reserv
+ * maps and global counts.
+ * hole punch is indicated if end is not LLONG_MAX
+ * In the hole punch case we scan the range and release found pages.
+ * Only when releasing a page is the associated region/reserv map
+ * deleted. The region/reserv map for ranges without associated
+ * pages are not modified.
+ * Note: If the passed end of range value is beyond the end of file, but
+ * not LLONG_MAX this routine still performs a hole punch operation.
+ */
+static void remove_inode_hugepages(struct inode *inode, loff_t lstart,
+ loff_t lend)
{
struct hstate *h = hstate_inode(inode);
struct address_space *mapping = &inode->i_data;
const pgoff_t start = lstart >> huge_page_shift(h);
+ const pgoff_t end = lend >> huge_page_shift(h);
+ struct vm_area_struct pseudo_vma;
struct pagevec pvec;
pgoff_t next;
int i, freed = 0;
+ long lookup_nr = PAGEVEC_SIZE;
+ bool truncate_op = (lend == LLONG_MAX);
+ memset(&pseudo_vma, 0, sizeof(struct vm_area_struct));
+ pseudo_vma.vm_flags = (VM_HUGETLB | VM_MAYSHARE | VM_SHARED);
pagevec_init(&pvec, 0);
next = start;
- while (1) {
- if (!pagevec_lookup(&pvec, mapping, next, PAGEVEC_SIZE)) {
+ while (next < end) {
+ /*
+ * Make sure to never grab more pages that we
+ * might possibly need.
+ */
+ if (end - next < lookup_nr)
+ lookup_nr = end - next;
+
+ /*
+ * This pagevec_lookup() may return pages past 'end',
+ * so we must check for page->index > end.
+ */
+ if (!pagevec_lookup(&pvec, mapping, next, lookup_nr)) {
if (next == start)
break;
next = start;
@@ -321,26 +356,69 @@ static void truncate_hugepages(struct inode *inode, loff_t lstart)
for (i = 0; i < pagevec_count(&pvec); ++i) {
struct page *page = pvec.pages[i];
+ u32 hash;
+
+ hash = hugetlb_fault_mutex_hash(h, current->mm,
+ &pseudo_vma,
+ mapping, next, 0);
+ mutex_lock(&hugetlb_fault_mutex_table[hash]);
lock_page(page);
+ if (page->index >= end) {
+ unlock_page(page);
+ mutex_unlock(&hugetlb_fault_mutex_table[hash]);
+ next = end; /* we are done */
+ break;
+ }
+
+ /*
+ * If page is mapped, it was faulted in after being
+ * unmapped. Do nothing in this race case. In the
+ * normal case page is not mapped.
+ */
+ if (!page_mapped(page)) {
+ bool rsv_on_error = !PagePrivate(page);
+ /*
+ * We must free the huge page and remove
+ * from page cache (remove_huge_page) BEFORE
+ * removing the region/reserve map
+ * (hugetlb_unreserve_pages). In rare out
+ * of memory conditions, removal of the
+ * region/reserve map could fail. Before
+ * free'ing the page, note PagePrivate which
+ * is used in case of error.
+ */
+ remove_huge_page(page);
+ freed++;
+ if (!truncate_op) {
+ if (unlikely(hugetlb_unreserve_pages(
+ inode, next,
+ next + 1, 1)))
+ hugetlb_fix_reserve_counts(
+ inode, rsv_on_error);
+ }
+ }
+
if (page->index > next)
next = page->index;
+
++next;
- truncate_huge_page(page);
unlock_page(page);
- freed++;
+
+ mutex_unlock(&hugetlb_fault_mutex_table[hash]);
}
huge_pagevec_release(&pvec);
}
- BUG_ON(!lstart && mapping->nrpages);
- hugetlb_unreserve_pages(inode, start, freed);
+
+ if (truncate_op)
+ (void)hugetlb_unreserve_pages(inode, start, LONG_MAX, freed);
}
static void hugetlbfs_evict_inode(struct inode *inode)
{
struct resv_map *resv_map;
- truncate_hugepages(inode, 0);
+ remove_inode_hugepages(inode, 0, LLONG_MAX);
resv_map = (struct resv_map *)inode->i_mapping->private_data;
/* root inode doesn't have the resv_map, so we should check it */
if (resv_map)
@@ -397,7 +475,7 @@ static int hugetlb_vmtruncate(struct inode *inode, loff_t offset)
if (!RB_EMPTY_ROOT(&mapping->i_mmap))
hugetlb_vmdelete_list(&mapping->i_mmap, pgoff, 0);
i_mmap_unlock_write(mapping);
- truncate_hugepages(inode, offset);
+ remove_inode_hugepages(inode, offset, LLONG_MAX);
return 0;
}
diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index 933da39..e7825c9 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -83,11 +83,13 @@ int hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,
int hugetlb_reserve_pages(struct inode *inode, long from, long to,
struct vm_area_struct *vma,
vm_flags_t vm_flags);
-void hugetlb_unreserve_pages(struct inode *inode, long offset, long freed);
+long hugetlb_unreserve_pages(struct inode *inode, long start, long end,
+ long freed);
int dequeue_hwpoisoned_huge_page(struct page *page);
bool isolate_huge_page(struct page *page, struct list_head *list);
void putback_active_hugepage(struct page *page);
void free_huge_page(struct page *page);
+void hugetlb_fix_reserve_counts(struct inode *inode, bool restore_reserve);
extern struct mutex *hugetlb_fault_mutex_table;
u32 hugetlb_fault_mutex_hash(struct hstate *h, struct mm_struct *mm,
struct vm_area_struct *vma,
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 52c2801..def39e3 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -548,6 +548,28 @@ retry:
}
/*
+ * A rare out of memory error was encountered which prevented removal of
+ * the reserve map region for a page. The huge page itself was free'ed
+ * and removed from the page cache. This routine will adjust the subpool
+ * usage count, and the global reserve count if needed. By incrementing
+ * these counts, the reserve map entry which could not be deleted will
+ * appear as a "reserved" entry instead of simply dangling with incorrect
+ * counts.
+ */
+void hugetlb_fix_reserve_counts(struct inode *inode, bool restore_reserve)
+{
+ struct hugepage_subpool *spool = subpool_inode(inode);
+ long rsv_adjust;
+
+ rsv_adjust = hugepage_subpool_get_pages(spool, 1);
+ if (restore_reserve && rsv_adjust) {
+ struct hstate *h = hstate_inode(inode);
+
+ hugetlb_acct_memory(h, 1);
+ }
+}
+
+/*
* Count and return the number of huge pages in the reserve map
* that intersect with the range [f, t).
*/
@@ -3909,7 +3931,8 @@ out_err:
return ret;
}
-void hugetlb_unreserve_pages(struct inode *inode, long offset, long freed)
+long hugetlb_unreserve_pages(struct inode *inode, long start, long end,
+ long freed)
{
struct hstate *h = hstate_inode(inode);
struct resv_map *resv_map = inode_resv_map(inode);
@@ -3917,8 +3940,17 @@ void hugetlb_unreserve_pages(struct inode *inode, long offset, long freed)
struct hugepage_subpool *spool = subpool_inode(inode);
long gbl_reserve;
- if (resv_map)
- chg = region_del(resv_map, offset, LONG_MAX);
+ if (resv_map) {
+ chg = region_del(resv_map, start, end);
+ /*
+ * region_del() can fail in the rare case where a region
+ * must be split and another region descriptor can not be
+ * allocated. If end == LONG_MAX, it will not fail.
+ */
+ if (chg < 0)
+ return chg;
+ }
+
spin_lock(&inode->i_lock);
inode->i_blocks -= (blocks_per_huge_page(h) * freed);
spin_unlock(&inode->i_lock);
@@ -3929,6 +3961,8 @@ void hugetlb_unreserve_pages(struct inode *inode, long offset, long freed)
*/
gbl_reserve = hugepage_subpool_put_pages(spool, (chg - freed));
hugetlb_acct_memory(h, -gbl_reserve);
+
+ return 0;
}
#ifdef CONFIG_ARCH_WANT_HUGE_PMD_SHARE
--
2.1.0
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [PATCH v3 04/10] hugetlbfs: hugetlb_vmtruncate_list() needs to take a range to delete
From: Mike Kravetz @ 2015-07-13 4:21 UTC (permalink / raw)
To: linux-mm, linux-kernel, linux-api
Cc: Dave Hansen, Naoya Horiguchi, David Rientjes, Hugh Dickins,
Davidlohr Bueso, Aneesh Kumar, Hillf Danton, Christoph Hellwig,
Andrew Morton, Michal Hocko, Mike Kravetz
In-Reply-To: <1436761268-6397-1-git-send-email-mike.kravetz@oracle.com>
fallocate hole punch will want to unmap a specific range of pages.
Modify the existing hugetlb_vmtruncate_list() routine to take a
start/end range. If end is 0, this indicates all pages after start
should be unmapped. This is the same as the existing truncate
functionality. Modify existing callers to add 0 as end of range.
Since the routine will be used in hole punch as well as truncate
operations, it is more appropriately renamed to hugetlb_vmdelete_list().
Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com>
---
fs/hugetlbfs/inode.c | 25 ++++++++++++++++++-------
1 file changed, 18 insertions(+), 7 deletions(-)
diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c
index 0cf74df..ed40f56 100644
--- a/fs/hugetlbfs/inode.c
+++ b/fs/hugetlbfs/inode.c
@@ -349,11 +349,15 @@ static void hugetlbfs_evict_inode(struct inode *inode)
}
static inline void
-hugetlb_vmtruncate_list(struct rb_root *root, pgoff_t pgoff)
+hugetlb_vmdelete_list(struct rb_root *root, pgoff_t start, pgoff_t end)
{
struct vm_area_struct *vma;
- vma_interval_tree_foreach(vma, root, pgoff, ULONG_MAX) {
+ /*
+ * end == 0 indicates that the entire range after
+ * start should be unmapped.
+ */
+ vma_interval_tree_foreach(vma, root, start, end ? end : ULONG_MAX) {
unsigned long v_offset;
/*
@@ -362,13 +366,20 @@ hugetlb_vmtruncate_list(struct rb_root *root, pgoff_t pgoff)
* which overlap the truncated area starting at pgoff,
* and no vma on a 32-bit arch can span beyond the 4GB.
*/
- if (vma->vm_pgoff < pgoff)
- v_offset = (pgoff - vma->vm_pgoff) << PAGE_SHIFT;
+ if (vma->vm_pgoff < start)
+ v_offset = (start - vma->vm_pgoff) << PAGE_SHIFT;
else
v_offset = 0;
- unmap_hugepage_range(vma, vma->vm_start + v_offset,
- vma->vm_end, NULL);
+ if (end) {
+ end = ((end - start) << PAGE_SHIFT) +
+ vma->vm_start + v_offset;
+ if (end > vma->vm_end)
+ end = vma->vm_end;
+ } else
+ end = vma->vm_end;
+
+ unmap_hugepage_range(vma, vma->vm_start + v_offset, end, NULL);
}
}
@@ -384,7 +395,7 @@ static int hugetlb_vmtruncate(struct inode *inode, loff_t offset)
i_size_write(inode, offset);
i_mmap_lock_write(mapping);
if (!RB_EMPTY_ROOT(&mapping->i_mmap))
- hugetlb_vmtruncate_list(&mapping->i_mmap, pgoff);
+ hugetlb_vmdelete_list(&mapping->i_mmap, pgoff, 0);
i_mmap_unlock_write(mapping);
truncate_hugepages(inode, offset);
return 0;
--
2.1.0
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox