* Re: [patch 2/2] MM: allow per-cpu vmstat_threshold and vmstat_worker configuration
From: Christoph Lameter @ 2017-05-16 13:37 UTC (permalink / raw)
To: Marcelo Tosatti
Cc: Luiz Capitulino, linux-kernel, linux-mm, Rik van Riel,
Linux RT Users, cmetcalf
In-Reply-To: <20170515191531.GA31483@amt.cnet>
On Mon, 15 May 2017, Marcelo Tosatti wrote:
> > NOHZ already does that. I wanted to know what your problem is that you
> > see. The latency issue has already been solved as far as I can tell .
> > Please tell me why the existing solutions are not sufficient for you.
>
> We don't want vmstat_worker to execute on a given CPU, even if the local
> CPU updates vm-statistics.
Instead of responding you repeat describing what you want.
> Because:
>
> vmstat_worker increases latency of the application
> (i can measure it if you want on a given CPU,
> how many ns's the following takes:
That still is no use case. Just a measurement of vmstat_worker. Pointless.
If you move the latency from the vmstat worker into the code thats
updating the counters then you will require increased use of atomics
which will increase contention which in turn will significantly
increase the overall latency.
> Why the existing solutions are not sufficient:
>
> 1) task-isolation patchset seems too heavy for our usecase (we do
> want IPIs, signals, etc).
Ok then minor delays from remote random events are tolerable?
Then you can also have a vmstat update.
> So this seems a little heavy for our usecase.
Sorry all of this does not make sense to me. Maybe get some numbers of of
an app with intensive OS access running with atomics vs vmstat worker?
NOHZ currently disables the vmstat worker when no updates occur. This is
applicable to DPDK and will provide a quiet vmstat worker free environment
if no statistics activity is occurring.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH] drm: use kvmalloc_array for drm_malloc*
From: Michal Hocko @ 2017-05-16 13:21 UTC (permalink / raw)
To: Daniel Vetter
Cc: dri-devel, linux-kernel, linux-mm, Daniel Vetter, Jani Nikula,
Sean Paul, David Airlie
In-Reply-To: <20170516130856.hvq62uuq6wmnhvpg@phenom.ffwll.local>
On Tue 16-05-17 15:08:56, Daniel Vetter wrote:
> On Tue, May 16, 2017 at 11:52:55AM +0200, Michal Hocko wrote:
> > On Tue 16-05-17 11:22:30, Daniel Vetter wrote:
> > > On Tue, May 16, 2017 at 11:06:06AM +0200, Michal Hocko wrote:
> > > > From: Michal Hocko <mhocko@suse.com>
> > > >
> > > > drm_malloc* has grown their own kmalloc with vmalloc fallback
> > > > implementations. MM has grown kvmalloc* helpers in the meantime. Let's
> > > > use those because it a) reduces the code and b) MM has a better idea
> > > > how to implement fallbacks (e.g. do not vmalloc before kmalloc is tried
> > > > with __GFP_NORETRY).
> > > >
> > > > Signed-off-by: Michal Hocko <mhocko@suse.com>
> > >
> > > Shouldn't we go one step further and just remove these wrappers, maybe
> > > with cocci?
> >
> > my cocci sucks...
> >
> > > Especially drm_malloc_gfp is surpremely pointless after this
> > > patch (and drm_malloc_ab probably not that useful either).
> >
> > So what about the following instead? It passes allyesconfig compilation.
>
> Yeah, looks good, but perhaps rebased onto your first patch. That way we
> split the functional change from the refactor (not the first time innocent
> looking changes in i915 gem code resulted in surprises).
OK, I will split it.
> Your patch also seems to need some stuff from -rc1, and atm drm-misc is
> still pre-rc1, so I'll pull both patches in once that's sorted (I can do
> the rebase myself, since it's rather trivial). But pls remind me in case
> it falls through the cracks and isn't in linux-next by end of this week
> :-)
I have based it on top of the current linux next (next-20170516). Let me
know if other tree is more appropriate.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH] mm: per-cgroup memory reclaim stats
From: Roman Gushchin @ 2017-05-16 13:13 UTC (permalink / raw)
To: Michal Hocko
Cc: Johannes Weiner, Tejun Heo, Li Zefan, Vladimir Davydov, cgroups,
linux-doc, linux-kernel, linux-mm
In-Reply-To: <20170516092956.GF2481@dhcp22.suse.cz>
Hi Michal!
On Tue, May 16, 2017 at 11:29:56AM +0200, Michal Hocko wrote:
> On Thu 11-05-17 20:16:23, Roman Gushchin wrote:
> > Track the following reclaim counters for every memory cgroup:
> > PGREFILL, PGSCAN, PGSTEAL, PGACTIVATE, PGDEACTIVATE, PGLAZYFREE and
> > PGLAZYFREED.
>
> yes, those are definitely useful. I have an old patch to add them as
> well but never managed to clean it up and post...
>
> > These values are exposed using the memory.stats interface of cgroup v2.
>
> Is there any reason to not add them to v1?
Not really, I'm just not sure, if it worth it to change v1 interface here.
If you want, I can add them.
> This should be rather trivial after recent changes from Johannes.
If you're about memcg1_events[]/memcg1_event_names[], they can't be reused,
because the pgscan and pgsteal values are both sums of direct and kswapd values:
e.g. events[PGSTEAL_KSWAPD] + events[PGSTEAL_DIRECT].
>
> > The meaning of each value is the same as for global counters,
> > available using /proc/vmstat.
> >
> > Also, for consistency, rename mem_cgroup_count_vm_event() to
> > count_memcg_event_mm().
> >
> > Signed-off-by: Roman Gushchin <guro@fb.com>
> > Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
> > Cc: Johannes Weiner <hannes@cmpxchg.org>
> > Cc: Tejun Heo <tj@kernel.org>
> > Cc: Li Zefan <lizefan@huawei.com>
> > Cc: Michal Hocko <mhocko@kernel.org>
> > Cc: Vladimir Davydov <vdavydov.dev@gmail.com>
> > Cc: cgroups@vger.kernel.org
> > Cc: linux-doc@vger.kernel.org
> > Cc: linux-kernel@vger.kernel.org
> > Cc: linux-mm@kvack.org
>
> the patch itself looks good to me. I will have to double check it after
> I am done with what I am doing currently and then will add my Acked-by
Thank you!
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH] drm: use kvmalloc_array for drm_malloc*
From: Daniel Vetter @ 2017-05-16 13:08 UTC (permalink / raw)
To: Michal Hocko
Cc: Daniel Vetter, dri-devel, linux-kernel, linux-mm, Daniel Vetter,
Jani Nikula, Sean Paul, David Airlie
In-Reply-To: <20170516095254.GG2481@dhcp22.suse.cz>
On Tue, May 16, 2017 at 11:52:55AM +0200, Michal Hocko wrote:
> On Tue 16-05-17 11:22:30, Daniel Vetter wrote:
> > On Tue, May 16, 2017 at 11:06:06AM +0200, Michal Hocko wrote:
> > > From: Michal Hocko <mhocko@suse.com>
> > >
> > > drm_malloc* has grown their own kmalloc with vmalloc fallback
> > > implementations. MM has grown kvmalloc* helpers in the meantime. Let's
> > > use those because it a) reduces the code and b) MM has a better idea
> > > how to implement fallbacks (e.g. do not vmalloc before kmalloc is tried
> > > with __GFP_NORETRY).
> > >
> > > Signed-off-by: Michal Hocko <mhocko@suse.com>
> >
> > Shouldn't we go one step further and just remove these wrappers, maybe
> > with cocci?
>
> my cocci sucks...
>
> > Especially drm_malloc_gfp is surpremely pointless after this
> > patch (and drm_malloc_ab probably not that useful either).
>
> So what about the following instead? It passes allyesconfig compilation.
Yeah, looks good, but perhaps rebased onto your first patch. That way we
split the functional change from the refactor (not the first time innocent
looking changes in i915 gem code resulted in surprises).
Your patch also seems to need some stuff from -rc1, and atm drm-misc is
still pre-rc1, so I'll pull both patches in once that's sorted (I can do
the rebase myself, since it's rather trivial). But pls remind me in case
it falls through the cracks and isn't in linux-next by end of this week
:-)
Thanks, Daniel
> ---
> From 6325ff674d9800e5648b9a86fdd36c9d5de9692f Mon Sep 17 00:00:00 2001
> From: Michal Hocko <mhocko@suse.com>
> Date: Tue, 16 May 2017 11:00:47 +0200
> Subject: [PATCH] drm: replace drm_[cm]alloc* by kvmalloc alternatives
>
> drm_[cm]alloc* has grown their own kmalloc with vmalloc fallback
> implementations. MM has grown kvmalloc* helpers in the meantime. Let's
> use those because it a) reduces the code and b) MM has a better idea
> how to implement fallbacks (e.g. do not vmalloc before kmalloc is tried
> with __GFP_NORETRY).
>
> drm_calloc_large needs to get __GFP_ZERO explicitly.
>
> drm_free_large can be replaced by kvfree directly.
>
> Signed-off-by: Michal Hocko <mhocko@suse.com>
> ---
> drivers/gpu/drm/amd/amdgpu/amdgpu_bo_list.c | 16 ++---
> drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 19 +++---
> drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 7 +-
> drivers/gpu/drm/drm_gem.c | 6 +-
> drivers/gpu/drm/etnaviv/etnaviv_gem.c | 12 ++--
> drivers/gpu/drm/etnaviv/etnaviv_gem_prime.c | 4 +-
> drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c | 12 ++--
> drivers/gpu/drm/exynos/exynos_drm_gem.c | 11 +--
> drivers/gpu/drm/i915/i915_debugfs.c | 4 +-
> drivers/gpu/drm/i915/i915_gem.c | 4 +-
> drivers/gpu/drm/i915/i915_gem_execbuffer.c | 34 +++++-----
> drivers/gpu/drm/i915/i915_gem_gtt.c | 6 +-
> drivers/gpu/drm/i915/i915_gem_userptr.c | 8 +--
> drivers/gpu/drm/i915/selftests/intel_breadcrumbs.c | 12 ++--
> drivers/gpu/drm/msm/msm_gem.c | 10 +--
> drivers/gpu/drm/radeon/radeon_cs.c | 11 +--
> drivers/gpu/drm/radeon/radeon_gem.c | 2 +-
> drivers/gpu/drm/radeon/radeon_ring.c | 4 +-
> drivers/gpu/drm/radeon/radeon_vm.c | 4 +-
> drivers/gpu/drm/ttm/ttm_tt.c | 13 ++--
> drivers/gpu/drm/udl/udl_dmabuf.c | 2 +-
> drivers/gpu/drm/udl/udl_gem.c | 2 +-
> drivers/gpu/drm/vc4/vc4_gem.c | 15 +++--
> drivers/gpu/drm/virtio/virtgpu_ioctl.c | 27 ++++----
> include/drm/drmP.h | 1 -
> include/drm/drm_mem_util.h | 78 ----------------------
> 26 files changed, 126 insertions(+), 198 deletions(-)
> delete mode 100644 include/drm/drm_mem_util.h
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_bo_list.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_bo_list.c
> index a6649874e6ce..9f0247cdda5e 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_bo_list.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_bo_list.c
> @@ -96,7 +96,7 @@ static int amdgpu_bo_list_set(struct amdgpu_device *adev,
> int r;
> unsigned long total_size = 0;
>
> - array = drm_malloc_ab(num_entries, sizeof(struct amdgpu_bo_list_entry));
> + array = kvmalloc_array(num_entries, sizeof(struct amdgpu_bo_list_entry), GFP_KERNEL);
> if (!array)
> return -ENOMEM;
> memset(array, 0, num_entries * sizeof(struct amdgpu_bo_list_entry));
> @@ -148,7 +148,7 @@ static int amdgpu_bo_list_set(struct amdgpu_device *adev,
> for (i = 0; i < list->num_entries; ++i)
> amdgpu_bo_unref(&list->array[i].robj);
>
> - drm_free_large(list->array);
> + kvfree(list->array);
>
> list->gds_obj = gds_obj;
> list->gws_obj = gws_obj;
> @@ -163,7 +163,7 @@ static int amdgpu_bo_list_set(struct amdgpu_device *adev,
> error_free:
> while (i--)
> amdgpu_bo_unref(&array[i].robj);
> - drm_free_large(array);
> + kvfree(array);
> return r;
> }
>
> @@ -224,7 +224,7 @@ void amdgpu_bo_list_free(struct amdgpu_bo_list *list)
> amdgpu_bo_unref(&list->array[i].robj);
>
> mutex_destroy(&list->lock);
> - drm_free_large(list->array);
> + kvfree(list->array);
> kfree(list);
> }
>
> @@ -244,8 +244,8 @@ int amdgpu_bo_list_ioctl(struct drm_device *dev, void *data,
>
> int r;
>
> - info = drm_malloc_ab(args->in.bo_number,
> - sizeof(struct drm_amdgpu_bo_list_entry));
> + info = kvmalloc_array(args->in.bo_number,
> + sizeof(struct drm_amdgpu_bo_list_entry), GFP_KERNEL);
> if (!info)
> return -ENOMEM;
>
> @@ -311,11 +311,11 @@ int amdgpu_bo_list_ioctl(struct drm_device *dev, void *data,
>
> memset(args, 0, sizeof(*args));
> args->out.list_handle = handle;
> - drm_free_large(info);
> + kvfree(info);
>
> return 0;
>
> error_free:
> - drm_free_large(info);
> + kvfree(info);
> return r;
> }
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c
> index 4e6b9501ab0a..5b3e0f63a115 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c
> @@ -194,7 +194,7 @@ int amdgpu_cs_parser_init(struct amdgpu_cs_parser *p, void *data)
> size = p->chunks[i].length_dw;
> cdata = (void __user *)(uintptr_t)user_chunk.chunk_data;
>
> - p->chunks[i].kdata = drm_malloc_ab(size, sizeof(uint32_t));
> + p->chunks[i].kdata = kvmalloc_array(size, sizeof(uint32_t), GFP_KERNEL);
> if (p->chunks[i].kdata == NULL) {
> ret = -ENOMEM;
> i--;
> @@ -247,7 +247,7 @@ int amdgpu_cs_parser_init(struct amdgpu_cs_parser *p, void *data)
> i = p->nchunks - 1;
> free_partial_kdata:
> for (; i >= 0; i--)
> - drm_free_large(p->chunks[i].kdata);
> + kvfree(p->chunks[i].kdata);
> kfree(p->chunks);
> p->chunks = NULL;
> p->nchunks = 0;
> @@ -505,7 +505,7 @@ static int amdgpu_cs_list_validate(struct amdgpu_cs_parser *p,
> return r;
>
> if (binding_userptr) {
> - drm_free_large(lobj->user_pages);
> + kvfree(lobj->user_pages);
> lobj->user_pages = NULL;
> }
> }
> @@ -571,7 +571,7 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p,
> release_pages(e->user_pages,
> e->robj->tbo.ttm->num_pages,
> false);
> - drm_free_large(e->user_pages);
> + kvfree(e->user_pages);
> e->user_pages = NULL;
> }
>
> @@ -601,8 +601,9 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p,
> list_for_each_entry(e, &need_pages, tv.head) {
> struct ttm_tt *ttm = e->robj->tbo.ttm;
>
> - e->user_pages = drm_calloc_large(ttm->num_pages,
> - sizeof(struct page*));
> + e->user_pages = kvmalloc_array(ttm->num_pages,
> + sizeof(struct page*),
> + GFP_KERNEL | __GFP_ZERO);
> if (!e->user_pages) {
> r = -ENOMEM;
> DRM_ERROR("calloc failure in %s\n", __func__);
> @@ -612,7 +613,7 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p,
> r = amdgpu_ttm_tt_get_user_pages(ttm, e->user_pages);
> if (r) {
> DRM_ERROR("amdgpu_ttm_tt_get_user_pages failed.\n");
> - drm_free_large(e->user_pages);
> + kvfree(e->user_pages);
> e->user_pages = NULL;
> goto error_free_pages;
> }
> @@ -708,7 +709,7 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p,
> release_pages(e->user_pages,
> e->robj->tbo.ttm->num_pages,
> false);
> - drm_free_large(e->user_pages);
> + kvfree(e->user_pages);
> }
> }
>
> @@ -761,7 +762,7 @@ static void amdgpu_cs_parser_fini(struct amdgpu_cs_parser *parser, int error, bo
> amdgpu_bo_list_put(parser->bo_list);
>
> for (i = 0; i < parser->nchunks; i++)
> - drm_free_large(parser->chunks[i].kdata);
> + kvfree(parser->chunks[i].kdata);
> kfree(parser->chunks);
> if (parser->job)
> amdgpu_job_free(parser->job);
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> index 07ff3b1514f1..749a6cde7985 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> @@ -279,8 +279,9 @@ static int amdgpu_vm_alloc_levels(struct amdgpu_device *adev,
> if (!parent->entries) {
> unsigned num_entries = amdgpu_vm_num_entries(adev, level);
>
> - parent->entries = drm_calloc_large(num_entries,
> - sizeof(struct amdgpu_vm_pt));
> + parent->entries = kvmalloc_array(num_entries,
> + sizeof(struct amdgpu_vm_pt),
> + GFP_KERNEL | __GFP_ZERO);
> if (!parent->entries)
> return -ENOMEM;
> memset(parent->entries, 0 , sizeof(struct amdgpu_vm_pt));
> @@ -2198,7 +2199,7 @@ static void amdgpu_vm_free_levels(struct amdgpu_vm_pt *level)
> for (i = 0; i <= level->last_entry_used; i++)
> amdgpu_vm_free_levels(&level->entries[i]);
>
> - drm_free_large(level->entries);
> + kvfree(level->entries);
> }
>
> /**
> diff --git a/drivers/gpu/drm/drm_gem.c b/drivers/gpu/drm/drm_gem.c
> index b1e28c944637..8dc11064253d 100644
> --- a/drivers/gpu/drm/drm_gem.c
> +++ b/drivers/gpu/drm/drm_gem.c
> @@ -521,7 +521,7 @@ struct page **drm_gem_get_pages(struct drm_gem_object *obj)
>
> npages = obj->size >> PAGE_SHIFT;
>
> - pages = drm_malloc_ab(npages, sizeof(struct page *));
> + pages = kvmalloc_array(npages, sizeof(struct page *), GFP_KERNEL);
> if (pages == NULL)
> return ERR_PTR(-ENOMEM);
>
> @@ -546,7 +546,7 @@ struct page **drm_gem_get_pages(struct drm_gem_object *obj)
> while (i--)
> put_page(pages[i]);
>
> - drm_free_large(pages);
> + kvfree(pages);
> return ERR_CAST(p);
> }
> EXPORT_SYMBOL(drm_gem_get_pages);
> @@ -582,7 +582,7 @@ void drm_gem_put_pages(struct drm_gem_object *obj, struct page **pages,
> put_page(pages[i]);
> }
>
> - drm_free_large(pages);
> + kvfree(pages);
> }
> EXPORT_SYMBOL(drm_gem_put_pages);
>
> diff --git a/drivers/gpu/drm/etnaviv/etnaviv_gem.c b/drivers/gpu/drm/etnaviv/etnaviv_gem.c
> index fd56f92f3469..d45dd9f3b00c 100644
> --- a/drivers/gpu/drm/etnaviv/etnaviv_gem.c
> +++ b/drivers/gpu/drm/etnaviv/etnaviv_gem.c
> @@ -748,7 +748,7 @@ static struct page **etnaviv_gem_userptr_do_get_pages(
> uintptr_t ptr;
> unsigned int flags = 0;
>
> - pvec = drm_malloc_ab(npages, sizeof(struct page *));
> + pvec = kvmallo_array(npages, sizeof(struct page *), GFP_KERNEL);
> if (!pvec)
> return ERR_PTR(-ENOMEM);
>
> @@ -772,7 +772,7 @@ static struct page **etnaviv_gem_userptr_do_get_pages(
>
> if (ret < 0) {
> release_pages(pvec, pinned, 0);
> - drm_free_large(pvec);
> + kvfree(pvec);
> return ERR_PTR(ret);
> }
>
> @@ -823,7 +823,7 @@ static int etnaviv_gem_userptr_get_pages(struct etnaviv_gem_object *etnaviv_obj)
> mm = get_task_mm(etnaviv_obj->userptr.task);
> pinned = 0;
> if (mm == current->mm) {
> - pvec = drm_malloc_ab(npages, sizeof(struct page *));
> + pvec = kvmalloc_array(npages, sizeof(struct page *), GFP_KERNEL);
> if (!pvec) {
> mmput(mm);
> return -ENOMEM;
> @@ -832,7 +832,7 @@ static int etnaviv_gem_userptr_get_pages(struct etnaviv_gem_object *etnaviv_obj)
> pinned = __get_user_pages_fast(etnaviv_obj->userptr.ptr, npages,
> !etnaviv_obj->userptr.ro, pvec);
> if (pinned < 0) {
> - drm_free_large(pvec);
> + kvfree(pvec);
> mmput(mm);
> return pinned;
> }
> @@ -845,7 +845,7 @@ static int etnaviv_gem_userptr_get_pages(struct etnaviv_gem_object *etnaviv_obj)
> }
>
> release_pages(pvec, pinned, 0);
> - drm_free_large(pvec);
> + kvfree(pvec);
>
> work = kmalloc(sizeof(*work), GFP_KERNEL);
> if (!work) {
> @@ -879,7 +879,7 @@ static void etnaviv_gem_userptr_release(struct etnaviv_gem_object *etnaviv_obj)
> int npages = etnaviv_obj->base.size >> PAGE_SHIFT;
>
> release_pages(etnaviv_obj->pages, npages, 0);
> - drm_free_large(etnaviv_obj->pages);
> + kvfree(etnaviv_obj->pages);
> }
> put_task_struct(etnaviv_obj->userptr.task);
> }
> diff --git a/drivers/gpu/drm/etnaviv/etnaviv_gem_prime.c b/drivers/gpu/drm/etnaviv/etnaviv_gem_prime.c
> index abed6f781281..e5da4f2300ba 100644
> --- a/drivers/gpu/drm/etnaviv/etnaviv_gem_prime.c
> +++ b/drivers/gpu/drm/etnaviv/etnaviv_gem_prime.c
> @@ -87,7 +87,7 @@ static void etnaviv_gem_prime_release(struct etnaviv_gem_object *etnaviv_obj)
> * ours, just free the array we allocated:
> */
> if (etnaviv_obj->pages)
> - drm_free_large(etnaviv_obj->pages);
> + kvfree(etnaviv_obj->pages);
>
> drm_prime_gem_destroy(&etnaviv_obj->base, etnaviv_obj->sgt);
> }
> @@ -128,7 +128,7 @@ struct drm_gem_object *etnaviv_gem_prime_import_sg_table(struct drm_device *dev,
> npages = size / PAGE_SIZE;
>
> etnaviv_obj->sgt = sgt;
> - etnaviv_obj->pages = drm_malloc_ab(npages, sizeof(struct page *));
> + etnaviv_obj->pages = kvmalloc_array(npages, sizeof(struct page *), GFP_KERNEL);
> if (!etnaviv_obj->pages) {
> ret = -ENOMEM;
> goto fail;
> diff --git a/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c b/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c
> index de80ee1b71df..ee7069e93eda 100644
> --- a/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c
> +++ b/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c
> @@ -345,9 +345,9 @@ int etnaviv_ioctl_gem_submit(struct drm_device *dev, void *data,
> * Copy the command submission and bo array to kernel space in
> * one go, and do this outside of any locks.
> */
> - bos = drm_malloc_ab(args->nr_bos, sizeof(*bos));
> - relocs = drm_malloc_ab(args->nr_relocs, sizeof(*relocs));
> - stream = drm_malloc_ab(1, args->stream_size);
> + bos = kvmalloc_array(args->nr_bos, sizeof(*bos), GFP_KERNEL);
> + relocs = kvmalloc_array(args->nr_relocs, sizeof(*relocs), GFP_KERNEL);
> + stream = kvmalloc_array(1, args->stream_size, GFP_KERNEL);
> cmdbuf = etnaviv_cmdbuf_new(gpu->cmdbuf_suballoc,
> ALIGN(args->stream_size, 8) + 8,
> args->nr_bos);
> @@ -489,11 +489,11 @@ int etnaviv_ioctl_gem_submit(struct drm_device *dev, void *data,
> if (cmdbuf)
> etnaviv_cmdbuf_free(cmdbuf);
> if (stream)
> - drm_free_large(stream);
> + kvfree(stream);
> if (bos)
> - drm_free_large(bos);
> + kvfree(bos);
> if (relocs)
> - drm_free_large(relocs);
> + kvfree(relocs);
>
> return ret;
> }
> diff --git a/drivers/gpu/drm/exynos/exynos_drm_gem.c b/drivers/gpu/drm/exynos/exynos_drm_gem.c
> index 55a1579d11b3..c23479be4850 100644
> --- a/drivers/gpu/drm/exynos/exynos_drm_gem.c
> +++ b/drivers/gpu/drm/exynos/exynos_drm_gem.c
> @@ -59,7 +59,8 @@ static int exynos_drm_alloc_buf(struct exynos_drm_gem *exynos_gem)
>
> nr_pages = exynos_gem->size >> PAGE_SHIFT;
>
> - exynos_gem->pages = drm_calloc_large(nr_pages, sizeof(struct page *));
> + exynos_gem->pages = kvmalloc_array(nr_pages, sizeof(struct page *),
> + GFP_KERNEL | __GFP_ZERO);
> if (!exynos_gem->pages) {
> DRM_ERROR("failed to allocate pages.\n");
> return -ENOMEM;
> @@ -101,7 +102,7 @@ static int exynos_drm_alloc_buf(struct exynos_drm_gem *exynos_gem)
> dma_free_attrs(to_dma_dev(dev), exynos_gem->size, exynos_gem->cookie,
> exynos_gem->dma_addr, exynos_gem->dma_attrs);
> err_free:
> - drm_free_large(exynos_gem->pages);
> + kvfree(exynos_gem->pages);
>
> return ret;
> }
> @@ -122,7 +123,7 @@ static void exynos_drm_free_buf(struct exynos_drm_gem *exynos_gem)
> (dma_addr_t)exynos_gem->dma_addr,
> exynos_gem->dma_attrs);
>
> - drm_free_large(exynos_gem->pages);
> + kvfree(exynos_gem->pages);
> }
>
> static int exynos_drm_gem_handle_create(struct drm_gem_object *obj,
> @@ -559,7 +560,7 @@ exynos_drm_gem_prime_import_sg_table(struct drm_device *dev,
> exynos_gem->dma_addr = sg_dma_address(sgt->sgl);
>
> npages = exynos_gem->size >> PAGE_SHIFT;
> - exynos_gem->pages = drm_malloc_ab(npages, sizeof(struct page *));
> + exynos_gem->pages = kvmalloc_array(npages, sizeof(struct page *), GFP_KERNEL);
> if (!exynos_gem->pages) {
> ret = -ENOMEM;
> goto err;
> @@ -588,7 +589,7 @@ exynos_drm_gem_prime_import_sg_table(struct drm_device *dev,
> return &exynos_gem->base;
>
> err_free_large:
> - drm_free_large(exynos_gem->pages);
> + kvfree(exynos_gem->pages);
> err:
> drm_gem_object_release(&exynos_gem->base);
> kfree(exynos_gem);
> diff --git a/drivers/gpu/drm/i915/i915_debugfs.c b/drivers/gpu/drm/i915/i915_debugfs.c
> index bd9abef40c66..c8f3c0cc79fb 100644
> --- a/drivers/gpu/drm/i915/i915_debugfs.c
> +++ b/drivers/gpu/drm/i915/i915_debugfs.c
> @@ -229,7 +229,7 @@ static int i915_gem_stolen_list_info(struct seq_file *m, void *data)
> int ret;
>
> total = READ_ONCE(dev_priv->mm.object_count);
> - objects = drm_malloc_ab(total, sizeof(*objects));
> + objects = kvmalloc_array(total, sizeof(*objects), GFP_KERNEL);
> if (!objects)
> return -ENOMEM;
>
> @@ -274,7 +274,7 @@ static int i915_gem_stolen_list_info(struct seq_file *m, void *data)
>
> mutex_unlock(&dev->struct_mutex);
> out:
> - drm_free_large(objects);
> + kvfree(objects);
> return ret;
> }
>
> diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
> index 0c1cbe98c994..aa790a6d38e2 100644
> --- a/drivers/gpu/drm/i915/i915_gem.c
> +++ b/drivers/gpu/drm/i915/i915_gem.c
> @@ -2556,7 +2556,7 @@ static void *i915_gem_object_map(const struct drm_i915_gem_object *obj,
>
> if (n_pages > ARRAY_SIZE(stack_pages)) {
> /* Too big for stack -- allocate temporary array instead */
> - pages = drm_malloc_gfp(n_pages, sizeof(*pages), GFP_TEMPORARY);
> + pages = kvmalloc_array(n_pages, sizeof(*pages), GFP_TEMPORARY);
> if (!pages)
> return NULL;
> }
> @@ -2578,7 +2578,7 @@ static void *i915_gem_object_map(const struct drm_i915_gem_object *obj,
> addr = vmap(pages, n_pages, 0, pgprot);
>
> if (pages != stack_pages)
> - drm_free_large(pages);
> + kvfree(pages);
>
> return addr;
> }
> diff --git a/drivers/gpu/drm/i915/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
> index af1965774e7b..04211c970b9f 100644
> --- a/drivers/gpu/drm/i915/i915_gem_execbuffer.c
> +++ b/drivers/gpu/drm/i915/i915_gem_execbuffer.c
> @@ -1019,11 +1019,11 @@ i915_gem_execbuffer_relocate_slow(struct drm_device *dev,
> for (i = 0; i < count; i++)
> total += exec[i].relocation_count;
>
> - reloc_offset = drm_malloc_ab(count, sizeof(*reloc_offset));
> - reloc = drm_malloc_ab(total, sizeof(*reloc));
> + reloc_offset = kvmalloc_array(count, sizeof(*reloc_offset), GFP_KERNEL);
> + reloc = kvmalloc_array(total, sizeof(*reloc), GFP_KERNEL);
> if (reloc == NULL || reloc_offset == NULL) {
> - drm_free_large(reloc);
> - drm_free_large(reloc_offset);
> + kvfree(reloc);
> + kvfree(reloc_offset);
> mutex_lock(&dev->struct_mutex);
> return -ENOMEM;
> }
> @@ -1099,8 +1099,8 @@ i915_gem_execbuffer_relocate_slow(struct drm_device *dev,
> */
>
> err:
> - drm_free_large(reloc);
> - drm_free_large(reloc_offset);
> + kvfree(reloc);
> + kvfree(reloc_offset);
> return ret;
> }
>
> @@ -1871,13 +1871,13 @@ i915_gem_execbuffer(struct drm_device *dev, void *data,
> }
>
> /* Copy in the exec list from userland */
> - exec_list = drm_malloc_ab(sizeof(*exec_list), args->buffer_count);
> - exec2_list = drm_malloc_ab(sizeof(*exec2_list), args->buffer_count);
> + exec_list = kvmalloc_array(sizeof(*exec_list), args->buffer_count, GFP_KERNEL);
> + exec2_list = kvmalloc_array(sizeof(*exec2_list), args->buffer_count, GFP_KERNEL);
> if (exec_list == NULL || exec2_list == NULL) {
> DRM_DEBUG("Failed to allocate exec list for %d buffers\n",
> args->buffer_count);
> - drm_free_large(exec_list);
> - drm_free_large(exec2_list);
> + kvfree(exec_list);
> + kvfree(exec2_list);
> return -ENOMEM;
> }
> ret = copy_from_user(exec_list,
> @@ -1886,8 +1886,8 @@ i915_gem_execbuffer(struct drm_device *dev, void *data,
> if (ret != 0) {
> DRM_DEBUG("copy %d exec entries failed %d\n",
> args->buffer_count, ret);
> - drm_free_large(exec_list);
> - drm_free_large(exec2_list);
> + kvfree(exec_list);
> + kvfree(exec2_list);
> return -EFAULT;
> }
>
> @@ -1936,8 +1936,8 @@ i915_gem_execbuffer(struct drm_device *dev, void *data,
> }
> }
>
> - drm_free_large(exec_list);
> - drm_free_large(exec2_list);
> + kvfree(exec_list);
> + kvfree(exec2_list);
> return ret;
> }
>
> @@ -1955,7 +1955,7 @@ i915_gem_execbuffer2(struct drm_device *dev, void *data,
> return -EINVAL;
> }
>
> - exec2_list = drm_malloc_gfp(args->buffer_count,
> + exec2_list = kvmalloc_array(args->buffer_count,
> sizeof(*exec2_list),
> GFP_TEMPORARY);
> if (exec2_list == NULL) {
> @@ -1969,7 +1969,7 @@ i915_gem_execbuffer2(struct drm_device *dev, void *data,
> if (ret != 0) {
> DRM_DEBUG("copy %d exec entries failed %d\n",
> args->buffer_count, ret);
> - drm_free_large(exec2_list);
> + kvfree(exec2_list);
> return -EFAULT;
> }
>
> @@ -1996,6 +1996,6 @@ i915_gem_execbuffer2(struct drm_device *dev, void *data,
> }
> }
>
> - drm_free_large(exec2_list);
> + kvfree(exec2_list);
> return ret;
> }
> diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.c b/drivers/gpu/drm/i915/i915_gem_gtt.c
> index bc3c63e92c16..899427863547 100644
> --- a/drivers/gpu/drm/i915/i915_gem_gtt.c
> +++ b/drivers/gpu/drm/i915/i915_gem_gtt.c
> @@ -3114,7 +3114,7 @@ intel_rotate_pages(struct intel_rotation_info *rot_info,
> int ret = -ENOMEM;
>
> /* Allocate a temporary list of source pages for random access. */
> - page_addr_list = drm_malloc_gfp(n_pages,
> + page_addr_list = kvmalloc_array(n_pages,
> sizeof(dma_addr_t),
> GFP_TEMPORARY);
> if (!page_addr_list)
> @@ -3147,14 +3147,14 @@ intel_rotate_pages(struct intel_rotation_info *rot_info,
> DRM_DEBUG_KMS("Created rotated page mapping for object size %zu (%ux%u tiles, %u pages)\n",
> obj->base.size, rot_info->plane[0].width, rot_info->plane[0].height, size);
>
> - drm_free_large(page_addr_list);
> + kvfree(page_addr_list);
>
> return st;
>
> err_sg_alloc:
> kfree(st);
> err_st_alloc:
> - drm_free_large(page_addr_list);
> + kvfree(page_addr_list);
>
> DRM_DEBUG_KMS("Failed to create rotated mapping for object size %zu! (%ux%u tiles, %u pages)\n",
> obj->base.size, rot_info->plane[0].width, rot_info->plane[0].height, size);
> diff --git a/drivers/gpu/drm/i915/i915_gem_userptr.c b/drivers/gpu/drm/i915/i915_gem_userptr.c
> index 58ccf8b8ca1c..1a0ce1dc68f5 100644
> --- a/drivers/gpu/drm/i915/i915_gem_userptr.c
> +++ b/drivers/gpu/drm/i915/i915_gem_userptr.c
> @@ -507,7 +507,7 @@ __i915_gem_userptr_get_pages_worker(struct work_struct *_work)
> ret = -ENOMEM;
> pinned = 0;
>
> - pvec = drm_malloc_gfp(npages, sizeof(struct page *), GFP_TEMPORARY);
> + pvec = kvmalloc_array(npages, sizeof(struct page *), GFP_TEMPORARY);
> if (pvec != NULL) {
> struct mm_struct *mm = obj->userptr.mm->mm;
> unsigned int flags = 0;
> @@ -555,7 +555,7 @@ __i915_gem_userptr_get_pages_worker(struct work_struct *_work)
> mutex_unlock(&obj->mm.lock);
>
> release_pages(pvec, pinned, 0);
> - drm_free_large(pvec);
> + kvfree(pvec);
>
> i915_gem_object_put(obj);
> put_task_struct(work->task);
> @@ -642,7 +642,7 @@ i915_gem_userptr_get_pages(struct drm_i915_gem_object *obj)
> pinned = 0;
>
> if (mm == current->mm) {
> - pvec = drm_malloc_gfp(num_pages, sizeof(struct page *),
> + pvec = kvmalloc_array(num_pages, sizeof(struct page *),
> GFP_TEMPORARY |
> __GFP_NORETRY |
> __GFP_NOWARN);
> @@ -669,7 +669,7 @@ i915_gem_userptr_get_pages(struct drm_i915_gem_object *obj)
>
> if (IS_ERR(pages))
> release_pages(pvec, pinned, 0);
> - drm_free_large(pvec);
> + kvfree(pvec);
>
> return pages;
> }
> diff --git a/drivers/gpu/drm/i915/selftests/intel_breadcrumbs.c b/drivers/gpu/drm/i915/selftests/intel_breadcrumbs.c
> index 19860a372d90..7276194c04f7 100644
> --- a/drivers/gpu/drm/i915/selftests/intel_breadcrumbs.c
> +++ b/drivers/gpu/drm/i915/selftests/intel_breadcrumbs.c
> @@ -117,7 +117,7 @@ static int igt_random_insert_remove(void *arg)
>
> mock_engine_reset(engine);
>
> - waiters = drm_malloc_gfp(count, sizeof(*waiters), GFP_TEMPORARY);
> + waiters = kvmalloc_array(count, sizeof(*waiters), GFP_TEMPORARY);
> if (!waiters)
> goto out_engines;
>
> @@ -169,7 +169,7 @@ static int igt_random_insert_remove(void *arg)
> out_bitmap:
> kfree(bitmap);
> out_waiters:
> - drm_free_large(waiters);
> + kvfree(waiters);
> out_engines:
> mock_engine_flush(engine);
> return err;
> @@ -187,7 +187,7 @@ static int igt_insert_complete(void *arg)
>
> mock_engine_reset(engine);
>
> - waiters = drm_malloc_gfp(count, sizeof(*waiters), GFP_TEMPORARY);
> + waiters = kvmalloc_array(count, sizeof(*waiters), GFP_TEMPORARY);
> if (!waiters)
> goto out_engines;
>
> @@ -254,7 +254,7 @@ static int igt_insert_complete(void *arg)
> out_bitmap:
> kfree(bitmap);
> out_waiters:
> - drm_free_large(waiters);
> + kvfree(waiters);
> out_engines:
> mock_engine_flush(engine);
> return err;
> @@ -368,7 +368,7 @@ static int igt_wakeup(void *arg)
>
> mock_engine_reset(engine);
>
> - waiters = drm_malloc_gfp(count, sizeof(*waiters), GFP_TEMPORARY);
> + waiters = kvmalloc_array(count, sizeof(*waiters), GFP_TEMPORARY);
> if (!waiters)
> goto out_engines;
>
> @@ -454,7 +454,7 @@ static int igt_wakeup(void *arg)
> put_task_struct(waiters[n].tsk);
> }
>
> - drm_free_large(waiters);
> + kvfree(waiters);
> out_engines:
> mock_engine_flush(engine);
> return err;
> diff --git a/drivers/gpu/drm/msm/msm_gem.c b/drivers/gpu/drm/msm/msm_gem.c
> index 68e509b3b9e4..465dab942afa 100644
> --- a/drivers/gpu/drm/msm/msm_gem.c
> +++ b/drivers/gpu/drm/msm/msm_gem.c
> @@ -50,13 +50,13 @@ static struct page **get_pages_vram(struct drm_gem_object *obj,
> struct page **p;
> int ret, i;
>
> - p = drm_malloc_ab(npages, sizeof(struct page *));
> + p = kvmalloc_array(npages, sizeof(struct page *), GFP_KERNEL);
> if (!p)
> return ERR_PTR(-ENOMEM);
>
> ret = drm_mm_insert_node(&priv->vram.mm, msm_obj->vram_node, npages);
> if (ret) {
> - drm_free_large(p);
> + kvfree(p);
> return ERR_PTR(ret);
> }
>
> @@ -127,7 +127,7 @@ static void put_pages(struct drm_gem_object *obj)
> drm_gem_put_pages(obj, msm_obj->pages, true, false);
> else {
> drm_mm_remove_node(msm_obj->vram_node);
> - drm_free_large(msm_obj->pages);
> + kvfree(msm_obj->pages);
> }
>
> msm_obj->pages = NULL;
> @@ -707,7 +707,7 @@ void msm_gem_free_object(struct drm_gem_object *obj)
> * ours, just free the array we allocated:
> */
> if (msm_obj->pages)
> - drm_free_large(msm_obj->pages);
> + kvfree(msm_obj->pages);
>
> drm_prime_gem_destroy(obj, msm_obj->sgt);
> } else {
> @@ -863,7 +863,7 @@ struct drm_gem_object *msm_gem_import(struct drm_device *dev,
>
> msm_obj = to_msm_bo(obj);
> msm_obj->sgt = sgt;
> - msm_obj->pages = drm_malloc_ab(npages, sizeof(struct page *));
> + msm_obj->pages = kvmalloc_array(npages, sizeof(struct page *), GFP_KERNEL);
> if (!msm_obj->pages) {
> ret = -ENOMEM;
> goto fail;
> diff --git a/drivers/gpu/drm/radeon/radeon_cs.c b/drivers/gpu/drm/radeon/radeon_cs.c
> index 3ac671f6c8e1..00b22af70f5c 100644
> --- a/drivers/gpu/drm/radeon/radeon_cs.c
> +++ b/drivers/gpu/drm/radeon/radeon_cs.c
> @@ -87,7 +87,8 @@ static int radeon_cs_parser_relocs(struct radeon_cs_parser *p)
> p->dma_reloc_idx = 0;
> /* FIXME: we assume that each relocs use 4 dwords */
> p->nrelocs = chunk->length_dw / 4;
> - p->relocs = drm_calloc_large(p->nrelocs, sizeof(struct radeon_bo_list));
> + p->relocs = kvmalloc_array(p->nrelocs, sizeof(struct radeon_bo_list),
> + GFP_KERNEL | __GFP_ZERO);
> if (p->relocs == NULL) {
> return -ENOMEM;
> }
> @@ -341,7 +342,7 @@ int radeon_cs_parser_init(struct radeon_cs_parser *p, void *data)
> continue;
> }
>
> - p->chunks[i].kdata = drm_malloc_ab(size, sizeof(uint32_t));
> + p->chunks[i].kdata = kvmalloc_array(size, sizeof(uint32_t), GFP_KERNEL);
> size *= sizeof(uint32_t);
> if (p->chunks[i].kdata == NULL) {
> return -ENOMEM;
> @@ -440,10 +441,10 @@ static void radeon_cs_parser_fini(struct radeon_cs_parser *parser, int error, bo
> }
> }
> kfree(parser->track);
> - drm_free_large(parser->relocs);
> - drm_free_large(parser->vm_bos);
> + kvfree(parser->relocs);
> + kvfree(parser->vm_bos);
> for (i = 0; i < parser->nchunks; i++)
> - drm_free_large(parser->chunks[i].kdata);
> + kvfree(parser->chunks[i].kdata);
> kfree(parser->chunks);
> kfree(parser->chunks_array);
> radeon_ib_free(parser->rdev, &parser->ib);
> diff --git a/drivers/gpu/drm/radeon/radeon_gem.c b/drivers/gpu/drm/radeon/radeon_gem.c
> index dddb372de2b9..574bf7e6b118 100644
> --- a/drivers/gpu/drm/radeon/radeon_gem.c
> +++ b/drivers/gpu/drm/radeon/radeon_gem.c
> @@ -587,7 +587,7 @@ static void radeon_gem_va_update_vm(struct radeon_device *rdev,
> ttm_eu_backoff_reservation(&ticket, &list);
>
> error_free:
> - drm_free_large(vm_bos);
> + kvfree(vm_bos);
>
> if (r && r != -ERESTARTSYS)
> DRM_ERROR("Couldn't update BO_VA (%d)\n", r);
> diff --git a/drivers/gpu/drm/radeon/radeon_ring.c b/drivers/gpu/drm/radeon/radeon_ring.c
> index 8c7872339c2a..84802b201bef 100644
> --- a/drivers/gpu/drm/radeon/radeon_ring.c
> +++ b/drivers/gpu/drm/radeon/radeon_ring.c
> @@ -314,7 +314,7 @@ unsigned radeon_ring_backup(struct radeon_device *rdev, struct radeon_ring *ring
> }
>
> /* and then save the content of the ring */
> - *data = drm_malloc_ab(size, sizeof(uint32_t));
> + *data = kvmalloc_array(size, sizeof(uint32_t), GFP_KERNEL);
> if (!*data) {
> mutex_unlock(&rdev->ring_lock);
> return 0;
> @@ -356,7 +356,7 @@ int radeon_ring_restore(struct radeon_device *rdev, struct radeon_ring *ring,
> }
>
> radeon_ring_unlock_commit(rdev, ring, false);
> - drm_free_large(data);
> + kvfree(data);
> return 0;
> }
>
> diff --git a/drivers/gpu/drm/radeon/radeon_vm.c b/drivers/gpu/drm/radeon/radeon_vm.c
> index a1358748cea5..5f68245579a3 100644
> --- a/drivers/gpu/drm/radeon/radeon_vm.c
> +++ b/drivers/gpu/drm/radeon/radeon_vm.c
> @@ -132,8 +132,8 @@ struct radeon_bo_list *radeon_vm_get_bos(struct radeon_device *rdev,
> struct radeon_bo_list *list;
> unsigned i, idx;
>
> - list = drm_malloc_ab(vm->max_pde_used + 2,
> - sizeof(struct radeon_bo_list));
> + list = kvmalloc_array(vm->max_pde_used + 2,
> + sizeof(struct radeon_bo_list), GFP_KERNEL);
> if (!list)
> return NULL;
>
> diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_tt.c
> index 5260179d788a..8ebc8d3560c3 100644
> --- a/drivers/gpu/drm/ttm/ttm_tt.c
> +++ b/drivers/gpu/drm/ttm/ttm_tt.c
> @@ -39,7 +39,6 @@
> #include <linux/slab.h>
> #include <linux/export.h>
> #include <drm/drm_cache.h>
> -#include <drm/drm_mem_util.h>
> #include <drm/ttm/ttm_module.h>
> #include <drm/ttm/ttm_bo_driver.h>
> #include <drm/ttm/ttm_placement.h>
> @@ -53,14 +52,16 @@
> */
> static void ttm_tt_alloc_page_directory(struct ttm_tt *ttm)
> {
> - ttm->pages = drm_calloc_large(ttm->num_pages, sizeof(void*));
> + ttm->pages = kvmalloc_array(ttm->num_pages, sizeof(void*),
> + GFP_KERNEL | __GFP_ZERO);
> }
>
> static void ttm_dma_tt_alloc_page_directory(struct ttm_dma_tt *ttm)
> {
> - ttm->ttm.pages = drm_calloc_large(ttm->ttm.num_pages,
> + ttm->ttm.pages = kvmalloc_array(ttm->ttm.num_pages,
> sizeof(*ttm->ttm.pages) +
> - sizeof(*ttm->dma_address));
> + sizeof(*ttm->dma_address),
> + GFP_KERNEL | __GFP_ZERO);
> ttm->dma_address = (void *) (ttm->ttm.pages + ttm->ttm.num_pages);
> }
>
> @@ -208,7 +209,7 @@ EXPORT_SYMBOL(ttm_tt_init);
>
> void ttm_tt_fini(struct ttm_tt *ttm)
> {
> - drm_free_large(ttm->pages);
> + kvfree(ttm->pages);
> ttm->pages = NULL;
> }
> EXPORT_SYMBOL(ttm_tt_fini);
> @@ -243,7 +244,7 @@ void ttm_dma_tt_fini(struct ttm_dma_tt *ttm_dma)
> {
> struct ttm_tt *ttm = &ttm_dma->ttm;
>
> - drm_free_large(ttm->pages);
> + kvfree(ttm->pages);
> ttm->pages = NULL;
> ttm_dma->dma_address = NULL;
> }
> diff --git a/drivers/gpu/drm/udl/udl_dmabuf.c b/drivers/gpu/drm/udl/udl_dmabuf.c
> index ed0e636243b2..2e031a894813 100644
> --- a/drivers/gpu/drm/udl/udl_dmabuf.c
> +++ b/drivers/gpu/drm/udl/udl_dmabuf.c
> @@ -228,7 +228,7 @@ static int udl_prime_create(struct drm_device *dev,
> return -ENOMEM;
>
> obj->sg = sg;
> - obj->pages = drm_malloc_ab(npages, sizeof(struct page *));
> + obj->pages = kvmalloc_array(npages, sizeof(struct page *), GFP_KERNEL);
> if (obj->pages == NULL) {
> DRM_ERROR("obj pages is NULL %d\n", npages);
> return -ENOMEM;
> diff --git a/drivers/gpu/drm/udl/udl_gem.c b/drivers/gpu/drm/udl/udl_gem.c
> index 775c50e4f02c..db9ceceba30e 100644
> --- a/drivers/gpu/drm/udl/udl_gem.c
> +++ b/drivers/gpu/drm/udl/udl_gem.c
> @@ -146,7 +146,7 @@ int udl_gem_get_pages(struct udl_gem_object *obj)
> void udl_gem_put_pages(struct udl_gem_object *obj)
> {
> if (obj->base.import_attach) {
> - drm_free_large(obj->pages);
> + kvfree(obj->pages);
> obj->pages = NULL;
> return;
> }
> diff --git a/drivers/gpu/drm/vc4/vc4_gem.c b/drivers/gpu/drm/vc4/vc4_gem.c
> index e9c381c42139..bf466674ca9b 100644
> --- a/drivers/gpu/drm/vc4/vc4_gem.c
> +++ b/drivers/gpu/drm/vc4/vc4_gem.c
> @@ -545,14 +545,15 @@ vc4_cl_lookup_bos(struct drm_device *dev,
> return -EINVAL;
> }
>
> - exec->bo = drm_calloc_large(exec->bo_count,
> - sizeof(struct drm_gem_cma_object *));
> + exec->bo = kvmalloc_array(exec->bo_count,
> + sizeof(struct drm_gem_cma_object *),
> + GFP_KERNEL | __GFP_ZERO);
> if (!exec->bo) {
> DRM_ERROR("Failed to allocate validated BO pointers\n");
> return -ENOMEM;
> }
>
> - handles = drm_malloc_ab(exec->bo_count, sizeof(uint32_t));
> + handles = kvmalloc_array(exec->bo_count, sizeof(uint32_t), GFP_KERNEL);
> if (!handles) {
> ret = -ENOMEM;
> DRM_ERROR("Failed to allocate incoming GEM handles\n");
> @@ -584,7 +585,7 @@ vc4_cl_lookup_bos(struct drm_device *dev,
> spin_unlock(&file_priv->table_lock);
>
> fail:
> - drm_free_large(handles);
> + kvfree(handles);
> return ret;
> }
>
> @@ -622,7 +623,7 @@ vc4_get_bcl(struct drm_device *dev, struct vc4_exec_info *exec)
> * read the contents back for validation, and I think the
> * bo->vaddr is uncached access.
> */
> - temp = drm_malloc_ab(temp_size, 1);
> + temp = kvmalloc_array(temp_size, 1, GFP_KERNEL);
> if (!temp) {
> DRM_ERROR("Failed to allocate storage for copying "
> "in bin/render CLs.\n");
> @@ -697,7 +698,7 @@ vc4_get_bcl(struct drm_device *dev, struct vc4_exec_info *exec)
> ret = vc4_wait_for_seqno(dev, exec->bin_dep_seqno, ~0ull, true);
>
> fail:
> - drm_free_large(temp);
> + kvfree(temp);
> return ret;
> }
>
> @@ -710,7 +711,7 @@ vc4_complete_exec(struct drm_device *dev, struct vc4_exec_info *exec)
> if (exec->bo) {
> for (i = 0; i < exec->bo_count; i++)
> drm_gem_object_unreference_unlocked(&exec->bo[i]->base);
> - drm_free_large(exec->bo);
> + kvfree(exec->bo);
> }
>
> while (!list_empty(&exec->unref_list)) {
> diff --git a/drivers/gpu/drm/virtio/virtgpu_ioctl.c b/drivers/gpu/drm/virtio/virtgpu_ioctl.c
> index 61f3a963af95..6ed4bfc9b82b 100644
> --- a/drivers/gpu/drm/virtio/virtgpu_ioctl.c
> +++ b/drivers/gpu/drm/virtio/virtgpu_ioctl.c
> @@ -119,13 +119,14 @@ static int virtio_gpu_execbuffer_ioctl(struct drm_device *dev, void *data,
> INIT_LIST_HEAD(&validate_list);
> if (exbuf->num_bo_handles) {
>
> - bo_handles = drm_malloc_ab(exbuf->num_bo_handles,
> - sizeof(uint32_t));
> - buflist = drm_calloc_large(exbuf->num_bo_handles,
> - sizeof(struct ttm_validate_buffer));
> + bo_handles = kvmalloc_array(exbuf->num_bo_handles,
> + sizeof(uint32_t), GFP_KERNEL);
> + buflist = kvmalloc_array(exbuf->num_bo_handles,
> + sizeof(struct ttm_validate_buffer),
> + GFP_KERNEL | __GFP_ZERO);
> if (!bo_handles || !buflist) {
> - drm_free_large(bo_handles);
> - drm_free_large(buflist);
> + kvfree(bo_handles);
> + kvfree(buflist);
> return -ENOMEM;
> }
>
> @@ -133,16 +134,16 @@ static int virtio_gpu_execbuffer_ioctl(struct drm_device *dev, void *data,
> if (copy_from_user(bo_handles, user_bo_handles,
> exbuf->num_bo_handles * sizeof(uint32_t))) {
> ret = -EFAULT;
> - drm_free_large(bo_handles);
> - drm_free_large(buflist);
> + kvfree(bo_handles);
> + kvfree(buflist);
> return ret;
> }
>
> for (i = 0; i < exbuf->num_bo_handles; i++) {
> gobj = drm_gem_object_lookup(drm_file, bo_handles[i]);
> if (!gobj) {
> - drm_free_large(bo_handles);
> - drm_free_large(buflist);
> + kvfree(bo_handles);
> + kvfree(buflist);
> return -ENOENT;
> }
>
> @@ -151,7 +152,7 @@ static int virtio_gpu_execbuffer_ioctl(struct drm_device *dev, void *data,
>
> list_add(&buflist[i].head, &validate_list);
> }
> - drm_free_large(bo_handles);
> + kvfree(bo_handles);
> }
>
> ret = virtio_gpu_object_list_validate(&ticket, &validate_list);
> @@ -171,7 +172,7 @@ static int virtio_gpu_execbuffer_ioctl(struct drm_device *dev, void *data,
>
> /* fence the command bo */
> virtio_gpu_unref_list(&validate_list);
> - drm_free_large(buflist);
> + kvfree(buflist);
> dma_fence_put(&fence->f);
> return 0;
>
> @@ -179,7 +180,7 @@ static int virtio_gpu_execbuffer_ioctl(struct drm_device *dev, void *data,
> ttm_eu_backoff_reservation(&ticket, &validate_list);
> out_free:
> virtio_gpu_unref_list(&validate_list);
> - drm_free_large(buflist);
> + kvfree(buflist);
> return ret;
> }
>
> diff --git a/include/drm/drmP.h b/include/drm/drmP.h
> index e1daa4f343cd..59df08d14b89 100644
> --- a/include/drm/drmP.h
> +++ b/include/drm/drmP.h
> @@ -70,7 +70,6 @@
> #include <drm/drm_fourcc.h>
> #include <drm/drm_global.h>
> #include <drm/drm_hashtab.h>
> -#include <drm/drm_mem_util.h>
> #include <drm/drm_mm.h>
> #include <drm/drm_os_linux.h>
> #include <drm/drm_sarea.h>
> diff --git a/include/drm/drm_mem_util.h b/include/drm/drm_mem_util.h
> deleted file mode 100644
> index d0f6cf2e5324..000000000000
> --- a/include/drm/drm_mem_util.h
> +++ /dev/null
> @@ -1,78 +0,0 @@
> -/*
> - * Copyright (C) 2008 Intel Corporation
> - *
> - * Permission is hereby granted, free of charge, to any person obtaining a
> - * copy of this software and associated documentation files (the "Software"),
> - * to deal in the Software without restriction, including without limitation
> - * the rights to use, copy, modify, merge, publish, distribute, sublicense,
> - * and/or sell copies of the Software, and to permit persons to whom the
> - * Software is furnished to do so, subject to the following conditions:
> - *
> - * The above copyright notice and this permission notice (including the next
> - * paragraph) shall be included in all copies or substantial portions of the
> - * Software.
> - *
> - * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
> - * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> - * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
> - * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
> - * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
> - * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
> - * IN THE SOFTWARE.
> - *
> - * Authors:
> - * Jesse Barnes <jbarnes@virtuousgeek.org>
> - *
> - */
> -#ifndef _DRM_MEM_UTIL_H_
> -#define _DRM_MEM_UTIL_H_
> -
> -#include <linux/vmalloc.h>
> -
> -static __inline__ void *drm_calloc_large(size_t nmemb, size_t size)
> -{
> - if (size != 0 && nmemb > SIZE_MAX / size)
> - return NULL;
> -
> - if (size * nmemb <= PAGE_SIZE)
> - return kcalloc(nmemb, size, GFP_KERNEL);
> -
> - return vzalloc(size * nmemb);
> -}
> -
> -/* Modeled after cairo's malloc_ab, it's like calloc but without the zeroing. */
> -static __inline__ void *drm_malloc_ab(size_t nmemb, size_t size)
> -{
> - if (size != 0 && nmemb > SIZE_MAX / size)
> - return NULL;
> -
> - if (size * nmemb <= PAGE_SIZE)
> - return kmalloc(nmemb * size, GFP_KERNEL);
> -
> - return vmalloc(size * nmemb);
> -}
> -
> -static __inline__ void *drm_malloc_gfp(size_t nmemb, size_t size, gfp_t gfp)
> -{
> - if (size != 0 && nmemb > SIZE_MAX / size)
> - return NULL;
> -
> - if (size * nmemb <= PAGE_SIZE)
> - return kmalloc(nmemb * size, gfp);
> -
> - if (gfp & __GFP_RECLAIMABLE) {
> - void *ptr = kmalloc(nmemb * size,
> - gfp | __GFP_NOWARN | __GFP_NORETRY);
> - if (ptr)
> - return ptr;
> - }
> -
> - return __vmalloc(size * nmemb, gfp, PAGE_KERNEL);
> -}
> -
> -static __inline void drm_free_large(void *ptr)
> -{
> - kvfree(ptr);
> -}
> -
> -#endif
> --
> 2.11.0
>
> --
> Michal Hocko
> SUSE Labs
--
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH] drm: use kvmalloc_array for drm_malloc*
From: Chris Wilson @ 2017-05-16 11:09 UTC (permalink / raw)
To: Michal Hocko
Cc: dri-devel, linux-kernel, linux-mm, Daniel Vetter, Jani Nikula,
Sean Paul, David Airlie
In-Reply-To: <20170516105352.GH2481@dhcp22.suse.cz>
On Tue, May 16, 2017 at 12:53:52PM +0200, Michal Hocko wrote:
> On Tue 16-05-17 10:31:19, Chris Wilson wrote:
> > On Tue, May 16, 2017 at 11:06:06AM +0200, Michal Hocko wrote:
> > > From: Michal Hocko <mhocko@suse.com>
> > >
> > > drm_malloc* has grown their own kmalloc with vmalloc fallback
> > > implementations. MM has grown kvmalloc* helpers in the meantime. Let's
> > > use those because it a) reduces the code and b) MM has a better idea
> > > how to implement fallbacks (e.g. do not vmalloc before kmalloc is tried
> > > with __GFP_NORETRY).
> >
> > Better? The same idea. The only difference I was reluctant to hand out
> > large pages for long lived objects. If that's the wisdom of the core mm,
> > so be it.
>
> vmalloc tends to fragment physical memory more os it is preferable to
> try the physically contiguous request first and only fall back to
> vmalloc if the first attempt would be too costly or it fails.
Not relevant for the changelog in this patch, but it would be nice to
have that written in kvmalloc() as to why the scatterring of 4k vmapped
pages prevents defragmentation when compared to allocating large pages.
I have vague recollections of seeing the conversation, but a summary as
to the reason why kvmalloc prefers large pages will be good for future
reference.
-Chris
--
Chris Wilson, Intel Open Source Technology Centre
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH] drm: use kvmalloc_array for drm_malloc*
From: Michal Hocko @ 2017-05-16 10:53 UTC (permalink / raw)
To: Chris Wilson
Cc: dri-devel, linux-kernel, linux-mm, Daniel Vetter, Jani Nikula,
Sean Paul, David Airlie
In-Reply-To: <20170516093119.GW19912@nuc-i3427.alporthouse.com>
On Tue 16-05-17 10:31:19, Chris Wilson wrote:
> On Tue, May 16, 2017 at 11:06:06AM +0200, Michal Hocko wrote:
> > From: Michal Hocko <mhocko@suse.com>
> >
> > drm_malloc* has grown their own kmalloc with vmalloc fallback
> > implementations. MM has grown kvmalloc* helpers in the meantime. Let's
> > use those because it a) reduces the code and b) MM has a better idea
> > how to implement fallbacks (e.g. do not vmalloc before kmalloc is tried
> > with __GFP_NORETRY).
>
> Better? The same idea. The only difference I was reluctant to hand out
> large pages for long lived objects. If that's the wisdom of the core mm,
> so be it.
vmalloc tends to fragment physical memory more os it is preferable to
try the physically contiguous request first and only fall back to
vmalloc if the first attempt would be too costly or it fails.
> Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk>
thanks!
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* [RFC PATCH 5/5] userfaultfd: non-cooperative: allow synchronous EVENT_REMOVE
From: Mike Rapoport @ 2017-05-16 10:36 UTC (permalink / raw)
To: Andrea Arcangeli; +Cc: Pavel Emelyanov, linux-mm, Mike Rapoport
In-Reply-To: <1494930962-3318-1-git-send-email-rppt@linux.vnet.ibm.com>
In non-cooperative case, multi-threaded userfaultfd monitor may encounter a
race between UFFDIO_COPY and the processing of UFFD_EVENT_REMOVE.
Unlike the page faults that suspend the faulting thread until the page
fault is resolved, other events resume exectution of the thread that caused
the event immediately after delivering the notification to the userfaultfd
monitor. The monitor may run UFFDIO_COPY in parallel with the event
processing and this may result in memory corruption.
With UFFD_EVENT_REMOVE_SYNC introduced by this patch, it would be possible
to block the non-cooperative thread until the userfaultfd monitor will
explicitly wake it.
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
---
fs/userfaultfd.c | 29 ++++++++++++++++++++++++++++-
include/uapi/linux/userfaultfd.h | 11 +++++++++++
2 files changed, 39 insertions(+), 1 deletion(-)
diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c
index 8868229..1167d0e 100644
--- a/fs/userfaultfd.c
+++ b/fs/userfaultfd.c
@@ -609,6 +609,14 @@ static void userfaultfd_event_complete(struct userfaultfd_ctx *ctx,
{
struct userfaultfd_wake_key key = { 0 };
+ /*
+ * For synchronous events we don't wake up the thread that
+ * caused the event. The userfault monitor has to explicitly
+ * wake it with ioctl(UFFDIO_WAKE_SYNC_EVENT)
+ */
+ if (ewq->msg.event & UFFD_EVENT_FLAG_SYNC)
+ return;
+
key.event = ewq->msg.event;
__wake_up_locked_key(&ctx->fault_pending_wqh, TASK_NORMAL, &key);
}
@@ -729,7 +737,8 @@ bool userfaultfd_remove(struct vm_area_struct *vma,
struct userfaultfd_wait_queue ewq;
ctx = vma->vm_userfaultfd_ctx.ctx;
- if (!ctx || !(ctx->features & UFFD_FEATURE_EVENT_REMOVE))
+ if (!ctx || !(ctx->features & UFFD_FEATURE_EVENT_REMOVE ||
+ ctx->features & UFFD_FEATURE_EVENT_REMOVE_SYNC))
return true;
userfaultfd_ctx_get(ctx);
@@ -738,6 +747,9 @@ bool userfaultfd_remove(struct vm_area_struct *vma,
msg_init(&ewq.msg);
ewq.msg.event = UFFD_EVENT_REMOVE;
+ if (ctx->features & UFFD_FEATURE_EVENT_REMOVE_SYNC)
+ ewq.msg.event |= UFFD_EVENT_FLAG_SYNC;
+
ewq.msg.arg.remove.start = start;
ewq.msg.arg.remove.end = end;
@@ -1564,6 +1576,18 @@ static int userfaultfd_wake(struct userfaultfd_ctx *ctx,
return ret;
}
+static int userfaultfd_wake_sync_event(struct userfaultfd_ctx *ctx,
+ unsigned long arg)
+{
+ struct userfaultfd_wake_key key = {
+ .event = arg,
+ };
+
+ wake_userfault(ctx, &key);
+
+ return 0;
+}
+
static int userfaultfd_copy(struct userfaultfd_ctx *ctx,
unsigned long arg)
{
@@ -1734,6 +1758,9 @@ static long userfaultfd_ioctl(struct file *file, unsigned cmd,
case UFFDIO_WAKE:
ret = userfaultfd_wake(ctx, arg);
break;
+ case UFFDIO_WAKE_SYNC_EVENT:
+ ret = userfaultfd_wake_sync_event(ctx, arg);
+ break;
case UFFDIO_COPY:
ret = userfaultfd_copy(ctx, arg);
break;
diff --git a/include/uapi/linux/userfaultfd.h b/include/uapi/linux/userfaultfd.h
index 3b05953..b1b15e4 100644
--- a/include/uapi/linux/userfaultfd.h
+++ b/include/uapi/linux/userfaultfd.h
@@ -21,6 +21,7 @@
#define UFFD_API_FEATURES (UFFD_FEATURE_EVENT_FORK | \
UFFD_FEATURE_EVENT_REMAP | \
UFFD_FEATURE_EVENT_REMOVE | \
+ UFFD_FEATURE_EVENT_REMOVE_SYNC | \
UFFD_FEATURE_EVENT_UNMAP | \
UFFD_FEATURE_MISSING_HUGETLBFS | \
UFFD_FEATURE_MISSING_SHMEM)
@@ -49,6 +50,7 @@
#define _UFFDIO_WAKE (0x02)
#define _UFFDIO_COPY (0x03)
#define _UFFDIO_ZEROPAGE (0x04)
+#define _UFFDIO_WAKE_SYNC_EVENT (0x05)
#define _UFFDIO_API (0x3F)
/* userfaultfd ioctl ids */
@@ -65,6 +67,7 @@
struct uffdio_copy)
#define UFFDIO_ZEROPAGE _IOWR(UFFDIO, _UFFDIO_ZEROPAGE, \
struct uffdio_zeropage)
+#define UFFDIO_WAKE_SYNC_EVENT _IOR(UFFDIO, _UFFDIO_WAKE_SYNC_EVENT, __u32)
/* read() structure */
struct uffd_msg {
@@ -113,6 +116,13 @@ struct uffd_msg {
#define UFFD_EVENT_REMOVE 0x15
#define UFFD_EVENT_UNMAP 0x16
+/*
+ * Events that are delivered synchronously. The causing thread is
+ * blocked until the event is handled by the userfault monitor
+ */
+#define UFFD_EVENT_FLAG_SYNC 0x80
+#define UFFD_EVENT_REMOVE_SYNC (UFFD_EVENT_REMOVE | UFFD_EVENT_FLAG_SYNC)
+
/* flags for UFFD_EVENT_PAGEFAULT */
#define UFFD_PAGEFAULT_FLAG_WRITE (1<<0) /* If this was a write fault */
#define UFFD_PAGEFAULT_FLAG_WP (1<<1) /* If reason is VM_UFFD_WP */
@@ -161,6 +171,7 @@ struct uffdio_api {
#define UFFD_FEATURE_MISSING_HUGETLBFS (1<<4)
#define UFFD_FEATURE_MISSING_SHMEM (1<<5)
#define UFFD_FEATURE_EVENT_UNMAP (1<<6)
+#define UFFD_FEATURE_EVENT_REMOVE_SYNC (1<<7)
__u64 features;
__u64 ioctls;
--
2.7.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [RFC PATCH 4/5] userfaultfd: non-cooperative: use fault_pending_wqh for all events
From: Mike Rapoport @ 2017-05-16 10:36 UTC (permalink / raw)
To: Andrea Arcangeli; +Cc: Pavel Emelyanov, linux-mm, Mike Rapoport
In-Reply-To: <1494930962-3318-1-git-send-email-rppt@linux.vnet.ibm.com>
Queuing page faults and non-cooperative events into different wait queues
does not have real value but rather makes the code more complicated.
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
---
fs/userfaultfd.c | 64 +++++++++++++++++++++-----------------------------------
1 file changed, 24 insertions(+), 40 deletions(-)
diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c
index 1bd772a..8868229 100644
--- a/fs/userfaultfd.c
+++ b/fs/userfaultfd.c
@@ -48,8 +48,6 @@ struct userfaultfd_ctx {
wait_queue_head_t fault_wqh;
/* waitqueue head for the pseudo fd to wakeup poll/read */
wait_queue_head_t fd_wqh;
- /* waitqueue head for events */
- wait_queue_head_t event_wqh;
/* a refile sequence protected by fault_pending_wqh lock */
struct seqcount refile_seq;
/* pseudo fd refcounting */
@@ -101,6 +99,9 @@ struct userfaultfd_wake_key {
static bool userfaultfd_should_wake(struct userfaultfd_wait_queue *uwq,
struct userfaultfd_wake_key *key)
{
+ if (key->event != uwq->msg.event)
+ return false;
+
if (key->event == UFFD_EVENT_PAGEFAULT) {
unsigned long start, len, address;
@@ -188,8 +189,6 @@ static void userfaultfd_ctx_put(struct userfaultfd_ctx *ctx)
VM_BUG_ON(waitqueue_active(&ctx->fault_pending_wqh));
VM_BUG_ON(spin_is_locked(&ctx->fault_wqh.lock));
VM_BUG_ON(waitqueue_active(&ctx->fault_wqh));
- VM_BUG_ON(spin_is_locked(&ctx->event_wqh.lock));
- VM_BUG_ON(waitqueue_active(&ctx->event_wqh));
VM_BUG_ON(spin_is_locked(&ctx->fd_wqh.lock));
VM_BUG_ON(waitqueue_active(&ctx->fd_wqh));
mmdrop(ctx->mm);
@@ -560,22 +559,21 @@ static void userfaultfd_event_wait_completion(struct userfaultfd_ctx *ctx,
if (WARN_ON_ONCE(current->flags & PF_EXITING))
goto out;
- ewq->ctx = ctx;
- init_waitqueue_entry(&ewq->wq, current);
+ userfaultfd_init_waitqueue(ctx, ewq);
- spin_lock(&ctx->event_wqh.lock);
+ spin_lock(&ctx->fault_pending_wqh.lock);
/*
* After the __add_wait_queue the uwq is visible to userland
* through poll/read().
*/
- __add_wait_queue(&ctx->event_wqh, &ewq->wq);
+ __add_wait_queue(&ctx->fault_pending_wqh, &ewq->wq);
for (;;) {
set_current_state(TASK_KILLABLE);
- if (ewq->msg.event == 0)
+ if (READ_ONCE(ewq->waken))
break;
if (ACCESS_ONCE(ctx->released) ||
fatal_signal_pending(current)) {
- __remove_wait_queue(&ctx->event_wqh, &ewq->wq);
+ __remove_wait_queue(&ctx->fault_pending_wqh, &ewq->wq);
if (ewq->msg.event == UFFD_EVENT_FORK) {
struct userfaultfd_ctx *new;
@@ -588,15 +586,15 @@ static void userfaultfd_event_wait_completion(struct userfaultfd_ctx *ctx,
break;
}
- spin_unlock(&ctx->event_wqh.lock);
+ spin_unlock(&ctx->fault_pending_wqh.lock);
wake_up_poll(&ctx->fd_wqh, POLLIN);
schedule();
- spin_lock(&ctx->event_wqh.lock);
+ spin_lock(&ctx->fault_pending_wqh.lock);
}
__set_current_state(TASK_RUNNING);
- spin_unlock(&ctx->event_wqh.lock);
+ spin_unlock(&ctx->fault_pending_wqh.lock);
/*
* ctx may go away after this if the userfault pseudo fd is
@@ -609,9 +607,10 @@ static void userfaultfd_event_wait_completion(struct userfaultfd_ctx *ctx,
static void userfaultfd_event_complete(struct userfaultfd_ctx *ctx,
struct userfaultfd_wait_queue *ewq)
{
- ewq->msg.event = 0;
- wake_up_locked(&ctx->event_wqh);
- __remove_wait_queue(&ctx->event_wqh, &ewq->wq);
+ struct userfaultfd_wake_key key = { 0 };
+
+ key.event = ewq->msg.event;
+ __wake_up_locked_key(&ctx->fault_pending_wqh, TASK_NORMAL, &key);
}
int dup_userfaultfd(struct vm_area_struct *vma, struct list_head *fcs)
@@ -898,12 +897,6 @@ static inline struct userfaultfd_wait_queue *find_userfault(
return find_userfault_in(&ctx->fault_pending_wqh);
}
-static inline struct userfaultfd_wait_queue *find_userfault_evt(
- struct userfaultfd_ctx *ctx)
-{
- return find_userfault_in(&ctx->event_wqh);
-}
-
static unsigned int userfaultfd_poll(struct file *file, poll_table *wait)
{
struct userfaultfd_ctx *ctx = file->private_data;
@@ -935,8 +928,6 @@ static unsigned int userfaultfd_poll(struct file *file, poll_table *wait)
smp_mb();
if (waitqueue_active(&ctx->fault_pending_wqh))
ret = POLLIN;
- else if (waitqueue_active(&ctx->event_wqh))
- ret = POLLIN;
return ret;
default:
@@ -981,7 +972,7 @@ static ssize_t userfaultfd_ctx_read(struct userfaultfd_ctx *ctx, int no_wait,
struct userfaultfd_wait_queue *uwq;
/*
* Handling fork event requires sleeping operations, so
- * we drop the event_wqh lock, then do these ops, then
+ * we drop the fault_pending_wqh lock, then do these ops, then
* lock it back and wake up the waiter. While the lock is
* dropped the ewq may go away so we keep track of it
* carefully.
@@ -996,7 +987,7 @@ static ssize_t userfaultfd_ctx_read(struct userfaultfd_ctx *ctx, int no_wait,
set_current_state(TASK_INTERRUPTIBLE);
spin_lock(&ctx->fault_pending_wqh.lock);
uwq = find_userfault(ctx);
- if (uwq) {
+ if (uwq && uwq->msg.event == UFFD_EVENT_PAGEFAULT) {
/*
* Use a seqcount to repeat the lockless check
* in wake_userfault() to avoid missing
@@ -1037,12 +1028,7 @@ static ssize_t userfaultfd_ctx_read(struct userfaultfd_ctx *ctx, int no_wait,
spin_unlock(&ctx->fault_pending_wqh.lock);
ret = 0;
break;
- }
- spin_unlock(&ctx->fault_pending_wqh.lock);
-
- spin_lock(&ctx->event_wqh.lock);
- uwq = find_userfault_evt(ctx);
- if (uwq) {
+ } else if (uwq) { /* non-pagefault event */
*msg = uwq->msg;
if (uwq->msg.event == UFFD_EVENT_FORK) {
@@ -1050,17 +1036,16 @@ static ssize_t userfaultfd_ctx_read(struct userfaultfd_ctx *ctx, int no_wait,
(unsigned long)
uwq->msg.arg.reserved.reserved1;
list_move(&uwq->wq.task_list, &fork_event);
- spin_unlock(&ctx->event_wqh.lock);
+ spin_unlock(&ctx->fault_pending_wqh.lock);
ret = 0;
break;
}
-
userfaultfd_event_complete(ctx, uwq);
- spin_unlock(&ctx->event_wqh.lock);
+ spin_unlock(&ctx->fault_pending_wqh.lock);
ret = 0;
break;
}
- spin_unlock(&ctx->event_wqh.lock);
+ spin_unlock(&ctx->fault_pending_wqh.lock);
if (signal_pending(current)) {
ret = -ERESTARTSYS;
@@ -1082,16 +1067,16 @@ static ssize_t userfaultfd_ctx_read(struct userfaultfd_ctx *ctx, int no_wait,
ret = resolve_userfault_fork(ctx, fork_nctx, msg);
if (!ret) {
- spin_lock(&ctx->event_wqh.lock);
+ spin_lock(&ctx->fault_pending_wqh.lock);
if (!list_empty(&fork_event)) {
uwq = list_first_entry(&fork_event,
typeof(*uwq),
wq.task_list);
list_del(&uwq->wq.task_list);
- __add_wait_queue(&ctx->event_wqh, &uwq->wq);
+ __add_wait_queue(&ctx->fault_pending_wqh, &uwq->wq);
userfaultfd_event_complete(ctx, uwq);
}
- spin_unlock(&ctx->event_wqh.lock);
+ spin_unlock(&ctx->fault_pending_wqh.lock);
}
}
@@ -1808,7 +1793,6 @@ static void init_once_userfaultfd_ctx(void *mem)
init_waitqueue_head(&ctx->fault_pending_wqh);
init_waitqueue_head(&ctx->fault_wqh);
- init_waitqueue_head(&ctx->event_wqh);
init_waitqueue_head(&ctx->fd_wqh);
seqcount_init(&ctx->refile_seq);
}
--
2.7.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [RFC PATCH 3/5] userfaultfd: non-cooperative: generalize wake key structure
From: Mike Rapoport @ 2017-05-16 10:36 UTC (permalink / raw)
To: Andrea Arcangeli; +Cc: Pavel Emelyanov, linux-mm, Mike Rapoport
In-Reply-To: <1494930962-3318-1-git-send-email-rppt@linux.vnet.ibm.com>
Upcoming support for synchronous non-page-fault events will require
userfaultfd_wake_function to be able to differentiate between the event
types. Depending on the event type, different parameters will define if the
wait queue element should be awaken. This requires usage of more general
structure than userfaultfd_wake_range to be used as the "key" parameter for
userfaultfd_wake_function.
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
---
fs/userfaultfd.c | 96 +++++++++++++++++++++++++++++++++-----------------------
1 file changed, 57 insertions(+), 39 deletions(-)
diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c
index fee5f08..1bd772a 100644
--- a/fs/userfaultfd.c
+++ b/fs/userfaultfd.c
@@ -91,31 +91,40 @@ struct userfaultfd_wake_range {
unsigned long len;
};
+struct userfaultfd_wake_key {
+ u8 event;
+ union {
+ struct userfaultfd_wake_range range;
+ } arg;
+};
+
static bool userfaultfd_should_wake(struct userfaultfd_wait_queue *uwq,
- struct userfaultfd_wake_range *range)
+ struct userfaultfd_wake_key *key)
{
- unsigned long start, len, address;
-
- /* len == 0 means wake all */
- address = uwq->msg.arg.pagefault.address;
- start = range->start;
- len = range->len;
- if (len && (start > address || start + len <= address))
- return false;
+ if (key->event == UFFD_EVENT_PAGEFAULT) {
+ unsigned long start, len, address;
+
+ /* len == 0 means wake all */
+ address = uwq->msg.arg.pagefault.address;
+ start = key->arg.range.start;
+ len = key->arg.range.len;
+ if (len && (start > address || start + len <= address))
+ return false;
+ }
return true;
}
static int userfaultfd_wake_function(wait_queue_t *wq, unsigned mode,
- int wake_flags, void *key)
+ int wake_flags, void *_key)
{
- struct userfaultfd_wake_range *range = key;
+ struct userfaultfd_wake_key *key = _key;
int ret;
struct userfaultfd_wait_queue *uwq;
uwq = container_of(wq, struct userfaultfd_wait_queue, wq);
ret = 0;
- if (!userfaultfd_should_wake(uwq, range))
+ if (!userfaultfd_should_wake(uwq, key))
goto out;
WRITE_ONCE(uwq->waken, true);
/*
@@ -802,7 +811,12 @@ static int userfaultfd_release(struct inode *inode, struct file *file)
struct mm_struct *mm = ctx->mm;
struct vm_area_struct *vma, *prev;
/* len == 0 means wake all */
- struct userfaultfd_wake_range range = { .len = 0, };
+ struct userfaultfd_wake_key key = {
+ .event = UFFD_EVENT_PAGEFAULT,
+ .arg.range = {
+ .len = 0,
+ },
+ };
unsigned long new_flags;
ACCESS_ONCE(ctx->released) = true;
@@ -850,8 +864,8 @@ static int userfaultfd_release(struct inode *inode, struct file *file)
* the fault_*wqh.
*/
spin_lock(&ctx->fault_pending_wqh.lock);
- __wake_up_locked_key(&ctx->fault_pending_wqh, TASK_NORMAL, &range);
- __wake_up_locked_key(&ctx->fault_wqh, TASK_NORMAL, &range);
+ __wake_up_locked_key(&ctx->fault_pending_wqh, TASK_NORMAL, &key);
+ __wake_up_locked_key(&ctx->fault_wqh, TASK_NORMAL, &key);
spin_unlock(&ctx->fault_pending_wqh.lock);
wake_up_poll(&ctx->fd_wqh, POLLHUP);
@@ -1115,20 +1129,20 @@ static ssize_t userfaultfd_read(struct file *file, char __user *buf,
}
static void __wake_userfault(struct userfaultfd_ctx *ctx,
- struct userfaultfd_wake_range *range)
+ struct userfaultfd_wake_key *key)
{
spin_lock(&ctx->fault_pending_wqh.lock);
/* wake all in the range and autoremove */
if (waitqueue_active(&ctx->fault_pending_wqh))
__wake_up_locked_key(&ctx->fault_pending_wqh, TASK_NORMAL,
- range);
+ key);
if (waitqueue_active(&ctx->fault_wqh))
- __wake_up_locked_key(&ctx->fault_wqh, TASK_NORMAL, range);
+ __wake_up_locked_key(&ctx->fault_wqh, TASK_NORMAL, key);
spin_unlock(&ctx->fault_pending_wqh.lock);
}
static __always_inline void wake_userfault(struct userfaultfd_ctx *ctx,
- struct userfaultfd_wake_range *range)
+ struct userfaultfd_wake_key *key)
{
unsigned seq;
bool need_wakeup;
@@ -1155,7 +1169,7 @@ static __always_inline void wake_userfault(struct userfaultfd_ctx *ctx,
cond_resched();
} while (read_seqcount_retry(&ctx->refile_seq, seq));
if (need_wakeup)
- __wake_userfault(ctx, range);
+ __wake_userfault(ctx, key);
}
static __always_inline int validate_range(struct mm_struct *mm,
@@ -1481,10 +1495,11 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx,
* permanently and it avoids userland to call
* UFFDIO_WAKE explicitly.
*/
- struct userfaultfd_wake_range range;
- range.start = start;
- range.len = vma_end - start;
- wake_userfault(vma->vm_userfaultfd_ctx.ctx, &range);
+ struct userfaultfd_wake_key key;
+ key.event = UFFD_EVENT_PAGEFAULT;
+ key.arg.range.start = start;
+ key.arg.range.len = vma_end - start;
+ wake_userfault(vma->vm_userfaultfd_ctx.ctx, &key);
}
new_flags = vma->vm_flags & ~(VM_UFFD_MISSING | VM_UFFD_WP);
@@ -1536,7 +1551,7 @@ static int userfaultfd_wake(struct userfaultfd_ctx *ctx,
{
int ret;
struct uffdio_range uffdio_wake;
- struct userfaultfd_wake_range range;
+ struct userfaultfd_wake_key key;
const void __user *buf = (void __user *)arg;
ret = -EFAULT;
@@ -1547,16 +1562,17 @@ static int userfaultfd_wake(struct userfaultfd_ctx *ctx,
if (ret)
goto out;
- range.start = uffdio_wake.start;
- range.len = uffdio_wake.len;
+ key.event = UFFD_EVENT_PAGEFAULT;
+ key.arg.range.start = uffdio_wake.start;
+ key.arg.range.len = uffdio_wake.len;
/*
* len == 0 means wake all and we don't want to wake all here,
* so check it again to be sure.
*/
- VM_BUG_ON(!range.len);
+ VM_BUG_ON(!key.arg.range.len);
- wake_userfault(ctx, &range);
+ wake_userfault(ctx, &key);
ret = 0;
out:
@@ -1569,7 +1585,7 @@ static int userfaultfd_copy(struct userfaultfd_ctx *ctx,
__s64 ret;
struct uffdio_copy uffdio_copy;
struct uffdio_copy __user *user_uffdio_copy;
- struct userfaultfd_wake_range range;
+ struct userfaultfd_wake_key key;
user_uffdio_copy = (struct uffdio_copy __user *) arg;
@@ -1605,12 +1621,13 @@ static int userfaultfd_copy(struct userfaultfd_ctx *ctx,
goto out;
BUG_ON(!ret);
/* len == 0 would wake all */
- range.len = ret;
+ key.event = UFFD_EVENT_PAGEFAULT;
+ key.arg.range.len = ret;
if (!(uffdio_copy.mode & UFFDIO_COPY_MODE_DONTWAKE)) {
- range.start = uffdio_copy.dst;
- wake_userfault(ctx, &range);
+ key.arg.range.start = uffdio_copy.dst;
+ wake_userfault(ctx, &key);
}
- ret = range.len == uffdio_copy.len ? 0 : -EAGAIN;
+ ret = key.arg.range.len == uffdio_copy.len ? 0 : -EAGAIN;
out:
return ret;
}
@@ -1621,7 +1638,7 @@ static int userfaultfd_zeropage(struct userfaultfd_ctx *ctx,
__s64 ret;
struct uffdio_zeropage uffdio_zeropage;
struct uffdio_zeropage __user *user_uffdio_zeropage;
- struct userfaultfd_wake_range range;
+ struct userfaultfd_wake_key key;
user_uffdio_zeropage = (struct uffdio_zeropage __user *) arg;
@@ -1650,12 +1667,13 @@ static int userfaultfd_zeropage(struct userfaultfd_ctx *ctx,
goto out;
/* len == 0 would wake all */
BUG_ON(!ret);
- range.len = ret;
+ key.event = UFFD_EVENT_PAGEFAULT;
+ key.arg.range.len = ret;
if (!(uffdio_zeropage.mode & UFFDIO_ZEROPAGE_MODE_DONTWAKE)) {
- range.start = uffdio_zeropage.range.start;
- wake_userfault(ctx, &range);
+ key.arg.range.start = uffdio_zeropage.range.start;
+ wake_userfault(ctx, &key);
}
- ret = range.len == uffdio_zeropage.range.len ? 0 : -EAGAIN;
+ ret = key.arg.range.len == uffdio_zeropage.range.len ? 0 : -EAGAIN;
out:
return ret;
}
--
2.7.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [RFC PATCH 1/5] userfaultfd: introduce userfault_init_waitqueue helper
From: Mike Rapoport @ 2017-05-16 10:35 UTC (permalink / raw)
To: Andrea Arcangeli; +Cc: Pavel Emelyanov, linux-mm, Mike Rapoport
In-Reply-To: <1494930962-3318-1-git-send-email-rppt@linux.vnet.ibm.com>
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
---
fs/userfaultfd.c | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)
diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c
index 1446e9d..b061e96 100644
--- a/fs/userfaultfd.c
+++ b/fs/userfaultfd.c
@@ -134,6 +134,15 @@ static int userfaultfd_wake_function(wait_queue_t *wq, unsigned mode,
return ret;
}
+static inline void userfaultfd_init_waitqueue(struct userfaultfd_ctx *ctx,
+ struct userfaultfd_wait_queue *uwq)
+{
+ init_waitqueue_func_entry(&uwq->wq, userfaultfd_wake_function);
+ uwq->wq.private = current;
+ uwq->ctx = ctx;
+ uwq->waken = false;
+}
+
/**
* userfaultfd_ctx_get - Acquires a reference to the internal userfaultfd
* context.
@@ -405,11 +414,8 @@ int handle_userfault(struct vm_fault *vmf, unsigned long reason)
/* take the reference before dropping the mmap_sem */
userfaultfd_ctx_get(ctx);
- init_waitqueue_func_entry(&uwq.wq, userfaultfd_wake_function);
- uwq.wq.private = current;
+ userfaultfd_init_waitqueue(ctx, &uwq);
uwq.msg = userfault_msg(vmf->address, vmf->flags, reason);
- uwq.ctx = ctx;
- uwq.waken = false;
return_to_userland =
(vmf->flags & (FAULT_FLAG_USER|FAULT_FLAG_KILLABLE)) ==
--
2.7.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [RFC PATCH 2/5] userfaultfd: introduce userfaultfd_should_wait helper
From: Mike Rapoport @ 2017-05-16 10:35 UTC (permalink / raw)
To: Andrea Arcangeli; +Cc: Pavel Emelyanov, linux-mm, Mike Rapoport
In-Reply-To: <1494930962-3318-1-git-send-email-rppt@linux.vnet.ibm.com>
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
---
fs/userfaultfd.c | 22 ++++++++++++++++------
1 file changed, 16 insertions(+), 6 deletions(-)
diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c
index b061e96..fee5f08 100644
--- a/fs/userfaultfd.c
+++ b/fs/userfaultfd.c
@@ -91,21 +91,31 @@ struct userfaultfd_wake_range {
unsigned long len;
};
+static bool userfaultfd_should_wake(struct userfaultfd_wait_queue *uwq,
+ struct userfaultfd_wake_range *range)
+{
+ unsigned long start, len, address;
+
+ /* len == 0 means wake all */
+ address = uwq->msg.arg.pagefault.address;
+ start = range->start;
+ len = range->len;
+ if (len && (start > address || start + len <= address))
+ return false;
+
+ return true;
+}
+
static int userfaultfd_wake_function(wait_queue_t *wq, unsigned mode,
int wake_flags, void *key)
{
struct userfaultfd_wake_range *range = key;
int ret;
struct userfaultfd_wait_queue *uwq;
- unsigned long start, len;
uwq = container_of(wq, struct userfaultfd_wait_queue, wq);
ret = 0;
- /* len == 0 means wake all */
- start = range->start;
- len = range->len;
- if (len && (start > uwq->msg.arg.pagefault.address ||
- start + len <= uwq->msg.arg.pagefault.address))
+ if (!userfaultfd_should_wake(uwq, range))
goto out;
WRITE_ONCE(uwq->waken, true);
/*
--
2.7.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [RFC PATCH 0/5] userfaultfd: non-cooperative: syncronous events
From: Mike Rapoport @ 2017-05-16 10:35 UTC (permalink / raw)
To: Andrea Arcangeli; +Cc: Pavel Emelyanov, linux-mm, Mike Rapoport
Hi,
These patches add ability to generate userfaultfd events so that thier
processing will be synchronized with the non-cooperative thread that caused
the event.
In the non-cooperative case userfaultfd resumes execution of the thread
that caused an event when the notification is read() by the uffd monitor.
In some cases, like, for example, madvise(MADV_REMOVE), it might be
desirable to keep the thread that caused the event suspended until the
uffd monitor had the event handled.
The first two patches just shuffle the code a bit to make subsequent
changes easier.
The patches 3 and 4 create some unification in the way the threads are
queued into waitqueues either after page fault or after a non-cooperative
event.
The fifth patch extends the userfaultfd API with an implementation of
UFFD_EVENT_REMOVE_SYNC that allows to keep the thread that triggered
UFFD_EVENT_REMOVE until the uffd monitor would not wake it explicitly.
Mike Rapoport (5):
userfaultfd: introduce userfault_init_waitqueue helper
userfaultfd: introduce userfaultfd_should_wait helper
userfaultfd: non-cooperative: generalize wake key structure
userfaultfd: non-cooperative: use fault_pending_wqh for all events
userfaultfd: non-cooperative: allow synchronous EVENT_REMOVE
fs/userfaultfd.c | 205 ++++++++++++++++++++++++---------------
include/uapi/linux/userfaultfd.h | 11 +++
2 files changed, 136 insertions(+), 80 deletions(-)
--
2.7.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* [PATCH] userfaultfd: drop dead code
From: Mike Rapoport @ 2017-05-16 10:35 UTC (permalink / raw)
To: Andrew Morton; +Cc: Andrea Arcangeli, linux-mm, lkml, Mike Rapoport
Calculation of start end end in __wake_userfault function are not used and
can be removed.
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
---
fs/userfaultfd.c | 5 -----
1 file changed, 5 deletions(-)
diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c
index f7555fc..1446e9d 100644
--- a/fs/userfaultfd.c
+++ b/fs/userfaultfd.c
@@ -1101,11 +1101,6 @@ static ssize_t userfaultfd_read(struct file *file, char __user *buf,
static void __wake_userfault(struct userfaultfd_ctx *ctx,
struct userfaultfd_wake_range *range)
{
- unsigned long start, end;
-
- start = range->start;
- end = range->start + range->len;
-
spin_lock(&ctx->fault_pending_wqh.lock);
/* wake all in the range and autoremove */
if (waitqueue_active(&ctx->fault_pending_wqh))
--
2.7.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [PATCH] exit: don't include unused userfaultfd_k.h
From: Mike Rapoport @ 2017-05-16 10:35 UTC (permalink / raw)
To: Andrew Morton; +Cc: Andrea Arcangeli, linux-mm, lkml, Mike Rapoport
Commit dd0db88d8094 (userfaultfd: non-cooperative: rollback
userfaultfd_exit) removed userfaultfd callback from exit() which makes
include of <linux/userfaultfd_k.h> unnecessary.
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
---
kernel/exit.c | 1 -
1 file changed, 1 deletion(-)
diff --git a/kernel/exit.c b/kernel/exit.c
index 516acdb..5b1dc84 100644
--- a/kernel/exit.c
+++ b/kernel/exit.c
@@ -51,7 +51,6 @@
#include <linux/task_io_accounting_ops.h>
#include <linux/tracehook.h>
#include <linux/fs_struct.h>
-#include <linux/userfaultfd_k.h>
#include <linux/init_task.h>
#include <linux/perf_event.h>
#include <trace/events/sched.h>
--
2.7.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* Re: [PATCH v2 2/2] powerpc/mm/hugetlb: Add support for 1G huge pages
From: Anshuman Khandual @ 2017-05-16 10:27 UTC (permalink / raw)
To: Aneesh Kumar K.V, akpm, mpe, Anshuman Khandual
Cc: linux-mm, linux-kernel, linuxppc-dev
In-Reply-To: <1494926264-22463-2-git-send-email-aneesh.kumar@linux.vnet.ibm.com>
On 05/16/2017 02:47 PM, Aneesh Kumar K.V wrote:
> POWER9 supports hugepages of size 2M and 1G in radix MMU mode. This patch
> enables the usage of 1G page size for hugetlbfs. This also update the helper
> such we can do 1G page allocation at runtime.
>
> We still don't enable 1G page size on DD1 version. This is to avoid doing
> workaround mentioned in commit: 6d3a0379ebdc8 (powerpc/mm: Add
> radix__tlb_flush_pte_p9_dd1()
>
> Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
Sounds good.
Reviewed-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH v2 1/2] mm/hugetlb: Cleanup ARCH_HAS_GIGANTIC_PAGE
From: Anshuman Khandual @ 2017-05-16 10:22 UTC (permalink / raw)
To: Aneesh Kumar K.V, akpm, mpe, Anshuman Khandual
Cc: linux-mm, linux-kernel, linuxppc-dev
In-Reply-To: <1494926264-22463-1-git-send-email-aneesh.kumar@linux.vnet.ibm.com>
On 05/16/2017 02:47 PM, Aneesh Kumar K.V wrote:
> This moves the #ifdef in C code to a Kconfig dependency. Also we move the
> gigantic_page_supported() function to be arch specific. This gives arch to
> conditionally enable runtime allocation of gigantic huge page. Architectures
> like ppc64 supports different gigantic huge page size (16G and 1G) based on the
> translation mode selected. This provides an opportunity for ppc64 to enable
> runtime allocation only w.r.t 1G hugepage.
Right.
>
> No functional change in this patch.
>
> Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
> ---
> arch/arm64/Kconfig | 2 +-
> arch/arm64/include/asm/hugetlb.h | 4 ++++
> arch/s390/Kconfig | 2 +-
> arch/s390/include/asm/hugetlb.h | 3 +++
> arch/x86/Kconfig | 2 +-
> mm/hugetlb.c | 7 ++-----
> 6 files changed, 12 insertions(+), 8 deletions(-)
>
> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> index 3741859765cf..1f8c1f73aada 100644
> --- a/arch/arm64/Kconfig
> +++ b/arch/arm64/Kconfig
> @@ -11,7 +11,7 @@ config ARM64
> select ARCH_HAS_ACPI_TABLE_UPGRADE if ACPI
> select ARCH_HAS_ELF_RANDOMIZE
> select ARCH_HAS_GCOV_PROFILE_ALL
> - select ARCH_HAS_GIGANTIC_PAGE
> + select ARCH_HAS_GIGANTIC_PAGE if MEMORY_ISOLATION && COMPACTION && CMA
> select ARCH_HAS_KCOV
> select ARCH_HAS_SET_MEMORY
> select ARCH_HAS_SG_CHAIN
> diff --git a/arch/arm64/include/asm/hugetlb.h b/arch/arm64/include/asm/hugetlb.h
> index bbc1e35aa601..793bd73b0d07 100644
> --- a/arch/arm64/include/asm/hugetlb.h
> +++ b/arch/arm64/include/asm/hugetlb.h
> @@ -83,4 +83,8 @@ extern void huge_ptep_set_wrprotect(struct mm_struct *mm,
> extern void huge_ptep_clear_flush(struct vm_area_struct *vma,
> unsigned long addr, pte_t *ptep);
>
> +#ifdef CONFIG_ARCH_HAS_GIGANTIC_PAGE
> +static inline bool gigantic_page_supported(void) { return true; }
> +#endif
> +
> #endif /* __ASM_HUGETLB_H */
> diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig
> index a2dcef0aacc7..a41bbf420dda 100644
> --- a/arch/s390/Kconfig
> +++ b/arch/s390/Kconfig
> @@ -67,7 +67,7 @@ config S390
> select ARCH_HAS_DEVMEM_IS_ALLOWED
> select ARCH_HAS_ELF_RANDOMIZE
> select ARCH_HAS_GCOV_PROFILE_ALL
> - select ARCH_HAS_GIGANTIC_PAGE
> + select ARCH_HAS_GIGANTIC_PAGE if MEMORY_ISOLATION && COMPACTION && CMA
> select ARCH_HAS_KCOV
> select ARCH_HAS_SET_MEMORY
> select ARCH_HAS_SG_CHAIN
> diff --git a/arch/s390/include/asm/hugetlb.h b/arch/s390/include/asm/hugetlb.h
> index cd546a245c68..89057b2cc8fe 100644
> --- a/arch/s390/include/asm/hugetlb.h
> +++ b/arch/s390/include/asm/hugetlb.h
> @@ -112,4 +112,7 @@ static inline pte_t huge_pte_modify(pte_t pte, pgprot_t newprot)
> return pte_modify(pte, newprot);
> }
>
> +#ifdef CONFIG_ARCH_HAS_GIGANTIC_PAGE
> +static inline bool gigantic_page_supported(void) { return true; }
> +#endif
> #endif /* _ASM_S390_HUGETLB_H */
> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> index cc98d5a294ee..30a6328136ac 100644
> --- a/arch/x86/Kconfig
> +++ b/arch/x86/Kconfig
> @@ -22,7 +22,7 @@ config X86_64
> def_bool y
> depends on 64BIT
> # Options that are inherently 64-bit kernel only:
> - select ARCH_HAS_GIGANTIC_PAGE
> + select ARCH_HAS_GIGANTIC_PAGE if MEMORY_ISOLATION && COMPACTION && CMA
> select ARCH_SUPPORTS_INT128
> select ARCH_USE_CMPXCHG_LOCKREF
> select HAVE_ARCH_SOFT_DIRTY
Should not we define gigantic_page_supported() function for X86 as well
like the other two archs above ?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* [PATCH V3] mm/madvise: Enable (soft|hard) offline of HugeTLB pages at PGD level
From: Anshuman Khandual @ 2017-05-16 10:05 UTC (permalink / raw)
To: linux-kernel, linux-mm; +Cc: akpm
In-Reply-To: <20170426035731.6924-1-khandual@linux.vnet.ibm.com>
Though migrating gigantic HugeTLB pages does not sound much like real
world use case, they can be affected by memory errors. Hence migration
at the PGD level HugeTLB pages should be supported just to enable soft
and hard offline use cases.
While allocating the new gigantic HugeTLB page, it should not matter
whether new page comes from the same node or not. There would be very
few gigantic pages on the system afterall, we should not be bothered
about node locality when trying to save a big page from crashing.
This change renames dequeu_huge_page_node() function as dequeue_huge
_page_node_exact() preserving it's original functionality. Now the new
dequeue_huge_page_node() function scans through all available online
nodes to allocate a huge page for the NUMA_NO_NODE case and just falls
back calling dequeu_huge_page_node_exact() for all other cases.
Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
---
Changes in V3:
* Dropped alloc_huge_page_nonid() as per Andrew
* Changed dequeue_huge_page_node() to accommodate NUMA_NO_NODE as per Andrew
* Added dequeue_huge_page_node_exact() which implements functionality for the
previous dequeue_huge_page_node() function
Changes in V2:
* Added hstate_is_gigantic() definition when !CONFIG_HUGETLB_PAGE
which takes care of the build failure reported earlier.
include/linux/hugetlb.h | 7 ++++++-
mm/hugetlb.c | 18 +++++++++++++++++-
mm/memory-failure.c | 13 +++++++++----
3 files changed, 32 insertions(+), 6 deletions(-)
diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index b857fc8cc2ec..614a0a40f1ef 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -466,7 +466,11 @@ extern int dissolve_free_huge_pages(unsigned long start_pfn,
static inline bool hugepage_migration_supported(struct hstate *h)
{
#ifdef CONFIG_ARCH_ENABLE_HUGEPAGE_MIGRATION
- return huge_page_shift(h) == PMD_SHIFT;
+ if ((huge_page_shift(h) == PMD_SHIFT) ||
+ (huge_page_shift(h) == PGDIR_SHIFT))
+ return true;
+ else
+ return false;
#else
return false;
#endif
@@ -518,6 +522,7 @@ struct hstate {};
#define vma_mmu_pagesize(v) PAGE_SIZE
#define huge_page_order(h) 0
#define huge_page_shift(h) PAGE_SHIFT
+#define hstate_is_gigantic(h) 0
static inline unsigned int pages_per_huge_page(struct hstate *h)
{
return 1;
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index e5828875f7bb..7cd0f09b8dd0 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -867,7 +867,7 @@ static void enqueue_huge_page(struct hstate *h, struct page *page)
h->free_huge_pages_node[nid]++;
}
-static struct page *dequeue_huge_page_node(struct hstate *h, int nid)
+static struct page *dequeue_huge_page_node_exact(struct hstate *h, int nid)
{
struct page *page;
@@ -887,6 +887,22 @@ static struct page *dequeue_huge_page_node(struct hstate *h, int nid)
return page;
}
+static struct page *dequeue_huge_page_node(struct hstate *h, int nid)
+{
+ struct page *page;
+ int node;
+
+ if (nid != NUMA_NO_NODE)
+ return dequeue_huge_page_node_exact(h, nid);
+
+ for_each_online_node(node) {
+ page = dequeue_huge_page_node_exact(h, node);
+ if (page)
+ return page;
+ }
+ return NULL;
+}
+
/* Movability of hugepages depends on migration support. */
static inline gfp_t htlb_alloc_mask(struct hstate *h)
{
diff --git a/mm/memory-failure.c b/mm/memory-failure.c
index 2527dfeddb00..f71efae2e494 100644
--- a/mm/memory-failure.c
+++ b/mm/memory-failure.c
@@ -1489,11 +1489,16 @@ EXPORT_SYMBOL(unpoison_memory);
static struct page *new_page(struct page *p, unsigned long private, int **x)
{
int nid = page_to_nid(p);
- if (PageHuge(p))
- return alloc_huge_page_node(page_hstate(compound_head(p)),
- nid);
- else
+ if (PageHuge(p)) {
+ struct hstate *hstate = page_hstate(compound_head(p));
+
+ if (hstate_is_gigantic(hstate))
+ return alloc_huge_page_node(hstate, NUMA_NO_NODE);
+
+ return alloc_huge_page_node(hstate, nid);
+ } else {
return __alloc_pages_node(nid, GFP_HIGHUSER_MOVABLE, 0);
+ }
}
/*
--
2.12.0
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* Re: [PATCH 06/17] mm/shmem: expose driver overridable huge option
From: Kirill A. Shutemov @ 2017-05-16 10:02 UTC (permalink / raw)
To: Matthew Auld
Cc: intel-gfx, Joonas Lahtinen, Dave Hansen, Daniel Vetter,
Hugh Dickins, linux-mm
In-Reply-To: <20170516082948.28090-7-matthew.auld@intel.com>
On Tue, May 16, 2017 at 09:29:37AM +0100, Matthew Auld wrote:
> In i915 we are aiming to support huge GTT pages for the GPU, and to
> complement this we also want to enable THP for our shmem backed objects.
> Even though THP is supported in shmemfs it can only be enabled through
> the huge= mount option, but for users of the kernel mounted shm_mnt like
> i915, we are a little stuck. There is the sysfs knob shmem_enabled to
> either forcefully enable/disable the feature, but that seems to only be
> useful for testing purposes. What we propose is to expose a driver
> overridable huge option as part of shmem_inode_info to control the use
> of THP for a given mapping.
I don't like this. It's kinda hacky.
Is there a reason why i915 cannot mount a new tmpfs for own use?
Or other option would be to change default to SHMEM_HUGE_ADVISE and wire
up fadvise handle to control per-file allocation policy.
--
Kirill A. Shutemov
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH] drm: use kvmalloc_array for drm_malloc*
From: Michal Hocko @ 2017-05-16 9:52 UTC (permalink / raw)
To: Daniel Vetter
Cc: dri-devel, linux-kernel, linux-mm, Daniel Vetter, Jani Nikula,
Sean Paul, David Airlie
In-Reply-To: <20170516092230.pzadndxm5gq4i4h6@phenom.ffwll.local>
On Tue 16-05-17 11:22:30, Daniel Vetter wrote:
> On Tue, May 16, 2017 at 11:06:06AM +0200, Michal Hocko wrote:
> > From: Michal Hocko <mhocko@suse.com>
> >
> > drm_malloc* has grown their own kmalloc with vmalloc fallback
> > implementations. MM has grown kvmalloc* helpers in the meantime. Let's
> > use those because it a) reduces the code and b) MM has a better idea
> > how to implement fallbacks (e.g. do not vmalloc before kmalloc is tried
> > with __GFP_NORETRY).
> >
> > Signed-off-by: Michal Hocko <mhocko@suse.com>
>
> Shouldn't we go one step further and just remove these wrappers, maybe
> with cocci?
my cocci sucks...
> Especially drm_malloc_gfp is surpremely pointless after this
> patch (and drm_malloc_ab probably not that useful either).
So what about the following instead? It passes allyesconfig compilation.
---
^ permalink raw reply
* Re: [PATCH] drm: use kvmalloc_array for drm_malloc*
From: Chris Wilson @ 2017-05-16 9:31 UTC (permalink / raw)
To: Michal Hocko
Cc: dri-devel, linux-kernel, linux-mm, Daniel Vetter, Jani Nikula,
Sean Paul, David Airlie, Michal Hocko
In-Reply-To: <20170516090606.5891-1-mhocko@kernel.org>
On Tue, May 16, 2017 at 11:06:06AM +0200, Michal Hocko wrote:
> From: Michal Hocko <mhocko@suse.com>
>
> drm_malloc* has grown their own kmalloc with vmalloc fallback
> implementations. MM has grown kvmalloc* helpers in the meantime. Let's
> use those because it a) reduces the code and b) MM has a better idea
> how to implement fallbacks (e.g. do not vmalloc before kmalloc is tried
> with __GFP_NORETRY).
Better? The same idea. The only difference I was reluctant to hand out
large pages for long lived objects. If that's the wisdom of the core mm,
so be it.
Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk>
-Chris
--
Chris Wilson, Intel Open Source Technology Centre
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH] mm: per-cgroup memory reclaim stats
From: Michal Hocko @ 2017-05-16 9:29 UTC (permalink / raw)
To: Roman Gushchin
Cc: Johannes Weiner, Tejun Heo, Li Zefan, Vladimir Davydov, cgroups,
linux-doc, linux-kernel, linux-mm
In-Reply-To: <1494530183-30808-1-git-send-email-guro@fb.com>
On Thu 11-05-17 20:16:23, Roman Gushchin wrote:
> Track the following reclaim counters for every memory cgroup:
> PGREFILL, PGSCAN, PGSTEAL, PGACTIVATE, PGDEACTIVATE, PGLAZYFREE and
> PGLAZYFREED.
yes, those are definitely useful. I have an old patch to add them as
well but never managed to clean it up and post...
> These values are exposed using the memory.stats interface of cgroup v2.
Is there any reason to not add them to v1? This should be rather trivial
after recent changes from Johannes.
> The meaning of each value is the same as for global counters,
> available using /proc/vmstat.
>
> Also, for consistency, rename mem_cgroup_count_vm_event() to
> count_memcg_event_mm().
>
> Signed-off-by: Roman Gushchin <guro@fb.com>
> Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
> Cc: Johannes Weiner <hannes@cmpxchg.org>
> Cc: Tejun Heo <tj@kernel.org>
> Cc: Li Zefan <lizefan@huawei.com>
> Cc: Michal Hocko <mhocko@kernel.org>
> Cc: Vladimir Davydov <vdavydov.dev@gmail.com>
> Cc: cgroups@vger.kernel.org
> Cc: linux-doc@vger.kernel.org
> Cc: linux-kernel@vger.kernel.org
> Cc: linux-mm@kvack.org
the patch itself looks good to me. I will have to double check it after
I am done with what I am doing currently and then will add my Acked-by
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* [PATCH v2 9/9] powerpc/hugetlb: Enable hugetlb migration for ppc64
From: Aneesh Kumar K.V @ 2017-05-16 9:23 UTC (permalink / raw)
To: akpm, mpe; +Cc: linux-mm, linux-kernel, linuxppc-dev, Aneesh Kumar K.V
In-Reply-To: <1494926612-23928-1-git-send-email-aneesh.kumar@linux.vnet.ibm.com>
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
---
arch/powerpc/platforms/Kconfig.cputype | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/arch/powerpc/platforms/Kconfig.cputype b/arch/powerpc/platforms/Kconfig.cputype
index 80175000042d..8acc4f27d101 100644
--- a/arch/powerpc/platforms/Kconfig.cputype
+++ b/arch/powerpc/platforms/Kconfig.cputype
@@ -351,6 +351,11 @@ config PPC_RADIX_MMU
is only implemented by IBM Power9 CPUs, if you don't have one of them
you can probably disable this.
+config ARCH_ENABLE_HUGEPAGE_MIGRATION
+ def_bool y
+ depends on PPC_BOOK3S_64 && HUGETLB_PAGE && MIGRATION
+
+
config PPC_MMU_NOHASH
def_bool y
depends on !PPC_STD_MMU
--
2.7.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [PATCH v2 5/9] mm/hugetlb: Move default definition of hugepd_t earlier in the header
From: Aneesh Kumar K.V @ 2017-05-16 9:23 UTC (permalink / raw)
To: akpm, mpe; +Cc: linux-mm, linux-kernel, linuxppc-dev, Aneesh Kumar K.V
In-Reply-To: <1494926612-23928-1-git-send-email-aneesh.kumar@linux.vnet.ibm.com>
This enable to use the hugepd_t type early. No functional change in this patch.
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
---
include/linux/hugetlb.h | 47 ++++++++++++++++++++++++-----------------------
1 file changed, 24 insertions(+), 23 deletions(-)
diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index edab98f0a7b8..f66c1d4e0d1f 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -14,6 +14,30 @@ struct ctl_table;
struct user_struct;
struct mmu_gather;
+#ifndef is_hugepd
+/*
+ * Some architectures requires a hugepage directory format that is
+ * required to support multiple hugepage sizes. For example
+ * a4fe3ce76 "powerpc/mm: Allow more flexible layouts for hugepage pagetables"
+ * introduced the same on powerpc. This allows for a more flexible hugepage
+ * pagetable layout.
+ */
+typedef struct { unsigned long pd; } hugepd_t;
+#define is_hugepd(hugepd) (0)
+#define __hugepd(x) ((hugepd_t) { (x) })
+static inline int gup_huge_pd(hugepd_t hugepd, unsigned long addr,
+ unsigned pdshift, unsigned long end,
+ int write, struct page **pages, int *nr)
+{
+ return 0;
+}
+#else
+extern int gup_huge_pd(hugepd_t hugepd, unsigned long addr,
+ unsigned pdshift, unsigned long end,
+ int write, struct page **pages, int *nr);
+#endif
+
+
#ifdef CONFIG_HUGETLB_PAGE
#include <linux/mempolicy.h>
@@ -222,29 +246,6 @@ static inline int pud_write(pud_t pud)
}
#endif
-#ifndef is_hugepd
-/*
- * Some architectures requires a hugepage directory format that is
- * required to support multiple hugepage sizes. For example
- * a4fe3ce76 "powerpc/mm: Allow more flexible layouts for hugepage pagetables"
- * introduced the same on powerpc. This allows for a more flexible hugepage
- * pagetable layout.
- */
-typedef struct { unsigned long pd; } hugepd_t;
-#define is_hugepd(hugepd) (0)
-#define __hugepd(x) ((hugepd_t) { (x) })
-static inline int gup_huge_pd(hugepd_t hugepd, unsigned long addr,
- unsigned pdshift, unsigned long end,
- int write, struct page **pages, int *nr)
-{
- return 0;
-}
-#else
-extern int gup_huge_pd(hugepd_t hugepd, unsigned long addr,
- unsigned pdshift, unsigned long end,
- int write, struct page **pages, int *nr);
-#endif
-
#define HUGETLB_ANON_FILE "anon_hugepage"
enum {
--
2.7.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [PATCH v2 8/9] powerpc/mm/hugetlb: Remove follow_huge_addr for powerpc
From: Aneesh Kumar K.V @ 2017-05-16 9:23 UTC (permalink / raw)
To: akpm, mpe; +Cc: linux-mm, linux-kernel, linuxppc-dev, Aneesh Kumar K.V
In-Reply-To: <1494926612-23928-1-git-send-email-aneesh.kumar@linux.vnet.ibm.com>
With generic code now handling hugetlb entries at pgd level and also
supporting hugepage directory format, we can now remove the powerpc
sepcific follow_huge_addr implementation.
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
---
arch/powerpc/mm/hugetlbpage.c | 64 -------------------------------------------
1 file changed, 64 deletions(-)
diff --git a/arch/powerpc/mm/hugetlbpage.c b/arch/powerpc/mm/hugetlbpage.c
index 5c829a83a4cc..1816b965a142 100644
--- a/arch/powerpc/mm/hugetlbpage.c
+++ b/arch/powerpc/mm/hugetlbpage.c
@@ -619,11 +619,6 @@ void hugetlb_free_pgd_range(struct mmu_gather *tlb,
} while (addr = next, addr != end);
}
-/*
- * 64 bit book3s use generic follow_page_mask
- */
-#ifdef CONFIG_PPC_BOOK3S_64
-
struct page *follow_huge_pd(struct vm_area_struct *vma,
unsigned long address, hugepd_t hpd,
int flags, int pdshift)
@@ -657,65 +652,6 @@ struct page *follow_huge_pd(struct vm_area_struct *vma,
return page;
}
-#else /* !CONFIG_PPC_BOOK3S_64 */
-
-/*
- * We are holding mmap_sem, so a parallel huge page collapse cannot run.
- * To prevent hugepage split, disable irq.
- */
-struct page *
-follow_huge_addr(struct mm_struct *mm, unsigned long address, int write)
-{
- bool is_thp;
- pte_t *ptep, pte;
- unsigned shift;
- unsigned long mask, flags;
- struct page *page = ERR_PTR(-EINVAL);
-
- local_irq_save(flags);
- ptep = find_linux_pte_or_hugepte(mm->pgd, address, &is_thp, &shift);
- if (!ptep)
- goto no_page;
- pte = READ_ONCE(*ptep);
- /*
- * Verify it is a huge page else bail.
- * Transparent hugepages are handled by generic code. We can skip them
- * here.
- */
- if (!shift || is_thp)
- goto no_page;
-
- if (!pte_present(pte)) {
- page = NULL;
- goto no_page;
- }
- mask = (1UL << shift) - 1;
- page = pte_page(pte);
- if (page)
- page += (address & mask) / PAGE_SIZE;
-
-no_page:
- local_irq_restore(flags);
- return page;
-}
-
-struct page *
-follow_huge_pmd(struct mm_struct *mm, unsigned long address,
- pmd_t *pmd, int write)
-{
- BUG();
- return NULL;
-}
-
-struct page *
-follow_huge_pud(struct mm_struct *mm, unsigned long address,
- pud_t *pud, int write)
-{
- BUG();
- return NULL;
-}
-#endif
-
static unsigned long hugepte_addr_end(unsigned long addr, unsigned long end,
unsigned long sz)
{
--
2.7.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [PATCH v2 7/9] powerpc/hugetlb: Add follow_huge_pd implementation for ppc64.
From: Aneesh Kumar K.V @ 2017-05-16 9:23 UTC (permalink / raw)
To: akpm, mpe; +Cc: linux-mm, linux-kernel, linuxppc-dev, Aneesh Kumar K.V
In-Reply-To: <1494926612-23928-1-git-send-email-aneesh.kumar@linux.vnet.ibm.com>
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
---
arch/powerpc/mm/hugetlbpage.c | 43 +++++++++++++++++++++++++++++++++++++++++++
1 file changed, 43 insertions(+)
diff --git a/arch/powerpc/mm/hugetlbpage.c b/arch/powerpc/mm/hugetlbpage.c
index 80f6d2ed551a..5c829a83a4cc 100644
--- a/arch/powerpc/mm/hugetlbpage.c
+++ b/arch/powerpc/mm/hugetlbpage.c
@@ -17,6 +17,8 @@
#include <linux/memblock.h>
#include <linux/bootmem.h>
#include <linux/moduleparam.h>
+#include <linux/swap.h>
+#include <linux/swapops.h>
#include <asm/pgtable.h>
#include <asm/pgalloc.h>
#include <asm/tlb.h>
@@ -618,6 +620,46 @@ void hugetlb_free_pgd_range(struct mmu_gather *tlb,
}
/*
+ * 64 bit book3s use generic follow_page_mask
+ */
+#ifdef CONFIG_PPC_BOOK3S_64
+
+struct page *follow_huge_pd(struct vm_area_struct *vma,
+ unsigned long address, hugepd_t hpd,
+ int flags, int pdshift)
+{
+ pte_t *ptep;
+ spinlock_t *ptl;
+ struct page *page = NULL;
+ unsigned long mask;
+ int shift = hugepd_shift(hpd);
+ struct mm_struct *mm = vma->vm_mm;
+
+retry:
+ ptl = &mm->page_table_lock;
+ spin_lock(ptl);
+
+ ptep = hugepte_offset(hpd, address, pdshift);
+ if (pte_present(*ptep)) {
+ mask = (1UL << shift) - 1;
+ page = pte_page(*ptep);
+ page += ((address & mask) >> PAGE_SHIFT);
+ if (flags & FOLL_GET)
+ get_page(page);
+ } else {
+ if (is_hugetlb_entry_migration(*ptep)) {
+ spin_unlock(ptl);
+ __migration_entry_wait(mm, ptep, ptl);
+ goto retry;
+ }
+ }
+ spin_unlock(ptl);
+ return page;
+}
+
+#else /* !CONFIG_PPC_BOOK3S_64 */
+
+/*
* We are holding mmap_sem, so a parallel huge page collapse cannot run.
* To prevent hugepage split, disable irq.
*/
@@ -672,6 +714,7 @@ follow_huge_pud(struct mm_struct *mm, unsigned long address,
BUG();
return NULL;
}
+#endif
static unsigned long hugepte_addr_end(unsigned long addr, unsigned long end,
unsigned long sz)
--
2.7.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox