From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 65863388369; Tue, 8 Sep 2026 20:05:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788897950; cv=none; b=A1Iva2UJlB4ac5QBDYSdPDGFOeyYfT1P6YxeKHsqFsdDWchDKX2Vi1rIfib7E2wp0VoM7q1jnegWdQxXW/QJ4LpAJIn3Zwj2JE3bJnJZBJGm4zVnjaOUxi9za6+Mwn43m5CHtXgQvfzMPBSmR/TUH9QcJtpLSAcBp1A1Bjxd2DE= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788897950; c=relaxed/simple; bh=RVcXI/9kSHuUo8O2heRZlHSZv1OgtVQ9ABB55RKE5TI=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=dFuJFN6YHG6ePkFH3p8AzsKqzHgHLIftJt8rQQZs2bz/Qb8t2m5ojgrMekcnX/R9p2Q3+/H8y3zNmJU9XrDdgrZerr7Bs5ck/R992302J8JRl8rA/q05Na9aQD/CtYaVf9rTLGokEjGGWRQA1KqdqhrFfIoijS59THR1FBVPOZ0= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Jc+25493; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Jc+25493" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 07C1F1F00A3A; Tue, 8 Sep 2026 20:05:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1788897948; bh=iscL7XQ2qz+Nzy4lBqsV3dTnKv6z1UKwiafW35/dBb4=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=Jc+25493tsckTbWCOZP+Zyv7md+VlPGAmEn3On6Sc+eAYFaK+dxOenSmzWLxaAOv8 i3nIqz8EUF8c80uLWntZ2aH9Lokq5iMJ74K7L9SPQTDc33lfTe5MZE/3gRynv7Y9JW Od6nbpeIIyAIOfstTWQdWkxXIeIQErrA+CfTpMp4DPftLLDpReZQ0x/FCFNigTagYU m78Su9hMxY5pV5va1RRv8CX4QDeKU3wEGpZI797elPK/sE2gaxlEp405CEO3TbJme1 akL1BLZnCJhjae4iUstmbk8Shgs+1z8bevPlT8sngmYD4yFsXVXTY4rdXE3RXyOJiN X4RroRrlG5uvg== From: "Lorenzo Stoakes (ARM)" Date: Tue, 08 Sep 2026 21:01:11 +0100 Subject: [PATCH 07/39] mm: add mmap action for discontiguous kernel page mapping Precedence: bulk X-Mailing-List: linux-trace-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20260908-b4-mmap-prepare-vma-flag-sanify-v1-7-dacf19cce22b@kernel.org> References: <20260908-b4-mmap-prepare-vma-flag-sanify-v1-0-dacf19cce22b@kernel.org> In-Reply-To: <20260908-b4-mmap-prepare-vma-flag-sanify-v1-0-dacf19cce22b@kernel.org> To: Andrew Morton , "Liam R. Howlett" , Vlastimil Babka , Jann Horn , Pedro Falcato , David Hildenbrand , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Jonathan Corbet , Greg Kroah-Hartman , Dennis Dalessandro , Jason Gunthorpe , Leon Romanovsky , Paul Moore , Stephen Smalley , Jaroslav Kysela , Takashi Iwai , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Kumar Kartikeya Dwivedi , Zi Yan , Baolin Wang , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Kiryl Shutsemau , Doug Gilbert , "James E.J. Bottomley" , "Martin K. Petersen" , Jaya Kumar , Simona Vetter , Helge Deller , Sebastian Reichel , John Hubbard , Peter Xu , Masami Hiramatsu , Oleg Nesterov , Peter Zijlstra , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Arnaldo Carvalho de Melo , Namhyung Kim , Mark Rutland , Rik van Riel , Harry Yoo , Juri Lelli , Vincent Guittot , Maarten Lankhorst , Maxime Ripard , Thomas Zimmermann , David Airlie , Will Deacon , "Aneesh Kumar K.V" , Nick Piggin , Arnd Bergmann , Muchun Song , Oscar Salvador , "Matthew Wilcox (Oracle)" , Jan Kara , Marc Zyngier , Oliver Upton , Catalin Marinas , Madhavan Srinivasan , Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , Christian Borntraeger , Janosch Frank , Claudio Imbrenda , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , "David S. Miller" , Andreas Larsson , Alexander Viro , Christian Brauner , Matthew Brost , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , Ying Huang , Alistair Popple , Chris Li , Kairui Song , Kemeng Shi , Nhat Pham , Baoquan He , Youngjun Park , Johannes Weiner , Qi Zheng , Shakeel Butt , Axel Rasmussen , Yuanchu Xie , Wei Xu , Xu Xin , Chengming Zhou , Michal Hocko , Miklos Szeredi Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-usb@vger.kernel.org, linux-rdma@vger.kernel.org, selinux@vger.kernel.org, linux-sound@vger.kernel.org, bpf@vger.kernel.org, linux-scsi@vger.kernel.org, linux-fbdev@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-trace-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linuxppc-dev@lists.ozlabs.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, fuse-devel@lists.linux.dev, "Lorenzo Stoakes (ARM)" X-Mailer: b4 0.14.3 X-Developer-Signature: v=1; a=openpgp-sha256; l=12539; i=ljs@kernel.org; h=from:subject:message-id; bh=RVcXI/9kSHuUo8O2heRZlHSZv1OgtVQ9ABB55RKE5TI=; b=owGbwMvMwCV2fu7ZrsZH9SKMp9WSGLIWZG4U2FvyUfRZ7qrcGX7ettfbfIQWMv3c5vsl01KS+ ZfSGYGjHaUsDGJcDLJiiizPv4jvDxIJm9d5wd8NZg4rE8gQBi5OAZjIotkM/9PrnrryT5T9JbHv 7FUt4W8ihz56XkprWBZecM9Gab5lYT8jw/XSN3Zb3/VGtSsx1gq/ucS4tTJ3k4J33a4fkycVXNm RwgwA X-Developer-Key: i=ljs@kernel.org; a=openpgp; fpr=E7F417BF5214569E89D04F46CF9DCD8A81E27F14 The existing kernel page mapping mmap actions allow for partial and full mapping of an array of struct page pointers. However some drivers require the mapping of discontiguous ranges. Permit this by providing discontig_kernel_page_ops which allows a driver to specify how the operation should begin and how batches of pages should be retrieved. It uses the minimum exposed interface to do so, providing address, page offset and both vm_private_data state and a local private state object. ops->init can establish state for the operation, and ops->get outputs the pages to map and their count. Should an error arise the core unmaps the VMA, invoking vm_ops->close, which is therefore where any state established by ops->init is released. The code carefully checks to ensure the user doesn't exceed VMA bounds. It intentionally allows mapping of less than the VMA range in case the driver wishes to allow the user to map an area larger than available data. To use it, users invoke mmap_action_map_discontig_kernel_pages() with initial local private state and a set of operations. Users can then use one of the provided helper functions to perform an action: * discontig_kernel_map_abort() - Abort and leave the mapping as it has been accumulated so far. * discontig_kernel_map_page() - Map a single page, or a compound page given its head page. * discontig_kernel_map_page_range() - Maps a struct page ** array of a specified count. The enum copy in the userland VMA tests is updated accordingly. Signed-off-by: Lorenzo Stoakes (ARM) --- include/linux/mm.h | 45 +++++++++++++++++ include/linux/mm_types.h | 44 +++++++++++++++- mm/internal.h | 3 ++ mm/memory.c | 108 ++++++++++++++++++++++++++++++++++++++-- mm/util.c | 7 +++ tools/testing/vma/include/dup.h | 5 +- 6 files changed, 203 insertions(+), 9 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 94b860b8c50a..4604cd011ca4 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -4647,10 +4647,55 @@ static inline void mmap_action_map_kernel_pages_full(struct vm_area_desc *desc, vma_desc_pages(desc)); } +static inline +void mmap_action_map_discontig_kernel_pages(struct vm_area_desc *desc, + void *init_private, const struct discontig_kernel_page_ops *ops) +{ + struct mmap_action *action = &desc->action; + + action->type = MMAP_DISCONTIG_KERNEL_PAGES; + action->map_kernel_discontig.init_private = init_private; + action->map_kernel_discontig.ops = ops; +} + int mmap_action_prepare(struct vm_area_desc *desc); int mmap_action_complete(struct vm_area_struct *vma, struct mmap_action *action, bool is_compat); +static inline void +discontig_kernel_map_abort(struct discontig_kernel_page_state *state) +{ + state->action = DISCONTIG_KERNEL_PAGE_ABORT; +} + +static inline void +discontig_kernel_map_page(struct discontig_kernel_page_state *state, + struct page *page) +{ + struct folio *folio = page_folio(page); + + if (folio_test_large(folio)) { + VM_WARN_ON_ONCE(page != folio_page(folio, 0)); + state->action = DISCONTIG_KERNEL_PAGE_MAP_COMPOUND_PAGE; + state->__folio = folio; + state->__nr_pages = min(state->nr_pages_remain, + folio_nr_pages(folio)); + } else { + state->action = DISCONTIG_KERNEL_PAGE_MAP_PAGE; + state->__page = page; + state->__nr_pages = 1; + } +} + +static inline void +discontig_kernel_map_page_range(struct discontig_kernel_page_state *state, + struct page **page_arr, unsigned long nr_pages) +{ + state->action = DISCONTIG_KERNEL_PAGE_MAP_PAGE_RANGE; + state->__page_arr = page_arr; + state->__nr_pages = nr_pages; +} + /* Look up the first VMA which exactly match the interval vm_start ... vm_end */ static inline struct vm_area_struct *find_exact_vma(struct mm_struct *mm, unsigned long vm_start, unsigned long vm_end) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 9ca2ea3664bc..0cb4f9603956 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -818,8 +818,44 @@ enum mmap_action_type { MMAP_NOTHING, MMAP_REMAP_PFN, MMAP_IO_REMAP_PFN, - MMAP_SIMPLE_IO_REMAP, /* I/O remap with guardrails. */ - MMAP_KERNEL_PAGES, /* Map kernel page range from array. */ + MMAP_SIMPLE_IO_REMAP, /* I/O remap with guardrails. */ + MMAP_KERNEL_PAGES, /* Map kernel page range from array. */ + MMAP_DISCONTIG_KERNEL_PAGES, /* Map kernel discontig page range. */ +}; + +enum discontig_kernel_page_action { + DISCONTIG_KERNEL_PAGE_ABORT, + DISCONTIG_KERNEL_PAGE_MAP_PAGE, + DISCONTIG_KERNEL_PAGE_MAP_COMPOUND_PAGE, + DISCONTIG_KERNEL_PAGE_MAP_PAGE_RANGE, +}; + +struct discontig_kernel_page_state { + /* Map state. */ + const unsigned long start; /* Start address of VMA. */ + const unsigned long end; /* End address of VMA. */ + unsigned long addr; /* The current address to be mapped. */ + pgoff_t pgoff; /* The current pgoff to be mapped. */ + unsigned long nr_pages_mapped; /* The number of pages mapped. */ + unsigned long nr_pages_remain; /* The number of pages remaining. */ + + /* User-defined state. */ + void *vm_private_data; /* VMA private data. */ + void *private; /* Mapping private data. */ + + /* Users should not touch these, use discontig_kernel_map_*() helpers. */ + enum discontig_kernel_page_action action; + union { + struct page *__page; + struct folio *__folio; + struct page **__page_arr; + }; + unsigned long __nr_pages; +}; + +struct discontig_kernel_page_ops { + int (*init)(void *vm_private_data, void **private); + int (*get)(struct discontig_kernel_page_state *state); }; /* @@ -844,6 +880,10 @@ struct mmap_action { unsigned long nr_pages; pgoff_t pgoff; } map_kernel; + struct { + void *init_private; + const struct discontig_kernel_page_ops *ops; + } map_kernel_discontig; }; enum mmap_action_type type; diff --git a/mm/internal.h b/mm/internal.h index fe93be1cea32..a86b9803a4c3 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1520,6 +1520,9 @@ int simple_ioremap_prepare(struct vm_area_desc *desc); int map_kernel_pages_prepare(struct vm_area_desc *desc); int map_kernel_pages_complete(struct vm_area_struct *vma, struct mmap_action *action); +int map_discontig_kernel_pages_prepare(struct vm_area_desc *desc); +int map_discontig_kernel_pages_complete(struct vm_area_struct *vma, + struct mmap_action *action); static inline int io_remap_pfn_range_prepare(struct vm_area_desc *desc) { diff --git a/mm/memory.c b/mm/memory.c index dc38cad1dcb7..8c9675451d4b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2609,17 +2609,23 @@ int vm_insert_pages(struct vm_area_struct *vma, unsigned long addr, } EXPORT_SYMBOL(vm_insert_pages); +static void __map_kernel_pages_prepare(struct vm_area_desc *desc) +{ + if (vma_desc_test(desc, VMA_MIXEDMAP_BIT)) + return; + + VM_WARN_ON_ONCE(mmap_read_trylock(desc->mm)); + VM_WARN_ON_ONCE(vma_desc_test(desc, VMA_PFNMAP_BIT)); + vma_desc_set_flags(desc, VMA_MIXEDMAP_BIT); +} + int map_kernel_pages_prepare(struct vm_area_desc *desc) { const struct mmap_action *action = &desc->action; const unsigned long addr = action->map_kernel.start; unsigned long nr_pages, end; - if (!vma_desc_test(desc, VMA_MIXEDMAP_BIT)) { - VM_WARN_ON_ONCE(mmap_read_trylock(desc->mm)); - VM_WARN_ON_ONCE(vma_desc_test(desc, VMA_PFNMAP_BIT)); - vma_desc_set_flags(desc, VMA_MIXEDMAP_BIT); - } + __map_kernel_pages_prepare(desc); nr_pages = action->map_kernel.nr_pages; end = addr + PAGE_SIZE * nr_pages; @@ -2640,6 +2646,98 @@ int map_kernel_pages_complete(struct vm_area_struct *vma, &nr_pages, vma->vm_page_prot); } +int map_discontig_kernel_pages_prepare(struct vm_area_desc *desc) +{ + const struct mmap_action *action = &desc->action; + const struct discontig_kernel_page_ops *ops = + action->map_kernel_discontig.ops; + + /* At minimum need to be able to get pages. */ + if (WARN_ON_ONCE(!ops->get)) + return -EINVAL; + + __map_kernel_pages_prepare(desc); + return 0; +} + +static int apply_discontig_action(struct vm_area_struct *vma, + struct discontig_kernel_page_state *state) +{ + unsigned long nr_pages = state->__nr_pages; + unsigned long addr = state->addr; + unsigned long i; + + if (state->action == DISCONTIG_KERNEL_PAGE_MAP_PAGE) + return insert_page(vma, addr, state->__page, + vma->vm_page_prot, /*mkwrite=*/false); + if (state->action == DISCONTIG_KERNEL_PAGE_MAP_PAGE_RANGE) + return insert_pages(vma, addr, state->__page_arr, + &nr_pages, vma->vm_page_prot); + + /* Compound folio - have to iterate through each page. */ + for (i = 0; i < nr_pages; i++, addr += PAGE_SIZE) { + struct page *page = folio_page(state->__folio, i); + int err; + + err = insert_page(vma, addr, page, vma->vm_page_prot, + /*mkwrite=*/false); + if (err) + return err; + } + return 0; +} + +int map_discontig_kernel_pages_complete(struct vm_area_struct *vma, + struct mmap_action *action) +{ + const struct discontig_kernel_page_ops *ops = + action->map_kernel_discontig.ops; + struct discontig_kernel_page_state state = { + .start = vma->vm_start, + .end = vma->vm_end, + .addr = vma->vm_start, + .pgoff = vma->vm_pgoff, + .nr_pages_mapped = 0, + .nr_pages_remain = vma_pages(vma), + .vm_private_data = vma->vm_private_data, + .private = action->map_kernel_discontig.init_private, + }; + int err = 0; + + if (ops->init) + err = ops->init(vma->vm_private_data, &state.private); + if (err) + return err; + + do { + unsigned long end, pgoff_end; + unsigned long nr_pages; + + /* Default to abort. */ + state.action = DISCONTIG_KERNEL_PAGE_ABORT; + err = ops->get(&state); + if (err || state.action == DISCONTIG_KERNEL_PAGE_ABORT) + return err; + nr_pages = state.__nr_pages; + + end = state.addr + PAGE_SIZE * nr_pages; + if (end > vma->vm_end) + return -EINVAL; + pgoff_end = state.pgoff + nr_pages; + + err = apply_discontig_action(vma, &state); + if (err) + return err; + + state.addr = end; + state.pgoff = pgoff_end; + state.nr_pages_mapped += nr_pages; + state.nr_pages_remain -= nr_pages; + } while (state.addr < vma->vm_end); + + return 0; +} + /** * vm_insert_page - insert single page into user vma * @vma: user vma to map to diff --git a/mm/util.c b/mm/util.c index b6f1bec9da15..5a1916d8fdc1 100644 --- a/mm/util.c +++ b/mm/util.c @@ -1464,6 +1464,8 @@ int mmap_action_prepare(struct vm_area_desc *desc) return simple_ioremap_prepare(desc); case MMAP_KERNEL_PAGES: return map_kernel_pages_prepare(desc); + case MMAP_DISCONTIG_KERNEL_PAGES: + return map_discontig_kernel_pages_prepare(desc); } WARN_ON_ONCE(1); @@ -1496,6 +1498,9 @@ int mmap_action_complete(struct vm_area_struct *vma, case MMAP_KERNEL_PAGES: err = map_kernel_pages_complete(vma, action); break; + case MMAP_DISCONTIG_KERNEL_PAGES: + err = map_discontig_kernel_pages_complete(vma, action); + break; case MMAP_IO_REMAP_PFN: case MMAP_SIMPLE_IO_REMAP: /* Should have been delegated. */ @@ -1517,6 +1522,7 @@ int mmap_action_prepare(struct vm_area_desc *desc) case MMAP_IO_REMAP_PFN: case MMAP_SIMPLE_IO_REMAP: case MMAP_KERNEL_PAGES: + case MMAP_DISCONTIG_KERNEL_PAGES: WARN_ON_ONCE(1); /* nommu cannot handle these. */ break; } @@ -1538,6 +1544,7 @@ int mmap_action_complete(struct vm_area_struct *vma, case MMAP_IO_REMAP_PFN: case MMAP_SIMPLE_IO_REMAP: case MMAP_KERNEL_PAGES: + case MMAP_DISCONTIG_KERNEL_PAGES: WARN_ON_ONCE(1); /* nommu cannot handle this. */ err = -EINVAL; diff --git a/tools/testing/vma/include/dup.h b/tools/testing/vma/include/dup.h index 1098655a5f4a..52eee05e6c32 100644 --- a/tools/testing/vma/include/dup.h +++ b/tools/testing/vma/include/dup.h @@ -457,8 +457,9 @@ enum mmap_action_type { MMAP_NOTHING, MMAP_REMAP_PFN, MMAP_IO_REMAP_PFN, - MMAP_SIMPLE_IO_REMAP, /* I/O remap with guardrails. */ - MMAP_KERNEL_PAGES, /* Map kernel page range from array. */ + MMAP_SIMPLE_IO_REMAP, /* I/O remap with guardrails. */ + MMAP_KERNEL_PAGES, /* Map kernel page range from array. */ + MMAP_DISCONTIG_KERNEL_PAGES, /* Map kernel discontig page range. */ }; /* -- 2.55.0