* [PATCH v6 05/30] mm: page_alloc: move prep_compound_page before post_alloc_hook
From: Michael S. Tsirkin @ 2026-05-11 8:53 UTC (permalink / raw)
To: linux-kernel
Cc: David Hildenbrand (Arm), Jason Wang, Xuan Zhuo,
Eugenio Pérez, Muchun Song, Oscar Salvador, Andrew Morton,
Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Brendan Jackman,
Johannes Weiner, Zi Yan, Baolin Wang, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Hugh Dickins, Matthew Brost,
Joshua Hahn, Rakie Kim, Byungchul Park, Gregory Price, Ying Huang,
Alistair Popple, Christoph Lameter, David Rientjes,
Roman Gushchin, Harry Yoo, Axel Rasmussen, Yuanchu Xie, Wei Xu,
Chris Li, Kairui Song, Kemeng Shi, Nhat Pham, Baoquan He,
virtualization, linux-mm, Andrea Arcangeli
In-Reply-To: <cover.1778488966.git.mst@redhat.com>
Move prep_compound_page() before post_alloc_hook() in prep_new_page().
The next patch adds a folio_zero_user() call to post_alloc_hook(),
which uses folio_nr_pages() to determine how many pages to zero.
Without compound metadata set up first, folio_nr_pages() returns 1
for higher-order allocations, so only the first page would be zeroed.
All other operations in post_alloc_hook() (arch_alloc_page, KASAN,
debug, page owner, etc.) use raw page pointers with explicit order
counts and are unaffected by this reordering.
Note: compaction_alloc_noprof() has the opposite ordering
(post_alloc_hook before prep_compound_page). This is fine because
compaction always passes USER_ADDR_NONE, so folio_zero_user() is
never called there and folio_nr_pages() is never reached inside
post_alloc_hook().
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Assisted-by: Claude:claude-opus-4-6
Assisted-by: cursor-agent:GPT-5.4-xhigh
---
mm/page_alloc.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index c3c0f4e2baa7..f76d5271b5c6 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1864,11 +1864,11 @@ static void prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags
unsigned int alloc_flags,
unsigned long user_addr)
{
- post_alloc_hook(page, order, gfp_flags, user_addr);
-
if (order && (gfp_flags & __GFP_COMP))
prep_compound_page(page, order);
+ post_alloc_hook(page, order, gfp_flags, user_addr);
+
/*
* page is set pfmemalloc when ALLOC_NO_WATERMARKS was necessary to
* allocate the page. The expectation is that the caller is taking
--
MST
^ permalink raw reply related
* [PATCH v6 04/30] mm: add folio_zero_user stub for configs without THP/HUGETLBFS
From: Michael S. Tsirkin @ 2026-05-11 8:53 UTC (permalink / raw)
To: linux-kernel
Cc: David Hildenbrand (Arm), Jason Wang, Xuan Zhuo,
Eugenio Pérez, Muchun Song, Oscar Salvador, Andrew Morton,
Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Brendan Jackman,
Johannes Weiner, Zi Yan, Baolin Wang, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Hugh Dickins, Matthew Brost,
Joshua Hahn, Rakie Kim, Byungchul Park, Gregory Price, Ying Huang,
Alistair Popple, Christoph Lameter, David Rientjes,
Roman Gushchin, Harry Yoo, Axel Rasmussen, Yuanchu Xie, Wei Xu,
Chris Li, Kairui Song, Kemeng Shi, Nhat Pham, Baoquan He,
virtualization, linux-mm, Andrea Arcangeli, Liam R. Howlett
In-Reply-To: <cover.1778488966.git.mst@redhat.com>
folio_zero_user() is defined in mm/memory.c under
CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_HUGETLBFS. A subsequent patch
will call it from post_alloc_hook() for all user page zeroing, so
configs without THP or HUGETLBFS will need a stub.
Add a macro in the #else branch that falls back to
clear_user_highpages(), which handles cache aliasing correctly on
VIPT architectures and is always available via highmem.h.
Without THP/HUGETLBFS, only order-0 user pages are allocated, so
the locality optimization in the real folio_zero_user() (zero near
the faulting address last) is not needed.
This also matches what vma_alloc_zeroed_movable_folio currently does.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Assisted-by: Claude:claude-opus-4-6
Assisted-by: cursor-agent:GPT-5.4-xhigh
---
include/linux/mm.h | 3 +++
1 file changed, 3 insertions(+)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index af23453e9dbd..3b1ca90fd435 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -5070,6 +5070,9 @@ long copy_folio_from_user(struct folio *dst_folio,
const void __user *usr_src,
bool allow_pagefault);
+#else /* !CONFIG_TRANSPARENT_HUGEPAGE && !CONFIG_HUGETLBFS */
+#define folio_zero_user(folio, addr_hint) \
+ clear_user_highpages(&(folio)->page, (addr_hint), folio_nr_pages(folio))
#endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_HUGETLBFS */
#if MAX_NUMNODES > 1
--
MST
^ permalink raw reply related
* [PATCH v6 03/30] mm: thread user_addr through page allocator for cache-friendly zeroing
From: Michael S. Tsirkin @ 2026-05-11 8:52 UTC (permalink / raw)
To: linux-kernel
Cc: David Hildenbrand (Arm), Jason Wang, Xuan Zhuo,
Eugenio Pérez, Muchun Song, Oscar Salvador, Andrew Morton,
Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Brendan Jackman,
Johannes Weiner, Zi Yan, Baolin Wang, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Hugh Dickins, Matthew Brost,
Joshua Hahn, Rakie Kim, Byungchul Park, Gregory Price, Ying Huang,
Alistair Popple, Christoph Lameter, David Rientjes,
Roman Gushchin, Harry Yoo, Axel Rasmussen, Yuanchu Xie, Wei Xu,
Chris Li, Kairui Song, Kemeng Shi, Nhat Pham, Baoquan He,
virtualization, linux-mm, Andrea Arcangeli, Liam R. Howlett,
Harry Yoo, Hao Li
In-Reply-To: <cover.1778488966.git.mst@redhat.com>
Thread a user virtual address from vma_alloc_folio() down through
the page allocator to post_alloc_hook(). This is plumbing
preparation for a subsequent patch that will use user_addr to
call folio_zero_user() for cache-friendly zeroing of user pages.
The user_addr is stored in struct alloc_context and flows through:
vma_alloc_folio -> folio_alloc_mpol -> __alloc_pages_mpol ->
__alloc_frozen_pages -> get_page_from_freelist -> prep_new_page ->
post_alloc_hook
USER_ADDR_NONE ((unsigned long)-1) is used for non-user
allocations, since address 0 is a valid userspace mapping.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Assisted-by: Claude:claude-opus-4-6
Assisted-by: cursor-agent:GPT-5.4-xhigh
---
include/linux/gfp.h | 2 +-
mm/compaction.c | 5 ++---
mm/hugetlb.c | 36 ++++++++++++++++++++----------------
mm/internal.h | 18 +++++++++++++++---
mm/mempolicy.c | 44 ++++++++++++++++++++++++++++++++------------
mm/page_alloc.c | 44 +++++++++++++++++++++++++++++---------------
mm/slub.c | 4 ++--
7 files changed, 101 insertions(+), 52 deletions(-)
diff --git a/include/linux/gfp.h b/include/linux/gfp.h
index 7ccbda35b9ad..ee35c5367abc 100644
--- a/include/linux/gfp.h
+++ b/include/linux/gfp.h
@@ -337,7 +337,7 @@ static inline struct folio *folio_alloc_noprof(gfp_t gfp, unsigned int order)
static inline struct folio *folio_alloc_mpol_noprof(gfp_t gfp, unsigned int order,
struct mempolicy *mpol, pgoff_t ilx, int nid)
{
- return folio_alloc_noprof(gfp, order);
+ return __folio_alloc_noprof(gfp, order, numa_node_id(), NULL);
}
#endif
diff --git a/mm/compaction.c b/mm/compaction.c
index 3648ce22c807..72684fe81e83 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -82,7 +82,7 @@ static inline bool is_via_compact_memory(int order) { return false; }
static struct page *mark_allocated_noprof(struct page *page, unsigned int order, gfp_t gfp_flags)
{
- post_alloc_hook(page, order, __GFP_MOVABLE);
+ post_alloc_hook(page, order, __GFP_MOVABLE, USER_ADDR_NONE);
set_page_refcounted(page);
return page;
}
@@ -1849,8 +1849,7 @@ static struct folio *compaction_alloc_noprof(struct folio *src, unsigned long da
set_page_private(&freepage[size], start_order);
}
dst = (struct folio *)freepage;
-
- post_alloc_hook(&dst->page, order, __GFP_MOVABLE);
+ post_alloc_hook(&dst->page, order, __GFP_MOVABLE, USER_ADDR_NONE);
set_page_refcounted(&dst->page);
if (order)
prep_compound_page(&dst->page, order);
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index f24bf49be047..a999f3ead852 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -1806,7 +1806,8 @@ struct address_space *hugetlb_folio_mapping_lock_write(struct folio *folio)
}
static struct folio *alloc_buddy_frozen_folio(int order, gfp_t gfp_mask,
- int nid, nodemask_t *nmask, nodemask_t *node_alloc_noretry)
+ int nid, nodemask_t *nmask, nodemask_t *node_alloc_noretry,
+ unsigned long addr)
{
struct folio *folio;
bool alloc_try_hard = true;
@@ -1823,7 +1824,7 @@ static struct folio *alloc_buddy_frozen_folio(int order, gfp_t gfp_mask,
if (alloc_try_hard)
gfp_mask |= __GFP_RETRY_MAYFAIL;
- folio = (struct folio *)__alloc_frozen_pages(gfp_mask, order, nid, nmask);
+ folio = (struct folio *)__alloc_frozen_pages(gfp_mask, order, nid, nmask, addr);
/*
* If we did not specify __GFP_RETRY_MAYFAIL, but still got a
@@ -1852,7 +1853,7 @@ static struct folio *alloc_buddy_frozen_folio(int order, gfp_t gfp_mask,
static struct folio *only_alloc_fresh_hugetlb_folio(struct hstate *h,
gfp_t gfp_mask, int nid, nodemask_t *nmask,
- nodemask_t *node_alloc_noretry)
+ nodemask_t *node_alloc_noretry, unsigned long addr)
{
struct folio *folio;
int order = huge_page_order(h);
@@ -1864,7 +1865,7 @@ static struct folio *only_alloc_fresh_hugetlb_folio(struct hstate *h,
folio = alloc_gigantic_frozen_folio(order, gfp_mask, nid, nmask);
else
folio = alloc_buddy_frozen_folio(order, gfp_mask, nid, nmask,
- node_alloc_noretry);
+ node_alloc_noretry, addr);
if (folio)
init_new_hugetlb_folio(folio);
return folio;
@@ -1878,11 +1879,12 @@ static struct folio *only_alloc_fresh_hugetlb_folio(struct hstate *h,
* pages is zero, and the accounting must be done in the caller.
*/
static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h,
- gfp_t gfp_mask, int nid, nodemask_t *nmask)
+ gfp_t gfp_mask, int nid, nodemask_t *nmask,
+ unsigned long addr)
{
struct folio *folio;
- folio = only_alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask, NULL);
+ folio = only_alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask, NULL, addr);
if (folio)
hugetlb_vmemmap_optimize_folio(h, folio);
return folio;
@@ -1922,7 +1924,7 @@ static struct folio *alloc_pool_huge_folio(struct hstate *h,
struct folio *folio;
folio = only_alloc_fresh_hugetlb_folio(h, gfp_mask, node,
- nodes_allowed, node_alloc_noretry);
+ nodes_allowed, node_alloc_noretry, USER_ADDR_NONE);
if (folio)
return folio;
}
@@ -2091,7 +2093,8 @@ int dissolve_free_hugetlb_folios(unsigned long start_pfn, unsigned long end_pfn)
* Allocates a fresh surplus page from the page allocator.
*/
static struct folio *alloc_surplus_hugetlb_folio(struct hstate *h,
- gfp_t gfp_mask, int nid, nodemask_t *nmask)
+ gfp_t gfp_mask, int nid, nodemask_t *nmask,
+ unsigned long addr)
{
struct folio *folio = NULL;
@@ -2103,7 +2106,7 @@ static struct folio *alloc_surplus_hugetlb_folio(struct hstate *h,
goto out_unlock;
spin_unlock_irq(&hugetlb_lock);
- folio = alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask);
+ folio = alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask, addr);
if (!folio)
return NULL;
@@ -2146,7 +2149,7 @@ static struct folio *alloc_migrate_hugetlb_folio(struct hstate *h, gfp_t gfp_mas
if (hstate_is_gigantic(h))
return NULL;
- folio = alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask);
+ folio = alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask, USER_ADDR_NONE);
if (!folio)
return NULL;
@@ -2182,14 +2185,14 @@ struct folio *alloc_buddy_hugetlb_folio_with_mpol(struct hstate *h,
if (mpol_is_preferred_many(mpol)) {
gfp_t gfp = gfp_mask & ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL);
- folio = alloc_surplus_hugetlb_folio(h, gfp, nid, nodemask);
+ folio = alloc_surplus_hugetlb_folio(h, gfp, nid, nodemask, addr);
/* Fallback to all nodes if page==NULL */
nodemask = NULL;
}
if (!folio)
- folio = alloc_surplus_hugetlb_folio(h, gfp_mask, nid, nodemask);
+ folio = alloc_surplus_hugetlb_folio(h, gfp_mask, nid, nodemask, addr);
mpol_cond_put(mpol);
return folio;
}
@@ -2296,7 +2299,8 @@ static int gather_surplus_pages(struct hstate *h, long delta)
* down the road to pick the current node if that is the case.
*/
folio = alloc_surplus_hugetlb_folio(h, htlb_alloc_mask(h),
- NUMA_NO_NODE, &alloc_nodemask);
+ NUMA_NO_NODE, &alloc_nodemask,
+ USER_ADDR_NONE);
if (!folio) {
alloc_ok = false;
break;
@@ -2702,7 +2706,7 @@ static int alloc_and_dissolve_hugetlb_folio(struct folio *old_folio,
spin_unlock_irq(&hugetlb_lock);
gfp_mask = htlb_alloc_mask(h) | __GFP_THISNODE;
new_folio = alloc_fresh_hugetlb_folio(h, gfp_mask,
- nid, NULL);
+ nid, NULL, USER_ADDR_NONE);
if (!new_folio)
return -ENOMEM;
goto retry;
@@ -3400,13 +3404,13 @@ static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid)
gfp_t gfp_mask = htlb_alloc_mask(h) | __GFP_THISNODE;
folio = only_alloc_fresh_hugetlb_folio(h, gfp_mask, nid,
- &node_states[N_MEMORY], NULL);
+ &node_states[N_MEMORY], NULL, USER_ADDR_NONE);
if (!folio && !list_empty(&folio_list) &&
hugetlb_vmemmap_optimizable_size(h)) {
prep_and_add_allocated_folios(h, &folio_list);
INIT_LIST_HEAD(&folio_list);
folio = only_alloc_fresh_hugetlb_folio(h, gfp_mask, nid,
- &node_states[N_MEMORY], NULL);
+ &node_states[N_MEMORY], NULL, USER_ADDR_NONE);
}
if (!folio)
break;
diff --git a/mm/internal.h b/mm/internal.h
index 5a2ddcf68e0b..751ae8911607 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -662,6 +662,12 @@ void calculate_min_free_kbytes(void);
int __meminit init_per_zone_wmark_min(void);
void page_alloc_sysctl_init(void);
+/*
+ * Sentinel for user_addr: indicates a non-user allocation.
+ * Cannot use 0 because address 0 is a valid userspace mapping.
+ */
+#define USER_ADDR_NONE ((unsigned long)-1)
+
/*
* Structure for holding the mostly immutable allocation parameters passed
* between functions involved in allocations, including the alloc_pages*
@@ -693,6 +699,7 @@ struct alloc_context {
*/
enum zone_type highest_zoneidx;
bool spread_dirty_pages;
+ unsigned long user_addr;
};
/*
@@ -916,24 +923,29 @@ static inline void init_compound_tail(struct page *tail,
prep_compound_tail(tail, head, order);
}
-void post_alloc_hook(struct page *page, unsigned int order, gfp_t gfp_flags);
+void post_alloc_hook(struct page *page, unsigned int order, gfp_t gfp_flags,
+ unsigned long user_addr);
extern bool free_pages_prepare(struct page *page, unsigned int order);
extern int user_min_free_kbytes;
struct page *__alloc_frozen_pages_noprof(gfp_t, unsigned int order, int nid,
- nodemask_t *);
+ nodemask_t *, unsigned long user_addr);
#define __alloc_frozen_pages(...) \
alloc_hooks(__alloc_frozen_pages_noprof(__VA_ARGS__))
void free_frozen_pages(struct page *page, unsigned int order);
+void free_frozen_pages_zeroed(struct page *page, unsigned int order);
void free_unref_folios(struct folio_batch *fbatch);
#ifdef CONFIG_NUMA
struct page *alloc_frozen_pages_noprof(gfp_t, unsigned int order);
+struct folio *folio_alloc_mpol_user_noprof(gfp_t gfp, unsigned int order,
+ struct mempolicy *pol, pgoff_t ilx, int nid,
+ unsigned long user_addr);
#else
static inline struct page *alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order)
{
- return __alloc_frozen_pages_noprof(gfp, order, numa_node_id(), NULL);
+ return __alloc_frozen_pages_noprof(gfp, order, numa_node_id(), NULL, USER_ADDR_NONE);
}
#endif
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 39e556e3d263..ea3043e0075b 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -2413,7 +2413,8 @@ bool mempolicy_in_oom_domain(struct task_struct *tsk,
}
static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order,
- int nid, nodemask_t *nodemask)
+ int nid, nodemask_t *nodemask,
+ unsigned long user_addr)
{
struct page *page;
gfp_t preferred_gfp;
@@ -2426,25 +2427,29 @@ static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order,
*/
preferred_gfp = gfp | __GFP_NOWARN;
preferred_gfp &= ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL);
- page = __alloc_frozen_pages_noprof(preferred_gfp, order, nid, nodemask);
+ page = __alloc_frozen_pages_noprof(preferred_gfp, order, nid,
+ nodemask, user_addr);
if (!page)
- page = __alloc_frozen_pages_noprof(gfp, order, nid, NULL);
+ page = __alloc_frozen_pages_noprof(gfp, order, nid, NULL,
+ user_addr);
return page;
}
/**
- * alloc_pages_mpol - Allocate pages according to NUMA mempolicy.
+ * __alloc_pages_mpol - Allocate pages according to NUMA mempolicy.
* @gfp: GFP flags.
* @order: Order of the page allocation.
* @pol: Pointer to the NUMA mempolicy.
* @ilx: Index for interleave mempolicy (also distinguishes alloc_pages()).
* @nid: Preferred node (usually numa_node_id() but @mpol may override it).
+ * @user_addr: User fault address for cache-friendly zeroing, or USER_ADDR_NONE.
*
* Return: The page on success or NULL if allocation fails.
*/
-static struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order,
- struct mempolicy *pol, pgoff_t ilx, int nid)
+static struct page *__alloc_pages_mpol(gfp_t gfp, unsigned int order,
+ struct mempolicy *pol, pgoff_t ilx, int nid,
+ unsigned long user_addr)
{
nodemask_t *nodemask;
struct page *page;
@@ -2452,7 +2457,8 @@ static struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order,
nodemask = policy_nodemask(gfp, pol, ilx, &nid);
if (pol->mode == MPOL_PREFERRED_MANY)
- return alloc_pages_preferred_many(gfp, order, nid, nodemask);
+ return alloc_pages_preferred_many(gfp, order, nid, nodemask,
+ user_addr);
if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) &&
/* filter "hugepage" allocation, unless from alloc_pages() */
@@ -2476,7 +2482,7 @@ static struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order,
*/
page = __alloc_frozen_pages_noprof(
gfp | __GFP_THISNODE | __GFP_NORETRY, order,
- nid, NULL);
+ nid, NULL, user_addr);
if (page || !(gfp & __GFP_DIRECT_RECLAIM))
return page;
/*
@@ -2488,7 +2494,7 @@ static struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order,
}
}
- page = __alloc_frozen_pages_noprof(gfp, order, nid, nodemask);
+ page = __alloc_frozen_pages_noprof(gfp, order, nid, nodemask, user_addr);
if (unlikely(pol->mode == MPOL_INTERLEAVE ||
pol->mode == MPOL_WEIGHTED_INTERLEAVE) && page) {
@@ -2504,11 +2510,18 @@ static struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order,
return page;
}
-struct folio *folio_alloc_mpol_noprof(gfp_t gfp, unsigned int order,
+static struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order,
struct mempolicy *pol, pgoff_t ilx, int nid)
{
- struct page *page = alloc_pages_mpol(gfp | __GFP_COMP, order, pol,
- ilx, nid);
+ return __alloc_pages_mpol(gfp, order, pol, ilx, nid, USER_ADDR_NONE);
+}
+
+struct folio *folio_alloc_mpol_user_noprof(gfp_t gfp, unsigned int order,
+ struct mempolicy *pol, pgoff_t ilx, int nid,
+ unsigned long user_addr)
+{
+ struct page *page = __alloc_pages_mpol(gfp | __GFP_COMP, order, pol,
+ ilx, nid, user_addr);
if (!page)
return NULL;
@@ -2516,6 +2529,13 @@ struct folio *folio_alloc_mpol_noprof(gfp_t gfp, unsigned int order,
return page_rmappable_folio(page);
}
+struct folio *folio_alloc_mpol_noprof(gfp_t gfp, unsigned int order,
+ struct mempolicy *pol, pgoff_t ilx, int nid)
+{
+ return folio_alloc_mpol_user_noprof(gfp, order, pol, ilx, nid,
+ USER_ADDR_NONE);
+}
+
struct page *alloc_frozen_pages_noprof(gfp_t gfp, unsigned order)
{
struct mempolicy *pol = &default_policy;
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index fc7327ebdf6c..c3c0f4e2baa7 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1806,7 +1806,7 @@ static inline bool should_skip_init(gfp_t flags)
}
inline void post_alloc_hook(struct page *page, unsigned int order,
- gfp_t gfp_flags)
+ gfp_t gfp_flags, unsigned long user_addr)
{
bool init = !want_init_on_free() && want_init_on_alloc(gfp_flags) &&
!should_skip_init(gfp_flags);
@@ -1861,9 +1861,10 @@ inline void post_alloc_hook(struct page *page, unsigned int order,
}
static void prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags,
- unsigned int alloc_flags)
+ unsigned int alloc_flags,
+ unsigned long user_addr)
{
- post_alloc_hook(page, order, gfp_flags);
+ post_alloc_hook(page, order, gfp_flags, user_addr);
if (order && (gfp_flags & __GFP_COMP))
prep_compound_page(page, order);
@@ -3943,7 +3944,8 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
page = rmqueue(zonelist_zone(ac->preferred_zoneref), zone, order,
gfp_mask, alloc_flags, ac->migratetype);
if (page) {
- prep_new_page(page, order, gfp_mask, alloc_flags);
+ prep_new_page(page, order, gfp_mask, alloc_flags,
+ ac->user_addr);
return page;
} else {
@@ -4171,7 +4173,8 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order,
/* Prep a captured page if available */
if (page)
- prep_new_page(page, order, gfp_mask, alloc_flags);
+ prep_new_page(page, order, gfp_mask, alloc_flags,
+ ac->user_addr);
/* Try get a page from the freelist if available */
if (!page)
@@ -5048,7 +5051,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
struct zoneref *z;
struct per_cpu_pages *pcp;
struct list_head *pcp_list;
- struct alloc_context ac;
+ struct alloc_context ac = { .user_addr = USER_ADDR_NONE };
gfp_t alloc_gfp;
unsigned int alloc_flags = ALLOC_WMARK_LOW;
int nr_populated = 0, nr_account = 0;
@@ -5163,7 +5166,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
}
nr_account++;
- prep_new_page(page, 0, gfp, 0);
+ prep_new_page(page, 0, gfp, 0, USER_ADDR_NONE);
set_page_refcounted(page);
page_array[nr_populated++] = page;
}
@@ -5188,12 +5191,13 @@ EXPORT_SYMBOL_GPL(alloc_pages_bulk_noprof);
* This is the 'heart' of the zoned buddy allocator.
*/
struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order,
- int preferred_nid, nodemask_t *nodemask)
+ int preferred_nid, nodemask_t *nodemask,
+ unsigned long user_addr)
{
struct page *page;
unsigned int alloc_flags = ALLOC_WMARK_LOW;
gfp_t alloc_gfp; /* The gfp_t that was actually used for allocation */
- struct alloc_context ac = { };
+ struct alloc_context ac = { .user_addr = user_addr };
/*
* There are several places where we assume that the order value is sane
@@ -5254,10 +5258,12 @@ EXPORT_SYMBOL(__alloc_frozen_pages_noprof);
struct page *__alloc_pages_noprof(gfp_t gfp, unsigned int order,
int preferred_nid, nodemask_t *nodemask)
+
{
struct page *page;
- page = __alloc_frozen_pages_noprof(gfp, order, preferred_nid, nodemask);
+ page = __alloc_frozen_pages_noprof(gfp, order, preferred_nid,
+ nodemask, USER_ADDR_NONE);
if (page)
set_page_refcounted(page);
return page;
@@ -5300,7 +5306,8 @@ struct folio *vma_alloc_folio_noprof(gfp_t gfp, int order,
gfp |= __GFP_NOWARN;
pol = get_vma_policy(vma, addr, order, &ilx);
- folio = folio_alloc_mpol_noprof(gfp, order, pol, ilx, numa_node_id());
+ folio = folio_alloc_mpol_user_noprof(gfp, order, pol, ilx,
+ numa_node_id(), addr);
mpol_cond_put(pol);
return folio;
}
@@ -5308,10 +5315,17 @@ struct folio *vma_alloc_folio_noprof(gfp_t gfp, int order,
struct folio *vma_alloc_folio_noprof(gfp_t gfp, int order,
struct vm_area_struct *vma, unsigned long addr)
{
+ struct page *page;
+
if (vma->vm_flags & VM_DROPPABLE)
gfp |= __GFP_NOWARN;
- return folio_alloc_noprof(gfp, order);
+ page = __alloc_frozen_pages_noprof(gfp | __GFP_COMP, order,
+ numa_node_id(), NULL, addr);
+ if (!page)
+ return NULL;
+ set_page_refcounted(page);
+ return page_rmappable_folio(page);
}
#endif
EXPORT_SYMBOL(vma_alloc_folio_noprof);
@@ -6892,7 +6906,7 @@ static void split_free_frozen_pages(struct list_head *list, gfp_t gfp_mask)
list_for_each_entry_safe(page, next, &list[order], lru) {
int i;
- post_alloc_hook(page, order, gfp_mask);
+ post_alloc_hook(page, order, gfp_mask, USER_ADDR_NONE);
if (!order)
continue;
@@ -7098,7 +7112,7 @@ int alloc_contig_frozen_range_noprof(unsigned long start, unsigned long end,
struct page *head = pfn_to_page(start);
check_new_pages(head, order);
- prep_new_page(head, order, gfp_mask, 0);
+ prep_new_page(head, order, gfp_mask, 0, USER_ADDR_NONE);
} else {
ret = -EINVAL;
WARN(true, "PFN range: requested [%lu, %lu), allocated [%lu, %lu)\n",
@@ -7763,7 +7777,7 @@ struct page *alloc_frozen_pages_nolock_noprof(gfp_t gfp_flags, int nid, unsigned
gfp_t alloc_gfp = __GFP_NOWARN | __GFP_ZERO | __GFP_NOMEMALLOC | __GFP_COMP
| gfp_flags;
unsigned int alloc_flags = ALLOC_TRYLOCK;
- struct alloc_context ac = { };
+ struct alloc_context ac = { .user_addr = USER_ADDR_NONE };
struct page *page;
VM_WARN_ON_ONCE(gfp_flags & ~__GFP_ACCOUNT);
diff --git a/mm/slub.c b/mm/slub.c
index 0baa906f39ab..74dd2d96941b 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -3275,7 +3275,7 @@ static inline struct slab *alloc_slab_page(gfp_t flags, int node,
else if (node == NUMA_NO_NODE)
page = alloc_frozen_pages(flags, order);
else
- page = __alloc_frozen_pages(flags, order, node, NULL);
+ page = __alloc_frozen_pages(flags, order, node, NULL, USER_ADDR_NONE);
if (!page)
return NULL;
@@ -5235,7 +5235,7 @@ static void *___kmalloc_large_node(size_t size, gfp_t flags, int node)
if (node == NUMA_NO_NODE)
page = alloc_frozen_pages_noprof(flags, order);
else
- page = __alloc_frozen_pages_noprof(flags, order, node, NULL);
+ page = __alloc_frozen_pages_noprof(flags, order, node, NULL, USER_ADDR_NONE);
if (page) {
ptr = page_address(page);
--
MST
^ permalink raw reply related
* [PATCH v6 02/30] mm: mempolicy: fix interleave index for unaligned VMA start
From: Michael S. Tsirkin @ 2026-05-11 8:52 UTC (permalink / raw)
To: linux-kernel
Cc: David Hildenbrand (Arm), Jason Wang, Xuan Zhuo,
Eugenio Pérez, Muchun Song, Oscar Salvador, Andrew Morton,
Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Brendan Jackman,
Johannes Weiner, Zi Yan, Baolin Wang, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Hugh Dickins, Matthew Brost,
Joshua Hahn, Rakie Kim, Byungchul Park, Gregory Price, Ying Huang,
Alistair Popple, Christoph Lameter, David Rientjes,
Roman Gushchin, Harry Yoo, Axel Rasmussen, Yuanchu Xie, Wei Xu,
Chris Li, Kairui Song, Kemeng Shi, Nhat Pham, Baoquan He,
virtualization, linux-mm, Andrea Arcangeli
In-Reply-To: <cover.1778488966.git.mst@redhat.com>
The NUMA interleave index formula (addr - vm_start) >> shift
gives wrong results when vm_start is not aligned to the folio
size: the subtraction before the shift allows low bits to
affect the result via borrows.
Use (addr >> shift) - (vm_start >> shift) instead, which
independently aligns both values before computing the
difference.
No functional change for current callers: the fix only affects
NUMA interleave and weighted-interleave policies. Current
large-order callers either pre-align the address
(vma_alloc_anon_folio_pmd) or do not use NUMA interleave
(drm_pagemap). All other callers use order 0 where the old
and new formulas are equivalent. However subsequent patches
in this series add large-order callers that pass unaligned
fault addresses, making this fix necessary.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
mm/mempolicy.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 6832cc68120f..39e556e3d263 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -2049,7 +2049,8 @@ struct mempolicy *get_vma_policy(struct vm_area_struct *vma,
if (pol->mode == MPOL_INTERLEAVE ||
pol->mode == MPOL_WEIGHTED_INTERLEAVE) {
*ilx += vma->vm_pgoff >> order;
- *ilx += (addr - vma->vm_start) >> (PAGE_SHIFT + order);
+ *ilx += (addr >> (PAGE_SHIFT + order)) -
+ (vma->vm_start >> (PAGE_SHIFT + order));
}
return pol;
}
--
MST
^ permalink raw reply related
* [PATCH v6 01/30] mm: move vma_alloc_folio_noprof to page_alloc.c
From: Michael S. Tsirkin @ 2026-05-11 8:52 UTC (permalink / raw)
To: linux-kernel
Cc: David Hildenbrand (Arm), Jason Wang, Xuan Zhuo,
Eugenio Pérez, Muchun Song, Oscar Salvador, Andrew Morton,
Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Brendan Jackman,
Johannes Weiner, Zi Yan, Baolin Wang, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Hugh Dickins, Matthew Brost,
Joshua Hahn, Rakie Kim, Byungchul Park, Gregory Price, Ying Huang,
Alistair Popple, Christoph Lameter, David Rientjes,
Roman Gushchin, Harry Yoo, Axel Rasmussen, Yuanchu Xie, Wei Xu,
Chris Li, Kairui Song, Kemeng Shi, Nhat Pham, Baoquan He,
virtualization, linux-mm, Andrea Arcangeli, Liam R. Howlett
In-Reply-To: <cover.1778488966.git.mst@redhat.com>
Move vma_alloc_folio_noprof() from an inline in gfp.h (for !NUMA)
and mempolicy.c (for NUMA) to page_alloc.c.
This prepares for a subsequent patch that will thread user_addr
through the allocator: having vma_alloc_folio_noprof in page_alloc.c
means user_addr can be passed to the internal allocation path
without changing public API signatures or duplicating plumbing
in both gfp.h and mempolicy.c.
The !NUMA path gains the VM_DROPPABLE -> __GFP_NOWARN check
that the NUMA path already had.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Assisted-by: Claude:claude-opus-4-6
Assisted-by: cursor-agent:GPT-5.4-xhigh
---
include/linux/gfp.h | 9 ++-------
mm/mempolicy.c | 32 --------------------------------
mm/page_alloc.c | 43 +++++++++++++++++++++++++++++++++++++++++++
3 files changed, 45 insertions(+), 39 deletions(-)
diff --git a/include/linux/gfp.h b/include/linux/gfp.h
index 51ef13ed756e..7ccbda35b9ad 100644
--- a/include/linux/gfp.h
+++ b/include/linux/gfp.h
@@ -318,13 +318,13 @@ static inline struct page *alloc_pages_node_noprof(int nid, gfp_t gfp_mask,
#define alloc_pages_node(...) alloc_hooks(alloc_pages_node_noprof(__VA_ARGS__))
+struct folio *vma_alloc_folio_noprof(gfp_t gfp, int order,
+ struct vm_area_struct *vma, unsigned long addr);
#ifdef CONFIG_NUMA
struct page *alloc_pages_noprof(gfp_t gfp, unsigned int order);
struct folio *folio_alloc_noprof(gfp_t gfp, unsigned int order);
struct folio *folio_alloc_mpol_noprof(gfp_t gfp, unsigned int order,
struct mempolicy *mpol, pgoff_t ilx, int nid);
-struct folio *vma_alloc_folio_noprof(gfp_t gfp, int order, struct vm_area_struct *vma,
- unsigned long addr);
#else
static inline struct page *alloc_pages_noprof(gfp_t gfp_mask, unsigned int order)
{
@@ -339,11 +339,6 @@ static inline struct folio *folio_alloc_mpol_noprof(gfp_t gfp, unsigned int orde
{
return folio_alloc_noprof(gfp, order);
}
-static inline struct folio *vma_alloc_folio_noprof(gfp_t gfp, int order,
- struct vm_area_struct *vma, unsigned long addr)
-{
- return folio_alloc_noprof(gfp, order);
-}
#endif
#define alloc_pages(...) alloc_hooks(alloc_pages_noprof(__VA_ARGS__))
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 4e4421b22b59..6832cc68120f 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -2515,38 +2515,6 @@ struct folio *folio_alloc_mpol_noprof(gfp_t gfp, unsigned int order,
return page_rmappable_folio(page);
}
-/**
- * vma_alloc_folio - Allocate a folio for a VMA.
- * @gfp: GFP flags.
- * @order: Order of the folio.
- * @vma: Pointer to VMA.
- * @addr: Virtual address of the allocation. Must be inside @vma.
- *
- * Allocate a folio for a specific address in @vma, using the appropriate
- * NUMA policy. The caller must hold the mmap_lock of the mm_struct of the
- * VMA to prevent it from going away. Should be used for all allocations
- * for folios that will be mapped into user space, excepting hugetlbfs, and
- * excepting where direct use of folio_alloc_mpol() is more appropriate.
- *
- * Return: The folio on success or NULL if allocation fails.
- */
-struct folio *vma_alloc_folio_noprof(gfp_t gfp, int order, struct vm_area_struct *vma,
- unsigned long addr)
-{
- struct mempolicy *pol;
- pgoff_t ilx;
- struct folio *folio;
-
- if (vma->vm_flags & VM_DROPPABLE)
- gfp |= __GFP_NOWARN;
-
- pol = get_vma_policy(vma, addr, order, &ilx);
- folio = folio_alloc_mpol_noprof(gfp, order, pol, ilx, numa_node_id());
- mpol_cond_put(pol);
- return folio;
-}
-EXPORT_SYMBOL(vma_alloc_folio_noprof);
-
struct page *alloc_frozen_pages_noprof(gfp_t gfp, unsigned order)
{
struct mempolicy *pol = &default_policy;
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 227d58dc3de6..fc7327ebdf6c 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -5273,6 +5273,49 @@ struct folio *__folio_alloc_noprof(gfp_t gfp, unsigned int order, int preferred_
}
EXPORT_SYMBOL(__folio_alloc_noprof);
+#ifdef CONFIG_NUMA
+/**
+ * vma_alloc_folio - Allocate a folio for a VMA.
+ * @gfp: GFP flags.
+ * @order: Order of the folio.
+ * @vma: Pointer to VMA.
+ * @addr: Virtual address of the allocation. Must be inside @vma.
+ *
+ * Allocate a folio for a specific address in @vma, using the appropriate
+ * NUMA policy. The caller must hold the mmap_lock of the mm_struct of the
+ * VMA to prevent it from going away. Should be used for all allocations
+ * for folios that will be mapped into user space, excepting hugetlbfs, and
+ * excepting where direct use of folio_alloc_mpol() is more appropriate.
+ *
+ * Return: The folio on success or NULL if allocation fails.
+ */
+struct folio *vma_alloc_folio_noprof(gfp_t gfp, int order,
+ struct vm_area_struct *vma, unsigned long addr)
+{
+ struct mempolicy *pol;
+ pgoff_t ilx;
+ struct folio *folio;
+
+ if (vma->vm_flags & VM_DROPPABLE)
+ gfp |= __GFP_NOWARN;
+
+ pol = get_vma_policy(vma, addr, order, &ilx);
+ folio = folio_alloc_mpol_noprof(gfp, order, pol, ilx, numa_node_id());
+ mpol_cond_put(pol);
+ return folio;
+}
+#else
+struct folio *vma_alloc_folio_noprof(gfp_t gfp, int order,
+ struct vm_area_struct *vma, unsigned long addr)
+{
+ if (vma->vm_flags & VM_DROPPABLE)
+ gfp |= __GFP_NOWARN;
+
+ return folio_alloc_noprof(gfp, order);
+}
+#endif
+EXPORT_SYMBOL(vma_alloc_folio_noprof);
+
/*
* Common helper functions. Never use with __GFP_HIGHMEM because the returned
* address cannot represent highmem pages. Use alloc_pages and then kmap if
--
MST
^ permalink raw reply related
* [PATCH v6 00/30] mm/virtio: skip redundant zeroing of host-zeroed pages
From: Michael S. Tsirkin @ 2026-05-11 8:50 UTC (permalink / raw)
To: linux-kernel
Cc: David Hildenbrand (Arm), Jason Wang, Xuan Zhuo,
Eugenio Pérez, Muchun Song, Oscar Salvador, Andrew Morton,
Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Brendan Jackman,
Johannes Weiner, Zi Yan, Baolin Wang, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Hugh Dickins, Matthew Brost,
Joshua Hahn, Rakie Kim, Byungchul Park, Gregory Price, Ying Huang,
Alistair Popple, Christoph Lameter, David Rientjes,
Roman Gushchin, Harry Yoo, Axel Rasmussen, Yuanchu Xie, Wei Xu,
Chris Li, Kairui Song, Kemeng Shi, Nhat Pham, Baoquan He,
virtualization, linux-mm, Andrea Arcangeli
When a guest reports free pages to the hypervisor via virtio-balloon's
free page reporting, the host typically zeros those pages when reclaiming
their backing memory (e.g., via MADV_DONTNEED on anonymous mappings).
When the guest later reallocates those pages, the kernel zeros them
again, redundantly.
Further, on architectures with aliasing caches, upstream with init_on_alloc
double-zeros user pages: once via kernel_init_pages() in
post_alloc_hook, and again via clear_user_highpage() at the
callsite (because user_alloc_needs_zeroing() returns true).
This series eliminates that double-zeroing by moving the zeroing
into the post_alloc_hook + propagating the "host
already zeroed this page" information through the buddy allocator.
For page reporting, VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED (bit 6)
is used. For the inflate/deflate path,
VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE (bit 7) is used.
Virtio spec: https://lore.kernel.org/all/cover.1778140241.git.mst@redhat.com
Based on v7.1-rc2. When applying on mm-unstable, two conflicts
are expected:
- kernel_init_pages() was renamed to clear_highpages_kasan_tagged()
in mm-unstable. Use clear_highpages_kasan_tagged() in the
post_alloc_hook else branch.
- FPI_PREPARED uses BIT(3) in mm-unstable. Bump FPI_ZEROED to
BIT(4).
Build-tested on mm-unstable at e9dd96806dbc:
https://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost.git zero-mm-unstable
The first 16 patches are independently mergeable cleanups + fixes:
- Patches 1-15: mm rework + cleanups + init_on_alloc double-zeroing fix.
- Patch 16: page_reporting capacity bugfix.
Patches 17-26: page reporting zeroing (DEVICE_INIT_REPORTED).
Patches 27-30: inflate/deflate zeroing (DEVICE_INIT_ON_INFLATE).
-------
Performance with THP enabled on a 2GB VM, 1 vCPU, allocating
256MB of anonymous pages:
metric baseline optimized delta
task-clock 232 +- 20 ms 51 +- 26 ms -78%
cache-misses 1.20M +- 248K 288K +- 102K -76%
instructions 16.3M +- 1.2M 13.8M +- 1.0M -15%
With hugetlb surplus pages:
metric baseline optimized delta
task-clock 219 +- 23 ms 65 +- 34 ms -70%
cache-misses 1.17M +- 391K 263K +- 36K -78%
instructions 17.9M +- 1.2M 15.1M +- 724K -16%
Two flags track known-zero pages:
PG_zeroed (aliased to PG_private) marks buddy allocator pages that
are known to contain all zeros -- either because the host zeroed
them during page reporting, or because they were freed via the
balloon deflate path. It lives on free-list pages and is consumed
by post_alloc_hook() on allocation.
HPG_zeroed (stored in hugetlb folio->private bits) serves the same
purpose for hugetlb pool pages, which are kept in a pool and may
be zeroed long after buddy allocation, so PG_zeroed (consumed at
allocation time) cannot track their state.
PG_zeroed lifecycle:
Sets PG_zeroed:
- page_reporting_drain: on reported pages when host zeroes them
- __free_pages_ok / __free_frozen_pages: when FPI_ZEROED is set
(balloon deflate path)
- buddy merge: on merged page if both buddies were zeroed
- expand(): propagate to split-off buddy sub-pages
Clears PG_zeroed:
- __free_pages_prepare: clears all PAGE_FLAGS_CHECK_AT_PREP flags
(PG_zeroed included), preventing PG_private aliasing leaks
- rmqueue_buddy / __rmqueue_pcplist: read-then-clear, passes
zeroed hint to prep_new_page -> post_alloc_hook
- __isolate_free_page: clear (compaction/page_reporting isolation)
- compaction, alloc_contig, split_free_frozen: clear before use
- buddy merge: clear both pages before merge, then conditionally
re-set on merged head if both were zeroed
HPG_zeroed lifecycle (hugetlb pool pages, stored in folio->private):
Sets HPG_zeroed:
- alloc_surplus_hugetlb_folio: after buddy allocation with
__GFP_ZERO, mark pool page as known-zero
Clears HPG_zeroed:
- free_huge_folio: page was mapped to userspace, no longer
known-zero when it returns to the pool
- alloc_hugetlb_folio / alloc_hugetlb_folio_reserve: clear
after reporting to caller via bool *zeroed output (consumed)
- The optimization is most effective with THP, where entire 2MB
pages are allocated directly from reported order-9+ buddy pages.
Without THP, only ~21% of order-0 allocations come from reported
pages due to low-order fragmentation.
- Persistent hugetlb pool pages are not covered: when freed by
userspace they return to the hugetlb free pool, not the buddy
allocator, so they are never reported to the host. Surplus
hugetlb pages are allocated from buddy and do benefit.
- PG_zeroed is aliased to PG_private. __free_pages_prepare() clears it
(preventing filesystem PG_private from leaking as false PG_zeroed).
FPI_ZEROED re-sets it after prepare for balloon deflate pages.
Is aliasing PG_private acceptable, or should a different bit be used?
- On architectures with aliasing caches, upstream with init_on_alloc
double-zeros user pages: once via kernel_init_pages() in
post_alloc_hook, and again via clear_user_highpage() at the
callsite (because user_alloc_needs_zeroing() returns true).
Our patches eliminate this by zeroing once via folio_zero_user()
in post_alloc_hook. Not a critical fix (people who set init_on_alloc
know they are paying performance) but a nice cleanup anyway.
Test program:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#ifndef MADV_POPULATE_WRITE
#define MADV_POPULATE_WRITE 23
#endif
#ifndef MAP_HUGETLB
#define MAP_HUGETLB 0x40000
#endif
int main(int argc, char **argv)
{
unsigned long size;
int flags = MAP_PRIVATE | MAP_ANONYMOUS;
void *p;
int r;
if (argc < 2) {
fprintf(stderr, "usage: %s <size_mb> [huge]\n", argv[0]);
return 1;
}
size = atol(argv[1]) * 1024UL * 1024;
if (argc >= 3 && strcmp(argv[2], "huge") == 0)
flags |= MAP_HUGETLB;
p = mmap(NULL, size, PROT_READ | PROT_WRITE, flags, -1, 0);
if (p == MAP_FAILED) {
perror("mmap");
return 1;
}
r = madvise(p, size, MADV_POPULATE_WRITE);
if (r) {
perror("madvise");
return 1;
}
munmap(p, size);
return 0;
}
Test script (bench.sh):
#!/bin/bash
# Usage: bench.sh <size_mb> <mode> <iterations> [huge]
# mode 0 = baseline, mode 1 = skip zeroing
SZ=${1:-256}; MODE=${2:-0}; ITER=${3:-10}; HUGE=${4:-}
FLUSH=/sys/module/page_reporting/parameters/flush
PERF_DATA=/tmp/perf-$MODE.csv
rmmod virtio_balloon 2>/dev/null
insmod virtio_balloon.ko host_zeroes_pages=$MODE
echo 512 > $FLUSH
[ "$HUGE" = "huge" ] && echo $((SZ/2)) > /proc/sys/vm/nr_overcommit_hugepages
rm -f $PERF_DATA
echo "=== sz=${SZ}MB mode=$MODE iter=$ITER $HUGE ==="
for i in $(seq 1 $ITER); do
echo 3 > /proc/sys/vm/drop_caches
echo 512 > $FLUSH
perf stat -e task-clock,instructions,cache-misses \
-x, -o $PERF_DATA --append -- ./alloc_once $SZ $HUGE
done
[ "$HUGE" = "huge" ] && echo 0 > /proc/sys/vm/nr_overcommit_hugepages
rmmod virtio_balloon
awk -F, '/^#/||/^$/{next}{v=$1+0;e=$3;gsub(/ /,"",e);s[e]+=v;ss[e]+=v*v;n[e]++}
END{for(e in s){a=s[e]/n[e];d=sqrt(ss[e]/n[e]-a*a);printf " %-16s %10.0f +- %8.0f (n=%d)\n",e,a,d,n[e]}}' $PERF_DATA
Compile and run:
gcc -static -O2 -o alloc_once alloc_once.c
bash bench.sh 256 0 10 # baseline (regular pages)
bash bench.sh 256 1 10 # optimized (regular pages)
bash bench.sh 256 0 10 huge # baseline (hugetlb surplus)
bash bench.sh 256 1 10 huge # optimized (hugetlb surplus)
Changes since v5:
- Rebased onto v7.1-rc2.
- Split alloc_anon_folio and alloc_swap_folio raw fault address
changes into separate patches.
- In virtio, move PAGE_POISON check for DEVICE_INIT_REPORTED
from probe() to validate(), clearing the feature instead of
just gating host_zeroes_pages. Same for confidential
computing check.
- Fix bisectability: FPI_ZEROED definition and usage now in
the same patch.
- Lots of commit log tweaks.
- Reorder: REPORTED before ON_INFLATE.
- Kerneldoc fixes.
Changes since v4:
With virtio spec posted, update to latest spec:
- Add VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED (bit 6) for reporting.
- Add VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE (bit 7) for inflate.
- Per-page virtqueue submission, per-page used_len feedback.
- Balloon migration preserves PageZeroed hint.
- Page_reporting capacity bugfix for small virtqueues.
- PG_zeroed propagation in split_large_buddy.
- Disable both features for confidential computing guests.
- Gate host_zeroes_pages on PAGE_POISON/poison_val: when PAGE_POISON
is negotiated with non-zero poison_val, device fills with poison
not zeros, so host_zeroes_pages must be false.
- Disable ON_INFLATE when PAGE_POISON with non-zero poison_val.
- Bound inflate bitmap reads by used_len from device.
- Move ON_INFLATE poison_val check to validate() for proper
feature negotiation.
- Fix NUMA interleave index for unaligned VMA start (new patch 1).
- Drop vma_alloc_folio_user_addr: with the ilx fix, callers can
pass raw fault address to vma_alloc_folio directly.
- Tested with DEBUG_VM, INIT_ON_ALLOC/FREE enabled.
Changes since v3 (address review by Gregory Price and David Hildenbrand):
- Keep user_addr threading internal: public APIs (__alloc_pages,
__folio_alloc, folio_alloc_mpol) are unchanged. Only internal
functions (__alloc_frozen_pages_noprof, __alloc_pages_mpol) carry
user_addr. This eliminates all API churn for external callers.
- Add vma_alloc_folio_user_addr() (2/22) to separate NUMA policy
address from the zeroing hint address. Fixes NUMA interleave
index corruption when passing unaligned fault address for
higher-order allocations.
- Add per-page zeroed_bitmap to page_reporting_dev_info (17/22).
The driver's report() callback manages the bitmap. Drain
checks it gated by the host_zeroes_pages static key. This
matches the proposed virtio balloon extension at
https://lore.kernel.org/all/cover.1776874126.git.mst@redhat.com/
- Clear PG_zeroed in __isolate_free_page() to prevent the aliased
PG_private flag from leaking to compaction/alloc_contig paths.
- Do not exclude PG_zeroed from PAGE_FLAGS_CHECK_AT_PREP macro.
Instead, __free_pages_prepare() clears it (preventing filesystem
PG_private leaking as false PG_zeroed), and FPI_ZEROED sets it
after prepare. Only buddy merge assertion is relaxed.
- Initialize alloc_context.user_addr in alloc_pages_bulk_noprof.
- Deflate and hugetlb changes are much smaller now. Still, the
patchset can be merged gradually, if desired.
Changes since v2 (address review by Gregory Price and David Hildenbrand):
- v2 used pghint_t / vma_alloc_folio_hints API. v3 switches to
threading user_addr through the page allocator and using __GFP_ZERO,
so post_alloc_hook() can use folio_zero_user() for cache-friendly
zeroing when the user fault address is known.
- Use FPI_ZEROED to set PG_zeroed after __free_pages_prepare() instead
of runtime masking in __free_one_page (further refined in v4).
- Drop redundant page_poisoning_enabled() check from mm core free
path -- already guarded at feature negotiation time in
virtio_balloon_validate. The balloon driver keeps its own
page_poisoning_enabled_static() check as defense in depth.
- Split free_frozen_pages_zeroed and put_page_zeroed into separate
patches. David Hildenbrand indicated he intends to rework balloon
pages to be frozen (no refcount), at which point put_page_zeroed
(21/22) can be dropped and the balloon can call
free_frozen_pages_zeroed directly.
- Use HPG_zeroed flag (in hugetlb folio->private) for hugetlb pool
pages instead of PG_zeroed, since pool pages are zeroed long after
buddy allocation and PG_zeroed is consumed at allocation time.
- syzbot CI found a PF_NO_COMPOUND BUG in the v2 pghint_t approach
where __ClearPageZeroed was called on compound hugetlb pages in
free_huge_folio. The v3 HPG_zeroed approach avoids this.
- Remove redundant arch vma_alloc_zeroed_movable_folio overrides
on x86, s390, m68k, and alpha (12/22). Suggested by David
Hildenbrand.
- Updated benchmarking script to compute per-run avg +- stddev
via awk on CSV output.
Changes v1->v2:
- Replaced __GFP_PREZEROED with PG_zeroed page flag (aliased PG_private)
- Added pghint_t type and vma_alloc_folio_hints() API
- Track PG_zeroed across buddy merges and splits
- Added post_alloc_hook integration (single consume/clear point)
- Added hugetlb support (pool pages + memfd)
- Added page_reporting flush parameter for deterministic testing
- Added free_frozen_pages_hint/put_page_hint for balloon deflate path
- Added try_to_claim_block PG_zeroed preservation
- Updated perf numbers with per-iteration flush methodology
Written with assistance from Claude (claude-opus-4-6).
Reviewed by cursor-agent (GPT-5.4-xhigh).
Everything manually read, patchset split and commit logs edited manually.
Michael S. Tsirkin (30):
mm: move vma_alloc_folio_noprof to page_alloc.c
mm: mempolicy: fix interleave index for unaligned VMA start
mm: thread user_addr through page allocator for cache-friendly zeroing
mm: add folio_zero_user stub for configs without THP/HUGETLBFS
mm: page_alloc: move prep_compound_page before post_alloc_hook
mm: use folio_zero_user for user pages in post_alloc_hook
mm: use __GFP_ZERO in vma_alloc_zeroed_movable_folio
mm: remove arch vma_alloc_zeroed_movable_folio overrides
mm: alloc_anon_folio: pass raw fault address to vma_alloc_folio
mm: alloc_swap_folio: pass raw fault address to vma_alloc_folio
mm: use __GFP_ZERO in alloc_anon_folio
mm: vma_alloc_anon_folio_pmd: pass raw fault address to
vma_alloc_folio
mm: use __GFP_ZERO in vma_alloc_anon_folio_pmd
mm: hugetlb: use __GFP_ZERO and skip zeroing for zeroed pages
mm: memfd: skip zeroing for zeroed hugetlb pool pages
mm: page_reporting: allow driver to set batch capacity
mm: page_alloc: propagate PageReported flag across buddy splits
mm: page_reporting: skip redundant zeroing of host-zeroed reported
pages
mm: page_reporting: add per-page zeroed bitmap for host feedback
mm: page_alloc: clear PG_zeroed on buddy merge if not both zero
mm: page_alloc: preserve PG_zeroed in page_del_and_expand
virtio_balloon: submit reported pages as individual buffers
mm: page_reporting: add flush parameter with page budget
mm: page_alloc: propagate PG_zeroed in split_large_buddy
virtio_balloon: skip zeroing for host-zeroed reported pages
virtio_balloon: disable reporting zeroed optimization for confidential
guests
mm: add free_frozen_pages_zeroed
mm: add put_page_zeroed and folio_put_zeroed
virtio_balloon: implement VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE
mm: balloon: use put_page_zeroed for zeroed balloon pages
arch/alpha/include/asm/page.h | 3 -
arch/m68k/include/asm/page_no.h | 3 -
arch/s390/include/asm/page.h | 3 -
arch/x86/include/asm/page.h | 3 -
drivers/virtio/virtio_balloon.c | 160 +++++++++++++++++----
fs/hugetlbfs/inode.c | 10 +-
include/linux/gfp.h | 12 +-
include/linux/highmem.h | 9 +-
include/linux/hugetlb.h | 14 +-
include/linux/mm.h | 15 ++
include/linux/page-flags.h | 9 ++
include/linux/page_reporting.h | 13 ++
include/uapi/linux/virtio_balloon.h | 2 +
mm/balloon.c | 7 +-
mm/compaction.c | 7 +-
mm/huge_memory.c | 12 +-
mm/hugetlb.c | 99 +++++++++----
mm/internal.h | 17 ++-
mm/memfd.c | 14 +-
mm/memory.c | 17 +--
mm/mempolicy.c | 73 ++++------
mm/page_alloc.c | 213 +++++++++++++++++++++++-----
mm/page_reporting.c | 88 ++++++++++--
mm/page_reporting.h | 12 ++
mm/slub.c | 4 +-
mm/swap.c | 18 ++-
26 files changed, 615 insertions(+), 222 deletions(-)
--
MST
^ permalink raw reply
* Re: [PATCH] drm/virtio: check virtio_gpu_array_lock_resv() return in cursor update
From: Dmitry Osipenko @ 2026-05-10 21:11 UTC (permalink / raw)
To: Deepanshu Kartikey, airlied, kraxel, gurchetansingh, olvaffe,
maarten.lankhorst, mripard, tzimmermann, simona, sumit.semwal,
christian.koenig
Cc: dri-devel, virtualization, linux-kernel, linux-media,
linaro-mm-sig, syzbot+72bd3dd3a5d5f39a0271, stable
In-Reply-To: <20260510053025.100224-1-kartikey406@gmail.com>
Hello,
On 5/10/26 08:30, Deepanshu Kartikey wrote:
> virtio_gpu_cursor_plane_update() calls virtio_gpu_array_lock_resv()
> but ignores its return value. The function can fail in two ways:
>
> - dma_resv_lock_interruptible() returns -ERESTARTSYS when a signal
> is delivered while waiting for the reservation lock.
> - dma_resv_reserve_fences() returns -ENOMEM if it fails to allocate
> a fence slot; in this case lock_resv unlocks before returning.
>
> In both cases the resv lock is not held on return. The cursor path
> proceeds to queue a fenced transfer command. The queue path then
> walks the object array and calls dma_resv_add_fence() on the cursor
> BO's reservation. dma_resv_add_fence() requires the resv lock to be
> held; with lockdep enabled the missing lock trips
> dma_resv_assert_held():
>
> WARNING: drivers/dma-buf/dma-resv.c:296 at dma_resv_add_fence+0x71e/0x840
> Call Trace:
> virtio_gpu_array_add_fence+0xcd/0x140
> virtio_gpu_queue_ctrl_sgs
> virtio_gpu_queue_fenced_ctrl_buffer+0x578/0xfb0
> virtio_gpu_cursor_plane_update+0x411/0xbc0
> drm_atomic_helper_commit_planes+0x497/0xf10
> ...
> drm_mode_cursor_ioctl+0xd4/0x110
> drm_ioctl+0x5e6/0xc60
> __x64_sys_ioctl+0x18e/0x210
>
> Beyond the WARN, mutating the dma_resv fence list without the lock
> races with concurrent readers/writers and can corrupt the list.
>
> Check the return value of virtio_gpu_array_lock_resv(). On failure,
> drop the references taken by virtio_gpu_array_add_obj() with
> virtio_gpu_array_put_free() (which does not unlock, matching the
> not-locked state) and return without queueing the command. A
> skipped cursor frame is harmless; the WARN and the underlying race
> are not.
>
> The bug was reported by syzbot, triggered via fault injection
> (fail_nth) on the DRM_IOCTL_MODE_CURSOR path, which forces the
> -ENOMEM branch in dma_resv_reserve_fences().
>
> Reported-by: syzbot+72bd3dd3a5d5f39a0271@syzkaller.appspotmail.com
> Closes: https://syzkaller.appspot.com/bug?extid=72bd3dd3a5d5f39a0271
> Fixes: 5cfd31c5b3a3 ("drm/virtio: fix virtio_gpu_cursor_plane_update().")
> Cc: stable@vger.kernel.org
> Signed-off-by: Deepanshu Kartikey <kartikey406@gmail.com>
> ---
> drivers/gpu/drm/virtio/virtgpu_plane.c | 5 ++++-
> 1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/virtio/virtgpu_plane.c b/drivers/gpu/drm/virtio/virtgpu_plane.c
> index a126d1b25f46..ca379b08b9ec 100644
> --- a/drivers/gpu/drm/virtio/virtgpu_plane.c
> +++ b/drivers/gpu/drm/virtio/virtgpu_plane.c
> @@ -459,7 +459,10 @@ static void virtio_gpu_cursor_plane_update(struct drm_plane *plane,
> if (!objs)
> return;
> virtio_gpu_array_add_obj(objs, vgfb->base.obj[0]);
> - virtio_gpu_array_lock_resv(objs);
> + if (virtio_gpu_array_lock_resv(objs)) {
> + virtio_gpu_array_put_free(objs);
> + return;
> + }
> virtio_gpu_cmd_transfer_to_host_2d
> (vgdev, 0,
> plane->state->crtc_w,
Thanks for the patch. Atomic update shouldn't fail due to non-critical
errors like on a signal interrupt. Could you please move this code that
may fail in update() to .prepare/cleanup_fb() callbacks?
--
Best regards,
Dmitry
^ permalink raw reply
* Re: [PATCH net-next v11 1/4] tun/tap: add ptr_ring consume helper with netdev queue wakeup
From: Michael S. Tsirkin @ 2026-05-10 18:27 UTC (permalink / raw)
To: Simon Schippers
Cc: willemdebruijn.kernel, jasowang, andrew+netdev, davem, edumazet,
kuba, pabeni, eperezma, leiyang, stephen, jon, tim.gebauer,
netdev, linux-kernel, kvm, virtualization
In-Reply-To: <5f39493e-b2f0-446b-9896-98a074f5ed6b@tu-dortmund.de>
On Sun, May 10, 2026 at 06:22:15PM +0200, Simon Schippers wrote:
> On 5/10/26 17:44, Michael S. Tsirkin wrote:
> > On Sun, May 10, 2026 at 04:01:39PM +0200, Simon Schippers wrote:
> >> On 5/10/26 15:40, Michael S. Tsirkin wrote:
> >>> On Sun, May 10, 2026 at 10:55:34AM +0200, Simon Schippers wrote:
> >>>> On 5/10/26 09:03, Simon Schippers wrote:
> >>>>> On 5/10/26 00:44, Michael S. Tsirkin wrote:
> >>>>>> On Sat, May 09, 2026 at 06:31:47PM +0200, Simon Schippers wrote:
> >>>>>>> On 5/8/26 17:10, Simon Schippers wrote:
> >>>>>>>> +static void tun_queue_purge(struct tun_struct *tun, struct tun_file *tfile)
> >>>>>>>> {
> >>>>>>>> void *ptr;
> >>>>>>>>
> >>>>>>>> - while ((ptr = ptr_ring_consume(&tfile->tx_ring)) != NULL)
> >>>>>>>> + while ((ptr = tun_ring_consume(tun, tfile)) != NULL)
> >>>>>>>> tun_ptr_free(ptr);
> >>>>>>>>
> >>>>>>>> skb_queue_purge(&tfile->sk.sk_write_queue);
> >>>>>>>
> >>>>>>> Sashiko is right once again. tun_ring_consume() in tun_queue_purge()
> >>>>>>> operates on a tfile that is being torn down. Its queue_index is no
> >>>>>>> longer valid. After the swap in __tun_detach(), it points to the
> >>>>>>> netdev subqueue of a different tfile.
> >>>>>>> --> We should not wake there.
> >>>>>>
> >>>>>> Does it not exactly point at ntfile which is what we want to wake?
> >>>>>>
> >>>>>
> >>>>> I see your point. But calling tun_ring_consume() as done here is
> >>>>> wrong, because it does not wake if the tx_ring of the tfile
> >>>>> (that is currently torn down) is empty. We could change
> >>>>> tun_ring_consume() to call __tun_wake_queue()
> >>>>> with consumed=0 if !ptr but I think this would slow down the consumer
> >>>>> path.
> >>>>>
> >>>>
> >>>> My statement is wrong:
> >>>> There is no way that the tx_ring is empty and the queue is stopped
> >>>> at the same time. So we do not need to touch tun_ring_consume() and
> >>>> this works just fine.
> >>>>
> >>>>>>
> >>>>>>> I will swap tun_ring_consume() with ptr_ring_consume() again and
> >>>>>>> submit a v12 :)
> >>>>>>
> >>>>>> If so then maybe
> >>>>>> netif_tx_wake_queue(netdev_get_tx_queue(tun->dev, index));
> >>>>>>
> >>>>>
> >>>>> But we should only do this if there is space in the ntfile.
> >>>>> My approach:
> >>>>>
> >>>>> @@ -586,12 +588,18 @@ static void __tun_detach(struct tun_file *tfile, bool clean)
> >>>>> BUG_ON(index >= tun->numqueues);
> >>>>>
> >>>>> rcu_assign_pointer(tun->tfiles[index],
> >>>>> tun->tfiles[tun->numqueues - 1]);
> >>>>> ntfile = rtnl_dereference(tun->tfiles[index]);
> >>>>> + spin_lock(&ntfile->tx_ring.consumer_lock);
> >>>>> ntfile->queue_index = index;
> >>>>> ntfile->xdp_rxq.queue_index = index;
> >>>>> + ntfile->cons_cnt = 0;
> >>>>> + if (__ptr_ring_empty(&ntfile->tx_ring)) {
> >>>>> + netif_wake_subqueue(tun->dev, index);
> >>>>> + }
> >>>>> + spin_unlock(&ntfile->tx_ring.consumer_lock);
> >>>>> rcu_assign_pointer(tun->tfiles[tun->numqueues - 1],
> >>>>> NULL);
> >>>>>
> >>>>> ntfile->cons_cnt is unvalid, because the new queue might not be stopped.
> >>>>> That is the reason why I reset it to 0.
> >>>>
> >>>> However, I still prefer this approach because the code is easier to
> >>>> understand.
> >>>
> >>>
> >>> So do you want me to finish review of this one and ack, or want to
> >>> post v12?
> >>>
> >>
> >> I will post a v12 with the proposed changes for patch 1.
> >> No other changes.
> >>
> >> Thanks!
> >
> > actually can you clarify? why only when ntfile ring is empty?
> >
>
> This avoids waking when ntfile->tx_ring is full. We can not use
> __ptr_ring_can_produce() with consumer locks, therefore I chose
> __ptr_ring_empty() instead.
>
> If there are any elements in ntfile->tx_ring we do not have to wake.
> This will be done by the consumer in tun_ring_consume() &
> __tun_wake_queue() after consuming those elements.
worth a code comment if you need to do v13.
^ permalink raw reply
* Re: [PATCH net-next v11 1/4] tun/tap: add ptr_ring consume helper with netdev queue wakeup
From: Simon Schippers @ 2026-05-10 16:22 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: willemdebruijn.kernel, jasowang, andrew+netdev, davem, edumazet,
kuba, pabeni, eperezma, leiyang, stephen, jon, tim.gebauer,
netdev, linux-kernel, kvm, virtualization
In-Reply-To: <20260510114401-mutt-send-email-mst@kernel.org>
On 5/10/26 17:44, Michael S. Tsirkin wrote:
> On Sun, May 10, 2026 at 04:01:39PM +0200, Simon Schippers wrote:
>> On 5/10/26 15:40, Michael S. Tsirkin wrote:
>>> On Sun, May 10, 2026 at 10:55:34AM +0200, Simon Schippers wrote:
>>>> On 5/10/26 09:03, Simon Schippers wrote:
>>>>> On 5/10/26 00:44, Michael S. Tsirkin wrote:
>>>>>> On Sat, May 09, 2026 at 06:31:47PM +0200, Simon Schippers wrote:
>>>>>>> On 5/8/26 17:10, Simon Schippers wrote:
>>>>>>>> +static void tun_queue_purge(struct tun_struct *tun, struct tun_file *tfile)
>>>>>>>> {
>>>>>>>> void *ptr;
>>>>>>>>
>>>>>>>> - while ((ptr = ptr_ring_consume(&tfile->tx_ring)) != NULL)
>>>>>>>> + while ((ptr = tun_ring_consume(tun, tfile)) != NULL)
>>>>>>>> tun_ptr_free(ptr);
>>>>>>>>
>>>>>>>> skb_queue_purge(&tfile->sk.sk_write_queue);
>>>>>>>
>>>>>>> Sashiko is right once again. tun_ring_consume() in tun_queue_purge()
>>>>>>> operates on a tfile that is being torn down. Its queue_index is no
>>>>>>> longer valid. After the swap in __tun_detach(), it points to the
>>>>>>> netdev subqueue of a different tfile.
>>>>>>> --> We should not wake there.
>>>>>>
>>>>>> Does it not exactly point at ntfile which is what we want to wake?
>>>>>>
>>>>>
>>>>> I see your point. But calling tun_ring_consume() as done here is
>>>>> wrong, because it does not wake if the tx_ring of the tfile
>>>>> (that is currently torn down) is empty. We could change
>>>>> tun_ring_consume() to call __tun_wake_queue()
>>>>> with consumed=0 if !ptr but I think this would slow down the consumer
>>>>> path.
>>>>>
>>>>
>>>> My statement is wrong:
>>>> There is no way that the tx_ring is empty and the queue is stopped
>>>> at the same time. So we do not need to touch tun_ring_consume() and
>>>> this works just fine.
>>>>
>>>>>>
>>>>>>> I will swap tun_ring_consume() with ptr_ring_consume() again and
>>>>>>> submit a v12 :)
>>>>>>
>>>>>> If so then maybe
>>>>>> netif_tx_wake_queue(netdev_get_tx_queue(tun->dev, index));
>>>>>>
>>>>>
>>>>> But we should only do this if there is space in the ntfile.
>>>>> My approach:
>>>>>
>>>>> @@ -586,12 +588,18 @@ static void __tun_detach(struct tun_file *tfile, bool clean)
>>>>> BUG_ON(index >= tun->numqueues);
>>>>>
>>>>> rcu_assign_pointer(tun->tfiles[index],
>>>>> tun->tfiles[tun->numqueues - 1]);
>>>>> ntfile = rtnl_dereference(tun->tfiles[index]);
>>>>> + spin_lock(&ntfile->tx_ring.consumer_lock);
>>>>> ntfile->queue_index = index;
>>>>> ntfile->xdp_rxq.queue_index = index;
>>>>> + ntfile->cons_cnt = 0;
>>>>> + if (__ptr_ring_empty(&ntfile->tx_ring)) {
>>>>> + netif_wake_subqueue(tun->dev, index);
>>>>> + }
>>>>> + spin_unlock(&ntfile->tx_ring.consumer_lock);
>>>>> rcu_assign_pointer(tun->tfiles[tun->numqueues - 1],
>>>>> NULL);
>>>>>
>>>>> ntfile->cons_cnt is unvalid, because the new queue might not be stopped.
>>>>> That is the reason why I reset it to 0.
>>>>
>>>> However, I still prefer this approach because the code is easier to
>>>> understand.
>>>
>>>
>>> So do you want me to finish review of this one and ack, or want to
>>> post v12?
>>>
>>
>> I will post a v12 with the proposed changes for patch 1.
>> No other changes.
>>
>> Thanks!
>
> actually can you clarify? why only when ntfile ring is empty?
>
This avoids waking when ntfile->tx_ring is full. We can not use
__ptr_ring_can_produce() with consumer locks, therefore I chose
__ptr_ring_empty() instead.
If there are any elements in ntfile->tx_ring we do not have to wake.
This will be done by the consumer in tun_ring_consume() &
__tun_wake_queue() after consuming those elements.
^ permalink raw reply
* Re: [PATCH net-next v11 1/4] tun/tap: add ptr_ring consume helper with netdev queue wakeup
From: Michael S. Tsirkin @ 2026-05-10 15:44 UTC (permalink / raw)
To: Simon Schippers
Cc: willemdebruijn.kernel, jasowang, andrew+netdev, davem, edumazet,
kuba, pabeni, eperezma, leiyang, stephen, jon, tim.gebauer,
netdev, linux-kernel, kvm, virtualization
In-Reply-To: <e5d69f44-4483-49f3-b7fc-aca2cdb24792@tu-dortmund.de>
On Sun, May 10, 2026 at 04:01:39PM +0200, Simon Schippers wrote:
> On 5/10/26 15:40, Michael S. Tsirkin wrote:
> > On Sun, May 10, 2026 at 10:55:34AM +0200, Simon Schippers wrote:
> >> On 5/10/26 09:03, Simon Schippers wrote:
> >>> On 5/10/26 00:44, Michael S. Tsirkin wrote:
> >>>> On Sat, May 09, 2026 at 06:31:47PM +0200, Simon Schippers wrote:
> >>>>> On 5/8/26 17:10, Simon Schippers wrote:
> >>>>>> +static void tun_queue_purge(struct tun_struct *tun, struct tun_file *tfile)
> >>>>>> {
> >>>>>> void *ptr;
> >>>>>>
> >>>>>> - while ((ptr = ptr_ring_consume(&tfile->tx_ring)) != NULL)
> >>>>>> + while ((ptr = tun_ring_consume(tun, tfile)) != NULL)
> >>>>>> tun_ptr_free(ptr);
> >>>>>>
> >>>>>> skb_queue_purge(&tfile->sk.sk_write_queue);
> >>>>>
> >>>>> Sashiko is right once again. tun_ring_consume() in tun_queue_purge()
> >>>>> operates on a tfile that is being torn down. Its queue_index is no
> >>>>> longer valid. After the swap in __tun_detach(), it points to the
> >>>>> netdev subqueue of a different tfile.
> >>>>> --> We should not wake there.
> >>>>
> >>>> Does it not exactly point at ntfile which is what we want to wake?
> >>>>
> >>>
> >>> I see your point. But calling tun_ring_consume() as done here is
> >>> wrong, because it does not wake if the tx_ring of the tfile
> >>> (that is currently torn down) is empty. We could change
> >>> tun_ring_consume() to call __tun_wake_queue()
> >>> with consumed=0 if !ptr but I think this would slow down the consumer
> >>> path.
> >>>
> >>
> >> My statement is wrong:
> >> There is no way that the tx_ring is empty and the queue is stopped
> >> at the same time. So we do not need to touch tun_ring_consume() and
> >> this works just fine.
> >>
> >>>>
> >>>>> I will swap tun_ring_consume() with ptr_ring_consume() again and
> >>>>> submit a v12 :)
> >>>>
> >>>> If so then maybe
> >>>> netif_tx_wake_queue(netdev_get_tx_queue(tun->dev, index));
> >>>>
> >>>
> >>> But we should only do this if there is space in the ntfile.
> >>> My approach:
> >>>
> >>> @@ -586,12 +588,18 @@ static void __tun_detach(struct tun_file *tfile, bool clean)
> >>> BUG_ON(index >= tun->numqueues);
> >>>
> >>> rcu_assign_pointer(tun->tfiles[index],
> >>> tun->tfiles[tun->numqueues - 1]);
> >>> ntfile = rtnl_dereference(tun->tfiles[index]);
> >>> + spin_lock(&ntfile->tx_ring.consumer_lock);
> >>> ntfile->queue_index = index;
> >>> ntfile->xdp_rxq.queue_index = index;
> >>> + ntfile->cons_cnt = 0;
> >>> + if (__ptr_ring_empty(&ntfile->tx_ring)) {
> >>> + netif_wake_subqueue(tun->dev, index);
> >>> + }
> >>> + spin_unlock(&ntfile->tx_ring.consumer_lock);
> >>> rcu_assign_pointer(tun->tfiles[tun->numqueues - 1],
> >>> NULL);
> >>>
> >>> ntfile->cons_cnt is unvalid, because the new queue might not be stopped.
> >>> That is the reason why I reset it to 0.
> >>
> >> However, I still prefer this approach because the code is easier to
> >> understand.
> >
> >
> > So do you want me to finish review of this one and ack, or want to
> > post v12?
> >
>
> I will post a v12 with the proposed changes for patch 1.
> No other changes.
>
> Thanks!
actually can you clarify? why only when ntfile ring is empty?
^ permalink raw reply
* [PATCH net-next v12 4/4] tun/tap & vhost-net: avoid ptr_ring tail-drop when a qdisc is present
From: Simon Schippers @ 2026-05-10 15:15 UTC (permalink / raw)
To: willemdebruijn.kernel, jasowang, andrew+netdev, davem, edumazet,
kuba, pabeni, mst, eperezma, leiyang, stephen, jon, tim.gebauer,
simon.schippers, netdev, linux-kernel, kvm, virtualization
In-Reply-To: <20260510151529.43895-1-simon.schippers@tu-dortmund.de>
This commit prevents tail-drop when a qdisc is present and the ptr_ring
becomes full. Once the ring reaches capacity after a produce attempt,
the netdev queue is stopped instead of dropping subsequent packets.
If no qdisc is present, the previous tail-drop behavior is preserved.
If producing an entry fails anyway due to a race, tun_net_xmit() drops
the packet. Such races are expected because LLTX is enabled and the
transmit path operates without the usual locking.
The __tun_wake_queue() function of the consumer races with the producer
for waking/stopping the netdev queue, which could result in a stalled
queue. Therefore, an smp_mb__after_atomic() is introduced that pairs
with the smp_mb() of the consumer. It follows the principle of store
buffering described in tools/memory-model/Documentation/recipes.txt:
- The producer in tun_net_xmit() first sets __QUEUE_STATE_DRV_XOFF,
followed by an smp_mb__after_atomic() (= smp_mb()), and then reads the
ring with __ptr_ring_check_produce().
- The consumer in __tun_wake_queue() first writes zero to the ring in
__ptr_ring_consume(), followed by an smp_mb(), and then reads the queue
status with netif_tx_queue_stopped().
=> Following the aforementioned principle, it is impossible for the
producer to see a full ring (and therefore not wake the queue on the
re-check) while the consumer simultaneously fails to see a stopped
queue (and therefore also does not wake it).
Benchmarks:
The benchmarks show a slight regression in raw transmission performance
when using two sending threads. Packet loss also occurs only in the
two-thread sending case; no packet loss was observed with a single
sending thread.
Test setup:
AMD Ryzen 5 5600X at 4.3 GHz, 3200 MHz RAM, isolated QEMU threads;
Average over 50 runs @ 100,000,000 packets. SRSO and spectre v2
mitigations disabled.
Note for tap+vhost-net:
XDP drop program active in VM -> ~2.5x faster; slower for tap due to
more syscalls (high utilization of entry_SYSRETQ_unsafe_stack in perf)
+--------------------------+--------------+----------------+----------+
| 1 thread | Stock | Patched with | diff |
| sending | | fq_codel qdisc | |
+------------+-------------+--------------+----------------+----------+
| TAP | Received | 1.132 Mpps | 1.123 Mpps | -0.8% |
| +-------------+--------------+----------------+----------+
| | Lost/s | 3.765 Mpps | 0 pps | |
+------------+-------------+--------------+----------------+----------+
| TAP | Received | 3.857 Mpps | 3.901 Mpps | +1.1% |
| +-------------+--------------+----------------+----------+
| +vhost-net | Lost/s | 0.802 Mpps | 0 pps | |
+------------+-------------+--------------+----------------+----------+
+--------------------------+--------------+----------------+----------+
| 2 threads | Stock | Patched with | diff |
| sending | | fq_codel qdisc | |
+------------+-------------+--------------+----------------+----------+
| TAP | Received | 1.115 Mpps | 1.081 Mpps | -3.0% |
| +-------------+--------------+----------------+----------+
| | Lost/s | 8.490 Mpps | 391 pps | |
+------------+-------------+--------------+----------------+----------+
| TAP | Received | 3.664 Mpps | 3.555 Mpps | -3.0% |
| +-------------+--------------+----------------+----------+
| +vhost-net | Lost/s | 5.330 Mpps | 938 pps | |
+------------+-------------+--------------+----------------+----------+
Co-developed-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
---
drivers/net/tun.c | 25 +++++++++++++++++++++++--
1 file changed, 23 insertions(+), 2 deletions(-)
diff --git a/drivers/net/tun.c b/drivers/net/tun.c
index cbec66646a4b..bfa49fa9e3a1 100644
--- a/drivers/net/tun.c
+++ b/drivers/net/tun.c
@@ -1018,6 +1018,7 @@ static netdev_tx_t tun_net_xmit(struct sk_buff *skb, struct net_device *dev)
struct netdev_queue *queue;
struct tun_file *tfile;
int len = skb->len;
+ int ret;
rcu_read_lock();
tfile = rcu_dereference(tun->tfiles[txq]);
@@ -1072,13 +1073,33 @@ static netdev_tx_t tun_net_xmit(struct sk_buff *skb, struct net_device *dev)
nf_reset_ct(skb);
- if (ptr_ring_produce(&tfile->tx_ring, skb)) {
+ queue = netdev_get_tx_queue(dev, txq);
+
+ spin_lock(&tfile->tx_ring.producer_lock);
+ ret = __ptr_ring_produce(&tfile->tx_ring, skb);
+ if (!qdisc_txq_has_no_queue(queue) &&
+ __ptr_ring_check_produce(&tfile->tx_ring) == -ENOSPC) {
+ netif_tx_stop_queue(queue);
+ /* Paired with smp_mb() in __tun_wake_queue() */
+ smp_mb__after_atomic();
+ if (!__ptr_ring_check_produce(&tfile->tx_ring))
+ netif_tx_wake_queue(queue);
+ }
+ spin_unlock(&tfile->tx_ring.producer_lock);
+
+ if (ret) {
+ /* This should be a rare case if a qdisc is present, but
+ * can happen due to lltx.
+ * Since skb_tx_timestamp(), skb_orphan(),
+ * run_ebpf_filter() and pskb_trim() could have tinkered
+ * with the SKB, returning NETDEV_TX_BUSY is unsafe and
+ * we must drop instead.
+ */
drop_reason = SKB_DROP_REASON_FULL_RING;
goto drop;
}
/* dev->lltx requires to do our own update of trans_start */
- queue = netdev_get_tx_queue(dev, txq);
txq_trans_cond_update(queue);
/* Notify and wake up reader process */
--
2.43.0
^ permalink raw reply related
* [PATCH net-next v12 3/4] ptr_ring: move free-space check into separate helper
From: Simon Schippers @ 2026-05-10 15:15 UTC (permalink / raw)
To: willemdebruijn.kernel, jasowang, andrew+netdev, davem, edumazet,
kuba, pabeni, mst, eperezma, leiyang, stephen, jon, tim.gebauer,
simon.schippers, netdev, linux-kernel, kvm, virtualization
In-Reply-To: <20260510151529.43895-1-simon.schippers@tu-dortmund.de>
This patch moves the check for available free space for a new entry into
a separate function. Existing callers that only check for a non-zero
return value are unaffected; __ptr_ring_produce() now returns -EINVAL
for a zero-size ring and -ENOSPC when full, whereas before both cases
returned -ENOSPC. The new helper allows callers to determine in advance
whether subsequent __ptr_ring_produce() calls will succeed. This
information can, for example, be used to temporarily stop producing until
__ptr_ring_check_produce() indicates that space is available again.
Co-developed-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
---
include/linux/ptr_ring.h | 20 ++++++++++++++++++--
1 file changed, 18 insertions(+), 2 deletions(-)
diff --git a/include/linux/ptr_ring.h b/include/linux/ptr_ring.h
index d2c3629bbe45..c95e891903f0 100644
--- a/include/linux/ptr_ring.h
+++ b/include/linux/ptr_ring.h
@@ -96,6 +96,20 @@ static inline bool ptr_ring_full_bh(struct ptr_ring *r)
return ret;
}
+/* Note: callers invoking this in a loop must use a compiler barrier,
+ * for example cpu_relax(). Callers must hold producer_lock.
+ */
+static inline int __ptr_ring_check_produce(struct ptr_ring *r)
+{
+ if (unlikely(!r->size))
+ return -EINVAL;
+
+ if (data_race(r->queue[r->producer]))
+ return -ENOSPC;
+
+ return 0;
+}
+
/* Note: callers invoking this in a loop must use a compiler barrier,
* for example cpu_relax(). Callers must hold producer_lock.
* Callers are responsible for making sure pointer that is being queued
@@ -103,8 +117,10 @@ static inline bool ptr_ring_full_bh(struct ptr_ring *r)
*/
static inline int __ptr_ring_produce(struct ptr_ring *r, void *ptr)
{
- if (unlikely(!r->size) || data_race(r->queue[r->producer]))
- return -ENOSPC;
+ int p = __ptr_ring_check_produce(r);
+
+ if (p)
+ return p;
/* Make sure the pointer we are storing points to a valid data. */
/* Pairs with the dependency ordering in __ptr_ring_consume. */
--
2.43.0
^ permalink raw reply related
* [PATCH net-next v12 1/4] tun/tap: add ptr_ring consume helper with netdev queue wakeup
From: Simon Schippers @ 2026-05-10 15:15 UTC (permalink / raw)
To: willemdebruijn.kernel, jasowang, andrew+netdev, davem, edumazet,
kuba, pabeni, mst, eperezma, leiyang, stephen, jon, tim.gebauer,
simon.schippers, netdev, linux-kernel, kvm, virtualization
In-Reply-To: <20260510151529.43895-1-simon.schippers@tu-dortmund.de>
Introduce tun_ring_consume() that wraps ptr_ring_consume() and calls
__tun_wake_queue(). The latter wakes the stopped netdev subqueue once
half of the ring capacity has been consumed, tracked via the new
cons_cnt field in tun_file. As a safety net, the queue is also woken on
the last consumed entry if it leaves the ring empty. The point is to
allow the queue to be stopped when it gets full, which is required for
traffic shaping - implemented by the following "avoid ptr_ring tail-drop
when a qdisc is present".
Some implementation details:
- tun_ring_recv() replaces ptr_ring_consume() with tun_ring_consume()
to properly wake the queue.
- __tun_detach() locks the tx_ring.consumer_lock to avoid races with
the consumer on the queue_index.
- The ptr_ring_consume() call in tun_queue_purge() is not replaced with
tun_ring_consume(). Instead, within the same tx_ring.consumer_lock
in __tun_detach(), the netdev queue is woken for the ntfile taking
it over, to avoid a possible stall. This does not matter for
tun_detach_all(), as it is called during device teardown and no tfile
takes over any queue.
- Reset cons_cnt in tun_attach() so the half-ring wake threshold is
valid for the new ring size after ptr_ring_resize().
- tun_queue_resize() wakes all queues after resizing with the proper
tx_ring.consumer_lock and resets the cons_cnt to avoid a possible
stale queue.
- The aforementioned upcoming patch explains the pairing of the smp_mb()
of __tun_wake_queue().
Without the corresponding queue stopping, this patch alone causes no
regression for a tap setup sending to a qemu VM: 1.132 Mpps
to 1.134 Mpps.
Details: AMD Ryzen 5 5600X at 4.3 GHz, 3200 MHz RAM, isolated QEMU
threads, pktgen sender; Avg over 50 runs @ 100,000,000 packets;
SRSO and spectre v2 mitigations disabled.
Co-developed-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
---
drivers/net/tun.c | 61 +++++++++++++++++++++++++++++++++++++++++++----
1 file changed, 57 insertions(+), 4 deletions(-)
diff --git a/drivers/net/tun.c b/drivers/net/tun.c
index b183189f1853..3dded7c7d12d 100644
--- a/drivers/net/tun.c
+++ b/drivers/net/tun.c
@@ -145,6 +145,8 @@ struct tun_file {
struct list_head next;
struct tun_struct *detached;
struct ptr_ring tx_ring;
+ /* Protected by tx_ring.consumer_lock */
+ int cons_cnt;
struct xdp_rxq_info xdp_rxq;
};
@@ -588,8 +590,13 @@ static void __tun_detach(struct tun_file *tfile, bool clean)
rcu_assign_pointer(tun->tfiles[index],
tun->tfiles[tun->numqueues - 1]);
ntfile = rtnl_dereference(tun->tfiles[index]);
+ spin_lock(&ntfile->tx_ring.consumer_lock);
ntfile->queue_index = index;
ntfile->xdp_rxq.queue_index = index;
+ ntfile->cons_cnt = 0;
+ if (__ptr_ring_empty(&ntfile->tx_ring))
+ netif_wake_subqueue(tun->dev, index);
+ spin_unlock(&ntfile->tx_ring.consumer_lock);
rcu_assign_pointer(tun->tfiles[tun->numqueues - 1],
NULL);
@@ -730,6 +737,9 @@ static int tun_attach(struct tun_struct *tun, struct file *file,
goto out;
}
+ spin_lock(&tfile->tx_ring.consumer_lock);
+ tfile->cons_cnt = 0;
+ spin_unlock(&tfile->tx_ring.consumer_lock);
tfile->queue_index = tun->numqueues;
tfile->socket.sk->sk_shutdown &= ~RCV_SHUTDOWN;
@@ -2115,13 +2125,46 @@ static ssize_t tun_put_user(struct tun_struct *tun,
return total;
}
-static void *tun_ring_recv(struct tun_file *tfile, int noblock, int *err)
+/* Callers must hold ring.consumer_lock */
+static void __tun_wake_queue(struct tun_struct *tun,
+ struct tun_file *tfile, int consumed)
+{
+ struct netdev_queue *txq = netdev_get_tx_queue(tun->dev,
+ tfile->queue_index);
+
+ /* Paired with smp_mb__after_atomic() in tun_net_xmit() */
+ smp_mb();
+ if (netif_tx_queue_stopped(txq)) {
+ tfile->cons_cnt += consumed;
+ if (tfile->cons_cnt >= tfile->tx_ring.size / 2 ||
+ __ptr_ring_empty(&tfile->tx_ring)) {
+ netif_tx_wake_queue(txq);
+ tfile->cons_cnt = 0;
+ }
+ }
+}
+
+static void *tun_ring_consume(struct tun_struct *tun, struct tun_file *tfile)
+{
+ void *ptr;
+
+ spin_lock(&tfile->tx_ring.consumer_lock);
+ ptr = __ptr_ring_consume(&tfile->tx_ring);
+ if (ptr)
+ __tun_wake_queue(tun, tfile, 1);
+
+ spin_unlock(&tfile->tx_ring.consumer_lock);
+ return ptr;
+}
+
+static void *tun_ring_recv(struct tun_struct *tun, struct tun_file *tfile,
+ int noblock, int *err)
{
DECLARE_WAITQUEUE(wait, current);
void *ptr = NULL;
int error = 0;
- ptr = ptr_ring_consume(&tfile->tx_ring);
+ ptr = tun_ring_consume(tun, tfile);
if (ptr)
goto out;
if (noblock) {
@@ -2133,7 +2176,7 @@ static void *tun_ring_recv(struct tun_file *tfile, int noblock, int *err)
while (1) {
set_current_state(TASK_INTERRUPTIBLE);
- ptr = ptr_ring_consume(&tfile->tx_ring);
+ ptr = tun_ring_consume(tun, tfile);
if (ptr)
break;
if (signal_pending(current)) {
@@ -2170,7 +2213,7 @@ static ssize_t tun_do_read(struct tun_struct *tun, struct tun_file *tfile,
if (!ptr) {
/* Read frames from ring */
- ptr = tun_ring_recv(tfile, noblock, &err);
+ ptr = tun_ring_recv(tun, tfile, noblock, &err);
if (!ptr)
return err;
}
@@ -3622,6 +3665,16 @@ static int tun_queue_resize(struct tun_struct *tun)
dev->tx_queue_len, GFP_KERNEL,
tun_ptr_free);
+ if (!ret) {
+ for (i = 0; i < tun->numqueues; i++) {
+ tfile = rtnl_dereference(tun->tfiles[i]);
+ spin_lock(&tfile->tx_ring.consumer_lock);
+ netif_wake_subqueue(tun->dev, tfile->queue_index);
+ tfile->cons_cnt = 0;
+ spin_unlock(&tfile->tx_ring.consumer_lock);
+ }
+ }
+
kfree(rings);
return ret;
}
--
2.43.0
^ permalink raw reply related
* [PATCH net-next v12 0/4] tun/tap & vhost-net: apply qdisc backpressure on full ptr_ring to reduce TX drops
From: Simon Schippers @ 2026-05-10 15:15 UTC (permalink / raw)
To: willemdebruijn.kernel, jasowang, andrew+netdev, davem, edumazet,
kuba, pabeni, mst, eperezma, leiyang, stephen, jon, tim.gebauer,
simon.schippers, netdev, linux-kernel, kvm, virtualization
This patch series deals with tun/tap & vhost-net which drop incoming
SKBs whenever their internal ptr_ring buffer is full. Instead, with this
patch series, the associated netdev queue is stopped - but only when a
qdisc is attached. If no qdisc is present the existing behavior is
preserved. The XDP transmit path is not affected. This patch series
touches tun/tap and vhost-net, as they share common logic and must be
updated together. Modifying only one of them would break the other.
By applying proper backpressure, this change allows the connected qdisc to
operate correctly, as reported in [1], and significantly improves
performance in real-world scenarios, as demonstrated in our paper [2]. For
example, we observed a 36% TCP throughput improvement for an OpenVPN
connection between Germany and the USA.
Synthetic pktgen benchmarks indicate a slight regression, and packet
loss is reduced to near zero. Pktgen benchmarks are provided per commit,
with the final commit showing the overall performance.
Thanks!
[1] Link: https://unix.stackexchange.com/questions/762935/traffic-shaping-ineffective-on-tun-device
[2] Link: https://cni.etit.tu-dortmund.de/storages/cni-etit/r/Research/Publications/2025/Gebauer_2025_VTCFall/Gebauer_VTCFall2025_AuthorsVersion.pdf
---
Changelog:
v12:
Patch 1:
- Revert tun_queue_purge() to plain ptr_ring_consume() and instead
explicitly wake the queue in __tun_detach() for the ntfile taking
over the queue slot (if its ring is empty).
- Inlined tun_reset_cons_cnt(), because only tun_attach() uses it.
- Patches 2-4 and cover letter unchanged.
- Compiled and short pktgen test.
v11:
- Renamed __ptr_ring_produce_peek() to __ptr_ring_check_produce()
(Sashiko)
- Add return code -EINVAL to __ptr_ring_check_produce() which lets
tun_net_xmit() stop the queue only on -ENOSPC. (MST)
- Resolve race on tfile->queue_index by locking tx_ring.consumer_lock
in __tun_detach(). (Sashiko)
- Wake the queue in tun_queue_resize() to avoid possible stalls.
- Other minor adjustments & reran the benchmarks.
v10: https://lore.kernel.org/netdev/20260506141033.180450-1-simon.schippers@tu-dortmund.de/
- Changed the term "Transmitted" to "Received" in the benchmarks,
as correctly pointed out by MST, and reran the benchmarks.
Addressed the Sashiko AI review:
- Avoid a data race on tfile->cons_cnt by always locking.
- Correctly count the number of consumed packets for vhost-net.
- Corrected a typo in the commit message of commit 3.
- Added a missing barrier on the consumer side.
--> The barriers now follow the "store buffering" principle.
- No longer return NETDEV_TX_BUSY at all, because it is unsafe.
--> Result: There are still a few drops with multiple senders, which
would be avoided by disabling LLTX.
V9: https://lore.kernel.org/netdev/20260428123859.19578-1-simon.schippers@tu-dortmund.de/
- Addressed minor nit by MST in patches 1 and 2.
- Rebased patch 3 because of commit d748047
("ptr_ring: disable KCSAN warnings").
- Documented the pair of the smp_mb__after_atomic() in tun_net_xmit()
with tun_ring_consume().
--> It simply pairs with the test_and_clear_bit() inside of
netif_wake_subqueue().
- Use 1 ptr_ring consumer spinlock instead of 2.
- Ran pktgen benchmarks with pg_set SHARED for 50 iterations on
latest kernel
--> No significant performance difference noticed
V8: https://lore.kernel.org/netdev/20260312130639.138988-1-simon.schippers@tu-dortmund.de/
- Drop code changes in drivers/net/tap.c; The code there deals with
ipvtap/macvtap which are unrelated to the goal of this patch series
and I did not realize that before
-> Greatly simplified logic, 4 instead of 9 commits
-> No more duplicated logics and distinction in vhost required
- Only wake after the queue stopped and half of the ring was consumed
as suggested by MST
-> Performance improvements for TAP, but still slightly slower
- Better benchmarking with pinned threads, XDP drop program for
tap+vhost-net and disabling CPU mitigations (and newer Ryzen 5 5600X
processor) as suggested by Jason Wang
V7: https://lore.kernel.org/netdev/20260107210448.37851-1-simon.schippers@tu-dortmund.de/
- Switch to an approach similar to veth (excluding the recently fixed
variant), as suggested by MST, with minor adjustments discussed in V6
- Rename the cover-letter title
- Add multithreaded pktgen and iperf3 benchmarks, as suggested by Jason
Wang
- Rework __ptr_ring_consume_created_space() so it can also be used after
batched consume
...
---
Simon Schippers (4):
tun/tap: add ptr_ring consume helper with netdev queue wakeup
vhost-net: wake queue of tun/tap after ptr_ring consume
ptr_ring: move free-space check into separate helper
tun/tap & vhost-net: avoid ptr_ring tail-drop when a qdisc is present
drivers/net/tun.c | 109 ++++++++++++++++++++++++++++++++++++---
drivers/vhost/net.c | 21 +++++---
include/linux/if_tun.h | 3 ++
include/linux/ptr_ring.h | 20 ++++++-
4 files changed, 139 insertions(+), 14 deletions(-)
--
2.43.0
^ permalink raw reply
* [PATCH net-next v12 2/4] vhost-net: wake queue of tun/tap after ptr_ring consume
From: Simon Schippers @ 2026-05-10 15:15 UTC (permalink / raw)
To: willemdebruijn.kernel, jasowang, andrew+netdev, davem, edumazet,
kuba, pabeni, mst, eperezma, leiyang, stephen, jon, tim.gebauer,
simon.schippers, netdev, linux-kernel, kvm, virtualization
In-Reply-To: <20260510151529.43895-1-simon.schippers@tu-dortmund.de>
Add tun_wake_queue() to tun.c and export it for use by vhost-net. The
function validates that the file belongs to a tun/tap device and that
the tfile exists, dereferences the tun_struct under RCU, and delegates
to __tun_wake_queue().
vhost_net_buf_produce() now calls tun_wake_queue() after a successful
batched consume of the ring to allow the netdev subqueue to be woken up.
The point is to allow the queue to be stopped when it gets full, which
is required for traffic shaping - implemented by the following
"avoid ptr_ring tail-drop when a qdisc is present".
Without the corresponding queue stopping, this patch alone causes no
throughput regression for a tap+vhost-net setup sending to a qemu VM:
3.857 Mpps to 3.891 Mpps.
Details: AMD Ryzen 5 5600X at 4.3 GHz, 3200 MHz RAM, isolated QEMU
threads, XDP drop program active in VM, pktgen sender; Avg over
50 runs @ 100,000,000 packets. SRSO and spectre v2 mitigations disabled.
Co-developed-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
---
drivers/net/tun.c | 23 +++++++++++++++++++++++
drivers/vhost/net.c | 21 +++++++++++++++------
include/linux/if_tun.h | 3 +++
3 files changed, 41 insertions(+), 6 deletions(-)
diff --git a/drivers/net/tun.c b/drivers/net/tun.c
index 3dded7c7d12d..cbec66646a4b 100644
--- a/drivers/net/tun.c
+++ b/drivers/net/tun.c
@@ -3783,6 +3783,29 @@ struct ptr_ring *tun_get_tx_ring(struct file *file)
}
EXPORT_SYMBOL_GPL(tun_get_tx_ring);
+/* Callers must hold ring.consumer_lock */
+void tun_wake_queue(struct file *file, int consumed)
+{
+ struct tun_file *tfile;
+ struct tun_struct *tun;
+
+ if (file->f_op != &tun_fops)
+ return;
+
+ tfile = file->private_data;
+ if (!tfile)
+ return;
+
+ rcu_read_lock();
+
+ tun = rcu_dereference(tfile->tun);
+ if (tun)
+ __tun_wake_queue(tun, tfile, consumed);
+
+ rcu_read_unlock();
+}
+EXPORT_SYMBOL_GPL(tun_wake_queue);
+
module_init(tun_init);
module_exit(tun_cleanup);
MODULE_DESCRIPTION(DRV_DESCRIPTION);
diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
index 80965181920c..ee583d6cc0fa 100644
--- a/drivers/vhost/net.c
+++ b/drivers/vhost/net.c
@@ -176,13 +176,21 @@ static void *vhost_net_buf_consume(struct vhost_net_buf *rxq)
return ret;
}
-static int vhost_net_buf_produce(struct vhost_net_virtqueue *nvq)
+static int vhost_net_buf_produce(struct sock *sk,
+ struct vhost_net_virtqueue *nvq)
{
+ struct file *file = sk->sk_socket->file;
struct vhost_net_buf *rxq = &nvq->rxq;
rxq->head = 0;
- rxq->tail = ptr_ring_consume_batched(nvq->rx_ring, rxq->queue,
- VHOST_NET_BATCH);
+ spin_lock(&nvq->rx_ring->consumer_lock);
+ rxq->tail = __ptr_ring_consume_batched(nvq->rx_ring, rxq->queue,
+ VHOST_NET_BATCH);
+
+ if (rxq->tail)
+ tun_wake_queue(file, rxq->tail);
+
+ spin_unlock(&nvq->rx_ring->consumer_lock);
return rxq->tail;
}
@@ -209,14 +217,15 @@ static int vhost_net_buf_peek_len(void *ptr)
return __skb_array_len_with_tag(ptr);
}
-static int vhost_net_buf_peek(struct vhost_net_virtqueue *nvq)
+static int vhost_net_buf_peek(struct sock *sk,
+ struct vhost_net_virtqueue *nvq)
{
struct vhost_net_buf *rxq = &nvq->rxq;
if (!vhost_net_buf_is_empty(rxq))
goto out;
- if (!vhost_net_buf_produce(nvq))
+ if (!vhost_net_buf_produce(sk, nvq))
return 0;
out:
@@ -995,7 +1004,7 @@ static int peek_head_len(struct vhost_net_virtqueue *rvq, struct sock *sk)
unsigned long flags;
if (rvq->rx_ring)
- return vhost_net_buf_peek(rvq);
+ return vhost_net_buf_peek(sk, rvq);
spin_lock_irqsave(&sk->sk_receive_queue.lock, flags);
head = skb_peek(&sk->sk_receive_queue);
diff --git a/include/linux/if_tun.h b/include/linux/if_tun.h
index 80166eb62f41..5f3e206c7a73 100644
--- a/include/linux/if_tun.h
+++ b/include/linux/if_tun.h
@@ -22,6 +22,7 @@ struct tun_msg_ctl {
#if defined(CONFIG_TUN) || defined(CONFIG_TUN_MODULE)
struct socket *tun_get_socket(struct file *);
struct ptr_ring *tun_get_tx_ring(struct file *file);
+void tun_wake_queue(struct file *file, int consumed);
static inline bool tun_is_xdp_frame(void *ptr)
{
@@ -55,6 +56,8 @@ static inline struct ptr_ring *tun_get_tx_ring(struct file *f)
return ERR_PTR(-EINVAL);
}
+static inline void tun_wake_queue(struct file *f, int consumed) {}
+
static inline bool tun_is_xdp_frame(void *ptr)
{
return false;
--
2.43.0
^ permalink raw reply related
* Re: [PATCH net-next v11 1/4] tun/tap: add ptr_ring consume helper with netdev queue wakeup
From: Simon Schippers @ 2026-05-10 14:01 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: willemdebruijn.kernel, jasowang, andrew+netdev, davem, edumazet,
kuba, pabeni, eperezma, leiyang, stephen, jon, tim.gebauer,
netdev, linux-kernel, kvm, virtualization
In-Reply-To: <20260510094020-mutt-send-email-mst@kernel.org>
On 5/10/26 15:40, Michael S. Tsirkin wrote:
> On Sun, May 10, 2026 at 10:55:34AM +0200, Simon Schippers wrote:
>> On 5/10/26 09:03, Simon Schippers wrote:
>>> On 5/10/26 00:44, Michael S. Tsirkin wrote:
>>>> On Sat, May 09, 2026 at 06:31:47PM +0200, Simon Schippers wrote:
>>>>> On 5/8/26 17:10, Simon Schippers wrote:
>>>>>> +static void tun_queue_purge(struct tun_struct *tun, struct tun_file *tfile)
>>>>>> {
>>>>>> void *ptr;
>>>>>>
>>>>>> - while ((ptr = ptr_ring_consume(&tfile->tx_ring)) != NULL)
>>>>>> + while ((ptr = tun_ring_consume(tun, tfile)) != NULL)
>>>>>> tun_ptr_free(ptr);
>>>>>>
>>>>>> skb_queue_purge(&tfile->sk.sk_write_queue);
>>>>>
>>>>> Sashiko is right once again. tun_ring_consume() in tun_queue_purge()
>>>>> operates on a tfile that is being torn down. Its queue_index is no
>>>>> longer valid. After the swap in __tun_detach(), it points to the
>>>>> netdev subqueue of a different tfile.
>>>>> --> We should not wake there.
>>>>
>>>> Does it not exactly point at ntfile which is what we want to wake?
>>>>
>>>
>>> I see your point. But calling tun_ring_consume() as done here is
>>> wrong, because it does not wake if the tx_ring of the tfile
>>> (that is currently torn down) is empty. We could change
>>> tun_ring_consume() to call __tun_wake_queue()
>>> with consumed=0 if !ptr but I think this would slow down the consumer
>>> path.
>>>
>>
>> My statement is wrong:
>> There is no way that the tx_ring is empty and the queue is stopped
>> at the same time. So we do not need to touch tun_ring_consume() and
>> this works just fine.
>>
>>>>
>>>>> I will swap tun_ring_consume() with ptr_ring_consume() again and
>>>>> submit a v12 :)
>>>>
>>>> If so then maybe
>>>> netif_tx_wake_queue(netdev_get_tx_queue(tun->dev, index));
>>>>
>>>
>>> But we should only do this if there is space in the ntfile.
>>> My approach:
>>>
>>> @@ -586,12 +588,18 @@ static void __tun_detach(struct tun_file *tfile, bool clean)
>>> BUG_ON(index >= tun->numqueues);
>>>
>>> rcu_assign_pointer(tun->tfiles[index],
>>> tun->tfiles[tun->numqueues - 1]);
>>> ntfile = rtnl_dereference(tun->tfiles[index]);
>>> + spin_lock(&ntfile->tx_ring.consumer_lock);
>>> ntfile->queue_index = index;
>>> ntfile->xdp_rxq.queue_index = index;
>>> + ntfile->cons_cnt = 0;
>>> + if (__ptr_ring_empty(&ntfile->tx_ring)) {
>>> + netif_wake_subqueue(tun->dev, index);
>>> + }
>>> + spin_unlock(&ntfile->tx_ring.consumer_lock);
>>> rcu_assign_pointer(tun->tfiles[tun->numqueues - 1],
>>> NULL);
>>>
>>> ntfile->cons_cnt is unvalid, because the new queue might not be stopped.
>>> That is the reason why I reset it to 0.
>>
>> However, I still prefer this approach because the code is easier to
>> understand.
>
>
> So do you want me to finish review of this one and ack, or want to
> post v12?
>
I will post a v12 with the proposed changes for patch 1.
No other changes.
Thanks!
^ permalink raw reply
* Re: [PATCH net-next v11 1/4] tun/tap: add ptr_ring consume helper with netdev queue wakeup
From: Michael S. Tsirkin @ 2026-05-10 13:40 UTC (permalink / raw)
To: Simon Schippers
Cc: willemdebruijn.kernel, jasowang, andrew+netdev, davem, edumazet,
kuba, pabeni, eperezma, leiyang, stephen, jon, tim.gebauer,
netdev, linux-kernel, kvm, virtualization
In-Reply-To: <9a4458fc-61f2-469b-8260-f144d3827b5d@tu-dortmund.de>
On Sun, May 10, 2026 at 10:55:34AM +0200, Simon Schippers wrote:
> On 5/10/26 09:03, Simon Schippers wrote:
> > On 5/10/26 00:44, Michael S. Tsirkin wrote:
> >> On Sat, May 09, 2026 at 06:31:47PM +0200, Simon Schippers wrote:
> >>> On 5/8/26 17:10, Simon Schippers wrote:
> >>>> +static void tun_queue_purge(struct tun_struct *tun, struct tun_file *tfile)
> >>>> {
> >>>> void *ptr;
> >>>>
> >>>> - while ((ptr = ptr_ring_consume(&tfile->tx_ring)) != NULL)
> >>>> + while ((ptr = tun_ring_consume(tun, tfile)) != NULL)
> >>>> tun_ptr_free(ptr);
> >>>>
> >>>> skb_queue_purge(&tfile->sk.sk_write_queue);
> >>>
> >>> Sashiko is right once again. tun_ring_consume() in tun_queue_purge()
> >>> operates on a tfile that is being torn down. Its queue_index is no
> >>> longer valid. After the swap in __tun_detach(), it points to the
> >>> netdev subqueue of a different tfile.
> >>> --> We should not wake there.
> >>
> >> Does it not exactly point at ntfile which is what we want to wake?
> >>
> >
> > I see your point. But calling tun_ring_consume() as done here is
> > wrong, because it does not wake if the tx_ring of the tfile
> > (that is currently torn down) is empty. We could change
> > tun_ring_consume() to call __tun_wake_queue()
> > with consumed=0 if !ptr but I think this would slow down the consumer
> > path.
> >
>
> My statement is wrong:
> There is no way that the tx_ring is empty and the queue is stopped
> at the same time. So we do not need to touch tun_ring_consume() and
> this works just fine.
>
> >>
> >>> I will swap tun_ring_consume() with ptr_ring_consume() again and
> >>> submit a v12 :)
> >>
> >> If so then maybe
> >> netif_tx_wake_queue(netdev_get_tx_queue(tun->dev, index));
> >>
> >
> > But we should only do this if there is space in the ntfile.
> > My approach:
> >
> > @@ -586,12 +588,18 @@ static void __tun_detach(struct tun_file *tfile, bool clean)
> > BUG_ON(index >= tun->numqueues);
> >
> > rcu_assign_pointer(tun->tfiles[index],
> > tun->tfiles[tun->numqueues - 1]);
> > ntfile = rtnl_dereference(tun->tfiles[index]);
> > + spin_lock(&ntfile->tx_ring.consumer_lock);
> > ntfile->queue_index = index;
> > ntfile->xdp_rxq.queue_index = index;
> > + ntfile->cons_cnt = 0;
> > + if (__ptr_ring_empty(&ntfile->tx_ring)) {
> > + netif_wake_subqueue(tun->dev, index);
> > + }
> > + spin_unlock(&ntfile->tx_ring.consumer_lock);
> > rcu_assign_pointer(tun->tfiles[tun->numqueues - 1],
> > NULL);
> >
> > ntfile->cons_cnt is unvalid, because the new queue might not be stopped.
> > That is the reason why I reset it to 0.
>
> However, I still prefer this approach because the code is easier to
> understand.
So do you want me to finish review of this one and ack, or want to
post v12?
^ permalink raw reply
* [PATCH RFC v3 2/6] rust/helpers: add virtio.c
From: Manos Pitsidianakis @ 2026-05-10 13:38 UTC (permalink / raw)
To: Miguel Ojeda
Cc: Manos Pitsidianakis, Peter Hilber, Stefano Garzarella,
Stefan Hajnoczi, Viresh Kumar, Michael S. Tsirkin, Boqun Feng,
Gary Guo, Björn Roy Baron, Benno Lossin, Andreas Hindborg,
Alice Ryhl, Trevor Gross, Danilo Krummrich, Daniel Almeida,
rust-for-linux, Jason Wang, Xuan Zhuo, Eugenio Pérez,
virtualization, linux-kernel, Manos Pitsidianakis
In-Reply-To: <20260510-rust-virtio-v3-0-1427f14d67e1@pitsidianak.is>
Some internal kernel virtio API functions are inline macros, so define
their symbols in a helper file.
Signed-off-by: Manos Pitsidianakis <manos@pitsidianak.is>
---
MAINTAINERS | 6 ++++++
rust/helpers/helpers.c | 1 +
rust/helpers/virtio.c | 37 +++++++++++++++++++++++++++++++++++++
3 files changed, 44 insertions(+)
diff --git a/MAINTAINERS b/MAINTAINERS
index d1cc0e12fe1f004da89b1aa339116908f642e894..48c9c666d90b5a256ab6fae1f42508b789a0ce50 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -27930,6 +27930,12 @@ F: include/uapi/linux/virtio_*.h
F: net/vmw_vsock/virtio*
F: tools/virtio/
+VIRTIO CORE API BINDINGS [RUST]
+M: Manos Pitsidianakis <manos@pitsidianak.is>
+L: virtualization@lists.linux.dev
+S: Maintained
+F: rust/helpers/virtio.c
+
VIRTIO CRYPTO DRIVER
M: Gonglei <arei.gonglei@huawei.com>
L: virtualization@lists.linux.dev
diff --git a/rust/helpers/helpers.c b/rust/helpers/helpers.c
index a3c42e51f00a0990bea81ebce6e99bb397ce7533..5dc0d2f2ee6bd2ae8e6abfe4baa247c1963967f6 100644
--- a/rust/helpers/helpers.c
+++ b/rust/helpers/helpers.c
@@ -61,6 +61,7 @@
#include "time.c"
#include "uaccess.c"
#include "usb.c"
+#include "virtio.c"
#include "vmalloc.c"
#include "wait.c"
#include "workqueue.c"
diff --git a/rust/helpers/virtio.c b/rust/helpers/virtio.c
new file mode 100644
index 0000000000000000000000000000000000000000..46aeeb063158823e66477777b3cd4bd1525df330
--- /dev/null
+++ b/rust/helpers/virtio.c
@@ -0,0 +1,37 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#ifdef CONFIG_VIRTIO
+#include <linux/virtio_config.h>
+
+__rust_helper bool
+rust_helper_virtio_has_feature(const struct virtio_device *vdev,
+ unsigned int fbit)
+{
+ return virtio_has_feature(vdev, fbit);
+}
+__rust_helper void rust_helper_virtio_get_features(struct virtio_device *vdev,
+ u64 *features_out)
+{
+ return virtio_get_features(vdev, features_out);
+}
+
+__rust_helper int rust_helper_virtio_find_vqs(struct virtio_device *vdev,
+ unsigned int nvqs,
+ struct virtqueue *vqs[],
+ struct virtqueue_info vqs_info[],
+ struct irq_affinity *desc)
+{
+ return virtio_find_vqs(vdev, nvqs, vqs, vqs_info, desc);
+}
+
+__rust_helper void rust_helper_virtio_device_ready(struct virtio_device *dev)
+{
+ return virtio_device_ready(dev);
+}
+
+__rust_helper bool
+rust_helper_virtio_is_little_endian(struct virtio_device *vdev)
+{
+ return virtio_is_little_endian(vdev);
+}
+#endif /* CONFIG_VIRTIO */
--
2.47.3
^ permalink raw reply related
* [PATCH RFC v3 6/6] samples/rust: Add sample virtio-rtc driver [WIP]
From: Manos Pitsidianakis @ 2026-05-10 13:38 UTC (permalink / raw)
To: Miguel Ojeda
Cc: Manos Pitsidianakis, Peter Hilber, Stefano Garzarella,
Stefan Hajnoczi, Viresh Kumar, Michael S. Tsirkin, Boqun Feng,
Gary Guo, Björn Roy Baron, Benno Lossin, Andreas Hindborg,
Alice Ryhl, Trevor Gross, Danilo Krummrich, Daniel Almeida,
rust-for-linux, Jason Wang, Xuan Zhuo, Eugenio Pérez,
virtualization, linux-kernel, Manos Pitsidianakis
In-Reply-To: <20260510-rust-virtio-v3-0-1427f14d67e1@pitsidianak.is>
While the driver queries clocks and capabilities for each clock, it
doesn't actually register them yet (TODO).
Until I implement missing functionality, there is some dead code and
some missing SAFETY comments.
Signed-off-by: Manos Pitsidianakis <manos@pitsidianak.is>
---
MAINTAINERS | 1 +
samples/rust/Kconfig | 15 ++
samples/rust/Makefile | 1 +
samples/rust/rust_virtio_rtc.rs | 403 ++++++++++++++++++++++++++++++++++++++++
4 files changed, 420 insertions(+)
diff --git a/MAINTAINERS b/MAINTAINERS
index e8012f708df5d4ee858c82aec3269e615fc8caad..3ed579e8d3cc64d1749cf261cd68f6338a830c4d 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -27937,6 +27937,7 @@ S: Maintained
F: rust/helpers/virtio.c
F: rust/kernel/virtio.rs
F: rust/kernel/virtio/
+F: samples/rust/rust_virtio_rtc.rs
VIRTIO CRYPTO DRIVER
M: Gonglei <arei.gonglei@huawei.com>
diff --git a/samples/rust/Kconfig b/samples/rust/Kconfig
index c49ab910634596aea4a1a73dac87585e084f420a..96a16aecc27198fd99f4ffd0ecdf0bc0876860c6 100644
--- a/samples/rust/Kconfig
+++ b/samples/rust/Kconfig
@@ -179,4 +179,19 @@ config SAMPLE_RUST_HOSTPROGS
If unsure, say N.
+config SAMPLE_RUST_VIRTIO_RTC
+ tristate "Rust Virtio RTC driver"
+ depends on VIRTIO
+ depends on PTP_1588_CLOCK_OPTIONAL
+ help
+ This driver provides current time from a Virtio RTC device. The driver
+ provides the time through one or more clocks. The Virtio RTC PTP
+ clocks and/or the Real Time Clock driver for Virtio RTC must be
+ enabled to expose the clocks to userspace.
+
+ To compile this code as a module, choose M here: the module will be
+ called rust_virtio_rtc.
+
+ If unsure, say M.
+
endif # SAMPLES_RUST
diff --git a/samples/rust/Makefile b/samples/rust/Makefile
index 6c0aaa58ccccfd12ef019f68ca784f6d977bc668..0142fd8656bb8cdc95b7ef54e3183b5e51358954 100644
--- a/samples/rust/Makefile
+++ b/samples/rust/Makefile
@@ -16,6 +16,7 @@ obj-$(CONFIG_SAMPLE_RUST_DRIVER_FAUX) += rust_driver_faux.o
obj-$(CONFIG_SAMPLE_RUST_DRIVER_AUXILIARY) += rust_driver_auxiliary.o
obj-$(CONFIG_SAMPLE_RUST_CONFIGFS) += rust_configfs.o
obj-$(CONFIG_SAMPLE_RUST_SOC) += rust_soc.o
+obj-$(CONFIG_SAMPLE_RUST_VIRTIO_RTC) += rust_virtio_rtc.o
rust_print-y := rust_print_main.o rust_print_events.o
diff --git a/samples/rust/rust_virtio_rtc.rs b/samples/rust/rust_virtio_rtc.rs
new file mode 100644
index 0000000000000000000000000000000000000000..81f8b377f6c92716c17ea11542fb4262cfd90e1c
--- /dev/null
+++ b/samples/rust/rust_virtio_rtc.rs
@@ -0,0 +1,403 @@
+// SPDX-License-Identifier: GPL-2.0
+
+//! Rust virtio driver sample.
+
+use core::{
+ ptr::NonNull, //
+ sync::atomic::{
+ AtomicU16,
+ Ordering, //
+ },
+};
+
+use kernel::{
+ device::{
+ Bound,
+ Core, //
+ },
+ new_mutex, //
+ new_spinlock, //
+ prelude::*,
+ sync::{
+ Completion,
+ Mutex,
+ SpinLock, //
+ },
+ virtio::{
+ self,
+ utils::*,
+ virtqueue::*, //
+ },
+};
+
+use pin_init::{
+ stack_pin_init,
+ stack_try_pin_init, //
+};
+
+#[pin_data]
+struct Token {
+ resp_actual_size: u32,
+ #[pin]
+ responded: Completion,
+}
+
+#[derive(Copy, Clone, Debug, Zeroable)]
+#[repr(C)]
+#[doc(alias = "virtio_rtc_req_head")]
+struct ReqHead {
+ msg_type: Le16,
+ reserved: [u8; 6],
+}
+
+#[derive(Copy, Clone, Debug, Zeroable)]
+#[repr(C)]
+#[doc(alias = "virtio_rtc_resp_head")]
+struct RespHead {
+ status: u8,
+ reserved: [u8; 7],
+}
+
+#[derive(Copy, Clone, Debug, Zeroable)]
+#[repr(C)]
+#[doc(alias = "virtio_rtc_resp_cfg")]
+struct RespCfg {
+ head: RespHead,
+ /** # of clocks -> clock ids < num_clocks are valid */
+ num_clocks: Le16,
+ reserved: [u8; 6],
+}
+
+#[derive(Debug, Zeroable)]
+#[repr(C)]
+#[doc(alias = "virtio_rtc_req_clock_cap")]
+struct ReqClockCap {
+ head: ReqHead,
+ clock_id: Le16,
+ reserved: [u8; 6],
+}
+
+#[derive(Copy, Clone, Debug, Zeroable)]
+#[repr(C)]
+#[doc(alias = "virtio_rtc_resp_clock_cap")]
+struct RespClockCap {
+ head: RespHead,
+ clock_type: u8,
+ leap_second_smearing: u8,
+ flags: u8,
+ reserved: [u8; 5],
+}
+
+#[derive(Debug, Zeroable)]
+#[repr(C)]
+#[doc(alias = "virtio_rtc_req_read")]
+struct ReqRead {
+ head: ReqHead,
+ clock_id: Le16,
+ reserved: [u8; 6],
+}
+
+#[derive(Copy, Clone, Debug, Zeroable)]
+#[repr(C)]
+#[doc(alias = "virtio_rtc_resp_read")]
+struct RespRead {
+ head: RespHead,
+ clock_reading: Le64,
+}
+
+#[repr(u8)]
+enum ClockType {
+ #[doc(alias = "VIRTIO_RTC_CLOCK_UTC")]
+ Utc = 0,
+ #[doc(alias = "VIRTIO_RTC_CLOCK_TAI")]
+ Tai = 1,
+ #[doc(alias = "VIRTIO_RTC_CLOCK_MONOTONIC")]
+ Monotonic = 2,
+ #[doc(alias = "VIRTIO_RTC_CLOCK_UTC_SMEARED")]
+ UtcSmeared = 3,
+ #[doc(alias = "VIRTIO_RTC_CLOCK_UTC_MAYBE_SMEARED")]
+ UtcMaybeSmeared = 4,
+}
+
+/// Send a message and receive reply
+fn send<Request: Zeroable + 'static, Response: Zeroable + Copy + 'static>(
+ req_data: Request,
+ vq: &SpinLock<VirtioRtcVq>,
+ timeout_jiffies: c_ulong,
+) -> Result<Response> {
+ // FIXME: This lock should also disable irqs.
+ let guard = vq.lock();
+
+ let req = VBox::<Request>::new(req_data, GFP_KERNEL)?;
+ let mut resp = VBox::<Response>::new_uninit(GFP_KERNEL)?;
+ let resp_ptr = NonNull::new(resp.as_mut_ptr()).unwrap();
+
+ stack_pin_init!(let token = pin_init!(Token {
+ resp_actual_size: 0,
+ responded <- Completion::new(),
+ }));
+ stack_try_pin_init!(let req_sgs = guard.reqvq().new_readable_sgtable(req, GFP_KERNEL));
+ let req_sgs: Pin<&mut _> = req_sgs?;
+ stack_try_pin_init!(let resp_sgs = guard.reqvq().new_writable_sgtable(resp, GFP_KERNEL));
+ let resp_sgs: Pin<&mut _> = resp_sgs?;
+
+ guard
+ .reqvq()
+ .add_sgs(&req_sgs, &resp_sgs, token.as_ref(), GFP_ATOMIC)?;
+
+ if guard.reqvq().kick_prepare() {
+ guard.reqvq().notify();
+ }
+ drop(guard);
+
+ if timeout_jiffies > 0 {
+ token
+ .responded
+ .wait_for_completion_interruptible_timeout(timeout_jiffies)?;
+ } else {
+ token.responded.wait_for_completion_interruptible()?;
+ }
+
+ if token.resp_actual_size as usize >= core::mem::size_of::<RespHead>() {
+ // SAFETY: all response types contain a `RespHead` header at the start.
+ let head: &RespHead = unsafe { resp_ptr.cast().as_ref() };
+ match head.status {
+ 0 => {
+ // OK, do nothing.
+ }
+ 1 => return Err(ENOTSUPP),
+ 2 => return Err(ENODEV),
+ 3 => return Err(EINVAL),
+ 4 | 5_u8..=u8::MAX => return Err(EIO),
+ }
+ } else if token.resp_actual_size as usize != core::mem::size_of::<Response>() {
+ return Err(EINVAL);
+ }
+ // SAFETY: we have checked that the device wrote the correct amount of bytes for this type and
+ // has returned a successful status code.
+ let resp = unsafe { *resp_ptr.as_ref() };
+
+ Ok(resp)
+}
+
+const VIRTIO_RTC_REQ_READ: u16 = 0x0001;
+const VIRTIO_RTC_REQ_CFG: u16 = 0x1000;
+const VIRTIO_RTC_REQ_CLOCK_CAP: u16 = 0x1001;
+
+struct VirtioRtcVq {
+ inner: Virtqueues,
+}
+
+// SAFETY: `VirtioRtcVq` is safe to be send to any task.
+unsafe impl Send for VirtioRtcVq {}
+
+impl VirtioRtcVq {
+ fn new(inner: Virtqueues) -> impl PinInit<SpinLock<Self>> {
+ new_spinlock!(Self { inner })
+ }
+
+ fn reqvq(&self) -> &Virtqueue {
+ unsafe { self.inner[0].as_ref() }
+ }
+}
+
+#[pin_data(PinnedDrop)]
+struct VirtioRtcDriver {
+ #[pin]
+ virtqueues: SpinLock<VirtioRtcVq>,
+ num_clocks: AtomicU16,
+ #[pin]
+ registered_clocks: Mutex<KVec<()>>,
+}
+
+#[pinned_drop]
+impl PinnedDrop for VirtioRtcDriver {
+ fn drop(self: Pin<&mut Self>) {
+ pr_info!("Remove Rust virtio driver sample.\n");
+ }
+}
+
+extern "C" fn vq_requestq_callback(vq: *mut kernel::bindings::virtqueue) {
+ // SAFETY: The kernel called this virtqueue callback and it must have provided a valid `vq`
+ // pointer
+ let vq = unsafe { Virtqueue::from_raw(vq) };
+ let dev: &virtio::Device<Bound> = vq.dev().expect("Could not get device");
+ let data = dev
+ .as_ref()
+ .drvdata::<VirtioRtcDriver>()
+ .expect("Could not borrow drvdata");
+ data.process_requestq();
+}
+
+impl VirtioRtcDriver {
+ /// Submit `VIRTIO_RTC_REQ_CFG` and return response (`num_clocks`)
+ fn req_cfg(&self) -> Result<u16> {
+ let head = ReqHead {
+ msg_type: VIRTIO_RTC_REQ_CFG.into(),
+ reserved: [0; 6],
+ };
+ let response: RespCfg = send(head, &self.virtqueues, 0)?;
+ pr_info!("Got response! {response:?}\n");
+
+ Ok(response.num_clocks.into())
+ }
+
+ fn process_requestq(&self) {
+ let mut cb_enabled = true;
+ loop {
+ // FIXME: This lock should also disable irqs.
+ let guard = self.virtqueues.lock();
+ if cb_enabled {
+ guard.reqvq().disable_cb();
+ cb_enabled = false;
+ }
+ if let Some((token, len)) = guard.reqvq().get_buf() {
+ drop(guard);
+ pr_info!("process_requestq got buf {len} bytes\n");
+ let mut token = token.cast::<Token>();
+ // SAFETY: pointer points to a valid Token that we have added to the virtqueue.
+ let token_ref = unsafe { token.as_mut() };
+ token_ref.resp_actual_size = len;
+ token_ref.responded.complete_all();
+ pr_info!("process_requestq ok\n");
+ } else {
+ if guard.reqvq().enable_cb() {
+ return;
+ }
+ cb_enabled = true;
+ }
+ }
+ }
+
+ fn clock_cap(&self, clock_id: u16) -> Result<RespClockCap> {
+ let req = ReqClockCap {
+ head: ReqHead {
+ msg_type: VIRTIO_RTC_REQ_CLOCK_CAP.into(),
+ reserved: [0; 6],
+ },
+ clock_id: clock_id.into(),
+ reserved: [0; 6],
+ };
+ let response: RespClockCap = send(req, &self.virtqueues, 0)?;
+ pr_info!("Got response: {response:?}\n");
+ Ok(response)
+ }
+
+ fn read(&self, clock_id: u16) -> Result<u64> {
+ let req = ReqRead {
+ head: ReqHead {
+ msg_type: VIRTIO_RTC_REQ_READ.into(),
+ reserved: [0; 6],
+ },
+ clock_id: clock_id.into(),
+ reserved: [0; 6],
+ };
+ let response: RespRead = send(req, &self.virtqueues, 0)?;
+ pr_info!("Got response: {response:?}\n");
+ Ok(response.clock_reading.into())
+ }
+}
+
+impl virtio::Driver for VirtioRtcDriver {
+ type IdInfo = ();
+
+ /// The table of device ids supported by the driver.
+ const ID_TABLE: virtio::IdTable<Self::IdInfo> = &VIRTIO_RTC_TABLE;
+
+ fn probe(vdev: &virtio::Device<Core>) -> impl PinInit<Self, Error> {
+ let vqs_info: [VirtqueueInfo; 1] = [
+ VirtqueueInfo::new(c"requestq", false, Some(vq_requestq_callback)),
+ //VirtqueueInfo::new(c"alarmq", false, vq_callback),
+ ];
+ try_pin_init!(Self {
+ num_clocks: AtomicU16::new(0),
+ virtqueues <- {
+ pr_info!("Probe Rust virtio driver sample.\n");
+ let vqs = match vdev.find_vqs(&vqs_info) {
+ Ok(vqs) => {
+ pr_info!("Found {} vqs.\n", vqs.len());
+ vqs
+ }
+ Err(err) => {
+ pr_info!("Could not find vqs: {err:?}.\n");
+
+ return Err(err);
+ }
+ };
+
+ VirtioRtcVq::new(vqs)
+ },
+ registered_clocks <- new_mutex!(KVec::with_capacity(0, GFP_KERNEL)?),
+ })
+ }
+
+ fn init(&self, vdev: &virtio::Device<Bound>) -> Result {
+ let num_clocks = self.req_cfg()?;
+ self.num_clocks.store(num_clocks, Ordering::SeqCst);
+ for i in 0..num_clocks {
+ let mut is_exposed = false;
+
+ let resp = self.clock_cap(i)?;
+ let (clock_type, leap_second_smearing, flags) =
+ (resp.clock_type, resp.leap_second_smearing, resp.flags);
+ if cfg!(CONFIG_VIRTIO_RTC_CLASS)
+ && (clock_type == ClockType::Utc as u8
+ || clock_type == ClockType::UtcSmeared as u8
+ || clock_type == ClockType::UtcMaybeSmeared as u8)
+ {
+ // TODO:
+
+ // ret = viortc_init_rtc_class_clock(viortc, vio_clk_id,
+ // clock_type, flags);
+ // if (ret < 0)
+ // return ret;
+ // if (ret > 0)
+ // is_exposed = true;
+ dev_warn!(vdev.as_ref(), "CONFIG_VIRTIO_RTC_CLASS TODO ");
+ }
+
+ if cfg!(CONFIG_VIRTIO_RTC_PTP) {
+ // TODO:
+
+ // ret = viortc_init_ptp_clock(viortc, vio_clk_id, clock_type,
+ // leap_second_smearing);
+ // if (ret < 0)
+ // return ret;
+ // if (ret > 0)
+ // is_exposed = true;
+ // todo!()
+ dev_warn!(vdev.as_ref(), "CONFIG_VIRTIO_RTC_PTP TODO ");
+ }
+
+ if !is_exposed {
+ dev_warn!(
+ vdev.as_ref(),
+ "cannot expose clock {i} (type {clock_type}, variant {leap_second_smearing}, \
+ flags {flags}) to userspace\n"
+ );
+ }
+ let clock_reading = self.read(i)?;
+ pr_info!("#{i} clock reading = {clock_reading}\n");
+ }
+ Ok(())
+ }
+
+ fn remove(_: &virtio::Device<Core>, _this: Pin<&Self>) {
+ pr_info!("Removing Rust virtio driver sample.\n");
+ }
+}
+
+kernel::virtio_device_table!(
+ VIRTIO_RTC_TABLE,
+ MODULE_VIRTIO_RTC_TABLE,
+ <VirtioRtcDriver as virtio::Driver>::IdInfo,
+ [(virtio::DeviceId::new(virtio::VirtioID::Clock), ())]
+);
+
+kernel::module_virtio_driver! {
+ type: VirtioRtcDriver,
+ name: "rust_virtio_rtc",
+ authors: ["Manos Pitsidianakis"],
+ description: "Rust virtio driver",
+ license: "GPL v2",
+}
--
2.47.3
^ permalink raw reply related
* [PATCH RFC v3 4/6] rust: add virtio module
From: Manos Pitsidianakis @ 2026-05-10 13:38 UTC (permalink / raw)
To: Miguel Ojeda
Cc: Manos Pitsidianakis, Peter Hilber, Stefano Garzarella,
Stefan Hajnoczi, Viresh Kumar, Michael S. Tsirkin, Boqun Feng,
Gary Guo, Björn Roy Baron, Benno Lossin, Andreas Hindborg,
Alice Ryhl, Trevor Gross, Danilo Krummrich, Daniel Almeida,
rust-for-linux, Jason Wang, Xuan Zhuo, Eugenio Pérez,
virtualization, linux-kernel, Manos Pitsidianakis
In-Reply-To: <20260510-rust-virtio-v3-0-1427f14d67e1@pitsidianak.is>
Add module that exposes bindings for the virtio API.
Signed-off-by: Manos Pitsidianakis <manos@pitsidianak.is>
---
MAINTAINERS | 2 +
rust/kernel/lib.rs | 2 +
rust/kernel/virtio.rs | 423 ++++++++++++++++++++++++++++++++++++++++
rust/kernel/virtio/utils.rs | 57 ++++++
rust/kernel/virtio/virtqueue.rs | 314 +++++++++++++++++++++++++++++
5 files changed, 798 insertions(+)
diff --git a/MAINTAINERS b/MAINTAINERS
index 48c9c666d90b5a256ab6fae1f42508b789a0ce50..e8012f708df5d4ee858c82aec3269e615fc8caad 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -27935,6 +27935,8 @@ M: Manos Pitsidianakis <manos@pitsidianak.is>
L: virtualization@lists.linux.dev
S: Maintained
F: rust/helpers/virtio.c
+F: rust/kernel/virtio.rs
+F: rust/kernel/virtio/
VIRTIO CRYPTO DRIVER
M: Gonglei <arei.gonglei@huawei.com>
diff --git a/rust/kernel/lib.rs b/rust/kernel/lib.rs
index d93292d47420f1f298a452ade5feefedce5ade86..061394f441dfa27f99939b5c4160e4161a7eaa1e 100644
--- a/rust/kernel/lib.rs
+++ b/rust/kernel/lib.rs
@@ -161,6 +161,8 @@
pub mod uaccess;
#[cfg(CONFIG_USB = "y")]
pub mod usb;
+#[cfg(CONFIG_VIRTIO = "y")]
+pub mod virtio;
pub mod workqueue;
pub mod xarray;
diff --git a/rust/kernel/virtio.rs b/rust/kernel/virtio.rs
new file mode 100644
index 0000000000000000000000000000000000000000..a5a4e2cfec55bc7cbca0d42b198fde6cd2b25f1c
--- /dev/null
+++ b/rust/kernel/virtio.rs
@@ -0,0 +1,423 @@
+// SPDX-License-Identifier: GPL-2.0
+
+//! VIRTIO abstraction.
+//!
+//! To implement a VIRTIO driver:
+//!
+//! - Implement the [`Driver`] trait for your driver type (use [`virtio_device_table`] macro to
+//! declare the `ID_TABLE` associated item)
+//! - Use the [`module_virtio_driver`] macro to declare your module
+
+use crate::{
+ bindings,
+ device_id::RawDeviceId,
+ error::{
+ from_result,
+ to_result,
+ Error,
+ Result, //
+ },
+ ffi::c_uint,
+ prelude::*,
+ types::Opaque, //
+};
+
+use core::{
+ marker::PhantomData,
+ pin::Pin,
+ ptr::NonNull, //
+};
+
+pub mod utils;
+pub mod virtqueue;
+
+/// IdTable type for virtio drivers.
+pub type IdTable<T> = &'static dyn crate::device_id::IdTable<DeviceId, T>;
+
+/// A VIRTIO device id.
+///
+/// [`struct virtio_device_id`]: srctree/include/linux/mod_devicetable.h
+#[repr(transparent)]
+#[derive(Clone, Copy)]
+pub struct DeviceId(bindings::virtio_device_id);
+
+// SAFETY: `DeviceId` is a `#[repr(transparent)]` wrapper of `struct virtio_device_id` and
+// does not add additional invariants, so it's safe to transmute to `RawType`.
+unsafe impl RawDeviceId for DeviceId {
+ type RawType = bindings::virtio_device_id;
+}
+
+impl DeviceId {
+ #[inline]
+ /// Create a new device id
+ pub const fn new(device: VirtioID) -> Self {
+ Self::new_with_vendor(device, VIRTIO_DEV_ANY_ID)
+ }
+
+ #[inline]
+ /// Create a new device id with vendor
+ pub const fn new_with_vendor(device: VirtioID, vendor: u32) -> Self {
+ // Replace with `bindings::virtio_device_id::default()` once stabilized for `const`.
+ // SAFETY: FFI type is valid to be zero-initialized.
+ let mut ret: bindings::virtio_device_id = unsafe { core::mem::zeroed() };
+ ret.device = device as u32;
+ ret.vendor = vendor;
+ Self(ret)
+ }
+}
+
+/// Create a virtio `IdTable` with its alias for modpost.
+#[macro_export]
+macro_rules! virtio_device_table {
+ ($table_name:ident, $module_table_name:ident, $id_info_type: ty, $table_data:expr) => {
+ const $table_name: $crate::device_id::IdArray<
+ $crate::virtio::DeviceId,
+ $id_info_type,
+ { $table_data.len() },
+ > = $crate::device_id::IdArray::new_without_index($table_data);
+
+ $crate::module_device_table!("virtio", $module_table_name, $table_name);
+ };
+}
+
+/// Declares a kernel module that exposes a single virtio driver.
+#[macro_export]
+macro_rules! module_virtio_driver {
+($($f:tt)*) => {
+ $crate::module_driver!(<T>, $crate::virtio::Adapter<T>, { $($f)* });
+};
+}
+
+/// The Virtio driver trait.
+///
+/// Drivers must implement this trait in order to get a virtio driver registered.
+pub trait Driver: Send {
+ /// The type holding information about each device id supported by the driver.
+ // TODO: Use `associated_type_defaults` once stabilized:
+ //
+ // ```
+ // type IdInfo: 'static = ();
+ // ```
+ type IdInfo: 'static;
+
+ /// The table of device ids supported by the driver.
+ const ID_TABLE: IdTable<Self::IdInfo>;
+
+ /// virtio driver probe.
+ ///
+ /// Called when a new virtio device is added or discovered. Implementers should
+ /// attempt to initialize the device here, but should try not sleep since driver data is set
+ /// after this method returns successfully.
+ fn probe(dev: &Device<crate::device::Core>) -> impl PinInit<Self, Error>;
+
+ /// virtio driver init.
+ ///
+ /// Called after a virtio device is probed successfully, can sleep.
+ fn init(&self, dev: &Device<crate::device::Bound>) -> Result;
+
+ /// virtio driver remove.
+ ///
+ /// Called when a [`Device`] is removed from its [`Driver`]. Implementing this callback
+ /// is optional.
+ ///
+ /// This callback serves as a place for drivers to perform teardown operations that require a
+ /// `&Device<Core>` or `&Device<Bound>` reference. For instance, drivers may try to perform I/O
+ /// operations to gracefully tear down the device.
+ ///
+ /// Otherwise, release operations for driver resources should be performed in `Self::drop`.
+ fn remove(dev: &Device<crate::device::Core>, this: Pin<&Self>) {
+ _ = (dev, this);
+ }
+}
+
+/// Abstraction for the virtio device structure (`struct virtio_device`).
+///
+/// [`struct virtio_device`]: srctree/include/linux/virtio.h
+#[repr(transparent)]
+pub struct Device<Ctx: crate::device::DeviceContext = crate::device::Normal>(
+ Opaque<bindings::virtio_device>,
+ PhantomData<Ctx>,
+);
+
+impl<Ctx: crate::device::DeviceContext> Device<Ctx> {
+ #[inline]
+ fn as_raw(&self) -> *mut bindings::virtio_device {
+ self.0.get()
+ }
+}
+
+// SAFETY: `virtio::Device` is a transparent wrapper of `struct virtio_device`.
+// The offset is guaranteed to point to a valid device field inside `virtio::Device`.
+unsafe impl<Ctx: crate::device::DeviceContext> crate::device::AsBusDevice<Ctx> for Device<Ctx> {
+ const OFFSET: usize = core::mem::offset_of!(bindings::virtio_device, dev);
+}
+
+// SAFETY: `Device` is a transparent wrapper of a type that doesn't depend on `Device`'s generic
+// argument.
+kernel::impl_device_context_deref!(unsafe { Device });
+
+impl<Ctx: crate::device::DeviceContext> Device<Ctx> {
+ // TODO: return VirtioID
+ /// Returns the virtio device ID.
+ #[inline]
+ pub fn device_id(&self) -> u32 {
+ // SAFETY: By its type invariant `self.as_raw` is always a valid pointer to a
+ // `struct virtio_device`.
+ unsafe { (*self.as_raw()).id.device }
+ }
+
+ /// Returns the virtio vendor ID.
+ #[inline]
+ pub fn vendor_id(&self) -> u32 {
+ // SAFETY: `self.as_raw` is a valid pointer to a `struct virtio_device`.
+ unsafe { (*self.as_raw()).id.vendor }
+ }
+
+ /// Reset device.
+ #[doc(alias = "virtio_reset_device")]
+ #[inline]
+ pub fn reset(&self) {
+ // SAFETY: By its type invariant `self.as_raw` is always a valid pointer to a
+ // `struct virtio_device`.
+ unsafe { bindings::virtio_reset_device(self.as_raw()) }
+ }
+
+ /// Mark device as ready.
+ #[doc(alias = "virtio_device_ready")]
+ #[inline]
+ pub fn ready(&self) {
+ // SAFETY: By its type invariant `self.as_raw` is always a valid pointer to a
+ // `struct virtio_device`.
+ unsafe { bindings::virtio_device_ready(self.as_raw()) }
+ }
+
+ /// Return virtqueues for this device.
+ #[doc(alias = "virtio_find_vqs")]
+ pub fn find_vqs(&self, info: &[virtqueue::VirtqueueInfo]) -> Result<virtqueue::Virtqueues> {
+ let mut vqs = KVec::with_capacity(info.len(), GFP_KERNEL)?;
+ // SAFETY: By its type invariant `self.as_raw` is always a valid pointer to a
+ // `struct virtio_device`.
+ to_result(unsafe {
+ bindings::virtio_find_vqs(
+ self.as_raw(),
+ info.len().try_into()?,
+ vqs.spare_capacity_mut().as_mut_ptr().cast(),
+ info.as_ptr().cast_mut().cast(),
+ core::ptr::null_mut(),
+ )
+ })?;
+ // SAFETY: virtio_find_vqs returned successfully so `vqs` must be populated.
+ unsafe { vqs.inc_len(info.len()) };
+ let mut inner = KVec::with_capacity(vqs.len(), GFP_KERNEL)?;
+ for vq in vqs {
+ inner.push(NonNull::new(vq).ok_or(EINVAL)?, GFP_KERNEL)?;
+ }
+ Ok(virtqueue::Virtqueues { inner })
+ }
+
+ /// Delete virtqueues from this device.
+ pub(crate) fn del_vqs(&self) {
+ // SAFETY: By its type invariant `self.as_raw` is always a valid pointer to a
+ // `struct virtio_device`.
+ let config = unsafe { (*self.as_raw()).config };
+ // SAFETY: `config` points to a valid virtqueue config struct.
+ if let Some(del_vqs) = unsafe { (*config).del_vqs } {
+ // SAFETY: By its type invariant `self.as_raw` is always a valid pointer to a
+ // `struct virtio_device`.
+ unsafe { del_vqs(self.as_raw()) }
+ }
+ }
+
+ /// Checks if the device has a feature bit.
+ #[inline]
+ pub fn has_feature(&self, fbit: c_uint) -> bool {
+ // SAFETY: By its type invariant `self.as_raw` is always a valid pointer to a
+ // `struct virtio_device`.
+ unsafe { bindings::virtio_has_feature(self.as_raw(), fbit) }
+ }
+}
+
+impl<Ctx: crate::device::DeviceContext> AsRef<crate::device::Device<Ctx>> for Device<Ctx> {
+ #[inline]
+ fn as_ref(&self) -> &crate::device::Device<Ctx> {
+ // SAFETY: By the type invariant of `Self`, `self.as_raw()` is a pointer to a valid
+ // `struct virtio_device`.
+ let dev = unsafe { core::ptr::addr_of_mut!((*self.as_raw()).dev) };
+
+ // SAFETY: `dev` points to a valid `struct device`.
+ unsafe { crate::device::Device::from_raw(dev) }
+ }
+}
+
+/// An adapter for the registration of virtio drivers.
+pub struct Adapter<T: Driver>(T);
+
+// SAFETY:
+// - `bindings::virtio_driver` is a C type declared as `repr(C)`.
+// - `T` is the type of the driver's device private data.
+// - `struct virtio_driver` embeds a `struct device_driver`.
+// - `DEVICE_DRIVER_OFFSET` is the correct byte offset to the embedded `struct device_driver`.
+unsafe impl<T: Driver + 'static> crate::driver::DriverLayout for Adapter<T> {
+ type DriverType = bindings::virtio_driver;
+ type DriverData = T;
+ const DEVICE_DRIVER_OFFSET: usize = core::mem::offset_of!(Self::DriverType, driver);
+}
+
+// SAFETY: A call to `unregister` for a given instance of `DriverType` is guaranteed to be valid if
+// a preceding call to `register` has been successful.
+unsafe impl<T: Driver + 'static> crate::driver::RegistrationOps for Adapter<T> {
+ unsafe fn register(
+ vdrv: &Opaque<Self::DriverType>,
+ name: &'static CStr,
+ module: &'static ThisModule,
+ ) -> Result {
+ // SAFETY: It's safe to set the fields of `struct virtio_driver` on initialization.
+ unsafe {
+ (*vdrv.get()).driver.name = name.as_char_ptr();
+ (*vdrv.get()).id_table = T::ID_TABLE.as_ptr();
+ (*vdrv.get()).probe = Some(Self::probe_callback);
+ (*vdrv.get()).remove = Some(Self::remove_callback);
+ }
+
+ // SAFETY: `vdrv` is guaranteed to be a valid `DriverType`.
+ to_result(unsafe { bindings::__register_virtio_driver(vdrv.get(), module.0) })
+ }
+
+ unsafe fn unregister(vdrv: &Opaque<Self::DriverType>) {
+ // SAFETY: `vdrv` is guaranteed to be a valid `DriverType`.
+ unsafe { bindings::unregister_virtio_driver(vdrv.get()) }
+ }
+}
+
+impl<T: Driver + 'static> Adapter<T> {
+ extern "C" fn probe_callback(vdev: *mut bindings::virtio_device) -> c_int {
+ // SAFETY: The kernel only ever calls the probe callback with a valid pointer to a `struct
+ // virtio_device`.
+ //
+ // INVARIANT: `vdev` is valid for the duration of `probe_callback()`.
+ let dev = unsafe { &*vdev.cast::<Device<crate::device::CoreInternal>>() };
+ from_result(|| {
+ let data = T::probe(dev);
+
+ dev.as_ref().set_drvdata(data)?;
+ // SAFETY: `Device::set_drvdata()` was just called so it's safe to borrow the data.
+ let data = unsafe { dev.as_ref().drvdata_borrow::<T>() };
+ dev.ready();
+ if let Err(err) = T::init(&data, dev) {
+ // SAFETY: `Device::set_drvdata()` was just called so it's safe to re-obtain the
+ // data.
+ let data = unsafe { dev.as_ref().drvdata_obtain::<T>() }.unwrap();
+ T::remove(dev, data.as_ref());
+ drop(data);
+ return Err(err);
+ }
+ Ok(0)
+ })
+ }
+
+ extern "C" fn remove_callback(vdev: *mut bindings::virtio_device) {
+ // SAFETY: The kernel only ever calls the remove callback with a valid pointer to a `struct
+ // virtio_device`.
+ //
+ // INVARIANT: `vdev` is valid for the duration of `remove_callback()`.
+ let dev = unsafe { &*vdev.cast::<Device<crate::device::CoreInternal>>() };
+
+ // SAFETY: `remove_callback` is only ever called after a successful call to
+ // `probe_callback`, hence it's guaranteed that `Device::set_drvdata()` has been called
+ // and stored a `Pin<KBox<T>>`.
+ let data = unsafe { dev.as_ref().drvdata_borrow::<T>() };
+
+ T::remove(dev, data);
+ dev.reset();
+ }
+}
+
+/// Any vendor
+pub const VIRTIO_DEV_ANY_ID: u32 = 0xffffffff;
+
+/// Virtio IDs
+///
+/// C header: [`include/uapi/linux/virtio_ids.h`](srctree/include/uapi/linux/virtio_ids.h)
+#[repr(u32)]
+pub enum VirtioID {
+ /// virtio net
+ Net = bindings::VIRTIO_ID_NET,
+ /// virtio block
+ Block = bindings::VIRTIO_ID_BLOCK,
+ /// virtio console
+ Console = bindings::VIRTIO_ID_CONSOLE,
+ /// virtio rng
+ Rng = bindings::VIRTIO_ID_RNG,
+ /// virtio balloon
+ Balloon = bindings::VIRTIO_ID_BALLOON,
+ /// virtio ioMemory
+ IOMem = bindings::VIRTIO_ID_IOMEM,
+ /// virtio remote processor messaging
+ RPMSG = bindings::VIRTIO_ID_RPMSG,
+ /// virtio scsi
+ Scsi = bindings::VIRTIO_ID_SCSI,
+ /// 9p virtio console
+ NineP = bindings::VIRTIO_ID_9P,
+ /// virtio WLAN MAC
+ Mac80211Wlan = bindings::VIRTIO_ID_MAC80211_WLAN,
+ /// virtio remoteproc serial link
+ RPROCSerial = bindings::VIRTIO_ID_RPROC_SERIAL,
+ /// Virtio caif
+ CAIF = bindings::VIRTIO_ID_CAIF,
+ /// virtio memory balloon
+ MemoryBalloon = bindings::VIRTIO_ID_MEMORY_BALLOON,
+ /// virtio GPU
+ GPU = bindings::VIRTIO_ID_GPU,
+ /// virtio clock/timer
+ Clock = bindings::VIRTIO_ID_CLOCK,
+ /// virtio input
+ Input = bindings::VIRTIO_ID_INPUT,
+ /// virtio vsock transport
+ VSock = bindings::VIRTIO_ID_VSOCK,
+ /// virtio crypto
+ Crypto = bindings::VIRTIO_ID_CRYPTO,
+ /// virtio signal distribution device
+ SignalDist = bindings::VIRTIO_ID_SIGNAL_DIST,
+ /// virtio pstore device
+ Pstore = bindings::VIRTIO_ID_PSTORE,
+ /// virtio IOMMU
+ Iommu = bindings::VIRTIO_ID_IOMMU,
+ /// virtio mem
+ Mem = bindings::VIRTIO_ID_MEM,
+ /// virtio sound
+ Sound = bindings::VIRTIO_ID_SOUND,
+ /// virtio filesystem
+ FS = bindings::VIRTIO_ID_FS,
+ /// virtio pmem
+ PMem = bindings::VIRTIO_ID_PMEM,
+ /// virtio rpmb
+ RPMB = bindings::VIRTIO_ID_RPMB,
+ /// virtio mac80211-hwsim
+ Mac80211Hwsim = bindings::VIRTIO_ID_MAC80211_HWSIM,
+ /// virtio video encoder
+ VideoEncoder = bindings::VIRTIO_ID_VIDEO_ENCODER,
+ /// virtio video decoder
+ VideoDecoder = bindings::VIRTIO_ID_VIDEO_DECODER,
+ /// virtio SCMI
+ SCMI = bindings::VIRTIO_ID_SCMI,
+ /// virtio nitro secure module
+ NitroSecMod = bindings::VIRTIO_ID_NITRO_SEC_MOD,
+ /// virtio i2c adapter
+ I2CAdapter = bindings::VIRTIO_ID_I2C_ADAPTER,
+ /// virtio watchdog
+ Watchdog = bindings::VIRTIO_ID_WATCHDOG,
+ /// virtio can
+ CAN = bindings::VIRTIO_ID_CAN,
+ /// virtio dmabuf
+ DMABuf = bindings::VIRTIO_ID_DMABUF,
+ /// virtio parameter server
+ ParamServ = bindings::VIRTIO_ID_PARAM_SERV,
+ /// virtio audio policy
+ AudioPolicy = bindings::VIRTIO_ID_AUDIO_POLICY,
+ /// virtio bluetooth
+ BT = bindings::VIRTIO_ID_BT,
+ /// virtio gpio
+ GPIO = bindings::VIRTIO_ID_GPIO,
+ /// virtio spi
+ SPI = bindings::VIRTIO_ID_SPI,
+}
diff --git a/rust/kernel/virtio/utils.rs b/rust/kernel/virtio/utils.rs
new file mode 100644
index 0000000000000000000000000000000000000000..8dca373f10a6906b891a9420c13cd8e9e929c412
--- /dev/null
+++ b/rust/kernel/virtio/utils.rs
@@ -0,0 +1,57 @@
+// SPDX-License-Identifier: GPL-2.0
+
+//! Helper types and utilities
+
+macro_rules! endian_type {
+ ($old_type:ident, $new_type:ident, $to_new:ident, $from_new:ident) => {
+ /// An unsigned integer type of with an explicit endianness.
+ #[derive(Copy, Clone, Eq, PartialEq, Debug, Default, pin_init::Zeroable)]
+ #[repr(transparent)]
+ pub struct $new_type($old_type);
+
+ $crate::static_assert!(
+ ::core::mem::align_of::<$new_type>() == ::core::mem::align_of::<$old_type>()
+ );
+ $crate::static_assert!(
+ ::core::mem::size_of::<$new_type>() == ::core::mem::size_of::<$old_type>()
+ );
+
+ impl $new_type {
+ /// Convert to CPU/native endianness.
+ pub const fn to_cpu(self) -> $old_type {
+ $old_type::$from_new(self.0)
+ }
+ }
+
+ impl PartialEq<$old_type> for $new_type {
+ fn eq(&self, other: &$old_type) -> bool {
+ self.0 == $old_type::$to_new(*other)
+ }
+ }
+
+ impl PartialEq<$new_type> for $old_type {
+ fn eq(&self, other: &$new_type) -> bool {
+ $old_type::$to_new(other.0) == *self
+ }
+ }
+
+ impl From<$new_type> for $old_type {
+ fn from(v: $new_type) -> $old_type {
+ v.to_cpu()
+ }
+ }
+
+ impl From<$old_type> for $new_type {
+ fn from(v: $old_type) -> $new_type {
+ $new_type($old_type::$to_new(v))
+ }
+ }
+ };
+}
+
+endian_type!(u16, Le16, to_le, from_le);
+endian_type!(u32, Le32, to_le, from_le);
+endian_type!(u64, Le64, to_le, from_le);
+endian_type!(u16, Be16, to_be, from_be);
+endian_type!(u32, Be32, to_be, from_be);
+endian_type!(u64, Be64, to_be, from_be);
diff --git a/rust/kernel/virtio/virtqueue.rs b/rust/kernel/virtio/virtqueue.rs
new file mode 100644
index 0000000000000000000000000000000000000000..781326c1723eb67a8c62524795ba431141fea202
--- /dev/null
+++ b/rust/kernel/virtio/virtqueue.rs
@@ -0,0 +1,314 @@
+// SPDX-License-Identifier: GPL-2.0
+
+//! Virtqueue functionality.
+//!
+//! # Discovering virtqueues
+//!
+//! Inside your driver's [`kernel::virtio::Driver::probe`] method, call
+//! [`kernel::virtio::Device::find_vqs`] method with your [`VirtqueueInfo`] struct.
+//!
+//! # Passing data to virtqueues
+//!
+//! Create your data as owned [`SGTable`] with:
+//!
+//! - [`Virtqueue::new_readable_sgtable`] for data that can be read from the device, and
+//! - [`Virtqueue::new_writable_sgtable`] for data that can be written from the device
+//!
+//! These methods will make sure to create the scatter-gather tables and DMA map them to the
+//! appropriate VIRTIO transport.
+//!
+//! To add the tables to the virtqueue, call [`Virtqueue::add_sgs`].
+
+use crate::{
+ alloc::{
+ allocator::VmallocPageIter,
+ Flags, //
+ },
+ bindings,
+ device::Bound,
+ dma::DataDirection,
+ error::{
+ code::{
+ EINVAL,
+ ENOENT, //
+ },
+ to_result,
+ Error,
+ Result, //
+ },
+ page::AsPageIter,
+ prelude::*,
+ scatterlist::{
+ Owned,
+ SGTable, //
+ },
+ str::{
+ self,
+ CStr, //
+ },
+ types::Opaque,
+ virtio::Device, //
+};
+
+use core::{
+ ptr::NonNull, //
+};
+
+/// Info for a virtqueue.
+///
+/// [`struct virtqueue_info`]: srctree/include/linux/virtio_config.h
+#[doc(alias = "virtqueue_info")]
+#[repr(transparent)]
+pub struct VirtqueueInfo(Opaque<bindings::virtqueue_info>);
+
+impl VirtqueueInfo {
+ #[inline]
+ /// Create a new [`VirtqueueInfo`]
+ pub const fn new(
+ name: &'static CStr,
+ ctx: bool,
+ callback: Option<unsafe extern "C" fn(*mut bindings::virtqueue)>,
+ ) -> Self {
+ Self(Opaque::new(bindings::virtqueue_info {
+ name: str::as_char_ptr_in_const_context(name),
+ ctx,
+ callback,
+ }))
+ }
+}
+
+/// A container for discovered virtqueues returned by [`Device::find_vqs`] method.
+///
+/// This type dereferences to a `NonNull<Virtqueue>` slice.
+///
+/// It deletes the virtqueues when dropped.
+pub struct Virtqueues {
+ pub(crate) inner: KVec<NonNull<Virtqueue>>,
+}
+
+impl Drop for Virtqueues {
+ fn drop(&mut self) {
+ let inner = core::mem::take(&mut self.inner);
+ let Some(first) = inner.into_iter().next() else {
+ return;
+ };
+ let first_ref = unsafe { first.as_ref() };
+ let Ok(vdev) = first_ref.dev() else {
+ return;
+ };
+ vdev.del_vqs();
+ }
+}
+
+impl core::ops::Deref for Virtqueues {
+ type Target = [NonNull<Virtqueue>];
+
+ #[inline]
+ fn deref(&self) -> &Self::Target {
+ &self.inner
+ }
+}
+
+/// An opaque handler for a virtqueue.
+///
+/// [`struct virtqueue`]: srctree/include/linux/virtio.h
+#[repr(transparent)]
+pub struct Virtqueue(Opaque<bindings::virtqueue>);
+
+impl Virtqueue {
+ /// Create a [`Virtqueue`] from a raw pointer.
+ ///
+ /// # Safety
+ ///
+ /// Callers must ensure that `ptr` is a properly initialized valid `virtqueue` pointer.
+ #[inline]
+ pub unsafe fn from_raw<'a>(ptr: *mut bindings::virtqueue) -> &'a Self {
+ // SAFETY: The safety requirements of this function guarantee that `ptr` is a valid
+ // pointer to a `struct virtqueue` for the duration of `'a`.
+ unsafe { &*ptr.cast() }
+ }
+
+ /// Obtain the raw `struct virtqueue *`.
+ #[inline]
+ pub(crate) fn as_raw(&self) -> *mut bindings::virtqueue {
+ self.0.get()
+ }
+
+ /// Get the [`Device`] associated with this virtqueue.
+ #[inline]
+ pub fn dev(&self) -> Result<&Device<Bound>> {
+ // SAFETY: By the type invariants, `self.as_raw()` is a valid pointer to a `struct
+ // virtqueue`.
+ if unsafe { (*self.as_raw()).vdev }.is_null() {
+ return Err(ENOENT);
+ }
+ // SAFETY: the pointer has been promised to be valid when self was created
+ Ok(unsafe { &*(&*self.as_raw()).vdev.cast::<Device<Bound>>() })
+ }
+
+ /// Get the vring size.
+ #[inline]
+ #[doc(alias = "virtqueue_get_vring_size")]
+ pub fn vring_size(&self) -> u32 {
+ // SAFETY: the pointer has been promised to be valid when self was created
+ unsafe { bindings::virtqueue_get_vring_size(self.as_raw()) }
+ }
+
+ /// Notify virtqueue.
+ #[inline]
+ #[doc(alias = "virtqueue_notify")]
+ pub fn notify(&self) -> bool {
+ // SAFETY: the pointer has been promised to be valid when self was created
+ unsafe { bindings::virtqueue_notify(self.as_raw()) }
+ }
+
+ /// Kick and prepare virtqueue.
+ #[inline]
+ #[doc(alias = "virtqueue_kick_prepare")]
+ pub fn kick_prepare(&self) -> bool {
+ // SAFETY: the pointer has been promised to be valid when self was created
+ unsafe { bindings::virtqueue_kick_prepare(self.as_raw()) }
+ }
+
+ /// Kick virtqueue.
+ #[inline]
+ #[doc(alias = "virtqueue_kick")]
+ pub fn kick(&self) -> bool {
+ // SAFETY: the pointer has been promised to be valid when self was created
+ unsafe { bindings::virtqueue_kick(self.as_raw()) }
+ }
+
+ /// Enable virtqueue's callback.
+ #[inline]
+ #[doc(alias = "virtqueue_enable_cb")]
+ pub fn enable_cb(&self) -> bool {
+ // SAFETY: the pointer has been promised to be valid when self was created
+ unsafe { bindings::virtqueue_enable_cb(self.as_raw()) }
+ }
+
+ /// Disable virtqueue's callback.
+ #[inline]
+ #[doc(alias = "virtqueue_disable_cb")]
+ pub fn disable_cb(&self) {
+ // SAFETY: the pointer has been promised to be valid when self was created
+ unsafe { bindings::virtqueue_disable_cb(self.as_raw()) }
+ }
+
+ /// Get a buffer from the virtqueue, if available.
+ ///
+ /// This method returns a pointer to the `token` value passed in [`Virtqueue::add_sgs`] method
+ /// and the amount of bytes that were written by the device.
+ #[inline]
+ #[doc(alias = "virtqueue_get_buf")]
+ pub fn get_buf(&'_ self) -> Option<(NonNull<u8>, u32)> {
+ let mut len = 0;
+ // SAFETY: the pointer has been promised to be valid when self was created
+ let ptr = unsafe { bindings::virtqueue_get_buf(self.as_raw(), &mut len) };
+ Some((NonNull::new(ptr.cast())?, len))
+ }
+
+ /// Add a list of scatter-gather lists to virtqueue.
+ #[inline]
+ #[doc(alias = "virtqueue_add_sgs")]
+ pub fn add_sgs<'token, PIn, POut, Token>(
+ &'_ self,
+ out_sgs: &'token SGTableReadable<POut>,
+ in_sgs: &'token SGTableWritable<PIn>,
+ token: Pin<&'token Token>,
+ gfp: Flags,
+ ) -> Result
+ where
+ for<'a> PIn: AsPageIter<Iter<'a> = VmallocPageIter<'a>> + 'static,
+ for<'a> POut: AsPageIter<Iter<'a> = VmallocPageIter<'a>> + 'static,
+ {
+ let out_sgs_num = u32::try_from(out_sgs.inner.iter().count())?;
+ let in_sgs_num = u32::try_from(in_sgs.inner.iter().count())?;
+
+ let Some(total_size) = out_sgs_num.checked_add(in_sgs_num) else {
+ return Err(EINVAL);
+ };
+
+ let mut sgs = KVec::with_capacity(2, GFP_KERNEL)?;
+
+ for entry in out_sgs.inner.iter() {
+ sgs.push(entry, GFP_KERNEL)?;
+ }
+ for entry in in_sgs.inner.iter() {
+ sgs.push(entry, GFP_KERNEL)?;
+ }
+
+ if usize::try_from(total_size) != Ok(sgs.len()) {
+ return Err(EINVAL);
+ }
+ // SAFETY: `self` has been promised to be valid when self was created
+ to_result(unsafe {
+ bindings::virtqueue_add_sgs(
+ self.as_raw(),
+ sgs.as_ptr().cast_mut().cast(),
+ out_sgs_num,
+ in_sgs_num,
+ NonNull::new(core::ptr::from_ref::<Token>(&*token.as_ref()).cast_mut())
+ .unwrap()
+ .as_ptr()
+ .cast(),
+ gfp.as_raw(),
+ )
+ })
+ }
+
+ /// Create a scatter-gather table readable by the device.
+ pub fn new_readable_sgtable<P>(
+ &self,
+ pages: P,
+ flags: Flags,
+ ) -> impl PinInit<SGTableReadable<P>, Error> + '_
+ where
+ for<'a> P: AsPageIter<Iter<'a> = VmallocPageIter<'a>> + 'static,
+ {
+ pin_init!(SGTableReadable {
+ inner <- SGTable::new(
+ self.dev().unwrap().as_ref().parent().unwrap(),
+ pages,
+ DataDirection::ToDevice,
+ flags,
+ ),
+ }? Error)
+ }
+
+ /// Create a scatter-gather table writable by the device.
+ pub fn new_writable_sgtable<P>(
+ &self,
+ pages: P,
+ flags: Flags,
+ ) -> impl PinInit<SGTableWritable<P>, Error> + '_
+ where
+ for<'a> P: AsPageIter<Iter<'a> = VmallocPageIter<'a>> + 'static,
+ {
+ pin_init!(SGTableWritable {
+ inner <- SGTable::new(
+ self.dev().unwrap().as_ref().parent().unwrap(),
+ pages,
+ DataDirection::FromDevice,
+ flags,
+ ),
+ }? Error)
+ }
+}
+
+/// An [`SGTable<Owned<P>>`] that is guaranteed to have been DMA-mapped as device-readable.
+///
+/// Created by [`Virtqueue::new_readable_sgtable`].
+#[pin_data]
+pub struct SGTableReadable<P> {
+ #[pin]
+ inner: SGTable<Owned<P>>,
+}
+
+/// An [`SGTable<Owned<P>>`] that is guaranteed to have been DMA-mapped as device-writable.
+///
+/// Created by [`Virtqueue::new_writable_sgtable`].
+#[pin_data]
+pub struct SGTableWritable<P> {
+ #[pin]
+ inner: SGTable<Owned<P>>,
+}
--
2.47.3
^ permalink raw reply related
* [PATCH RFC v3 5/6] rust: impl interruptible waits for Completion
From: Manos Pitsidianakis @ 2026-05-10 13:38 UTC (permalink / raw)
To: Miguel Ojeda
Cc: Manos Pitsidianakis, Peter Hilber, Stefano Garzarella,
Stefan Hajnoczi, Viresh Kumar, Michael S. Tsirkin, Boqun Feng,
Gary Guo, Björn Roy Baron, Benno Lossin, Andreas Hindborg,
Alice Ryhl, Trevor Gross, Danilo Krummrich, Daniel Almeida,
rust-for-linux, Jason Wang, Xuan Zhuo, Eugenio Pérez,
virtualization, linux-kernel, Manos Pitsidianakis
In-Reply-To: <20260510-rust-virtio-v3-0-1427f14d67e1@pitsidianak.is>
Allow Completion to wait interruptibly.
Signed-off-by: Manos Pitsidianakis <manos@pitsidianak.is>
---
rust/kernel/sync/completion.rs | 42 +++++++++++++++++++++++++++++++++++++++++-
1 file changed, 41 insertions(+), 1 deletion(-)
diff --git a/rust/kernel/sync/completion.rs b/rust/kernel/sync/completion.rs
index c50012a940a3c7a3e0edf302c8f833bdc4415200..17c9e48a5359c0c885be9ebf6843e74d5abe56e5 100644
--- a/rust/kernel/sync/completion.rs
+++ b/rust/kernel/sync/completion.rs
@@ -6,7 +6,12 @@
//!
//! C header: [`include/linux/completion.h`](srctree/include/linux/completion.h)
-use crate::{bindings, prelude::*, types::Opaque};
+use crate::{
+ bindings,
+ prelude::*,
+ time::Jiffies,
+ types::Opaque, //
+};
/// Synchronization primitive to signal when a certain task has been completed.
///
@@ -109,4 +114,39 @@ pub fn wait_for_completion(&self) {
// SAFETY: `self.as_raw()` is a pointer to a valid `struct completion`.
unsafe { bindings::wait_for_completion(self.as_raw()) };
}
+
+ /// Wait for completion of an interruptible task without a timeout.
+ ///
+ /// See also [`Completion::complete_all`].
+ #[inline]
+ pub fn wait_for_completion_interruptible(&self) -> Result {
+ // SAFETY: `self.as_raw()` is a pointer to a valid `struct completion`.
+ let err = unsafe { bindings::wait_for_completion_interruptible(self.as_raw()) };
+ if err < 0 {
+ Err(Error::from_errno(err))
+ } else {
+ Ok(())
+ }
+ }
+
+ /// Wait for completion of an interruptible task with a timeout.
+ ///
+ /// See also [`Completion::complete_all`].
+ #[inline]
+ pub fn wait_for_completion_interruptible_timeout(
+ &self,
+ timeout_jiffies: Jiffies,
+ ) -> Result<Jiffies> {
+ // SAFETY: `self.as_raw()` is a pointer to a valid `struct completion`.
+ let ret: c_long = unsafe {
+ bindings::wait_for_completion_interruptible_timeout(self.as_raw(), timeout_jiffies)
+ };
+ if ret == 0 {
+ return Err(ETIMEDOUT);
+ }
+ match Jiffies::try_from(ret) {
+ Ok(ret) => Ok(ret),
+ Err(_) => Err(Error::from_errno(ret as c_int)),
+ }
+ }
}
--
2.47.3
^ permalink raw reply related
* [PATCH RFC v3 3/6] rust/kernel/device: return parent at same context
From: Manos Pitsidianakis @ 2026-05-10 13:38 UTC (permalink / raw)
To: Miguel Ojeda
Cc: Manos Pitsidianakis, Peter Hilber, Stefano Garzarella,
Stefan Hajnoczi, Viresh Kumar, Michael S. Tsirkin, Boqun Feng,
Gary Guo, Björn Roy Baron, Benno Lossin, Andreas Hindborg,
Alice Ryhl, Trevor Gross, Danilo Krummrich, Daniel Almeida,
rust-for-linux, Jason Wang, Xuan Zhuo, Eugenio Pérez,
virtualization, linux-kernel, Manos Pitsidianakis
In-Reply-To: <20260510-rust-virtio-v3-0-1427f14d67e1@pitsidianak.is>
The return value of `Device::parent` method was defaulting to `Normal`
context, instead of having the same context as `self`.
Signed-off-by: Manos Pitsidianakis <manos@pitsidianak.is>
---
rust/kernel/device.rs | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/rust/kernel/device.rs b/rust/kernel/device.rs
index 94e0548e76871d8b7de309c1f1c7b77bb49738ed..07d2f1225cd87f4fc091a5d6cb626726ae175894 100644
--- a/rust/kernel/device.rs
+++ b/rust/kernel/device.rs
@@ -341,7 +341,7 @@ pub(crate) fn as_raw(&self) -> *mut bindings::device {
/// Returns a reference to the parent device, if any.
#[cfg_attr(not(CONFIG_AUXILIARY_BUS), expect(dead_code))]
- pub(crate) fn parent(&self) -> Option<&Device> {
+ pub(crate) fn parent(&self) -> Option<&Device<Ctx>> {
// SAFETY:
// - By the type invariant `self.as_raw()` is always valid.
// - The parent device is only ever set at device creation.
--
2.47.3
^ permalink raw reply related
* [PATCH RFC v3 0/6] Add Rust virtio bindings and sample device
From: Manos Pitsidianakis @ 2026-05-10 13:38 UTC (permalink / raw)
To: Miguel Ojeda
Cc: Manos Pitsidianakis, Peter Hilber, Stefano Garzarella,
Stefan Hajnoczi, Viresh Kumar, Michael S. Tsirkin, Boqun Feng,
Gary Guo, Björn Roy Baron, Benno Lossin, Andreas Hindborg,
Alice Ryhl, Trevor Gross, Danilo Krummrich, Daniel Almeida,
rust-for-linux, Jason Wang, Xuan Zhuo, Eugenio Pérez,
virtualization, linux-kernel, Manos Pitsidianakis
Hi all, this RFC series adds Rust bindings for Virtio drivers
(frontends in virtio parlance).
As a PoC, it also adds a sample virtio-rtc driver which performs
capability discovery through the virtqueue without registering any clock.
Before I send a cleaned-up non-RFC I would like some initial feedback
(i.e. is it something the upstream wants?)
This was tested with the rust-vmm vhost-device-rtc device backend that I
wrote[^0]:
[^0]: https://github.com/rust-vmm/vhost-device/tree/main/vhost-device-rtc
Instructions:
Run the daemon in a separate terminal:
$ cargo run --bin vhost-device-rtc -- -s /tmp/rtc.sock
Then run the VM:
$ qemu-system-aarch64 \
-machine type=virt,virtualization=off,acpi=on \
-cpu host \
-smp 8 \
-accel kvm \
-drive if=virtio,format=qcow2,file=./debian-13-nocloud-arm64-daily.qcow2 \
-device virtio-net-pci,netdev=unet \
-device virtio-scsi-pci \
-serial mon:stdio \
-m 8192 \
-object memory-backend-memfd,id=mem,size=8G,share=on \
-numa node,memdev=mem \
-display none \
-vga none \
-kernel /path/to/linux/build/arch/arm64/boot/Image \
-device vhost-user-test-device,chardev=rtc,id=rtc,virtio-id=17,num_vqs=2,vq_size=1024 \
-chardev socket,path=/tmp/rtc.sock,id=rtc \
...
Example output:
[ 1.105238] rust_virtio_rtc: Probe Rust virtio driver sample.
[ 1.105645] rust_virtio_rtc: Found 1 vqs.
[ 1.136050] rust_virtio_rtc: process_requestq got buf 16 bytes
[ 1.136125] rust_virtio_rtc: Got response! Ok(RespCfg { head: ReqHead { msg_type: Le16(0), reserved: [0, 0, 0, 0, 0, 0] }, num_clocks: Le16(3), reserved: [0, 0, 0, 0, 0, 0] })
[ 1.136701] rust_virtio_rtc: Got response! Ok(RespClockCap { head: ReqHead { msg_type: Le16(0), reserved: [0, 0, 0, 0, 0, 0] }, clock_type: 3, leap_second_smearing: 0, flags: 0, reserved: [0, 0, 0, 0, 0] })
[ 1.136724] rust_virtio_rtc virtio0: cannot expose clock 0 (type 3, variant 0, flags 0) to userspace
[ 1.137259] rust_virtio_rtc: Got response! Ok(RespRead { head: ReqHead { msg_type: Le16(0), reserved: [0, 0, 0, 0, 0, 0] }, clock_reading: Le64(1777890485031060388) })
[ 1.137277] rust_virtio_rtc: #0 clock reading = 1777890485031060388
[ 1.137749] rust_virtio_rtc: Got response! Ok(RespClockCap { head: ReqHead { msg_type: Le16(0), reserved: [0, 0, 0, 0, 0, 0] }, clock_type: 1, leap_second_smearing: 0, flags: 0, reserved: [0, 0, 0, 0, 0] })
[ 1.137769] rust_virtio_rtc virtio0: cannot expose clock 1 (type 1, variant 0, flags 0) to userspace
[ 1.138247] rust_virtio_rtc: Got response! Ok(RespRead { head: ReqHead { msg_type: Le16(0), reserved: [0, 0, 0, 0, 0, 0] }, clock_reading: Le64(1777890485032086075) })
[ 1.138264] rust_virtio_rtc: #1 clock reading = 1777890485032086075
[ 1.138730] rust_virtio_rtc: Got response! Ok(RespClockCap { head: ReqHead { msg_type: Le16(0), reserved: [0, 0, 0, 0, 0, 0] }, clock_type: 2, leap_second_smearing: 0, flags: 0, reserved: [0, 0, 0, 0, 0] })
[ 1.138751] rust_virtio_rtc virtio0: cannot expose clock 2 (type 2, variant 0, flags 0) to userspace
[ 1.139253] rust_virtio_rtc: Got response! Ok(RespRead { head: ReqHead { msg_type: Le16(0), reserved: [0, 0, 0, 0, 0, 0] }, clock_reading: Le64(338567896865557) })
[ 1.139270] rust_virtio_rtc: #2 clock reading = 338567896865557
Concerns - Notes - TODOs
========================
- Virtqueue lifetimes don't neatly apply to Rust as expected, so a lot
of times we have to go through unsafe pointer dereferences (though
which are guaranteed by Virtio subsystem to be valid, for example when
a callback is called with the vq argument). There's a potential for
misuse and definitely could use better thinking.
- `struct virtio_device` is not reference-counted like other implemented
device types in rust/kernel. Maybe we need to change C API first to
make them reference counted, assuming this doesn't break anything?
- The sample driver obviously conflicts with the C implementation, so
this would either need to move out of samples/ or figure out some way
to handle this in kbuild.
- kernel::virtio module and its types need a few rustdoc examples that I
will add in followup series
- Note that the registration of RTC clocks etc in the sample driver is
not done, I'm putting it off until I receive some feedback first. The
sample driver otherwise does send and receive data from the virtqueue
as a PoC.
PS: No LLMs used so any mistakes and goofs are solely written by me.
Signed-off-by: Manos Pitsidianakis <manos@pitsidianak.is>
---
Changes in v3:
- Removed unused methods from virtio API
- Clean up how scattergather lists are added to virtqueues by using
owned SGTables only, and make the API safe(r)
- Add RAII cleanup for find_vqs return value that calls del_vqs
- Reset device after remove callback
- Significantly clean up sample driver as a result of the other cleanups
- Link to v2: https://lore.kernel.org/r/20260509-rust-virtio-v2-0-c1e30ec2bd21@pitsidianak.is
Changes in v2:
- Move helper ifdefs to helper file (thanks Alice)
- Changed CONFIG checks to IS_ENABLED to allow for CONFIG_VIRTIO=m
- Split all use imports to one item per line according to style guide
- Fixed wait_for_completion_interruptible*() rustdocs
- Use Jiffy type alias in wait_for_completion_interruptible_timeout()
- Pepper and salt #[inline]s wherever appropriate as per style guide
- Split probe() into probe() and init() to allow cleaning up if init
fails
- Remove unnecessary Send and Sync unsafe impls for
kernel::virtio::Device
- Remove unnecessary LeSize and BeSize
- Accept Option<_> for virtqueue callback when creating a VirtqueueInfo
- Made all vq buffer adding operations unsafe
- Use AtomicU16 instead of Cell<u16> for sample virtio driver
- Fix RespHead field types in sample virtio driver
- Fix response error checking in sample virtio driver
- Change some device contexts in method signatures
- Link to v1: https://lore.kernel.org/r/20260505-rust-virtio-v1-0-9563383909e4@pitsidianak.is
---
Manos Pitsidianakis (6):
rust/bindings: generate virtio bindings
rust/helpers: add virtio.c
rust/kernel/device: return parent at same context
rust: add virtio module
rust: impl interruptible waits for Completion
samples/rust: Add sample virtio-rtc driver [WIP]
MAINTAINERS | 9 +
rust/bindings/bindings_helper.h | 5 +
rust/helpers/helpers.c | 1 +
rust/helpers/virtio.c | 37 ++++
rust/kernel/device.rs | 2 +-
rust/kernel/lib.rs | 2 +
rust/kernel/sync/completion.rs | 42 +++-
rust/kernel/virtio.rs | 423 ++++++++++++++++++++++++++++++++++++++++
rust/kernel/virtio/utils.rs | 57 ++++++
rust/kernel/virtio/virtqueue.rs | 314 +++++++++++++++++++++++++++++
samples/rust/Kconfig | 15 ++
samples/rust/Makefile | 1 +
samples/rust/rust_virtio_rtc.rs | 403 ++++++++++++++++++++++++++++++++++++++
13 files changed, 1309 insertions(+), 2 deletions(-)
---
base-commit: 028ef9c96e96197026887c0f092424679298aae8
change-id: 20260504-rust-virtio-8523b01dfdc2
Best regards,
--
Manos Pitsidianakis <manos@pitsidianak.is>
^ permalink raw reply
* [PATCH RFC v3 1/6] rust/bindings: generate virtio bindings
From: Manos Pitsidianakis @ 2026-05-10 13:38 UTC (permalink / raw)
To: Miguel Ojeda
Cc: Manos Pitsidianakis, Peter Hilber, Stefano Garzarella,
Stefan Hajnoczi, Viresh Kumar, Michael S. Tsirkin, Boqun Feng,
Gary Guo, Björn Roy Baron, Benno Lossin, Andreas Hindborg,
Alice Ryhl, Trevor Gross, Danilo Krummrich, Daniel Almeida,
rust-for-linux, Jason Wang, Xuan Zhuo, Eugenio Pérez,
virtualization, linux-kernel, Manos Pitsidianakis
In-Reply-To: <20260510-rust-virtio-v3-0-1427f14d67e1@pitsidianak.is>
Add virtio headers if CONFIG_VIRTIO is enabled.
Signed-off-by: Manos Pitsidianakis <manos@pitsidianak.is>
---
rust/bindings/bindings_helper.h | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/rust/bindings/bindings_helper.h b/rust/bindings/bindings_helper.h
index 083cc44aa952c2b29ab82d5d481063a1cf48bccf..1cbe1a4b5647fd646c1be3c5c80fb24ae7b97a4a 100644
--- a/rust/bindings/bindings_helper.h
+++ b/rust/bindings/bindings_helper.h
@@ -151,3 +151,8 @@ const vm_flags_t RUST_CONST_HELPER_VM_NOHUGEPAGE = VM_NOHUGEPAGE;
#include "../../drivers/android/binder/rust_binder_events.h"
#include "../../drivers/android/binder/page_range_helper.h"
#endif
+
+#if IS_ENABLED(CONFIG_VIRTIO)
+#include <linux/virtio_config.h>
+#include <uapi/linux/virtio_ids.h>
+#endif /* IS_ENABLED(CONFIG_VIRTIO) */
--
2.47.3
^ permalink raw reply related
* Re: [PATCH net-next v11 1/4] tun/tap: add ptr_ring consume helper with netdev queue wakeup
From: Simon Schippers @ 2026-05-10 8:55 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: willemdebruijn.kernel, jasowang, andrew+netdev, davem, edumazet,
kuba, pabeni, eperezma, leiyang, stephen, jon, tim.gebauer,
netdev, linux-kernel, kvm, virtualization
In-Reply-To: <a14b700a-94e8-44c0-bde0-d8307761c0f1@tu-dortmund.de>
On 5/10/26 09:03, Simon Schippers wrote:
> On 5/10/26 00:44, Michael S. Tsirkin wrote:
>> On Sat, May 09, 2026 at 06:31:47PM +0200, Simon Schippers wrote:
>>> On 5/8/26 17:10, Simon Schippers wrote:
>>>> +static void tun_queue_purge(struct tun_struct *tun, struct tun_file *tfile)
>>>> {
>>>> void *ptr;
>>>>
>>>> - while ((ptr = ptr_ring_consume(&tfile->tx_ring)) != NULL)
>>>> + while ((ptr = tun_ring_consume(tun, tfile)) != NULL)
>>>> tun_ptr_free(ptr);
>>>>
>>>> skb_queue_purge(&tfile->sk.sk_write_queue);
>>>
>>> Sashiko is right once again. tun_ring_consume() in tun_queue_purge()
>>> operates on a tfile that is being torn down. Its queue_index is no
>>> longer valid. After the swap in __tun_detach(), it points to the
>>> netdev subqueue of a different tfile.
>>> --> We should not wake there.
>>
>> Does it not exactly point at ntfile which is what we want to wake?
>>
>
> I see your point. But calling tun_ring_consume() as done here is
> wrong, because it does not wake if the tx_ring of the tfile
> (that is currently torn down) is empty. We could change
> tun_ring_consume() to call __tun_wake_queue()
> with consumed=0 if !ptr but I think this would slow down the consumer
> path.
>
My statement is wrong:
There is no way that the tx_ring is empty and the queue is stopped
at the same time. So we do not need to touch tun_ring_consume() and
this works just fine.
>>
>>> I will swap tun_ring_consume() with ptr_ring_consume() again and
>>> submit a v12 :)
>>
>> If so then maybe
>> netif_tx_wake_queue(netdev_get_tx_queue(tun->dev, index));
>>
>
> But we should only do this if there is space in the ntfile.
> My approach:
>
> @@ -586,12 +588,18 @@ static void __tun_detach(struct tun_file *tfile, bool clean)
> BUG_ON(index >= tun->numqueues);
>
> rcu_assign_pointer(tun->tfiles[index],
> tun->tfiles[tun->numqueues - 1]);
> ntfile = rtnl_dereference(tun->tfiles[index]);
> + spin_lock(&ntfile->tx_ring.consumer_lock);
> ntfile->queue_index = index;
> ntfile->xdp_rxq.queue_index = index;
> + ntfile->cons_cnt = 0;
> + if (__ptr_ring_empty(&ntfile->tx_ring)) {
> + netif_wake_subqueue(tun->dev, index);
> + }
> + spin_unlock(&ntfile->tx_ring.consumer_lock);
> rcu_assign_pointer(tun->tfiles[tun->numqueues - 1],
> NULL);
>
> ntfile->cons_cnt is unvalid, because the new queue might not be stopped.
> That is the reason why I reset it to 0.
However, I still prefer this approach because the code is easier to
understand.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox