* [PATCH v6 1/2] Drivers: hv: Export some symbols for mshv_vtl
From: Naman Jain @ 2025-07-24 8:25 UTC (permalink / raw)
To: K . Y . Srinivasan, Haiyang Zhang, Wei Liu, Dexuan Cui,
Michael Kelley
Cc: Roman Kisel, Anirudh Rayabharam, Saurabh Sengar,
Stanislav Kinsburskii, Nuno Das Neves, ALOK TIWARI,
Markus Elfring, linux-kernel, linux-hyperv
In-Reply-To: <20250724082547.195235-1-namjain@linux.microsoft.com>
MSHV_VTL driver is going to be introduced, which is supposed to
provide interface for Virtual Machine Monitors (VMMs) to control
Virtual Trust Level (VTL). Export the symbols needed
to make it work (vmbus_isr, hv_context and hv_post_message).
Co-developed-by: Roman Kisel <romank@linux.microsoft.com>
Signed-off-by: Roman Kisel <romank@linux.microsoft.com>
Co-developed-by: Saurabh Sengar <ssengar@linux.microsoft.com>
Signed-off-by: Saurabh Sengar <ssengar@linux.microsoft.com>
Reviewed-by: Roman Kisel <romank@linux.microsoft.com>
Reviewed-by: Saurabh Sengar <ssengar@linux.microsoft.com>
Message-ID: <20250512140432.2387503-2-namjain@linux.microsoft.com>
Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/oe-kbuild-all/202506110544.q0NDMQVc-lkp@intel.com/
Signed-off-by: Naman Jain <namjain@linux.microsoft.com>
---
drivers/hv/hv.c | 3 +++
drivers/hv/hyperv_vmbus.h | 1 +
drivers/hv/vmbus_drv.c | 4 +++-
3 files changed, 7 insertions(+), 1 deletion(-)
diff --git a/drivers/hv/hv.c b/drivers/hv/hv.c
index b14c5f9e0ef2..b16e94daa270 100644
--- a/drivers/hv/hv.c
+++ b/drivers/hv/hv.c
@@ -18,6 +18,7 @@
#include <linux/clockchips.h>
#include <linux/delay.h>
#include <linux/interrupt.h>
+#include <linux/export.h>
#include <clocksource/hyperv_timer.h>
#include <asm/mshyperv.h>
#include <linux/set_memory.h>
@@ -25,6 +26,7 @@
/* The one and only */
struct hv_context hv_context;
+EXPORT_SYMBOL_GPL(hv_context);
/*
* hv_init - Main initialization routine.
@@ -95,6 +97,7 @@ int hv_post_message(union hv_connection_id connection_id,
return hv_result(status);
}
+EXPORT_SYMBOL_GPL(hv_post_message);
int hv_synic_alloc(void)
{
diff --git a/drivers/hv/hyperv_vmbus.h b/drivers/hv/hyperv_vmbus.h
index 0b450e53161e..b61f01fc1960 100644
--- a/drivers/hv/hyperv_vmbus.h
+++ b/drivers/hv/hyperv_vmbus.h
@@ -32,6 +32,7 @@
*/
#define HV_UTIL_NEGO_TIMEOUT 55
+void vmbus_isr(void);
/* Definitions for the monitored notification facility */
union hv_monitor_trigger_group {
diff --git a/drivers/hv/vmbus_drv.c b/drivers/hv/vmbus_drv.c
index 2ed5a1e89d69..a366365f2c49 100644
--- a/drivers/hv/vmbus_drv.c
+++ b/drivers/hv/vmbus_drv.c
@@ -36,6 +36,7 @@
#include <linux/syscore_ops.h>
#include <linux/dma-map-ops.h>
#include <linux/pci.h>
+#include <linux/export.h>
#include <clocksource/hyperv_timer.h>
#include <asm/mshyperv.h>
#include "hyperv_vmbus.h"
@@ -1306,7 +1307,7 @@ static void vmbus_chan_sched(struct hv_per_cpu_context *hv_cpu)
}
}
-static void vmbus_isr(void)
+void vmbus_isr(void)
{
struct hv_per_cpu_context *hv_cpu
= this_cpu_ptr(hv_context.cpu_context);
@@ -1329,6 +1330,7 @@ static void vmbus_isr(void)
add_interrupt_randomness(vmbus_interrupt);
}
+EXPORT_SYMBOL_GPL(vmbus_isr);
static irqreturn_t vmbus_percpu_isr(int irq, void *dev_id)
{
--
2.34.1
^ permalink raw reply related
* [PATCH v6 0/2] Drivers: hv: Introduce new driver - mshv_vtl
From: Naman Jain @ 2025-07-24 8:25 UTC (permalink / raw)
To: K . Y . Srinivasan, Haiyang Zhang, Wei Liu, Dexuan Cui,
Michael Kelley
Cc: Roman Kisel, Anirudh Rayabharam, Saurabh Sengar,
Stanislav Kinsburskii, Nuno Das Neves, ALOK TIWARI,
Markus Elfring, linux-kernel, linux-hyperv
Introduce a new mshv_vtl driver to provide an interface for Virtual
Machine Monitor like OpenVMM and its use as OpenHCL paravisor to
control VTL0 (Virtual trust Level).
Expose devices and support IOCTLs for features like VTL creation,
VTL0 memory management, context switch, making hypercalls,
mapping VTL0 address space to VTL2 userspace, getting new VMBus
messages and channel events in VTL2 etc.
OpenVMM : https://openvmm.dev/guide/
Changes since v5:
https://lore.kernel.org/all/20250611072704.83199-1-namjain@linux.microsoft.com/
Addressed Michael Kelley's suggestions:
* Added "depends on HYPERV_VTL_MODE", removed "depends on HYPERV" in Kconfig
* Removed unused macro MAX_GUEST_MEM_SIZE
* Made macro dependency explicit: MSHV_PG_OFF_CPU_MASK and MSHV_REAL_OFF_SHIFT
* Refactored and corrected how allow_bitmap is used and defined. Removed PAGE_SIZE dependency.
* Added __packed for structure definitions wherever it was missing.
* Moved hv_register_vsm_* union definitions to hvgdk_mini.h, kept mshv_synic_overlay_page_msr
in the driver, renamed it and added a comment. (Nuno)
* Introduced global variables input_vtl_zero and input_vtl_normal and used them everywhere these
were defined locally
* s/"page_to_phys(reg_page) >> HV_HYP_PAGE_SHIFT"/"page_to_hvpfn(reg_page)" in
mshv_vtl_configure_reg_page
* Refactored mshv_vtl_vmbus_isr() to reduce complexity in finding and resetting bits similar to
how vmbus_chan_sched is implemented.
* Used __get_free_page() instead in mshv_vtl_alloc_context()
* Added fallback hv_setup_vmbus_handler(vmbus_isr) in hv_vtl_setup_synic() and in
hv_vtl_remove_synic().
* Maintained symmetry of functions in hv_vtl_remove_synic
* Added a note for explanation of excluding last PFN in the range provided in
mshv_vtl_ioctl_add_vtl0_mem()
* Added comments for hotplug being not supported, wherever cpu_online() was used to check if CPU
is online or not.
* Added a check for input.cpu to make sure it's less than nr_cpu_ids in
mshv_vtl_ioctl_set_poll_file()
* Removed switch-case and implemented static tables in mshv_vtl_(get|set)_reg for reducing LOC
* Simplified mshv_vtl_ioctl_(get|set)_regs to process one register at a time, and fixed earlier
bug with array of registers processing.
* Used hv_result_to_errno() in mshv_vtl_sint_ioctl_signal_event()
* Added a READ_ONCE() while reading old_eventfd in mshv_vtl_sint_ioctl_set_eventfd()
* Renamed mshv_vtl_hvcall and mshv_vtl_hvcall_setup to remove ambiguity
* Took care of latest mm patches regarding PFN_DEV, pfn_t deprecation
* Few other minor changes while reorganizing code.
Addressed Markus Elfring's suggestions:
* Used guard(mutex) for better mutex handling.
Changes since v4:
https://lore.kernel.org/all/20250610052435.1660967-1-namjain@linux.microsoft.com/
* Fixed warnings from kernel test robot for missing export.h when the
kernel is compiled with W=1 option.
Some recent changes in kernel flags these warnings and that's why it
was not seen in previous runs. Warnings in other Hyper-V drivers
will be fixed separately.
* No functional changes
Changes since v3:
https://lore.kernel.org/all/20250519045642.50609-1-namjain@linux.microsoft.com/
Addressed Stanislav's, Nuno's comments.
* Change data types for different variables, excluding the ones in uapi headers
* Added comment for the need of HUGEPAGES config in Kconfig.
* generalized new IOCTL names by removing VTL in their name.
* Rebased and added Saurabh's Reviewed-by tag
Changes since v2:
https://lore.kernel.org/all/20250512140432.2387503-1-namjain@linux.microsoft.com/
* Removed CONFIG_OF dependency (addressed Saurabh's comments)
* Fixed typo in "allow_map_intialized" variable name
Changes since v1:
https://lore.kernel.org/all/20250506084937.624680-1-namjain@linux.microsoft.com/
Addressed Saurabh's comments:
* Split the patch in 2 to keep export symbols separate
* Make MSHV_VTL module tristate and fixed compilation warning that would come when HYPERV is
compiled as a module.
* Remove the use of ref_count
* Split functionality of mshv_vtl_ioctl_get_set_regs to different functions
mshv_vtl_ioctl_(get|set)_regs as it actually make things simpler
* Fixed use of copy_from_user in atomic context in mshv_vtl_hvcall_call.
Added ToDo comment for info.
* Added extra code to free memory for vtl in error scenarios in mshv_ioctl_create_vtl()
Addressed Alok's comments regarding:
* Additional conditional checks
* corrected typo in HV_X64_REGISTER_MSR_MTRR_PHYS_MASKB case
* empty lines before return statement
* Added/edited comments, variable names, structure field names as suggested to improve
documentation - no functional change here.
Naman Jain (2):
Drivers: hv: Export some symbols for mshv_vtl
Drivers: hv: Introduce mshv_vtl driver
drivers/hv/Kconfig | 22 +
drivers/hv/Makefile | 7 +-
drivers/hv/hv.c | 3 +
drivers/hv/hyperv_vmbus.h | 1 +
drivers/hv/mshv_vtl.h | 52 ++
drivers/hv/mshv_vtl_main.c | 1508 +++++++++++++++++++++++++++++++++++
drivers/hv/vmbus_drv.c | 4 +-
include/hyperv/hvgdk_mini.h | 106 +++
include/uapi/linux/mshv.h | 80 ++
9 files changed, 1781 insertions(+), 2 deletions(-)
create mode 100644 drivers/hv/mshv_vtl.h
create mode 100644 drivers/hv/mshv_vtl_main.c
base-commit: a933d3dc1968fcfb0ab72879ec304b1971ed1b9a
--
2.34.1
^ permalink raw reply
* Re: [PATCH v3 07/16] x86/kvm/emulate: Introduce EM_ASM_1SRC2
From: Sean Christopherson @ 2025-07-24 0:16 UTC (permalink / raw)
To: Peter Zijlstra
Cc: x86, kys, haiyangz, wei.liu, decui, tglx, mingo, bp, dave.hansen,
hpa, pbonzini, ardb, kees, Arnd Bergmann, gregkh, jpoimboe,
linux-hyperv, linux-kernel, kvm, linux-efi, samitolvanen, ojeda
In-Reply-To: <20250714103440.394654786@infradead.org>
For all of the KVM patches, please use
KVM: x86:
"x86/kvm" is used for guest-side code, and while I hope no one will conflate this
with guest code, the consistency is helpful.
On Mon, Jul 14, 2025, Peter Zijlstra wrote:
> Replace the FASTOP1SRC2*() instructions.
>
> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> ---
> arch/x86/kvm/emulate.c | 34 ++++++++++++++++++++++++++--------
> 1 file changed, 26 insertions(+), 8 deletions(-)
>
> --- a/arch/x86/kvm/emulate.c
> +++ b/arch/x86/kvm/emulate.c
> @@ -317,6 +317,24 @@ static int em_##op(struct x86_emulate_ct
> ON64(case 8: __EM_ASM_1(op##q, rax); break;) \
> EM_ASM_END
>
> +/* 1-operand, using "c" (src2) */
> +#define EM_ASM_1SRC2(op, name) \
> + EM_ASM_START(name) \
> + case 1: __EM_ASM_1(op##b, cl); break; \
> + case 2: __EM_ASM_1(op##w, cx); break; \
> + case 4: __EM_ASM_1(op##l, ecx); break; \
> + ON64(case 8: __EM_ASM_1(op##q, rcx); break;) \
> + EM_ASM_END
> +
> +/* 1-operand, using "c" (src2) with exception */
> +#define EM_ASM_1SRC2EX(op, name) \
> + EM_ASM_START(name) \
> + case 1: __EM_ASM_1_EX(op##b, cl); break; \
> + case 2: __EM_ASM_1_EX(op##w, cx); break; \
> + case 4: __EM_ASM_1_EX(op##l, ecx); break; \
> + ON64(case 8: __EM_ASM_1(op##q, rcx); break;) \
This needs to be __EM_ASM_1_EX(). Luckily, KVM-Unit-Tests actually has testcase
for divq (somewhere in the morass of testcases). I also now have an extension to
the fastops selftest to explicitly test all four flavors of div-by-zero; I'll get
it posted tomorrow.
(also, don't also me how long it took me to spot the copy+paste typo; I was full
on debugging the exception fixup code before I realized my local diff looked
"odd", *sigh*)
^ permalink raw reply
* Re: [PATCH RESEND] netvsc: transfer lower device max tso size
From: Jakub Kicinski @ 2025-07-23 22:16 UTC (permalink / raw)
To: Haiyang Zhang
Cc: Jason Wang, Stephen Hemminger, Cindy Lu, KY Srinivasan, Wei Liu,
Dexuan Cui, Andrew Lunn, David S. Miller, Eric Dumazet,
Paolo Abeni, Simon Horman, Michael Kelley, Shradha Gupta,
Kees Cook, Stanislav Fomichev, Kuniyuki Iwashima,
Alexander Lobakin, Guillaume Nault, Joe Damato, Ahmed Zaki,
open list:Hyper-V/Azure CORE AND DRIVERS,
open list:NETWORKING DRIVERS, open list
In-Reply-To: <SJ2PR21MB40138F71138A809C3A2D903BCA5FA@SJ2PR21MB4013.namprd21.prod.outlook.com>
On Wed, 23 Jul 2025 20:18:03 +0000 Haiyang Zhang wrote:
> > > Btw, if I understand this correctly. This is for future development so
> > > it's not a blocker for this patch?
> >
> > Not a blocker, I'm just giving an example of the netvsc auto-weirdness
> > being a source of tech debt and bugs. Commit d7501e076d859d is another
> > recent one off the top of my head. IIUC systemd-networkd is broadly
> > deployed now. It'd be great if there was some migration plan for moving
> > this sort of VM auto-bonding to user space (with the use of the common
> > bonding driver, not each hypervisor rolling its own).
>
> Actually, we had used the common bonding driver 9 years ago. But it's
> replaced by this kernel/netvsc based "transparent" bonding mode. See
> the patches listed below.
>
> The user mode bonding scripts were unstable, and difficult to deliver
> & update for various distros. So Stephen developed the new "transparent"
> bonding mode, which greatly improves the situation.
I specifically highlighted systemd-networkd as the change in the user
space landscape.
^ permalink raw reply
* RE: [EXTERNAL] Re: [PATCH net, 0/2] net: Add llist_node init and fix hv_netvsc namespace error
From: Haiyang Zhang @ 2025-07-23 20:49 UTC (permalink / raw)
To: Jakub Kicinski, Haiyang Zhang
Cc: linux-hyperv@vger.kernel.org, netdev@vger.kernel.org,
KY Srinivasan, wei.liu@kernel.org, Dexuan Cui,
andrew+netdev@lunn.ch, sd@queasysnail.net,
viro@zeniv.linux.org.uk, chuck.lever@oracle.com, neil@brown.name,
edumazet@google.com, pabeni@redhat.com, horms@kernel.org,
davem@davemloft.net, kuniyu@google.com,
linux-kernel@vger.kernel.org
In-Reply-To: <20250722184433.2b951171@kernel.org>
> -----Original Message-----
> From: Jakub Kicinski <kuba@kernel.org>
> Sent: Tuesday, July 22, 2025 9:45 PM
> To: Haiyang Zhang <haiyangz@linux.microsoft.com>
> Cc: linux-hyperv@vger.kernel.org; netdev@vger.kernel.org; Haiyang Zhang
> <haiyangz@microsoft.com>; KY Srinivasan <kys@microsoft.com>;
> wei.liu@kernel.org; Dexuan Cui <decui@microsoft.com>;
> andrew+netdev@lunn.ch; sd@queasysnail.net; viro@zeniv.linux.org.uk;
> chuck.lever@oracle.com; neil@brown.name; edumazet@google.com;
> pabeni@redhat.com; horms@kernel.org; davem@davemloft.net;
> kuniyu@google.com; linux-kernel@vger.kernel.org
> Subject: [EXTERNAL] Re: [PATCH net, 0/2] net: Add llist_node init and fix
> hv_netvsc namespace error
>
> On Tue, 22 Jul 2025 16:50:46 -0700 Haiyang Zhang wrote:
> > Add llist_node init to setup_net(), so we can check if the node is on
> list.
> > Then, fix the namespace callback function in hv_netvsc.
>
> Can you not do the moving from a workqueue? Schedule a work, let the
> stack finish what it's doing, take rtnl_lock, do you your own moving?
Thanks for the suggestion! I will implement and test it.
- Haiyang
^ permalink raw reply
* Re: [PATCH] net: mana: Use page pool fragments for RX buffers instead of full pages to improve memory efficiency and throughput.
From: Jacob Keller @ 2025-07-23 20:22 UTC (permalink / raw)
To: Dipayaan Roy
Cc: kuba, kys, haiyangz, wei.liu, decui, andrew+netdev, davem,
edumazet, pabeni, longli, kotaranov, horms, ast, daniel, hawk,
john.fastabend, sdf, lorenzo, michal.kubiak, ernis, shradhagupta,
shirazsaleem, rosenp, netdev, linux-hyperv, linux-rdma, bpf
In-Reply-To: <20250723182224.GA25631@linuxonhyperv3.guj3yctzbm1etfxqx2vob5hsef.xx.internal.cloudapp.net>
[-- Attachment #1.1: Type: text/plain, Size: 425 bytes --]
On 7/23/2025 11:22 AM, Dipayaan Roy wrote:
> Hi Jacob,
>
> Thanks for the review, I tested this patch previously
> for 4kb page size systems. The throughput improvement is around
> 10 to 15%. Also sligthly better memory utilization as for
> typical 1500 MTU size we are able to fit 2 Rx buffer in
> a single 4kb page as compared to 1 per page previously.
>
> Thanks
> Dipayaan Roy
>
Fantastic
-Jake
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]
^ permalink raw reply
* Re: [PATCH RESEND] netvsc: transfer lower device max tso size
From: Haiyang Zhang @ 2025-07-23 20:18 UTC (permalink / raw)
To: Jakub Kicinski, Jason Wang, Stephen Hemminger
Cc: Cindy Lu, KY Srinivasan, Wei Liu, Dexuan Cui, Andrew Lunn,
David S. Miller, Eric Dumazet, Paolo Abeni, Simon Horman,
Michael Kelley, Shradha Gupta, Kees Cook, Stanislav Fomichev,
Kuniyuki Iwashima, Alexander Lobakin, Guillaume Nault, Joe Damato,
Ahmed Zaki, open list:Hyper-V/Azure CORE AND DRIVERS,
open list:NETWORKING DRIVERS, open list
In-Reply-To: <20250723080532.53ecc4f1@kernel.org>
> -----Original Message-----
> From: Jakub Kicinski <kuba@kernel.org>
> Sent: Wednesday, July 23, 2025 11:06 AM
> To: Jason Wang <jasowang@redhat.com>
> Cc: Cindy Lu <lulu@redhat.com>; KY Srinivasan <kys@microsoft.com>; Haiyang
> Zhang <haiyangz@microsoft.com>; Wei Liu <wei.liu@kernel.org>; Dexuan Cui
> <decui@microsoft.com>; Andrew Lunn <andrew+netdev@lunn.ch>; David S.
> Miller <davem@davemloft.net>; Eric Dumazet <edumazet@google.com>; Paolo
> Abeni <pabeni@redhat.com>; Simon Horman <horms@kernel.org>; Michael Kelley
> <mhklinux@outlook.com>; Shradha Gupta <shradhagupta@linux.microsoft.com>;
> Kees Cook <kees@kernel.org>; Stanislav Fomichev <sdf@fomichev.me>;
> Kuniyuki Iwashima <kuniyu@google.com>; Alexander Lobakin
> <aleksander.lobakin@intel.com>; Guillaume Nault <gnault@redhat.com>; Joe
> Damato <jdamato@fastly.com>; Ahmed Zaki <ahmed.zaki@intel.com>; open
> list:Hyper-V/Azure CORE AND DRIVERS <linux-hyperv@vger.kernel.org>; open
> list:NETWORKING DRIVERS <netdev@vger.kernel.org>; open list <linux-
> kernel@vger.kernel.org>
> Subject: [EXTERNAL] Re: [PATCH RESEND] netvsc: transfer lower device max
> tso size
>
> On Wed, 23 Jul 2025 14:00:47 +0800 Jason Wang wrote:
> > > > But this fixes a real problem, otherwise nested VM performance will
> be
> > > > broken due to the GSO software segmentation.
> > >
> > > Perhaps, possibly, a migration plan can be devised, away from the
> > > netvsc model, so we don't have to deal with nuggets of joy like:
> > >
> https://lore.ker/
> nel.org%2Fall%2F1752870014-28909-1-git-send-email-
> haiyangz%40linux.microsoft.com%2F&data=05%7C02%7Chaiyangz%40microsoft.com%
> 7C27aa73c4b6e9446c7f6508ddc9fa6053%7C72f988bf86f141af91ab2d7cd011db47%7C1%
> 7C0%7C638888799449774753%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsI
> lYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C
> %7C%7C&sdata=HYX4dDie8QlEepzwPiiI3eYASe7WN5o%2BWTXmMGoyFOM%3D&reserved=0
> >
> > Btw, if I understand this correctly. This is for future development so
> > it's not a blocker for this patch?
>
> Not a blocker, I'm just giving an example of the netvsc auto-weirdness
> being a source of tech debt and bugs. Commit d7501e076d859d is another
> recent one off the top of my head. IIUC systemd-networkd is broadly
> deployed now. It'd be great if there was some migration plan for moving
> this sort of VM auto-bonding to user space (with the use of the common
> bonding driver, not each hypervisor rolling its own).
Actually, we had used the common bonding driver 9 years ago. But it's
replaced by this kernel/netvsc based "transparent" bonding mode. See
the patches listed below.
The user mode bonding scripts were unstable, and difficult to deliver
& update for various distros. So Stephen developed the new "transparent"
bonding mode, which greatly improves the situation.
@Stephen Hemminger <stephen@networkplumber.org>, in case he wants to
add more regarding the history.
Related patches:
author Haiyang Zhang <haiyangz@microsoft.com> 2016-07-11 17:06:42 -0700
committer David S. Miller <davem@davemloft.net> 2016-07-12 10:41:53 -0700
commit 178cd55f086629cf0bad9c66c793a7e2bcc3abb6 (patch)
tools: hv: Add a script to help bonding synthetic and VF NICs
author stephen hemminger <stephen@networkplumber.org> 2017-08-01 19:58:55 -0700
committer David S. Miller <davem@davemloft.net> 2017-08-02 16:55:33 -0700
commit 12aa7469d101e139b3728e540884bc7d72dca70a (patch)
netvsc: remove bonding setup script
No longer needed, now all managed by transparent VF logic.
Thanks,
- Haiyang
^ permalink raw reply
* RE: [PATCH v2] net: mana: Use page pool fragments for RX buffers instead of full pages to improve memory efficiency.
From: Haiyang Zhang @ 2025-07-23 19:55 UTC (permalink / raw)
To: Dipayaan Roy, horms@kernel.org, kuba@kernel.org, KY Srinivasan,
wei.liu@kernel.org, Dexuan Cui, andrew+netdev@lunn.ch,
davem@davemloft.net, edumazet@google.com, pabeni@redhat.com,
Long Li, Konstantin Taranov, ast@kernel.org, daniel@iogearbox.net,
hawk@kernel.org, john.fastabend@gmail.com, sdf@fomichev.me,
lorenzo@kernel.org, michal.kubiak@intel.com,
ernis@linux.microsoft.com, shradhagupta@linux.microsoft.com,
Shiraz Saleem, rosenp@gmail.com, netdev@vger.kernel.org,
linux-hyperv@vger.kernel.org, linux-rdma@vger.kernel.org,
bpf@vger.kernel.org, linux-kernel@vger.kernel.org,
ssengar@linux.microsoft.com, Dipayaan Roy
In-Reply-To: <20250723190706.GA5291@linuxonhyperv3.guj3yctzbm1etfxqx2vob5hsef.xx.internal.cloudapp.net>
> -----Original Message-----
> From: Dipayaan Roy <dipayanroy@linux.microsoft.com>
> Sent: Wednesday, July 23, 2025 3:07 PM
> To: horms@kernel.org; kuba@kernel.org; KY Srinivasan <kys@microsoft.com>;
> Haiyang Zhang <haiyangz@microsoft.com>; wei.liu@kernel.org; Dexuan Cui
> <decui@microsoft.com>; andrew+netdev@lunn.ch; davem@davemloft.net;
> edumazet@google.com; pabeni@redhat.com; Long Li <longli@microsoft.com>;
> Konstantin Taranov <kotaranov@microsoft.com>; ast@kernel.org;
> daniel@iogearbox.net; hawk@kernel.org; john.fastabend@gmail.com;
> sdf@fomichev.me; lorenzo@kernel.org; michal.kubiak@intel.com;
> ernis@linux.microsoft.com; shradhagupta@linux.microsoft.com; Shiraz Saleem
> <shirazsaleem@microsoft.com>; rosenp@gmail.com; netdev@vger.kernel.org;
> linux-hyperv@vger.kernel.org; linux-rdma@vger.kernel.org;
> bpf@vger.kernel.org; linux-kernel@vger.kernel.org;
> ssengar@linux.microsoft.com; Dipayaan Roy <dipayanroy@microsoft.com>
> Subject: [PATCH v2] net: mana: Use page pool fragments for RX buffers
> instead of full pages to improve memory efficiency.
>
> This patch enhances RX buffer handling in the mana driver by allocating
> pages from a page pool and slicing them into MTU-sized fragments, rather
> than dedicating a full page per packet. This approach is especially
> beneficial on systems with large page sizes like 64KB.
>
> Key improvements:
>
> - Proper integration of page pool for RX buffer allocations.
> - MTU-sized buffer slicing to improve memory utilization.
> - Reduce overall per Rx queue memory footprint.
> - Automatic fallback to full-page buffers when:
> * Jumbo frames are enabled (MTU > PAGE_SIZE / 2).
> * The XDP path is active, to avoid complexities with fragment reuse.
> - Removal of redundant pre-allocated RX buffers used in scenarios like MTU
> changes, ensuring consistency in RX buffer allocation.
>
> Testing on VMs with 64KB pages shows around 200% throughput improvement.
> Memory efficiency is significantly improved due to reduced wastage in page
> allocations. Example: We are now able to fit 35 rx buffers in a single
> 64kb
> page for MTU size of 1500, instead of 1 rx buffer per page previously.
>
> Tested:
>
> - iperf3, iperf2, and nttcp benchmarks.
> - Jumbo frames with MTU 9000.
> - Native XDP programs (XDP_PASS, XDP_DROP, XDP_TX, XDP_REDIRECT) for
> testing the XDP path in driver.
> - Page leak detection (kmemleak).
> - Driver load/unload, reboot, and stress scenarios.
>
> Signed-off-by: Dipayaan Roy <dipayanroy@linux.microsoft.com>
>
> Reviewed-by: Jacob Keller <jacob.e.keller@intel.com>
> Reviewed-by: Saurabh Sengar <ssengar@linux.microsoft.com>
Reviewed-by: Haiyang Zhang <haiyangz@microsoft.com>
^ permalink raw reply
* [PATCH v2] net: mana: Use page pool fragments for RX buffers instead of full pages to improve memory efficiency.
From: Dipayaan Roy @ 2025-07-23 19:07 UTC (permalink / raw)
To: horms, kuba, kys, haiyangz, wei.liu, decui, andrew+netdev, davem,
edumazet, pabeni, longli, kotaranov, ast, daniel, hawk,
john.fastabend, sdf, lorenzo, michal.kubiak, ernis, shradhagupta,
shirazsaleem, rosenp, netdev, linux-hyperv, linux-rdma, bpf,
linux-kernel, ssengar, dipayanroy
This patch enhances RX buffer handling in the mana driver by allocating
pages from a page pool and slicing them into MTU-sized fragments, rather
than dedicating a full page per packet. This approach is especially
beneficial on systems with large page sizes like 64KB.
Key improvements:
- Proper integration of page pool for RX buffer allocations.
- MTU-sized buffer slicing to improve memory utilization.
- Reduce overall per Rx queue memory footprint.
- Automatic fallback to full-page buffers when:
* Jumbo frames are enabled (MTU > PAGE_SIZE / 2).
* The XDP path is active, to avoid complexities with fragment reuse.
- Removal of redundant pre-allocated RX buffers used in scenarios like MTU
changes, ensuring consistency in RX buffer allocation.
Testing on VMs with 64KB pages shows around 200% throughput improvement.
Memory efficiency is significantly improved due to reduced wastage in page
allocations. Example: We are now able to fit 35 rx buffers in a single 64kb
page for MTU size of 1500, instead of 1 rx buffer per page previously.
Tested:
- iperf3, iperf2, and nttcp benchmarks.
- Jumbo frames with MTU 9000.
- Native XDP programs (XDP_PASS, XDP_DROP, XDP_TX, XDP_REDIRECT) for
testing the XDP path in driver.
- Page leak detection (kmemleak).
- Driver load/unload, reboot, and stress scenarios.
Signed-off-by: Dipayaan Roy <dipayanroy@linux.microsoft.com>
Reviewed-by: Jacob Keller <jacob.e.keller@intel.com>
Reviewed-by: Saurabh Sengar <ssengar@linux.microsoft.com>
---
Changes in v2:
- Fixed mana_xdp_set() to return error code on failure instead of
always returning 0.
- Moved all local variable declarations to the start of functions
in mana_get_rxbuf_cfg.
- Removed unnecessary parentheses and wrapped lines to <= 80 chars.
- Use mana_xdp_get() for checking bpf_prog.
- Factored repeated page put logic into a static helper function.
---
.../net/ethernet/microsoft/mana/mana_bpf.c | 25 +-
drivers/net/ethernet/microsoft/mana/mana_en.c | 286 ++++++------------
.../ethernet/microsoft/mana/mana_ethtool.c | 13 -
include/net/mana/mana.h | 13 +-
4 files changed, 119 insertions(+), 218 deletions(-)
diff --git a/drivers/net/ethernet/microsoft/mana/mana_bpf.c b/drivers/net/ethernet/microsoft/mana/mana_bpf.c
index d30721d4516f..f2dee2655352 100644
--- a/drivers/net/ethernet/microsoft/mana/mana_bpf.c
+++ b/drivers/net/ethernet/microsoft/mana/mana_bpf.c
@@ -174,6 +174,7 @@ static int mana_xdp_set(struct net_device *ndev, struct bpf_prog *prog,
struct mana_port_context *apc = netdev_priv(ndev);
struct bpf_prog *old_prog;
struct gdma_context *gc;
+ int err = 0;
gc = apc->ac->gdma_dev->gdma_context;
@@ -198,15 +199,33 @@ static int mana_xdp_set(struct net_device *ndev, struct bpf_prog *prog,
if (old_prog)
bpf_prog_put(old_prog);
- if (apc->port_is_up)
+ if (apc->port_is_up) {
+ /* Re-create rxq's after xdp prog was loaded or unloaded.
+ * Ex: re create rxq's to switch from full pages to smaller
+ * size page fragments when xdp prog is unloaded and vice-versa.
+ */
+
+ err = mana_detach(ndev, false);
+ if (err) {
+ netdev_err(ndev, "mana_detach failed at xdp set: %d\n", err);
+ goto out;
+ }
+
+ err = mana_attach(ndev);
+ if (err) {
+ netdev_err(ndev, "mana_attach failed at xdp set: %d\n", err);
+ goto out;
+ }
+
mana_chn_setxdp(apc, prog);
+ }
if (prog)
ndev->max_mtu = MANA_XDP_MTU_MAX;
else
ndev->max_mtu = gc->adapter_mtu - ETH_HLEN;
-
- return 0;
+out:
+ return err;
}
int mana_bpf(struct net_device *ndev, struct netdev_bpf *bpf)
diff --git a/drivers/net/ethernet/microsoft/mana/mana_en.c b/drivers/net/ethernet/microsoft/mana/mana_en.c
index a7973651ae51..9276833e434b 100644
--- a/drivers/net/ethernet/microsoft/mana/mana_en.c
+++ b/drivers/net/ethernet/microsoft/mana/mana_en.c
@@ -56,6 +56,14 @@ static bool mana_en_need_log(struct mana_port_context *apc, int err)
return true;
}
+static void mana_put_rx_page(struct mana_rxq *rxq, struct page *page, bool from_pool)
+{
+ if (from_pool)
+ page_pool_put_full_page(rxq->page_pool, page, false);
+ else
+ put_page(page);
+}
+
/* Microsoft Azure Network Adapter (MANA) functions */
static int mana_open(struct net_device *ndev)
@@ -548,171 +556,48 @@ static u16 mana_select_queue(struct net_device *ndev, struct sk_buff *skb,
return txq;
}
-/* Release pre-allocated RX buffers */
-void mana_pre_dealloc_rxbufs(struct mana_port_context *mpc)
-{
- struct device *dev;
- int i;
-
- dev = mpc->ac->gdma_dev->gdma_context->dev;
-
- if (!mpc->rxbufs_pre)
- goto out1;
-
- if (!mpc->das_pre)
- goto out2;
-
- while (mpc->rxbpre_total) {
- i = --mpc->rxbpre_total;
- dma_unmap_single(dev, mpc->das_pre[i], mpc->rxbpre_datasize,
- DMA_FROM_DEVICE);
- put_page(virt_to_head_page(mpc->rxbufs_pre[i]));
- }
-
- kfree(mpc->das_pre);
- mpc->das_pre = NULL;
-
-out2:
- kfree(mpc->rxbufs_pre);
- mpc->rxbufs_pre = NULL;
-
-out1:
- mpc->rxbpre_datasize = 0;
- mpc->rxbpre_alloc_size = 0;
- mpc->rxbpre_headroom = 0;
-}
-
-/* Get a buffer from the pre-allocated RX buffers */
-static void *mana_get_rxbuf_pre(struct mana_rxq *rxq, dma_addr_t *da)
-{
- struct net_device *ndev = rxq->ndev;
- struct mana_port_context *mpc;
- void *va;
-
- mpc = netdev_priv(ndev);
-
- if (!mpc->rxbufs_pre || !mpc->das_pre || !mpc->rxbpre_total) {
- netdev_err(ndev, "No RX pre-allocated bufs\n");
- return NULL;
- }
-
- /* Check sizes to catch unexpected coding error */
- if (mpc->rxbpre_datasize != rxq->datasize) {
- netdev_err(ndev, "rxbpre_datasize mismatch: %u: %u\n",
- mpc->rxbpre_datasize, rxq->datasize);
- return NULL;
- }
-
- if (mpc->rxbpre_alloc_size != rxq->alloc_size) {
- netdev_err(ndev, "rxbpre_alloc_size mismatch: %u: %u\n",
- mpc->rxbpre_alloc_size, rxq->alloc_size);
- return NULL;
- }
-
- if (mpc->rxbpre_headroom != rxq->headroom) {
- netdev_err(ndev, "rxbpre_headroom mismatch: %u: %u\n",
- mpc->rxbpre_headroom, rxq->headroom);
- return NULL;
- }
-
- mpc->rxbpre_total--;
-
- *da = mpc->das_pre[mpc->rxbpre_total];
- va = mpc->rxbufs_pre[mpc->rxbpre_total];
- mpc->rxbufs_pre[mpc->rxbpre_total] = NULL;
-
- /* Deallocate the array after all buffers are gone */
- if (!mpc->rxbpre_total)
- mana_pre_dealloc_rxbufs(mpc);
-
- return va;
-}
-
/* Get RX buffer's data size, alloc size, XDP headroom based on MTU */
-static void mana_get_rxbuf_cfg(int mtu, u32 *datasize, u32 *alloc_size,
- u32 *headroom)
+static void mana_get_rxbuf_cfg(struct mana_port_context *apc,
+ int mtu, u32 *datasize, u32 *alloc_size,
+ u32 *headroom, u32 *frag_count)
{
- if (mtu > MANA_XDP_MTU_MAX)
- *headroom = 0; /* no support for XDP */
- else
- *headroom = XDP_PACKET_HEADROOM;
-
- *alloc_size = SKB_DATA_ALIGN(mtu + MANA_RXBUF_PAD + *headroom);
-
- /* Using page pool in this case, so alloc_size is PAGE_SIZE */
- if (*alloc_size < PAGE_SIZE)
- *alloc_size = PAGE_SIZE;
+ u32 len, buf_size;
+ /* Calculate datasize first (consistent across all cases) */
*datasize = mtu + ETH_HLEN;
-}
-
-int mana_pre_alloc_rxbufs(struct mana_port_context *mpc, int new_mtu, int num_queues)
-{
- struct device *dev;
- struct page *page;
- dma_addr_t da;
- int num_rxb;
- void *va;
- int i;
-
- mana_get_rxbuf_cfg(new_mtu, &mpc->rxbpre_datasize,
- &mpc->rxbpre_alloc_size, &mpc->rxbpre_headroom);
-
- dev = mpc->ac->gdma_dev->gdma_context->dev;
-
- num_rxb = num_queues * mpc->rx_queue_size;
-
- WARN(mpc->rxbufs_pre, "mana rxbufs_pre exists\n");
- mpc->rxbufs_pre = kmalloc_array(num_rxb, sizeof(void *), GFP_KERNEL);
- if (!mpc->rxbufs_pre)
- goto error;
-
- mpc->das_pre = kmalloc_array(num_rxb, sizeof(dma_addr_t), GFP_KERNEL);
- if (!mpc->das_pre)
- goto error;
-
- mpc->rxbpre_total = 0;
-
- for (i = 0; i < num_rxb; i++) {
- page = dev_alloc_pages(get_order(mpc->rxbpre_alloc_size));
- if (!page)
- goto error;
- va = page_to_virt(page);
-
- da = dma_map_single(dev, va + mpc->rxbpre_headroom,
- mpc->rxbpre_datasize, DMA_FROM_DEVICE);
- if (dma_mapping_error(dev, da)) {
- put_page(page);
- goto error;
+ /* For xdp and jumbo frames make sure only one packet fits per page */
+ if (mtu + MANA_RXBUF_PAD > PAGE_SIZE / 2 || mana_xdp_get(apc)) {
+ if (mana_xdp_get(apc)) {
+ *headroom = XDP_PACKET_HEADROOM;
+ *alloc_size = PAGE_SIZE;
+ } else {
+ *headroom = 0; /* no support for XDP */
+ *alloc_size = SKB_DATA_ALIGN(mtu + MANA_RXBUF_PAD +
+ *headroom);
}
- mpc->rxbufs_pre[i] = va;
- mpc->das_pre[i] = da;
- mpc->rxbpre_total = i + 1;
+ *frag_count = 1;
+ return;
}
- return 0;
+ /* Standard MTU case - optimize for multiple packets per page */
+ *headroom = 0;
-error:
- netdev_err(mpc->ndev, "Failed to pre-allocate RX buffers for %d queues\n", num_queues);
- mana_pre_dealloc_rxbufs(mpc);
- return -ENOMEM;
+ /* Calculate base buffer size needed */
+ len = SKB_DATA_ALIGN(mtu + MANA_RXBUF_PAD + *headroom);
+ buf_size = ALIGN(len, MANA_RX_FRAG_ALIGNMENT);
+
+ /* Calculate how many packets can fit in a page */
+ *frag_count = PAGE_SIZE / buf_size;
+ *alloc_size = buf_size;
}
static int mana_change_mtu(struct net_device *ndev, int new_mtu)
{
- struct mana_port_context *mpc = netdev_priv(ndev);
unsigned int old_mtu = ndev->mtu;
int err;
- /* Pre-allocate buffers to prevent failure in mana_attach later */
- err = mana_pre_alloc_rxbufs(mpc, new_mtu, mpc->num_queues);
- if (err) {
- netdev_err(ndev, "Insufficient memory for new MTU\n");
- return err;
- }
-
err = mana_detach(ndev, false);
if (err) {
netdev_err(ndev, "mana_detach failed: %d\n", err);
@@ -728,7 +613,6 @@ static int mana_change_mtu(struct net_device *ndev, int new_mtu)
}
out:
- mana_pre_dealloc_rxbufs(mpc);
return err;
}
@@ -1841,8 +1725,11 @@ static void mana_rx_skb(void *buf_va, bool from_pool,
drop:
if (from_pool) {
- page_pool_recycle_direct(rxq->page_pool,
- virt_to_head_page(buf_va));
+ if (rxq->frag_count == 1)
+ page_pool_recycle_direct(rxq->page_pool,
+ virt_to_head_page(buf_va));
+ else
+ page_pool_free_va(rxq->page_pool, buf_va, true);
} else {
WARN_ON_ONCE(rxq->xdp_save_va);
/* Save for reuse */
@@ -1854,37 +1741,47 @@ static void mana_rx_skb(void *buf_va, bool from_pool,
return;
}
-static void *mana_get_rxfrag(struct mana_rxq *rxq, struct device *dev,
- dma_addr_t *da, bool *from_pool)
+static void *mana_get_rxfrag(struct mana_rxq *rxq,
+ struct device *dev, dma_addr_t *da, bool *from_pool)
{
struct page *page;
+ u32 offset;
void *va;
-
*from_pool = false;
- /* Reuse XDP dropped page if available */
- if (rxq->xdp_save_va) {
- va = rxq->xdp_save_va;
- rxq->xdp_save_va = NULL;
- } else {
- page = page_pool_dev_alloc_pages(rxq->page_pool);
- if (!page)
+ /* Don't use fragments for jumbo frames or XDP (i.e when fragment = 1 per page) */
+ if (rxq->frag_count == 1) {
+ /* Reuse XDP dropped page if available */
+ if (rxq->xdp_save_va) {
+ va = rxq->xdp_save_va;
+ page = virt_to_head_page(va);
+ rxq->xdp_save_va = NULL;
+ } else {
+ page = page_pool_dev_alloc_pages(rxq->page_pool);
+ if (!page)
+ return NULL;
+
+ *from_pool = true;
+ va = page_to_virt(page);
+ }
+
+ *da = dma_map_single(dev, va + rxq->headroom, rxq->datasize,
+ DMA_FROM_DEVICE);
+ if (dma_mapping_error(dev, *da)) {
+ mana_put_rx_page(rxq, page, *from_pool);
return NULL;
+ }
- *from_pool = true;
- va = page_to_virt(page);
+ return va;
}
- *da = dma_map_single(dev, va + rxq->headroom, rxq->datasize,
- DMA_FROM_DEVICE);
- if (dma_mapping_error(dev, *da)) {
- if (*from_pool)
- page_pool_put_full_page(rxq->page_pool, page, false);
- else
- put_page(virt_to_head_page(va));
-
+ page = page_pool_dev_alloc_frag(rxq->page_pool, &offset, rxq->alloc_size);
+ if (!page)
return NULL;
- }
+
+ va = page_to_virt(page) + offset;
+ *da = page_pool_get_dma_addr(page) + offset + rxq->headroom;
+ *from_pool = true;
return va;
}
@@ -1901,9 +1798,9 @@ static void mana_refill_rx_oob(struct device *dev, struct mana_rxq *rxq,
va = mana_get_rxfrag(rxq, dev, &da, &from_pool);
if (!va)
return;
-
- dma_unmap_single(dev, rxoob->sgl[0].address, rxq->datasize,
- DMA_FROM_DEVICE);
+ if (!rxoob->from_pool || rxq->frag_count == 1)
+ dma_unmap_single(dev, rxoob->sgl[0].address, rxq->datasize,
+ DMA_FROM_DEVICE);
*old_buf = rxoob->buf_va;
*old_fp = rxoob->from_pool;
@@ -2314,15 +2211,15 @@ static void mana_destroy_rxq(struct mana_port_context *apc,
if (!rx_oob->buf_va)
continue;
- dma_unmap_single(dev, rx_oob->sgl[0].address,
- rx_oob->sgl[0].size, DMA_FROM_DEVICE);
-
page = virt_to_head_page(rx_oob->buf_va);
- if (rx_oob->from_pool)
- page_pool_put_full_page(rxq->page_pool, page, false);
- else
- put_page(page);
+ if (rxq->frag_count == 1) {
+ dma_unmap_single(dev, rx_oob->sgl[0].address, rx_oob->sgl[0].size,
+ DMA_FROM_DEVICE);
+ mana_put_rx_page(rxq, page, rx_oob->from_pool);
+ } else {
+ page_pool_free_va(rxq->page_pool, rx_oob->buf_va, true);
+ }
rx_oob->buf_va = NULL;
}
@@ -2338,16 +2235,11 @@ static void mana_destroy_rxq(struct mana_port_context *apc,
static int mana_fill_rx_oob(struct mana_recv_buf_oob *rx_oob, u32 mem_key,
struct mana_rxq *rxq, struct device *dev)
{
- struct mana_port_context *mpc = netdev_priv(rxq->ndev);
bool from_pool = false;
dma_addr_t da;
void *va;
- if (mpc->rxbufs_pre)
- va = mana_get_rxbuf_pre(rxq, &da);
- else
- va = mana_get_rxfrag(rxq, dev, &da, &from_pool);
-
+ va = mana_get_rxfrag(rxq, dev, &da, &from_pool);
if (!va)
return -ENOMEM;
@@ -2428,11 +2320,22 @@ static int mana_create_page_pool(struct mana_rxq *rxq, struct gdma_context *gc)
struct page_pool_params pprm = {};
int ret;
- pprm.pool_size = mpc->rx_queue_size;
+ pprm.pool_size = mpc->rx_queue_size / rxq->frag_count + 1;
pprm.nid = gc->numa_node;
pprm.napi = &rxq->rx_cq.napi;
pprm.netdev = rxq->ndev;
pprm.order = get_order(rxq->alloc_size);
+ pprm.queue_idx = rxq->rxq_idx;
+ pprm.dev = gc->dev;
+
+ /* Let the page pool do the dma map when page sharing with multiple fragments
+ * enabled for rx buffers.
+ */
+ if (rxq->frag_count > 1) {
+ pprm.flags = PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV;
+ pprm.max_len = PAGE_SIZE;
+ pprm.dma_dir = DMA_FROM_DEVICE;
+ }
rxq->page_pool = page_pool_create(&pprm);
@@ -2471,9 +2374,8 @@ static struct mana_rxq *mana_create_rxq(struct mana_port_context *apc,
rxq->rxq_idx = rxq_idx;
rxq->rxobj = INVALID_MANA_HANDLE;
- mana_get_rxbuf_cfg(ndev->mtu, &rxq->datasize, &rxq->alloc_size,
- &rxq->headroom);
-
+ mana_get_rxbuf_cfg(apc, ndev->mtu, &rxq->datasize, &rxq->alloc_size,
+ &rxq->headroom, &rxq->frag_count);
/* Create page pool for RX queue */
err = mana_create_page_pool(rxq, gc);
if (err) {
diff --git a/drivers/net/ethernet/microsoft/mana/mana_ethtool.c b/drivers/net/ethernet/microsoft/mana/mana_ethtool.c
index a1afa75a9463..7ede03c74fb9 100644
--- a/drivers/net/ethernet/microsoft/mana/mana_ethtool.c
+++ b/drivers/net/ethernet/microsoft/mana/mana_ethtool.c
@@ -396,12 +396,6 @@ static int mana_set_channels(struct net_device *ndev,
unsigned int old_count = apc->num_queues;
int err;
- err = mana_pre_alloc_rxbufs(apc, ndev->mtu, new_count);
- if (err) {
- netdev_err(ndev, "Insufficient memory for new allocations");
- return err;
- }
-
err = mana_detach(ndev, false);
if (err) {
netdev_err(ndev, "mana_detach failed: %d\n", err);
@@ -416,7 +410,6 @@ static int mana_set_channels(struct net_device *ndev,
}
out:
- mana_pre_dealloc_rxbufs(apc);
return err;
}
@@ -465,12 +458,7 @@ static int mana_set_ringparam(struct net_device *ndev,
/* pre-allocating new buffers to prevent failures in mana_attach() later */
apc->rx_queue_size = new_rx;
- err = mana_pre_alloc_rxbufs(apc, ndev->mtu, apc->num_queues);
apc->rx_queue_size = old_rx;
- if (err) {
- netdev_err(ndev, "Insufficient memory for new allocations\n");
- return err;
- }
err = mana_detach(ndev, false);
if (err) {
@@ -488,7 +476,6 @@ static int mana_set_ringparam(struct net_device *ndev,
apc->rx_queue_size = old_rx;
}
out:
- mana_pre_dealloc_rxbufs(apc);
return err;
}
diff --git a/include/net/mana/mana.h b/include/net/mana/mana.h
index e1030a7d2daa..99a3847b0f9d 100644
--- a/include/net/mana/mana.h
+++ b/include/net/mana/mana.h
@@ -65,6 +65,8 @@ enum TRI_STATE {
#define MANA_STATS_RX_COUNT 5
#define MANA_STATS_TX_COUNT 11
+#define MANA_RX_FRAG_ALIGNMENT 64
+
struct mana_stats_rx {
u64 packets;
u64 bytes;
@@ -328,6 +330,7 @@ struct mana_rxq {
u32 datasize;
u32 alloc_size;
u32 headroom;
+ u32 frag_count;
mana_handle_t rxobj;
@@ -503,14 +506,6 @@ struct mana_port_context {
/* This points to an array of num_queues of RQ pointers. */
struct mana_rxq **rxqs;
- /* pre-allocated rx buffer array */
- void **rxbufs_pre;
- dma_addr_t *das_pre;
- int rxbpre_total;
- u32 rxbpre_datasize;
- u32 rxbpre_alloc_size;
- u32 rxbpre_headroom;
-
struct bpf_prog *bpf_prog;
/* Create num_queues EQs, SQs, SQ-CQs, RQs and RQ-CQs, respectively. */
@@ -574,8 +569,6 @@ int mana_query_link_cfg(struct mana_port_context *apc);
int mana_set_bw_clamp(struct mana_port_context *apc, u32 speed,
int enable_clamping);
void mana_query_phy_stats(struct mana_port_context *apc);
-int mana_pre_alloc_rxbufs(struct mana_port_context *apc, int mtu, int num_queues);
-void mana_pre_dealloc_rxbufs(struct mana_port_context *apc);
extern const struct ethtool_ops mana_ethtool_ops;
extern struct dentry *mana_debugfs_root;
--
2.43.0
^ permalink raw reply related
* [RFC PATCH V3 4/4] x86/hyperv: Allow Hyper-V to inject STIMER0 interrupts
From: Tianyu Lan @ 2025-07-23 19:03 UTC (permalink / raw)
To: kys, haiyangz, wei.liu, decui, tglx, mingo, bp, dave.hansen, x86,
hpa, arnd, Neeraj.Upadhyay, kvijayab
Cc: Tianyu Lan, linux-arch, linux-hyperv, linux-kernel
In-Reply-To: <20250723190308.5945-1-ltykernel@gmail.com>
From: Tianyu Lan <tiala@microsoft.com>
When Secure AVIC is enabled, call Secure AVIC
function to allow Hyper-V to inject STIMER0 interrupt.
Signed-off-by: Tianyu Lan <tiala@microsoft.com>
---
arch/x86/hyperv/hv_init.c | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/arch/x86/hyperv/hv_init.c b/arch/x86/hyperv/hv_init.c
index 3d1d3547095a..591338162420 100644
--- a/arch/x86/hyperv/hv_init.c
+++ b/arch/x86/hyperv/hv_init.c
@@ -132,6 +132,10 @@ static int hv_cpu_init(unsigned int cpu)
wrmsrq(HV_X64_MSR_VP_ASSIST_PAGE, msr.as_uint64);
}
+ /* Allow Hyper-V stimer vector to be injected from Hypervisor. */
+ if (ms_hyperv.misc_features & HV_STIMER_DIRECT_MODE_AVAILABLE)
+ apic_update_vector(cpu, HYPERV_STIMER0_VECTOR, true);
+
return hyperv_init_ghcb();
}
@@ -239,6 +243,9 @@ static int hv_cpu_die(unsigned int cpu)
*ghcb_va = NULL;
}
+ if (ms_hyperv.misc_features & HV_STIMER_DIRECT_MODE_AVAILABLE)
+ apic_update_vector(cpu, HYPERV_STIMER0_VECTOR, false);
+
hv_common_cpu_die(cpu);
if (hv_vp_assist_page && hv_vp_assist_page[cpu]) {
--
2.25.1
^ permalink raw reply related
* [RFC PATCH V3 3/4] x86/Hyper-V: Don't use auto-eoi when Secure AVIC is available
From: Tianyu Lan @ 2025-07-23 19:03 UTC (permalink / raw)
To: kys, haiyangz, wei.liu, decui, tglx, mingo, bp, dave.hansen, x86,
hpa, arnd, Neeraj.Upadhyay, kvijayab
Cc: Tianyu Lan, linux-arch, linux-hyperv, linux-kernel
In-Reply-To: <20250723190308.5945-1-ltykernel@gmail.com>
From: Tianyu Lan <tiala@microsoft.com>
Hyper-V doesn't support auto-eoi with Secure AVIC.
So set the HV_DEPRECATING_AEOI_RECOMMENDED flag
to force writing the EIO register after handling an interrupt.
Signed-off-by: Tianyu Lan <tiala@microsoft.com>
---
arch/x86/kernel/cpu/mshyperv.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/arch/x86/kernel/cpu/mshyperv.c b/arch/x86/kernel/cpu/mshyperv.c
index c78f860419d6..8f029650f16c 100644
--- a/arch/x86/kernel/cpu/mshyperv.c
+++ b/arch/x86/kernel/cpu/mshyperv.c
@@ -463,6 +463,8 @@ static void __init ms_hyperv_init_platform(void)
ms_hyperv.max_vp_index, ms_hyperv.max_lp_index);
hv_identify_partition_type();
+ if (cc_platform_has(CC_ATTR_SNP_SECURE_AVIC))
+ ms_hyperv.hints |= HV_DEPRECATING_AEOI_RECOMMENDED;
if (ms_hyperv.hints & HV_X64_HYPERV_NESTED) {
hv_nested = true;
--
2.25.1
^ permalink raw reply related
* [RFC PATCH V3 2/4] Drivers: hv: Allow vmbus message synic interrupt injected from Hyper-V
From: Tianyu Lan @ 2025-07-23 19:03 UTC (permalink / raw)
To: kys, haiyangz, wei.liu, decui, tglx, mingo, bp, dave.hansen, x86,
hpa, arnd, Neeraj.Upadhyay, kvijayab
Cc: Tianyu Lan, linux-arch, linux-hyperv, linux-kernel
In-Reply-To: <20250723190308.5945-1-ltykernel@gmail.com>
From: Tianyu Lan <tiala@microsoft.com>
When Secure AVIC is enabled, VMBus driver should
call x2apic Secure AVIC interface to allow Hyper-V
to inject VMBus message interrupt.
Signed-off-by: Tianyu Lan <tiala@microsoft.com>
---
Change since v3
- Add hv_enable_coco_interrupt() as wrapper
of apic_update_vector()
arch/x86/hyperv/hv_apic.c | 5 +++++
drivers/hv/hv.c | 2 ++
drivers/hv/hv_common.c | 5 +++++
include/asm-generic/mshyperv.h | 1 +
4 files changed, 13 insertions(+)
diff --git a/arch/x86/hyperv/hv_apic.c b/arch/x86/hyperv/hv_apic.c
index 1c48396e5389..dd6829440ea2 100644
--- a/arch/x86/hyperv/hv_apic.c
+++ b/arch/x86/hyperv/hv_apic.c
@@ -53,6 +53,11 @@ static void hv_apic_icr_write(u32 low, u32 id)
wrmsrq(HV_X64_MSR_ICR, reg_val);
}
+void hv_enable_coco_interrupt(unsigned int cpu, unsigned int vector, bool set)
+{
+ apic_update_vector(cpu, vector, set);
+}
+
static u32 hv_apic_read(u32 reg)
{
u32 reg_val, hi;
diff --git a/drivers/hv/hv.c b/drivers/hv/hv.c
index 308c8f279df8..2aafe8946e5b 100644
--- a/drivers/hv/hv.c
+++ b/drivers/hv/hv.c
@@ -20,6 +20,7 @@
#include <linux/interrupt.h>
#include <clocksource/hyperv_timer.h>
#include <asm/mshyperv.h>
+#include <asm/apic.h>
#include <linux/set_memory.h>
#include "hyperv_vmbus.h"
@@ -310,6 +311,7 @@ void hv_synic_enable_regs(unsigned int cpu)
if (vmbus_irq != -1)
enable_percpu_irq(vmbus_irq, 0);
shared_sint.as_uint64 = hv_get_msr(HV_MSR_SINT0 + VMBUS_MESSAGE_SINT);
+ hv_enable_coco_interrupt(cpu, vmbus_interrupt, true);
shared_sint.vector = vmbus_interrupt;
shared_sint.masked = false;
diff --git a/drivers/hv/hv_common.c b/drivers/hv/hv_common.c
index 49898d10faff..0f024ab3d360 100644
--- a/drivers/hv/hv_common.c
+++ b/drivers/hv/hv_common.c
@@ -716,6 +716,11 @@ u64 __weak hv_tdx_hypercall(u64 control, u64 param1, u64 param2)
}
EXPORT_SYMBOL_GPL(hv_tdx_hypercall);
+void __weak hv_enable_coco_interrupt(unsigned int cpu, unsigned int vector, bool set)
+{
+}
+EXPORT_SYMBOL_GPL(hv_enable_coco_interrupt);
+
void hv_identify_partition_type(void)
{
/* Assume guest role */
diff --git a/include/asm-generic/mshyperv.h b/include/asm-generic/mshyperv.h
index a729b77983fa..7907c9878369 100644
--- a/include/asm-generic/mshyperv.h
+++ b/include/asm-generic/mshyperv.h
@@ -333,6 +333,7 @@ bool hv_is_isolation_supported(void);
bool hv_isolation_type_snp(void);
u64 hv_ghcb_hypercall(u64 control, void *input, void *output, u32 input_size);
u64 hv_tdx_hypercall(u64 control, u64 param1, u64 param2);
+void hv_enable_coco_interrupt(unsigned int cpu, unsigned int vector, bool set);
void hyperv_cleanup(void);
bool hv_query_ext_cap(u64 cap_query);
void hv_setup_dma_ops(struct device *dev, bool coherent);
--
2.25.1
^ permalink raw reply related
* [RFC PATCH V3 1/4] x86/Hyper-V: Not use hv apic driver when Secure AVIC is available
From: Tianyu Lan @ 2025-07-23 19:03 UTC (permalink / raw)
To: kys, haiyangz, wei.liu, decui, tglx, mingo, bp, dave.hansen, x86,
hpa, arnd, Neeraj.Upadhyay, kvijayab
Cc: Tianyu Lan, linux-arch, linux-hyperv, linux-kernel
In-Reply-To: <20250723190308.5945-1-ltykernel@gmail.com>
From: Tianyu Lan <tiala@microsoft.com>
When Secure AVIC is available, AMD x2apic Secure
AVIC driver should be selected and return directly
in the hv_apic_init().
Signed-off-by: Tianyu Lan <tiala@microsoft.com>
---
arch/x86/hyperv/hv_apic.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/arch/x86/hyperv/hv_apic.c b/arch/x86/hyperv/hv_apic.c
index bfde0a3498b9..1c48396e5389 100644
--- a/arch/x86/hyperv/hv_apic.c
+++ b/arch/x86/hyperv/hv_apic.c
@@ -293,6 +293,9 @@ static void hv_send_ipi_self(int vector)
void __init hv_apic_init(void)
{
+ if (cc_platform_has(CC_ATTR_SNP_SECURE_AVIC))
+ return;
+
if (ms_hyperv.hints & HV_X64_CLUSTER_IPI_RECOMMENDED) {
pr_info("Hyper-V: Using IPI hypercalls\n");
/*
--
2.25.1
^ permalink raw reply related
* [RFC PATCH V3 0/4] x86/Hyper-V: Add AMD Secure AVIC for Hyper-V platform
From: Tianyu Lan @ 2025-07-23 19:03 UTC (permalink / raw)
To: kys, haiyangz, wei.liu, decui, tglx, mingo, bp, dave.hansen, x86,
hpa, arnd, Neeraj.Upadhyay, kvijayab
Cc: Tianyu Lan, linux-arch, linux-hyperv, linux-kernel
From: Tianyu Lan <tiala@microsoft.com>
Secure AVIC is a new hardware feature in the AMD64
architecture to allow SEV-SNP guests to prevent the
hypervisor from generating unexpected interrupts to
a vCPU or otherwise violate architectural assumptions
around APIC behavior.
Each vCPU has a guest-allocated APIC backing page of
size 4K, which maintains APIC state for that vCPU.
APIC backing page's ALLOWED_IRR field indicates the
interrupt vectors which the guest allows the hypervisor
to send.
This patchset is to enable the feature for Hyper-V
platform. Patch "Drivers: hv: Allow vmbus message
synic interrupt injected from Hyper-V" is to expose
new fucntion hv_enable_coco_interrupt() and device
driver and arch code may update AVIC backing page
ALLOWED_IRR field to allow Hyper-V inject associated
vector.
This patchset is based on the AMD patchset "AMD: Add
Secure AVIC Guest Support"
https://lkml.org/lkml/2025/6/10/1579
Change since v2:
- Add hv_enable_coco_interrupt() as wrapper
of apic_update_vector()
- Re-work change logs
Change since v1:
- Remove the check of Secure AVIC when set APIC backing page
- Use apic_update_vector() instead of exposing new interface
from Secure AVIC driver to update APIC backing page and allow
associated interrupt to be injected by hypervisor.
Tianyu Lan (4):
x86/Hyper-V: Not use hv apic driver when Secure AVIC is available
drivers/hv: Allow vmbus message synic interrupt injected from Hyper-V
x86/Hyper-V: Not use auto-eoi when Secure AVIC is available
x86/hyperv: Allow Hyper-V to inject STIMER0 interrupts
arch/x86/hyperv/hv_apic.c | 8 ++++++++
arch/x86/hyperv/hv_init.c | 7 +++++++
arch/x86/kernel/cpu/mshyperv.c | 2 ++
drivers/hv/hv.c | 2 ++
drivers/hv/hv_common.c | 5 +++++
include/asm-generic/mshyperv.h | 1 +
6 files changed, 25 insertions(+)
--
2.25.1
^ permalink raw reply
* Re: [PATCH] net: mana: Use page pool fragments for RX buffers instead of full pages to improve memory efficiency and throughput.
From: Dipayaan Roy @ 2025-07-23 18:57 UTC (permalink / raw)
To: Saurabh Singh Sengar
Cc: kuba, kys, haiyangz, wei.liu, decui, andrew+netdev, davem,
edumazet, pabeni, longli, kotaranov, horms, ast, daniel, hawk,
john.fastabend, sdf, lorenzo, michal.kubiak, ernis, shradhagupta,
shirazsaleem, rosenp, netdev, linux-hyperv, linux-rdma, bpf
In-Reply-To: <20250723073804.GA19640@linuxonhyperv3.guj3yctzbm1etfxqx2vob5hsef.xx.internal.cloudapp.net>
Thanks Saurabh,
I am sending a v2.
Regards
Dipayaan Roy
On Wed, Jul 23, 2025 at 12:38:04AM -0700, Saurabh Singh Sengar wrote:
> On Mon, Jul 21, 2025 at 03:14:17AM -0700, Dipayaan Roy wrote:
> > This patch enhances RX buffer handling in the mana driver by allocating
> > pages from a page pool and slicing them into MTU-sized fragments, rather
> > than dedicating a full page per packet. This approach is especially
> > beneficial on systems with 64KB page sizes.
> >
> > Key improvements:
> >
> > - Proper integration of page pool for RX buffer allocations.
> > - MTU-sized buffer slicing to improve memory utilization.
> > - Reduce overall per Rx queue memory footprint.
> > - Automatic fallback to full-page buffers when:
> > * Jumbo frames are enabled (MTU > PAGE_SIZE / 2).
> > * The XDP path is active, to avoid complexities with fragment reuse.
> > - Removal of redundant pre-allocated RX buffers used in scenarios like MTU
> > changes, ensuring consistency in RX buffer allocation.
> >
> > Testing on VMs with 64KB pages shows around 200% throughput improvement.
> > Memory efficiency is significantly improved due to reduced wastage in page
> > allocations. Example: We are now able to fit 35 Rx buffers in a single 64KB
> > page for MTU size of 1500, instead of 1 Rx buffer per page previously.
> >
> > Tested:
> >
> > - iperf3, iperf2, and nttcp benchmarks.
> > - Jumbo frames with MTU 9000.
> > - Native XDP programs (XDP_PASS, XDP_DROP, XDP_TX, XDP_REDIRECT) for
> > testing the driver???s XDP path.
> > - Page leak detection (kmemleak).
> > - Driver load/unload, reboot, and stress scenarios.
> >
> > Signed-off-by: Dipayaan Roy <dipayanroy@linux.microsoft.com>
> > ---
> > .../net/ethernet/microsoft/mana/mana_bpf.c | 22 +-
> > drivers/net/ethernet/microsoft/mana/mana_en.c | 284 ++++++------------
> > .../ethernet/microsoft/mana/mana_ethtool.c | 13 -
> > include/net/mana/mana.h | 13 +-
> > 4 files changed, 115 insertions(+), 217 deletions(-)
> >
> > diff --git a/drivers/net/ethernet/microsoft/mana/mana_bpf.c b/drivers/net/ethernet/microsoft/mana/mana_bpf.c
> > index d30721d4516f..96813b6c184f 100644
> > --- a/drivers/net/ethernet/microsoft/mana/mana_bpf.c
> > +++ b/drivers/net/ethernet/microsoft/mana/mana_bpf.c
> > @@ -174,6 +174,7 @@ static int mana_xdp_set(struct net_device *ndev, struct bpf_prog *prog,
> > struct mana_port_context *apc = netdev_priv(ndev);
> > struct bpf_prog *old_prog;
> > struct gdma_context *gc;
> > + int err;
> >
> > gc = apc->ac->gdma_dev->gdma_context;
> >
> > @@ -198,14 +199,33 @@ static int mana_xdp_set(struct net_device *ndev, struct bpf_prog *prog,
> > if (old_prog)
> > bpf_prog_put(old_prog);
> >
> > - if (apc->port_is_up)
> > + if (apc->port_is_up) {
> > + /* Re-create rxq's after xdp prog was loaded or unloaded.
> > + * Ex: re create rxq's to switch from full pages to smaller
> > + * size page fragments when xdp prog is unloaded and vice-versa.
> > + */
> > +
> > + err = mana_detach(ndev, false);
> > + if (err) {
> > + netdev_err(ndev, "mana_detach failed at xdp set: %d\n", err);
> > + goto out;
>
> You should return err. At out we are always returning 0 which is wrong.
>
> > + }
> > +
> > + err = mana_attach(ndev);
> > + if (err) {
> > + netdev_err(ndev, "mana_attach failed at xdp set: %d\n", err);
> > + goto out;
>
> same here.
>
> > + }
> > +
> > mana_chn_setxdp(apc, prog);
> > + }
> >
> > if (prog)
> > ndev->max_mtu = MANA_XDP_MTU_MAX;
> > else
> > ndev->max_mtu = gc->adapter_mtu - ETH_HLEN;
> >
> > +out:
> > return 0;
> > }
> >
> > diff --git a/drivers/net/ethernet/microsoft/mana/mana_en.c b/drivers/net/ethernet/microsoft/mana/mana_en.c
> > index a7973651ae51..a474c59c907c 100644
> > --- a/drivers/net/ethernet/microsoft/mana/mana_en.c
> > +++ b/drivers/net/ethernet/microsoft/mana/mana_en.c
> > @@ -548,171 +548,45 @@ static u16 mana_select_queue(struct net_device *ndev, struct sk_buff *skb,
> > return txq;
> > }
> >
> > -/* Release pre-allocated RX buffers */
> > -void mana_pre_dealloc_rxbufs(struct mana_port_context *mpc)
> > -{
> > - struct device *dev;
> > - int i;
> > -
> > - dev = mpc->ac->gdma_dev->gdma_context->dev;
> > -
> > - if (!mpc->rxbufs_pre)
> > - goto out1;
> > -
> > - if (!mpc->das_pre)
> > - goto out2;
> > -
> > - while (mpc->rxbpre_total) {
> > - i = --mpc->rxbpre_total;
> > - dma_unmap_single(dev, mpc->das_pre[i], mpc->rxbpre_datasize,
> > - DMA_FROM_DEVICE);
> > - put_page(virt_to_head_page(mpc->rxbufs_pre[i]));
> > - }
> > -
> > - kfree(mpc->das_pre);
> > - mpc->das_pre = NULL;
> > -
> > -out2:
> > - kfree(mpc->rxbufs_pre);
> > - mpc->rxbufs_pre = NULL;
> > -
> > -out1:
> > - mpc->rxbpre_datasize = 0;
> > - mpc->rxbpre_alloc_size = 0;
> > - mpc->rxbpre_headroom = 0;
> > -}
> > -
> > -/* Get a buffer from the pre-allocated RX buffers */
> > -static void *mana_get_rxbuf_pre(struct mana_rxq *rxq, dma_addr_t *da)
> > -{
> > - struct net_device *ndev = rxq->ndev;
> > - struct mana_port_context *mpc;
> > - void *va;
> > -
> > - mpc = netdev_priv(ndev);
> > -
> > - if (!mpc->rxbufs_pre || !mpc->das_pre || !mpc->rxbpre_total) {
> > - netdev_err(ndev, "No RX pre-allocated bufs\n");
> > - return NULL;
> > - }
> > -
> > - /* Check sizes to catch unexpected coding error */
> > - if (mpc->rxbpre_datasize != rxq->datasize) {
> > - netdev_err(ndev, "rxbpre_datasize mismatch: %u: %u\n",
> > - mpc->rxbpre_datasize, rxq->datasize);
> > - return NULL;
> > - }
> > -
> > - if (mpc->rxbpre_alloc_size != rxq->alloc_size) {
> > - netdev_err(ndev, "rxbpre_alloc_size mismatch: %u: %u\n",
> > - mpc->rxbpre_alloc_size, rxq->alloc_size);
> > - return NULL;
> > - }
> > -
> > - if (mpc->rxbpre_headroom != rxq->headroom) {
> > - netdev_err(ndev, "rxbpre_headroom mismatch: %u: %u\n",
> > - mpc->rxbpre_headroom, rxq->headroom);
> > - return NULL;
> > - }
> > -
> > - mpc->rxbpre_total--;
> > -
> > - *da = mpc->das_pre[mpc->rxbpre_total];
> > - va = mpc->rxbufs_pre[mpc->rxbpre_total];
> > - mpc->rxbufs_pre[mpc->rxbpre_total] = NULL;
> > -
> > - /* Deallocate the array after all buffers are gone */
> > - if (!mpc->rxbpre_total)
> > - mana_pre_dealloc_rxbufs(mpc);
> > -
> > - return va;
> > -}
> > -
> > /* Get RX buffer's data size, alloc size, XDP headroom based on MTU */
> > -static void mana_get_rxbuf_cfg(int mtu, u32 *datasize, u32 *alloc_size,
> > - u32 *headroom)
> > +static void mana_get_rxbuf_cfg(struct mana_port_context *apc,
> > + int mtu, u32 *datasize, u32 *alloc_size,
> > + u32 *headroom, u32 *frag_count)
> > {
> > - if (mtu > MANA_XDP_MTU_MAX)
> > - *headroom = 0; /* no support for XDP */
> > - else
> > - *headroom = XDP_PACKET_HEADROOM;
> > -
> > - *alloc_size = SKB_DATA_ALIGN(mtu + MANA_RXBUF_PAD + *headroom);
> > -
> > - /* Using page pool in this case, so alloc_size is PAGE_SIZE */
> > - if (*alloc_size < PAGE_SIZE)
> > - *alloc_size = PAGE_SIZE;
> > -
> > + /* Calculate datasize first (consistent across all cases) */
> > *datasize = mtu + ETH_HLEN;
> > -}
> > -
> > -int mana_pre_alloc_rxbufs(struct mana_port_context *mpc, int new_mtu, int num_queues)
> > -{
> > - struct device *dev;
> > - struct page *page;
> > - dma_addr_t da;
> > - int num_rxb;
> > - void *va;
> > - int i;
> > -
> > - mana_get_rxbuf_cfg(new_mtu, &mpc->rxbpre_datasize,
> > - &mpc->rxbpre_alloc_size, &mpc->rxbpre_headroom);
> > -
> > - dev = mpc->ac->gdma_dev->gdma_context->dev;
> > -
> > - num_rxb = num_queues * mpc->rx_queue_size;
> > -
> > - WARN(mpc->rxbufs_pre, "mana rxbufs_pre exists\n");
> > - mpc->rxbufs_pre = kmalloc_array(num_rxb, sizeof(void *), GFP_KERNEL);
> > - if (!mpc->rxbufs_pre)
> > - goto error;
> >
> > - mpc->das_pre = kmalloc_array(num_rxb, sizeof(dma_addr_t), GFP_KERNEL);
> > - if (!mpc->das_pre)
> > - goto error;
> > -
> > - mpc->rxbpre_total = 0;
> > -
> > - for (i = 0; i < num_rxb; i++) {
> > - page = dev_alloc_pages(get_order(mpc->rxbpre_alloc_size));
> > - if (!page)
> > - goto error;
> > -
> > - va = page_to_virt(page);
> > -
> > - da = dma_map_single(dev, va + mpc->rxbpre_headroom,
> > - mpc->rxbpre_datasize, DMA_FROM_DEVICE);
> > - if (dma_mapping_error(dev, da)) {
> > - put_page(page);
> > - goto error;
> > + /* For xdp and jumbo frames make sure only one packet fits per page */
> > + if (((mtu + MANA_RXBUF_PAD) > PAGE_SIZE / 2) || rcu_access_pointer(apc->bpf_prog)) {
> > + if (rcu_access_pointer(apc->bpf_prog)) {
> > + *headroom = XDP_PACKET_HEADROOM;
> > + *alloc_size = PAGE_SIZE;
> > + } else {
> > + *headroom = 0; /* no support for XDP */
> > + *alloc_size = SKB_DATA_ALIGN(mtu + MANA_RXBUF_PAD + *headroom);
> > }
> >
> > - mpc->rxbufs_pre[i] = va;
> > - mpc->das_pre[i] = da;
> > - mpc->rxbpre_total = i + 1;
> > + *frag_count = 1;
> > + return;
> > }
> >
> > - return 0;
> > + /* Standard MTU case - optimize for multiple packets per page */
> > + *headroom = 0;
> >
> > -error:
> > - netdev_err(mpc->ndev, "Failed to pre-allocate RX buffers for %d queues\n", num_queues);
> > - mana_pre_dealloc_rxbufs(mpc);
> > - return -ENOMEM;
> > + /* Calculate base buffer size needed */
> > + u32 len = SKB_DATA_ALIGN(mtu + MANA_RXBUF_PAD + *headroom);
> > + u32 buf_size = ALIGN(len, MANA_RX_FRAG_ALIGNMENT);
>
> its good to have all the declaration at start of function.
>
> > +
> > + /* Calculate how many packets can fit in a page */
> > + *frag_count = PAGE_SIZE / buf_size;
> > + *alloc_size = buf_size;
> > }
> >
> > static int mana_change_mtu(struct net_device *ndev, int new_mtu)
> > {
> > - struct mana_port_context *mpc = netdev_priv(ndev);
> > unsigned int old_mtu = ndev->mtu;
> > int err;
> >
> > - /* Pre-allocate buffers to prevent failure in mana_attach later */
> > - err = mana_pre_alloc_rxbufs(mpc, new_mtu, mpc->num_queues);
> > - if (err) {
> > - netdev_err(ndev, "Insufficient memory for new MTU\n");
> > - return err;
> > - }
> > -
> > err = mana_detach(ndev, false);
> > if (err) {
> > netdev_err(ndev, "mana_detach failed: %d\n", err);
> > @@ -728,7 +602,6 @@ static int mana_change_mtu(struct net_device *ndev, int new_mtu)
> > }
> >
> > out:
> > - mana_pre_dealloc_rxbufs(mpc);
> > return err;
> > }
> >
> > @@ -1841,8 +1714,11 @@ static void mana_rx_skb(void *buf_va, bool from_pool,
> >
> > drop:
> > if (from_pool) {
> > - page_pool_recycle_direct(rxq->page_pool,
> > - virt_to_head_page(buf_va));
> > + if (rxq->frag_count == 1)
> > + page_pool_recycle_direct(rxq->page_pool,
> > + virt_to_head_page(buf_va));
> > + else
> > + page_pool_free_va(rxq->page_pool, buf_va, true);
> > } else {
> > WARN_ON_ONCE(rxq->xdp_save_va);
> > /* Save for reuse */
> > @@ -1854,37 +1730,50 @@ static void mana_rx_skb(void *buf_va, bool from_pool,
> > return;
> > }
> >
> > -static void *mana_get_rxfrag(struct mana_rxq *rxq, struct device *dev,
> > - dma_addr_t *da, bool *from_pool)
> > +static void *mana_get_rxfrag(struct mana_rxq *rxq,
> > + struct device *dev, dma_addr_t *da, bool *from_pool)
> > {
> > struct page *page;
> > + u32 offset;
> > void *va;
> > -
> > *from_pool = false;
> >
> > - /* Reuse XDP dropped page if available */
> > - if (rxq->xdp_save_va) {
> > - va = rxq->xdp_save_va;
> > - rxq->xdp_save_va = NULL;
> > - } else {
> > - page = page_pool_dev_alloc_pages(rxq->page_pool);
> > - if (!page)
> > + /* Don't use fragments for jumbo frames or XDP (i.e when fragment = 1 per page) */
> > + if (rxq->frag_count == 1) {
> > + /* Reuse XDP dropped page if available */
> > + if (rxq->xdp_save_va) {
> > + va = rxq->xdp_save_va;
> > + rxq->xdp_save_va = NULL;
> > + } else {
> > + page = page_pool_dev_alloc_pages(rxq->page_pool);
> > + if (!page)
> > + return NULL;
> > +
> > + *from_pool = true;
> > + va = page_to_virt(page);
> > + }
> > +
> > + *da = dma_map_single(dev, va + rxq->headroom, rxq->datasize,
> > + DMA_FROM_DEVICE);
> > + if (dma_mapping_error(dev, *da)) {
> > + if (*from_pool)
> > + page_pool_put_full_page(rxq->page_pool, page, false);
> > + else
> > + put_page(virt_to_head_page(va));
> > +
> > return NULL;
> > + }
> >
> > - *from_pool = true;
> > - va = page_to_virt(page);
> > + return va;
> > }
> >
> > - *da = dma_map_single(dev, va + rxq->headroom, rxq->datasize,
> > - DMA_FROM_DEVICE);
> > - if (dma_mapping_error(dev, *da)) {
> > - if (*from_pool)
> > - page_pool_put_full_page(rxq->page_pool, page, false);
> > - else
> > - put_page(virt_to_head_page(va));
> > -
> > + page = page_pool_dev_alloc_frag(rxq->page_pool, &offset, rxq->alloc_size);
> > + if (!page)
> > return NULL;
> > - }
> > +
> > + va = page_to_virt(page) + offset;
> > + *da = page_pool_get_dma_addr(page) + offset + rxq->headroom;
> > + *from_pool = true;
> >
> > return va;
> > }
> > @@ -1901,9 +1790,9 @@ static void mana_refill_rx_oob(struct device *dev, struct mana_rxq *rxq,
> > va = mana_get_rxfrag(rxq, dev, &da, &from_pool);
> > if (!va)
> > return;
> > -
> > - dma_unmap_single(dev, rxoob->sgl[0].address, rxq->datasize,
> > - DMA_FROM_DEVICE);
> > + if (!rxoob->from_pool || rxq->frag_count == 1)
> > + dma_unmap_single(dev, rxoob->sgl[0].address, rxq->datasize,
> > + DMA_FROM_DEVICE);
> > *old_buf = rxoob->buf_va;
> > *old_fp = rxoob->from_pool;
> >
> > @@ -2314,15 +2203,19 @@ static void mana_destroy_rxq(struct mana_port_context *apc,
> > if (!rx_oob->buf_va)
> > continue;
> >
> > - dma_unmap_single(dev, rx_oob->sgl[0].address,
> > - rx_oob->sgl[0].size, DMA_FROM_DEVICE);
> > -
> > page = virt_to_head_page(rx_oob->buf_va);
> >
> > - if (rx_oob->from_pool)
> > - page_pool_put_full_page(rxq->page_pool, page, false);
> > - else
> > - put_page(page);
> > + if (rxq->frag_count == 1) {
> > + dma_unmap_single(dev, rx_oob->sgl[0].address, rx_oob->sgl[0].size,
> > + DMA_FROM_DEVICE);
> > +
> > + if (rx_oob->from_pool)
> > + page_pool_put_full_page(rxq->page_pool, page, false);
> > + else
> > + put_page(page);
> > + } else {
> > + page_pool_free_va(rxq->page_pool, rx_oob->buf_va, true);
> > + }
> >
> > rx_oob->buf_va = NULL;
> > }
> > @@ -2338,16 +2231,11 @@ static void mana_destroy_rxq(struct mana_port_context *apc,
> > static int mana_fill_rx_oob(struct mana_recv_buf_oob *rx_oob, u32 mem_key,
> > struct mana_rxq *rxq, struct device *dev)
> > {
> > - struct mana_port_context *mpc = netdev_priv(rxq->ndev);
> > bool from_pool = false;
> > dma_addr_t da;
> > void *va;
> >
> > - if (mpc->rxbufs_pre)
> > - va = mana_get_rxbuf_pre(rxq, &da);
> > - else
> > - va = mana_get_rxfrag(rxq, dev, &da, &from_pool);
> > -
> > + va = mana_get_rxfrag(rxq, dev, &da, &from_pool);
> > if (!va)
> > return -ENOMEM;
> >
> > @@ -2428,11 +2316,22 @@ static int mana_create_page_pool(struct mana_rxq *rxq, struct gdma_context *gc)
> > struct page_pool_params pprm = {};
> > int ret;
> >
> > - pprm.pool_size = mpc->rx_queue_size;
> > + pprm.pool_size = mpc->rx_queue_size / rxq->frag_count + 1;
> > pprm.nid = gc->numa_node;
> > pprm.napi = &rxq->rx_cq.napi;
> > pprm.netdev = rxq->ndev;
> > pprm.order = get_order(rxq->alloc_size);
> > + pprm.queue_idx = rxq->rxq_idx;
> > + pprm.dev = gc->dev;
> > +
> > + /* Let the page pool do the dma map when page sharing with multiple fragments
> > + * enabled for rx buffers.
> > + */
> > + if (rxq->frag_count > 1) {
> > + pprm.flags = PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV;
> > + pprm.max_len = PAGE_SIZE;
> > + pprm.dma_dir = DMA_FROM_DEVICE;
> > + }
> >
> > rxq->page_pool = page_pool_create(&pprm);
> >
> > @@ -2471,9 +2370,8 @@ static struct mana_rxq *mana_create_rxq(struct mana_port_context *apc,
> > rxq->rxq_idx = rxq_idx;
> > rxq->rxobj = INVALID_MANA_HANDLE;
> >
> > - mana_get_rxbuf_cfg(ndev->mtu, &rxq->datasize, &rxq->alloc_size,
> > - &rxq->headroom);
> > -
> > + mana_get_rxbuf_cfg(apc, ndev->mtu, &rxq->datasize, &rxq->alloc_size,
> > + &rxq->headroom, &rxq->frag_count);
> > /* Create page pool for RX queue */
> > err = mana_create_page_pool(rxq, gc);
> > if (err) {
> > diff --git a/drivers/net/ethernet/microsoft/mana/mana_ethtool.c b/drivers/net/ethernet/microsoft/mana/mana_ethtool.c
> > index a1afa75a9463..7ede03c74fb9 100644
> > --- a/drivers/net/ethernet/microsoft/mana/mana_ethtool.c
> > +++ b/drivers/net/ethernet/microsoft/mana/mana_ethtool.c
> > @@ -396,12 +396,6 @@ static int mana_set_channels(struct net_device *ndev,
> > unsigned int old_count = apc->num_queues;
> > int err;
> >
> > - err = mana_pre_alloc_rxbufs(apc, ndev->mtu, new_count);
> > - if (err) {
> > - netdev_err(ndev, "Insufficient memory for new allocations");
> > - return err;
> > - }
> > -
> > err = mana_detach(ndev, false);
> > if (err) {
> > netdev_err(ndev, "mana_detach failed: %d\n", err);
> > @@ -416,7 +410,6 @@ static int mana_set_channels(struct net_device *ndev,
> > }
> >
> > out:
> > - mana_pre_dealloc_rxbufs(apc);
> > return err;
> > }
> >
> > @@ -465,12 +458,7 @@ static int mana_set_ringparam(struct net_device *ndev,
> >
> > /* pre-allocating new buffers to prevent failures in mana_attach() later */
> > apc->rx_queue_size = new_rx;
> > - err = mana_pre_alloc_rxbufs(apc, ndev->mtu, apc->num_queues);
> > apc->rx_queue_size = old_rx;
> > - if (err) {
> > - netdev_err(ndev, "Insufficient memory for new allocations\n");
> > - return err;
> > - }
> >
> > err = mana_detach(ndev, false);
> > if (err) {
> > @@ -488,7 +476,6 @@ static int mana_set_ringparam(struct net_device *ndev,
> > apc->rx_queue_size = old_rx;
> > }
> > out:
> > - mana_pre_dealloc_rxbufs(apc);
> > return err;
> > }
> >
> > diff --git a/include/net/mana/mana.h b/include/net/mana/mana.h
> > index e1030a7d2daa..99a3847b0f9d 100644
> > --- a/include/net/mana/mana.h
> > +++ b/include/net/mana/mana.h
> > @@ -65,6 +65,8 @@ enum TRI_STATE {
> > #define MANA_STATS_RX_COUNT 5
> > #define MANA_STATS_TX_COUNT 11
> >
> > +#define MANA_RX_FRAG_ALIGNMENT 64
> > +
> > struct mana_stats_rx {
> > u64 packets;
> > u64 bytes;
> > @@ -328,6 +330,7 @@ struct mana_rxq {
> > u32 datasize;
> > u32 alloc_size;
> > u32 headroom;
> > + u32 frag_count;
> >
> > mana_handle_t rxobj;
> >
> > @@ -503,14 +506,6 @@ struct mana_port_context {
> > /* This points to an array of num_queues of RQ pointers. */
> > struct mana_rxq **rxqs;
> >
> > - /* pre-allocated rx buffer array */
> > - void **rxbufs_pre;
> > - dma_addr_t *das_pre;
> > - int rxbpre_total;
> > - u32 rxbpre_datasize;
> > - u32 rxbpre_alloc_size;
> > - u32 rxbpre_headroom;
> > -
> > struct bpf_prog *bpf_prog;
> >
> > /* Create num_queues EQs, SQs, SQ-CQs, RQs and RQ-CQs, respectively. */
> > @@ -574,8 +569,6 @@ int mana_query_link_cfg(struct mana_port_context *apc);
> > int mana_set_bw_clamp(struct mana_port_context *apc, u32 speed,
> > int enable_clamping);
> > void mana_query_phy_stats(struct mana_port_context *apc);
> > -int mana_pre_alloc_rxbufs(struct mana_port_context *apc, int mtu, int num_queues);
> > -void mana_pre_dealloc_rxbufs(struct mana_port_context *apc);
> >
> > extern const struct ethtool_ops mana_ethtool_ops;
> > extern struct dentry *mana_debugfs_root;
> > --
> > 2.43.0
>
>
> Rest looks good. After fixing above,
> Reviewed-by: Saurabh Sengar <ssengar@linux.microsoft.com>
^ permalink raw reply
* Re: [PATCH] net: mana: Use page pool fragments for RX buffers instead of full pages to improve memory efficiency and throughput.
From: Dipayaan Roy @ 2025-07-23 18:35 UTC (permalink / raw)
To: Simon Horman
Cc: kuba, kys, haiyangz, wei.liu, decui, andrew+netdev, davem,
edumazet, pabeni, longli, kotaranov, horms, ast, daniel, hawk,
john.fastabend, sdf, lorenzo, michal.kubiak, ernis, shradhagupta,
shirazsaleem, rosenp, netdev, linux-hyperv, linux-rdma, bpf,
dipayanroy, ssengar
In-Reply-To: <20250722111929.GE2459@horms.kernel.org>
Hi Simon,
Thank you for the review, I am sending
out v2 to address these comments.
Regards
Dipayaan Roy
On Tue, Jul 22, 2025 at 12:19:29PM +0100, Simon Horman wrote:
> On Mon, Jul 21, 2025 at 03:14:17AM -0700, Dipayaan Roy wrote:
> > This patch enhances RX buffer handling in the mana driver by allocating
> > pages from a page pool and slicing them into MTU-sized fragments, rather
> > than dedicating a full page per packet. This approach is especially
> > beneficial on systems with 64KB page sizes.
> >
> > Key improvements:
> >
> > - Proper integration of page pool for RX buffer allocations.
> > - MTU-sized buffer slicing to improve memory utilization.
> > - Reduce overall per Rx queue memory footprint.
> > - Automatic fallback to full-page buffers when:
> > * Jumbo frames are enabled (MTU > PAGE_SIZE / 2).
> > * The XDP path is active, to avoid complexities with fragment reuse.
> > - Removal of redundant pre-allocated RX buffers used in scenarios like MTU
> > changes, ensuring consistency in RX buffer allocation.
> >
> > Testing on VMs with 64KB pages shows around 200% throughput improvement.
> > Memory efficiency is significantly improved due to reduced wastage in page
> > allocations. Example: We are now able to fit 35 Rx buffers in a single 64KB
> > page for MTU size of 1500, instead of 1 Rx buffer per page previously.
> >
> > Tested:
> >
> > - iperf3, iperf2, and nttcp benchmarks.
> > - Jumbo frames with MTU 9000.
> > - Native XDP programs (XDP_PASS, XDP_DROP, XDP_TX, XDP_REDIRECT) for
> > testing the driver???s XDP path.
> > - Page leak detection (kmemleak).
> > - Driver load/unload, reboot, and stress scenarios.
> >
> > Signed-off-by: Dipayaan Roy <dipayanroy@linux.microsoft.com>
>
> Hi,
>
> Some minor feedback from my side.
>
> > diff --git a/drivers/net/ethernet/microsoft/mana/mana_en.c b/drivers/net/ethernet/microsoft/mana/mana_en.c
>
> ...
>
> > -int mana_pre_alloc_rxbufs(struct mana_port_context *mpc, int new_mtu, int num_queues)
> > -{
> > - struct device *dev;
> > - struct page *page;
> > - dma_addr_t da;
> > - int num_rxb;
> > - void *va;
> > - int i;
> > -
> > - mana_get_rxbuf_cfg(new_mtu, &mpc->rxbpre_datasize,
> > - &mpc->rxbpre_alloc_size, &mpc->rxbpre_headroom);
> > -
> > - dev = mpc->ac->gdma_dev->gdma_context->dev;
> > -
> > - num_rxb = num_queues * mpc->rx_queue_size;
> > -
> > - WARN(mpc->rxbufs_pre, "mana rxbufs_pre exists\n");
> > - mpc->rxbufs_pre = kmalloc_array(num_rxb, sizeof(void *), GFP_KERNEL);
> > - if (!mpc->rxbufs_pre)
> > - goto error;
> >
> > - mpc->das_pre = kmalloc_array(num_rxb, sizeof(dma_addr_t), GFP_KERNEL);
> > - if (!mpc->das_pre)
> > - goto error;
> > -
> > - mpc->rxbpre_total = 0;
> > -
> > - for (i = 0; i < num_rxb; i++) {
> > - page = dev_alloc_pages(get_order(mpc->rxbpre_alloc_size));
> > - if (!page)
> > - goto error;
> > -
> > - va = page_to_virt(page);
> > -
> > - da = dma_map_single(dev, va + mpc->rxbpre_headroom,
> > - mpc->rxbpre_datasize, DMA_FROM_DEVICE);
> > - if (dma_mapping_error(dev, da)) {
> > - put_page(page);
> > - goto error;
> > + /* For xdp and jumbo frames make sure only one packet fits per page */
> > + if (((mtu + MANA_RXBUF_PAD) > PAGE_SIZE / 2) || rcu_access_pointer(apc->bpf_prog)) {
>
> The line above seems to have unnecessary parentheses.
> And should be line wrapped to be 80 columns wide or less,
> as is still preferred by Networking code.
> The latter condition is flagged by checkpatch.pl --max-line-length=80
>
> if (mtu + MANA_RXBUF_PAD > PAGE_SIZE / 2 ||
> rcu_access_pointer(apc->bpf_prog)) {
>
> (The above is completely untested)
>
> Also, I am a little confused by the use of rcu_access_pointer()
> above and below, as bpf_prog does not seem to be managed by RCU.
>
> Flagged by Sparse.
>
> > + if (rcu_access_pointer(apc->bpf_prog)) {
> > + *headroom = XDP_PACKET_HEADROOM;
> > + *alloc_size = PAGE_SIZE;
> > + } else {
> > + *headroom = 0; /* no support for XDP */
> > + *alloc_size = SKB_DATA_ALIGN(mtu + MANA_RXBUF_PAD + *headroom);
> > }
> >
> > - mpc->rxbufs_pre[i] = va;
> > - mpc->das_pre[i] = da;
> > - mpc->rxbpre_total = i + 1;
> > + *frag_count = 1;
> > + return;
> > }
> >
> > - return 0;
> > + /* Standard MTU case - optimize for multiple packets per page */
> > + *headroom = 0;
> >
> > -error:
> > - netdev_err(mpc->ndev, "Failed to pre-allocate RX buffers for %d queues\n", num_queues);
> > - mana_pre_dealloc_rxbufs(mpc);
> > - return -ENOMEM;
> > + /* Calculate base buffer size needed */
> > + u32 len = SKB_DATA_ALIGN(mtu + MANA_RXBUF_PAD + *headroom);
> > + u32 buf_size = ALIGN(len, MANA_RX_FRAG_ALIGNMENT);
> > +
> > + /* Calculate how many packets can fit in a page */
> > + *frag_count = PAGE_SIZE / buf_size;
> > + *alloc_size = buf_size;
> > }
>
> ...
>
> > -static void *mana_get_rxfrag(struct mana_rxq *rxq, struct device *dev,
> > - dma_addr_t *da, bool *from_pool)
> > +static void *mana_get_rxfrag(struct mana_rxq *rxq,
> > + struct device *dev, dma_addr_t *da, bool *from_pool)
> > {
> > struct page *page;
> > + u32 offset;
> > void *va;
> > -
> > *from_pool = false;
> >
> > - /* Reuse XDP dropped page if available */
> > - if (rxq->xdp_save_va) {
> > - va = rxq->xdp_save_va;
> > - rxq->xdp_save_va = NULL;
> > - } else {
> > - page = page_pool_dev_alloc_pages(rxq->page_pool);
> > - if (!page)
> > + /* Don't use fragments for jumbo frames or XDP (i.e when fragment = 1 per page) */
> > + if (rxq->frag_count == 1) {
> > + /* Reuse XDP dropped page if available */
> > + if (rxq->xdp_save_va) {
> > + va = rxq->xdp_save_va;
> > + rxq->xdp_save_va = NULL;
> > + } else {
> > + page = page_pool_dev_alloc_pages(rxq->page_pool);
> > + if (!page)
> > + return NULL;
> > +
> > + *from_pool = true;
> > + va = page_to_virt(page);
> > + }
> > +
> > + *da = dma_map_single(dev, va + rxq->headroom, rxq->datasize,
> > + DMA_FROM_DEVICE);
> > + if (dma_mapping_error(dev, *da)) {
> > + if (*from_pool)
> > + page_pool_put_full_page(rxq->page_pool, page, false);
> > + else
> > + put_page(virt_to_head_page(va));
>
> The put logic above seems to appear in this patch
> more than once. IMHO a helper would be nice.
>
> > +
> > return NULL;
> > + }
> >
> > - *from_pool = true;
> > - va = page_to_virt(page);
> > + return va;
> > }
>
> ...
>
> --
> pw-bot: changes-requested
^ permalink raw reply
* Re: [PATCH] net: mana: Use page pool fragments for RX buffers instead of full pages to improve memory efficiency and throughput.
From: Dipayaan Roy @ 2025-07-23 18:22 UTC (permalink / raw)
To: Jacob Keller
Cc: kuba, kys, haiyangz, wei.liu, decui, andrew+netdev, davem,
edumazet, pabeni, longli, kotaranov, horms, ast, daniel, hawk,
john.fastabend, sdf, lorenzo, michal.kubiak, ernis, shradhagupta,
shirazsaleem, rosenp, netdev, linux-hyperv, linux-rdma, bpf
In-Reply-To: <1d18f7dc-eca5-4daa-98fb-5c8d20ef6ac4@intel.com>
Hi Jacob,
Thanks for the review, I tested this patch previously
for 4kb page size systems. The throughput improvement is around
10 to 15%. Also sligthly better memory utilization as for
typical 1500 MTU size we are able to fit 2 Rx buffer in
a single 4kb page as compared to 1 per page previously.
Thanks
Dipayaan Roy
On Mon, Jul 21, 2025 at 04:51:10PM -0700, Jacob Keller wrote:
>
>
> On 7/21/2025 3:14 AM, Dipayaan Roy wrote:
> > This patch enhances RX buffer handling in the mana driver by allocating
> > pages from a page pool and slicing them into MTU-sized fragments, rather
> > than dedicating a full page per packet. This approach is especially
> > beneficial on systems with 64KB page sizes.
> >
> > Key improvements:
> >
> > - Proper integration of page pool for RX buffer allocations.
> > - MTU-sized buffer slicing to improve memory utilization.
> > - Reduce overall per Rx queue memory footprint.
> > - Automatic fallback to full-page buffers when:
> > * Jumbo frames are enabled (MTU > PAGE_SIZE / 2).
> > * The XDP path is active, to avoid complexities with fragment reuse.
> > - Removal of redundant pre-allocated RX buffers used in scenarios like MTU
> > changes, ensuring consistency in RX buffer allocation.
> >
> > Testing on VMs with 64KB pages shows around 200% throughput improvement.
> > Memory efficiency is significantly improved due to reduced wastage in page
> > allocations. Example: We are now able to fit 35 Rx buffers in a single 64KB
> > page for MTU size of 1500, instead of 1 Rx buffer per page previously.
> >
> Nice to see such improvements while also reducing the overall driver
> code footprint!
>
> Do you happen to have numbers on some of the smaller page sizes? I'm not
> sure how common 64KB is, but it seems like this would still be quite
> beneficial even if you had 4K or 8K pages when operating at 1500 MTU.
>
> Reviewed-by: Jacob Keller <jacob.e.keller@intel.com>
^ permalink raw reply
* Re: [PATCH RESEND] netvsc: transfer lower device max tso size
From: Jakub Kicinski @ 2025-07-23 15:05 UTC (permalink / raw)
To: Jason Wang
Cc: Cindy Lu, K. Y. Srinivasan, Haiyang Zhang, Wei Liu, Dexuan Cui,
Andrew Lunn, David S. Miller, Eric Dumazet, Paolo Abeni,
Simon Horman, Michael Kelley, Shradha Gupta, Kees Cook,
Stanislav Fomichev, Kuniyuki Iwashima, Alexander Lobakin,
Guillaume Nault, Joe Damato, Ahmed Zaki,
open list:Hyper-V/Azure CORE AND DRIVERS,
open list:NETWORKING DRIVERS, open list
In-Reply-To: <CACGkMEtEvkSaYP1s+jq-3RPrX_GAr1gQ+b=b4oytw9_dGnSc_w@mail.gmail.com>
On Wed, 23 Jul 2025 14:00:47 +0800 Jason Wang wrote:
> > > But this fixes a real problem, otherwise nested VM performance will be
> > > broken due to the GSO software segmentation.
> >
> > Perhaps, possibly, a migration plan can be devised, away from the
> > netvsc model, so we don't have to deal with nuggets of joy like:
> > https://lore.kernel.org/all/1752870014-28909-1-git-send-email-haiyangz@linux.microsoft.com/
>
> Btw, if I understand this correctly. This is for future development so
> it's not a blocker for this patch?
Not a blocker, I'm just giving an example of the netvsc auto-weirdness
being a source of tech debt and bugs. Commit d7501e076d859d is another
recent one off the top of my head. IIUC systemd-networkd is broadly
deployed now. It'd be great if there was some migration plan for moving
this sort of VM auto-bonding to user space (with the use of the common
bonding driver, not each hypervisor rolling its own).
^ permalink raw reply
* RE: [PATCH v5 2/2] Drivers: hv: Introduce mshv_vtl driver
From: Michael Kelley @ 2025-07-23 14:13 UTC (permalink / raw)
To: Naman Jain, K . Y . Srinivasan, Haiyang Zhang, Wei Liu,
Dexuan Cui
Cc: Roman Kisel, Anirudh Rayabharam, Saurabh Sengar,
Stanislav Kinsburskii, Nuno Das Neves, ALOK TIWARI,
linux-kernel@vger.kernel.org, linux-hyperv@vger.kernel.org
In-Reply-To: <829f0716-d2f2-449f-8f49-397a46046c69@linux.microsoft.com>
From: Naman Jain <namjain@linux.microsoft.com> Sent: Wednesday, July 23, 2025 2:59 AM
>
> On 7/22/2025 10:39 PM, Michael Kelley wrote:
> > From: Naman Jain <namjain@linux.microsoft.com> Sent: Tuesday, July 22, 2025 4:09 AM
> >>
> >> On 7/18/2025 8:37 PM, Michael Kelley wrote:
> >>> From: Naman Jain <namjain@linux.microsoft.com> Sent: Thursday, July 17, 2025 9:36 PM
> >>>>
> >>>> On 7/9/2025 10:49 PM, Michael Kelley wrote:
> >>>>> From: Naman Jain <namjain@linux.microsoft.com> Sent: Wednesday, June 11, 2025 12:27 AM
> >>>
> >>> [snip]
> >>>
> >>>>
> >>>>> Separately, "allow_bitmap" size is 64K bytes, or 512K bits. Is that the
> >>>>> correct size? From looking at mshv_vtl_hvcall_is_allowed(), I think this
> >>>>> bitmap is indexed by the HV call code, which is a 16 bit value. So you
> >>>>> only need 64K bits, and the size is too big by a factor of 8. In any case,
> >>>>> it seems like the size should not be expressed in terms of PAGE_SIZE.
> >>>>
> >>>> There are HVcall codes which are of type u16. So max(HVcall code) =
> >>>> 0xffff.
> >>>>
> >>>> For every HVcall that needs to be allowed, we are saving HVcall code
> >>>> info in a bitmap in below fashion:
> >>>> if x = HVCall code and bitmap is an array of u64, of size
> >>>> ((0xffff/64=1023) + 1)
> >>>>
> >>>> bitmap[x / 64] = (u64)1 << (x%64);
> >>>>
> >>>> Later on in mshv_vtl_hvcall_is_allowed(), we calculate the array index
> >>>> by dividing it by 64, and then see if call_code/64 bit is set.
> >>>
> >>> I didn't add comments in mshv_vtl_hvcall_is_allowed(), but that code
> >>> can be simplified by recognizing that the Linux kernel bitmap utilities
> >>> can operate on bitmaps that are much larger than just 64 bits. Let's
> >>> assume that the allow_bitmap field in struct mshv_vtl_hvcall_fds has
> >>> 64K bits, regardless of whether it is declared as an array of u64,
> >>> an array of u16, or an array of u8. Then mshv_vtl_hvcall_is_allowed()
> >>> can be implemented as a single line:
> >>>
> >>> return test_bit(call_code, fd->allow_bitmap);
> >>>
> >>> There's no need to figure out which array element contains the bit,
> >>> or to construct a mask to select that particular bit in the array element.
> >>> And since call_code is a u16, test_bit won't access outside the allocated
> >>> 64K bits.
> >>>
> >>
> >> I understood it now. This works and is much better. Will incorporate it
> >> in next patch.
> >>
> >>>>
> >>>> Coming to size of allow_bitmap[], it is independent of PAGE_SIZE, and
> >>>> can be safely initialized to 1024 (reducing by a factor of 8).
> >>>> bitmap_size's maximum value is going to be 1024 in current
> >>>> implementation, picking u64 was not mandatory, u16 will also work. Also,
> >>>> item_index is also u16, so I should make bitmap_size as u16.
> >>>
> >>> The key question for me is whether bitmap_size describes the number
> >>> of bits in allow_bitmap, or whether it describes the number of array
> >>> elements in the declared allow_bitmap array. It's more typical to
> >>> describe a bitmap size as the number of bits. Then the value is
> >>> independent of the array element size, as the array element size
> >>> usually doesn't really matter anyway if using the Linux kernel's
> >>> bitmap utilities. The array element size only matters in allocating
> >>> the correct amount of space is for whatever number of bits are
> >>> needed in the bitmap.
> >>>
> >>
> >> I tried to put your suggestions in code. Please let me know if below
> >> works. I tested this and it works. Just that, I am a little hesitant in
> >> changing things on Userspace side of it, which passes these parameters
> >> in IOCTL. This way, userspace remains the same, the confusion of names
> >> may go away, and the code becomes simpler.
> >
> > Yes, I suspected there might be constraints due to not wanting
> > to disturb existing user space code.
> >
> >>
> >> --- a/include/uapi/linux/mshv.h
> >> +++ b/include/uapi/linux/mshv.h
> >> @@ -332,7 +332,7 @@ struct mshv_vtl_set_poll_file {
> >> };
> >>
> >> struct mshv_vtl_hvcall_setup {
> >> - __u64 bitmap_size;
> >> + __u64 bitmap_array_size;
> >> __u64 allow_bitmap_ptr; /* pointer to __u64 */
> >> };
> >>
> >>
> >> --- a/drivers/hv/mshv_vtl_main.c
> >> +++ b/drivers/hv/mshv_vtl_main.c
> >> @@ -52,10 +52,12 @@ static bool has_message;
> >> static struct eventfd_ctx *flag_eventfds[HV_EVENT_FLAGS_COUNT];
> >> static DEFINE_MUTEX(flag_lock);
> >> static bool __read_mostly mshv_has_reg_page;
> >> -#define MAX_BITMAP_SIZE 1024
> >> +
> >> +/* hvcall code is of type u16, allocate a bitmap of size (1 << 16) to accomodate it */
> >
> > s/accomodate/accommodate/
>
> Acked.
>
> >
> >> +#define MAX_BITMAP_SIZE (1 << 16)
> >>
> >> struct mshv_vtl_hvcall_fd {
> >> - u64 allow_bitmap[MAX_BITMAP_SIZE];
> >> + u64 allow_bitmap[MAX_BITMAP_SIZE / 64];
> >
> > OK, this seems like a reasonable approach to get the correct
> > number of bits.
> >
> >> bool allow_map_initialized;
> >> /*
> >> * Used to protect hvcall setup in IOCTLs
> >>
> >> @@ -1204,12 +1207,12 @@ static int mshv_vtl_hvcall_do_setup(struct mshv_vtl_hvcall_fd *fd,
> >> sizeof(struct mshv_vtl_hvcall_setup))) {
> >> return -EFAULT;
> >> }
> >> - if (hvcall_setup.bitmap_size > ARRAY_SIZE(fd->allow_bitmap)) {
> >> + if (hvcall_setup.bitmap_array_size > ARRAY_SIZE(fd->allow_bitmap)) {
> >> return -EINVAL;
> >> }
> >> if (copy_from_user(&fd->allow_bitmap,
> >> (void __user *)hvcall_setup.allow_bitmap_ptr,
> >> - hvcall_setup.bitmap_size)) {
> >> + hvcall_setup.bitmap_array_size)) {
> >
> > This still doesn't work. copy_from_user() expects a byte count as its
> > 3rd argument. If we have 64K bits in the bitmap, that means 8K bytes,
> > and 1K for bitmap_array_size. So the value of the 3rd argument
> > here is 1K, and you'll be copying 1K bytes when you want to be
> > copying 8K bytes. The 3rd argument needs to be:
> >
> > hv_call_setup.bitmap_array_size * sizeof(u64)
> >
> > It's a bit messy to have to add this multiply, and in the bigger
> > picture it might be cleaner to have the bitmap_array be
> > declared in units of u8 instead of u64, but that would affect
> > user space in a way that you probably want to avoid.
> >
> > Have you checked that user space is passing in the correct values
> > to the ioctl? If the kernel side code is wrong, user space might be
> > wrong as well. And if user space is wrong, then you might have
> > the flexibility to change everything to use u8 instead of u64.
> >
>
>
> This is correct in Userspace:
> https://github.com/microsoft/openvmm/blob/main/openhcl/hcl/src/ioctl.rs#L905
>
I'm not really Rust literate, but it does look like userspace is
setting the bitmap_size to the number of *bytes*. User space has
used u64 as its underlying type, which is OK as long as the ioctl
API is understood to take a byte count in the bitmap_size field,
and user space populates that field with a byte count, which it does.
> This was wrong in kernel code internally from the beginning. This did
> not get caught because we took a large array size (2 * PAGE_SIZE) which
> never failed the check
> (hvcall_setup.bitmap_array_size > ARRAY_SIZE(fd->allow_bitmap))
> and hvcall_setup.bitmap_size always had number of bytes to copy.
>
> We can make allow_bitmap as an array of u8 here, irrespective of how
> userspace is setting the bits in the bitmap because it is finally doing
> the same thing. I'll make the change.
Yep. So on the kernel side, bitmap_array_size is a *byte* count,
and user space and the kernel side are in agreement. ;-)
>
>
> >> return -EFAULT;
> >> }
> >>
> >> @@ -1221,11 +1224,7 @@ static int mshv_vtl_hvcall_do_setup(struct mshv_vtl_hvcall_fd *fd,
> >>
> >> static bool mshv_vtl_hvcall_is_allowed(struct mshv_vtl_hvcall_fd *fd, u16 call_code)
> >> {
> >> - u8 bits_per_item = 8 * sizeof(fd->allow_bitmap[0]);
> >> - u16 item_index = call_code / bits_per_item;
> >> - u64 mask = 1ULL << (call_code % bits_per_item);
> >> -
> >> - return fd->allow_bitmap[item_index] & mask;
> >> + return test_bit(call_code, (unsigned long *)fd->allow_bitmap);
> >> }
> >
> > Yes, the rest of this looks good.
> >
> > Michael
>
> For CPU hotplug, I checked internally and we do not support hotplug in
> VTL2 for OpenHCL. Hotplugging vCPUs in VTL0 is transparent to VTL2 and
> these CPUs remain online in VTL2. VTL2 also won't be hotplugging CPUs
> when VTL0 is running.
Good to know.
>
> I am planning to put a comment mentioning this in the two places where
> we check for cpu_online() in the driver. Preventing hotplug from
> happening through hotplug notifiers is a bit overkill for me, as of now,
> but if you feel that should be done, I will work on it.
I'm marginally OK with just a comment because VTL2 code is not
running general-purpose workloads.
>
> Since good enough changes are done in this version, I will send the next
> version soon, unless there are follow-up comments here which needs
> something to change. We can then have a fresh look at it and work
> together to enhance it.
Sounds good.
Michael
^ permalink raw reply
* Re: [PATCH v5 2/2] Drivers: hv: Introduce mshv_vtl driver
From: Naman Jain @ 2025-07-23 9:58 UTC (permalink / raw)
To: Michael Kelley, K . Y . Srinivasan, Haiyang Zhang, Wei Liu,
Dexuan Cui
Cc: Roman Kisel, Anirudh Rayabharam, Saurabh Sengar,
Stanislav Kinsburskii, Nuno Das Neves, ALOK TIWARI,
linux-kernel@vger.kernel.org, linux-hyperv@vger.kernel.org
In-Reply-To: <SN6PR02MB41578D18024EC5339690046CD45CA@SN6PR02MB4157.namprd02.prod.outlook.com>
On 7/22/2025 10:39 PM, Michael Kelley wrote:
> From: Naman Jain <namjain@linux.microsoft.com> Sent: Tuesday, July 22, 2025 4:09 AM
>>
>> On 7/18/2025 8:37 PM, Michael Kelley wrote:
>>> From: Naman Jain <namjain@linux.microsoft.com> Sent: Thursday, July 17, 2025 9:36 PM
>>>>
>>>> On 7/9/2025 10:49 PM, Michael Kelley wrote:
>>>>> From: Naman Jain <namjain@linux.microsoft.com> Sent: Wednesday, June 11, 2025 12:27 AM
>>>
>>> [snip]
>>>
>>>>
>>>>> Separately, "allow_bitmap" size is 64K bytes, or 512K bits. Is that the
>>>>> correct size? From looking at mshv_vtl_hvcall_is_allowed(), I think this
>>>>> bitmap is indexed by the HV call code, which is a 16 bit value. So you
>>>>> only need 64K bits, and the size is too big by a factor of 8. In any case,
>>>>> it seems like the size should not be expressed in terms of PAGE_SIZE.
>>>>
>>>> There are HVcall codes which are of type u16. So max(HVcall code) =
>>>> 0xffff.
>>>>
>>>> For every HVcall that needs to be allowed, we are saving HVcall code
>>>> info in a bitmap in below fashion:
>>>> if x = HVCall code and bitmap is an array of u64, of size
>>>> ((0xffff/64=1023) + 1)
>>>>
>>>> bitmap[x / 64] = (u64)1 << (x%64);
>>>>
>>>> Later on in mshv_vtl_hvcall_is_allowed(), we calculate the array index
>>>> by dividing it by 64, and then see if call_code/64 bit is set.
>>>
>>> I didn't add comments in mshv_vtl_hvcall_is_allowed(), but that code
>>> can be simplified by recognizing that the Linux kernel bitmap utilities
>>> can operate on bitmaps that are much larger than just 64 bits. Let's
>>> assume that the allow_bitmap field in struct mshv_vtl_hvcall_fds has
>>> 64K bits, regardless of whether it is declared as an array of u64,
>>> an array of u16, or an array of u8. Then mshv_vtl_hvcall_is_allowed()
>>> can be implemented as a single line:
>>>
>>> return test_bit(call_code, fd->allow_bitmap);
>>>
>>> There's no need to figure out which array element contains the bit,
>>> or to construct a mask to select that particular bit in the array element.
>>> And since call_code is a u16, test_bit won't access outside the allocated
>>> 64K bits.
>>>
>>
>> I understood it now. This works and is much better. Will incorporate it
>> in next patch.
>>
>>>>
>>>> Coming to size of allow_bitmap[], it is independent of PAGE_SIZE, and
>>>> can be safely initialized to 1024 (reducing by a factor of 8).
>>>> bitmap_size's maximum value is going to be 1024 in current
>>>> implementation, picking u64 was not mandatory, u16 will also work. Also,
>>>> item_index is also u16, so I should make bitmap_size as u16.
>>>
>>> The key question for me is whether bitmap_size describes the number
>>> of bits in allow_bitmap, or whether it describes the number of array
>>> elements in the declared allow_bitmap array. It's more typical to
>>> describe a bitmap size as the number of bits. Then the value is
>>> independent of the array element size, as the array element size
>>> usually doesn't really matter anyway if using the Linux kernel's
>>> bitmap utilities. The array element size only matters in allocating
>>> the correct amount of space is for whatever number of bits are
>>> needed in the bitmap.
>>>
>>
>> I tried to put your suggestions in code. Please let me know if below
>> works. I tested this and it works. Just that, I am a little hesitant in
>> changing things on Userspace side of it, which passes these parameters
>> in IOCTL. This way, userspace remains the same, the confusion of names
>> may go away, and the code becomes simpler.
>
> Yes, I suspected there might be constraints due to not wanting
> to disturb existing user space code.
>
>>
>> --- a/include/uapi/linux/mshv.h
>> +++ b/include/uapi/linux/mshv.h
>> @@ -332,7 +332,7 @@ struct mshv_vtl_set_poll_file {
>> };
>>
>> struct mshv_vtl_hvcall_setup {
>> - __u64 bitmap_size;
>> + __u64 bitmap_array_size;
>> __u64 allow_bitmap_ptr; /* pointer to __u64 */
>> };
>>
>>
>> --- a/drivers/hv/mshv_vtl_main.c
>> +++ b/drivers/hv/mshv_vtl_main.c
>> @@ -52,10 +52,12 @@ static bool has_message;
>> static struct eventfd_ctx *flag_eventfds[HV_EVENT_FLAGS_COUNT];
>> static DEFINE_MUTEX(flag_lock);
>> static bool __read_mostly mshv_has_reg_page;
>> -#define MAX_BITMAP_SIZE 1024
>> +
>> +/* hvcall code is of type u16, allocate a bitmap of size (1 << 16) to accomodate it */
>
> s/accomodate/accommodate/
Acked.
>
>> +#define MAX_BITMAP_SIZE (1 << 16)
>>
>> struct mshv_vtl_hvcall_fd {
>> - u64 allow_bitmap[MAX_BITMAP_SIZE];
>> + u64 allow_bitmap[MAX_BITMAP_SIZE / 64];
>
> OK, this seems like a reasonable approach to get the correct
> number of bits.
>
>> bool allow_map_initialized;
>> /*
>> * Used to protect hvcall setup in IOCTLs
>>
>> @@ -1204,12 +1207,12 @@ static int mshv_vtl_hvcall_do_setup(struct mshv_vtl_hvcall_fd *fd,
>> sizeof(struct mshv_vtl_hvcall_setup))) {
>> return -EFAULT;
>> }
>> - if (hvcall_setup.bitmap_size > ARRAY_SIZE(fd->allow_bitmap)) {
>> + if (hvcall_setup.bitmap_array_size > ARRAY_SIZE(fd->allow_bitmap)) {
>> return -EINVAL;
>> }
>> if (copy_from_user(&fd->allow_bitmap,
>> (void __user *)hvcall_setup.allow_bitmap_ptr,
>> - hvcall_setup.bitmap_size)) {
>> + hvcall_setup.bitmap_array_size)) {
>
> This still doesn't work. copy_from_user() expects a byte count as its
> 3rd argument. If we have 64K bits in the bitmap, that means 8K bytes,
> and 1K for bitmap_array_size. So the value of the 3rd argument
> here is 1K, and you'll be copying 1K bytes when you want to be
> copying 8K bytes. The 3rd argument needs to be:
>
> hv_call_setup.bitmap_array_size * sizeof(u64)
>
> It's a bit messy to have to add this multiply, and in the bigger
> picture it might be cleaner to have the bitmap_array be
> declared in units of u8 instead of u64, but that would affect
> user space in a way that you probably want to avoid.
>
> Have you checked that user space is passing in the correct values
> to the ioctl? If the kernel side code is wrong, user space might be
> wrong as well. And if user space is wrong, then you might have
> the flexibility to change everything to use u8 instead of u64.
>
This is correct in Userspace:
https://github.com/microsoft/openvmm/blob/main/openhcl/hcl/src/ioctl.rs#L905
This was wrong in kernel code internally from the beginning. This did
not get caught because we took a large array size (2 * PAGE_SIZE) which
never failed the check
(hvcall_setup.bitmap_array_size > ARRAY_SIZE(fd->allow_bitmap))
and hvcall_setup.bitmap_size always had number of bytes to copy.
We can make allow_bitmap as an array of u8 here, irrespective of how
userspace is setting the bits in the bitmap because it is finally doing
the same thing. I'll make the change.
>> return -EFAULT;
>> }
>>
>> @@ -1221,11 +1224,7 @@ static int mshv_vtl_hvcall_do_setup(struct mshv_vtl_hvcall_fd *fd,
>>
>> static bool mshv_vtl_hvcall_is_allowed(struct mshv_vtl_hvcall_fd *fd, u16 call_code)
>> {
>> - u8 bits_per_item = 8 * sizeof(fd->allow_bitmap[0]);
>> - u16 item_index = call_code / bits_per_item;
>> - u64 mask = 1ULL << (call_code % bits_per_item);
>> -
>> - return fd->allow_bitmap[item_index] & mask;
>> + return test_bit(call_code, (unsigned long *)fd->allow_bitmap);
>> }
>
> Yes, the rest of this looks good.
>
> Michael
For CPU hotplug, I checked internally and we do not support hotplug in
VTL2 for OpenHCL. Hotplugging vCPUs in VTL0 is transparent to VTL2 and
these CPUs remain online in VTL2. VTL2 also won't be hotplugging CPUs
when VTL0 is running.
I am planning to put a comment mentioning this in the two places where
we check for cpu_online() in the driver. Preventing hotplug from
happening through hotplug notifiers is a bit overkill for me, as of now,
but if you feel that should be done, I will work on it.
Since good enough changes are done in this version, I will send the next
version soon, unless there are follow-up comments here which needs
something to change. We can then have a fresh look at it and work
together to enhance it.
Thanks for your time and guidance Michael.
Thanks,
Naman
^ permalink raw reply
* Re: [PATCH] net: mana: Use page pool fragments for RX buffers instead of full pages to improve memory efficiency and throughput.
From: Saurabh Singh Sengar @ 2025-07-23 7:38 UTC (permalink / raw)
To: Dipayaan Roy
Cc: kuba, kys, haiyangz, wei.liu, decui, andrew+netdev, davem,
edumazet, pabeni, longli, kotaranov, horms, ast, daniel, hawk,
john.fastabend, sdf, lorenzo, michal.kubiak, ernis, shradhagupta,
shirazsaleem, rosenp, netdev, linux-hyperv, linux-rdma, bpf
In-Reply-To: <20250721101417.GA18873@linuxonhyperv3.guj3yctzbm1etfxqx2vob5hsef.xx.internal.cloudapp.net>
On Mon, Jul 21, 2025 at 03:14:17AM -0700, Dipayaan Roy wrote:
> This patch enhances RX buffer handling in the mana driver by allocating
> pages from a page pool and slicing them into MTU-sized fragments, rather
> than dedicating a full page per packet. This approach is especially
> beneficial on systems with 64KB page sizes.
>
> Key improvements:
>
> - Proper integration of page pool for RX buffer allocations.
> - MTU-sized buffer slicing to improve memory utilization.
> - Reduce overall per Rx queue memory footprint.
> - Automatic fallback to full-page buffers when:
> * Jumbo frames are enabled (MTU > PAGE_SIZE / 2).
> * The XDP path is active, to avoid complexities with fragment reuse.
> - Removal of redundant pre-allocated RX buffers used in scenarios like MTU
> changes, ensuring consistency in RX buffer allocation.
>
> Testing on VMs with 64KB pages shows around 200% throughput improvement.
> Memory efficiency is significantly improved due to reduced wastage in page
> allocations. Example: We are now able to fit 35 Rx buffers in a single 64KB
> page for MTU size of 1500, instead of 1 Rx buffer per page previously.
>
> Tested:
>
> - iperf3, iperf2, and nttcp benchmarks.
> - Jumbo frames with MTU 9000.
> - Native XDP programs (XDP_PASS, XDP_DROP, XDP_TX, XDP_REDIRECT) for
> testing the driver’s XDP path.
> - Page leak detection (kmemleak).
> - Driver load/unload, reboot, and stress scenarios.
>
> Signed-off-by: Dipayaan Roy <dipayanroy@linux.microsoft.com>
> ---
> .../net/ethernet/microsoft/mana/mana_bpf.c | 22 +-
> drivers/net/ethernet/microsoft/mana/mana_en.c | 284 ++++++------------
> .../ethernet/microsoft/mana/mana_ethtool.c | 13 -
> include/net/mana/mana.h | 13 +-
> 4 files changed, 115 insertions(+), 217 deletions(-)
>
> diff --git a/drivers/net/ethernet/microsoft/mana/mana_bpf.c b/drivers/net/ethernet/microsoft/mana/mana_bpf.c
> index d30721d4516f..96813b6c184f 100644
> --- a/drivers/net/ethernet/microsoft/mana/mana_bpf.c
> +++ b/drivers/net/ethernet/microsoft/mana/mana_bpf.c
> @@ -174,6 +174,7 @@ static int mana_xdp_set(struct net_device *ndev, struct bpf_prog *prog,
> struct mana_port_context *apc = netdev_priv(ndev);
> struct bpf_prog *old_prog;
> struct gdma_context *gc;
> + int err;
>
> gc = apc->ac->gdma_dev->gdma_context;
>
> @@ -198,14 +199,33 @@ static int mana_xdp_set(struct net_device *ndev, struct bpf_prog *prog,
> if (old_prog)
> bpf_prog_put(old_prog);
>
> - if (apc->port_is_up)
> + if (apc->port_is_up) {
> + /* Re-create rxq's after xdp prog was loaded or unloaded.
> + * Ex: re create rxq's to switch from full pages to smaller
> + * size page fragments when xdp prog is unloaded and vice-versa.
> + */
> +
> + err = mana_detach(ndev, false);
> + if (err) {
> + netdev_err(ndev, "mana_detach failed at xdp set: %d\n", err);
> + goto out;
You should return err. At out we are always returning 0 which is wrong.
> + }
> +
> + err = mana_attach(ndev);
> + if (err) {
> + netdev_err(ndev, "mana_attach failed at xdp set: %d\n", err);
> + goto out;
same here.
> + }
> +
> mana_chn_setxdp(apc, prog);
> + }
>
> if (prog)
> ndev->max_mtu = MANA_XDP_MTU_MAX;
> else
> ndev->max_mtu = gc->adapter_mtu - ETH_HLEN;
>
> +out:
> return 0;
> }
>
> diff --git a/drivers/net/ethernet/microsoft/mana/mana_en.c b/drivers/net/ethernet/microsoft/mana/mana_en.c
> index a7973651ae51..a474c59c907c 100644
> --- a/drivers/net/ethernet/microsoft/mana/mana_en.c
> +++ b/drivers/net/ethernet/microsoft/mana/mana_en.c
> @@ -548,171 +548,45 @@ static u16 mana_select_queue(struct net_device *ndev, struct sk_buff *skb,
> return txq;
> }
>
> -/* Release pre-allocated RX buffers */
> -void mana_pre_dealloc_rxbufs(struct mana_port_context *mpc)
> -{
> - struct device *dev;
> - int i;
> -
> - dev = mpc->ac->gdma_dev->gdma_context->dev;
> -
> - if (!mpc->rxbufs_pre)
> - goto out1;
> -
> - if (!mpc->das_pre)
> - goto out2;
> -
> - while (mpc->rxbpre_total) {
> - i = --mpc->rxbpre_total;
> - dma_unmap_single(dev, mpc->das_pre[i], mpc->rxbpre_datasize,
> - DMA_FROM_DEVICE);
> - put_page(virt_to_head_page(mpc->rxbufs_pre[i]));
> - }
> -
> - kfree(mpc->das_pre);
> - mpc->das_pre = NULL;
> -
> -out2:
> - kfree(mpc->rxbufs_pre);
> - mpc->rxbufs_pre = NULL;
> -
> -out1:
> - mpc->rxbpre_datasize = 0;
> - mpc->rxbpre_alloc_size = 0;
> - mpc->rxbpre_headroom = 0;
> -}
> -
> -/* Get a buffer from the pre-allocated RX buffers */
> -static void *mana_get_rxbuf_pre(struct mana_rxq *rxq, dma_addr_t *da)
> -{
> - struct net_device *ndev = rxq->ndev;
> - struct mana_port_context *mpc;
> - void *va;
> -
> - mpc = netdev_priv(ndev);
> -
> - if (!mpc->rxbufs_pre || !mpc->das_pre || !mpc->rxbpre_total) {
> - netdev_err(ndev, "No RX pre-allocated bufs\n");
> - return NULL;
> - }
> -
> - /* Check sizes to catch unexpected coding error */
> - if (mpc->rxbpre_datasize != rxq->datasize) {
> - netdev_err(ndev, "rxbpre_datasize mismatch: %u: %u\n",
> - mpc->rxbpre_datasize, rxq->datasize);
> - return NULL;
> - }
> -
> - if (mpc->rxbpre_alloc_size != rxq->alloc_size) {
> - netdev_err(ndev, "rxbpre_alloc_size mismatch: %u: %u\n",
> - mpc->rxbpre_alloc_size, rxq->alloc_size);
> - return NULL;
> - }
> -
> - if (mpc->rxbpre_headroom != rxq->headroom) {
> - netdev_err(ndev, "rxbpre_headroom mismatch: %u: %u\n",
> - mpc->rxbpre_headroom, rxq->headroom);
> - return NULL;
> - }
> -
> - mpc->rxbpre_total--;
> -
> - *da = mpc->das_pre[mpc->rxbpre_total];
> - va = mpc->rxbufs_pre[mpc->rxbpre_total];
> - mpc->rxbufs_pre[mpc->rxbpre_total] = NULL;
> -
> - /* Deallocate the array after all buffers are gone */
> - if (!mpc->rxbpre_total)
> - mana_pre_dealloc_rxbufs(mpc);
> -
> - return va;
> -}
> -
> /* Get RX buffer's data size, alloc size, XDP headroom based on MTU */
> -static void mana_get_rxbuf_cfg(int mtu, u32 *datasize, u32 *alloc_size,
> - u32 *headroom)
> +static void mana_get_rxbuf_cfg(struct mana_port_context *apc,
> + int mtu, u32 *datasize, u32 *alloc_size,
> + u32 *headroom, u32 *frag_count)
> {
> - if (mtu > MANA_XDP_MTU_MAX)
> - *headroom = 0; /* no support for XDP */
> - else
> - *headroom = XDP_PACKET_HEADROOM;
> -
> - *alloc_size = SKB_DATA_ALIGN(mtu + MANA_RXBUF_PAD + *headroom);
> -
> - /* Using page pool in this case, so alloc_size is PAGE_SIZE */
> - if (*alloc_size < PAGE_SIZE)
> - *alloc_size = PAGE_SIZE;
> -
> + /* Calculate datasize first (consistent across all cases) */
> *datasize = mtu + ETH_HLEN;
> -}
> -
> -int mana_pre_alloc_rxbufs(struct mana_port_context *mpc, int new_mtu, int num_queues)
> -{
> - struct device *dev;
> - struct page *page;
> - dma_addr_t da;
> - int num_rxb;
> - void *va;
> - int i;
> -
> - mana_get_rxbuf_cfg(new_mtu, &mpc->rxbpre_datasize,
> - &mpc->rxbpre_alloc_size, &mpc->rxbpre_headroom);
> -
> - dev = mpc->ac->gdma_dev->gdma_context->dev;
> -
> - num_rxb = num_queues * mpc->rx_queue_size;
> -
> - WARN(mpc->rxbufs_pre, "mana rxbufs_pre exists\n");
> - mpc->rxbufs_pre = kmalloc_array(num_rxb, sizeof(void *), GFP_KERNEL);
> - if (!mpc->rxbufs_pre)
> - goto error;
>
> - mpc->das_pre = kmalloc_array(num_rxb, sizeof(dma_addr_t), GFP_KERNEL);
> - if (!mpc->das_pre)
> - goto error;
> -
> - mpc->rxbpre_total = 0;
> -
> - for (i = 0; i < num_rxb; i++) {
> - page = dev_alloc_pages(get_order(mpc->rxbpre_alloc_size));
> - if (!page)
> - goto error;
> -
> - va = page_to_virt(page);
> -
> - da = dma_map_single(dev, va + mpc->rxbpre_headroom,
> - mpc->rxbpre_datasize, DMA_FROM_DEVICE);
> - if (dma_mapping_error(dev, da)) {
> - put_page(page);
> - goto error;
> + /* For xdp and jumbo frames make sure only one packet fits per page */
> + if (((mtu + MANA_RXBUF_PAD) > PAGE_SIZE / 2) || rcu_access_pointer(apc->bpf_prog)) {
> + if (rcu_access_pointer(apc->bpf_prog)) {
> + *headroom = XDP_PACKET_HEADROOM;
> + *alloc_size = PAGE_SIZE;
> + } else {
> + *headroom = 0; /* no support for XDP */
> + *alloc_size = SKB_DATA_ALIGN(mtu + MANA_RXBUF_PAD + *headroom);
> }
>
> - mpc->rxbufs_pre[i] = va;
> - mpc->das_pre[i] = da;
> - mpc->rxbpre_total = i + 1;
> + *frag_count = 1;
> + return;
> }
>
> - return 0;
> + /* Standard MTU case - optimize for multiple packets per page */
> + *headroom = 0;
>
> -error:
> - netdev_err(mpc->ndev, "Failed to pre-allocate RX buffers for %d queues\n", num_queues);
> - mana_pre_dealloc_rxbufs(mpc);
> - return -ENOMEM;
> + /* Calculate base buffer size needed */
> + u32 len = SKB_DATA_ALIGN(mtu + MANA_RXBUF_PAD + *headroom);
> + u32 buf_size = ALIGN(len, MANA_RX_FRAG_ALIGNMENT);
its good to have all the declaration at start of function.
> +
> + /* Calculate how many packets can fit in a page */
> + *frag_count = PAGE_SIZE / buf_size;
> + *alloc_size = buf_size;
> }
>
> static int mana_change_mtu(struct net_device *ndev, int new_mtu)
> {
> - struct mana_port_context *mpc = netdev_priv(ndev);
> unsigned int old_mtu = ndev->mtu;
> int err;
>
> - /* Pre-allocate buffers to prevent failure in mana_attach later */
> - err = mana_pre_alloc_rxbufs(mpc, new_mtu, mpc->num_queues);
> - if (err) {
> - netdev_err(ndev, "Insufficient memory for new MTU\n");
> - return err;
> - }
> -
> err = mana_detach(ndev, false);
> if (err) {
> netdev_err(ndev, "mana_detach failed: %d\n", err);
> @@ -728,7 +602,6 @@ static int mana_change_mtu(struct net_device *ndev, int new_mtu)
> }
>
> out:
> - mana_pre_dealloc_rxbufs(mpc);
> return err;
> }
>
> @@ -1841,8 +1714,11 @@ static void mana_rx_skb(void *buf_va, bool from_pool,
>
> drop:
> if (from_pool) {
> - page_pool_recycle_direct(rxq->page_pool,
> - virt_to_head_page(buf_va));
> + if (rxq->frag_count == 1)
> + page_pool_recycle_direct(rxq->page_pool,
> + virt_to_head_page(buf_va));
> + else
> + page_pool_free_va(rxq->page_pool, buf_va, true);
> } else {
> WARN_ON_ONCE(rxq->xdp_save_va);
> /* Save for reuse */
> @@ -1854,37 +1730,50 @@ static void mana_rx_skb(void *buf_va, bool from_pool,
> return;
> }
>
> -static void *mana_get_rxfrag(struct mana_rxq *rxq, struct device *dev,
> - dma_addr_t *da, bool *from_pool)
> +static void *mana_get_rxfrag(struct mana_rxq *rxq,
> + struct device *dev, dma_addr_t *da, bool *from_pool)
> {
> struct page *page;
> + u32 offset;
> void *va;
> -
> *from_pool = false;
>
> - /* Reuse XDP dropped page if available */
> - if (rxq->xdp_save_va) {
> - va = rxq->xdp_save_va;
> - rxq->xdp_save_va = NULL;
> - } else {
> - page = page_pool_dev_alloc_pages(rxq->page_pool);
> - if (!page)
> + /* Don't use fragments for jumbo frames or XDP (i.e when fragment = 1 per page) */
> + if (rxq->frag_count == 1) {
> + /* Reuse XDP dropped page if available */
> + if (rxq->xdp_save_va) {
> + va = rxq->xdp_save_va;
> + rxq->xdp_save_va = NULL;
> + } else {
> + page = page_pool_dev_alloc_pages(rxq->page_pool);
> + if (!page)
> + return NULL;
> +
> + *from_pool = true;
> + va = page_to_virt(page);
> + }
> +
> + *da = dma_map_single(dev, va + rxq->headroom, rxq->datasize,
> + DMA_FROM_DEVICE);
> + if (dma_mapping_error(dev, *da)) {
> + if (*from_pool)
> + page_pool_put_full_page(rxq->page_pool, page, false);
> + else
> + put_page(virt_to_head_page(va));
> +
> return NULL;
> + }
>
> - *from_pool = true;
> - va = page_to_virt(page);
> + return va;
> }
>
> - *da = dma_map_single(dev, va + rxq->headroom, rxq->datasize,
> - DMA_FROM_DEVICE);
> - if (dma_mapping_error(dev, *da)) {
> - if (*from_pool)
> - page_pool_put_full_page(rxq->page_pool, page, false);
> - else
> - put_page(virt_to_head_page(va));
> -
> + page = page_pool_dev_alloc_frag(rxq->page_pool, &offset, rxq->alloc_size);
> + if (!page)
> return NULL;
> - }
> +
> + va = page_to_virt(page) + offset;
> + *da = page_pool_get_dma_addr(page) + offset + rxq->headroom;
> + *from_pool = true;
>
> return va;
> }
> @@ -1901,9 +1790,9 @@ static void mana_refill_rx_oob(struct device *dev, struct mana_rxq *rxq,
> va = mana_get_rxfrag(rxq, dev, &da, &from_pool);
> if (!va)
> return;
> -
> - dma_unmap_single(dev, rxoob->sgl[0].address, rxq->datasize,
> - DMA_FROM_DEVICE);
> + if (!rxoob->from_pool || rxq->frag_count == 1)
> + dma_unmap_single(dev, rxoob->sgl[0].address, rxq->datasize,
> + DMA_FROM_DEVICE);
> *old_buf = rxoob->buf_va;
> *old_fp = rxoob->from_pool;
>
> @@ -2314,15 +2203,19 @@ static void mana_destroy_rxq(struct mana_port_context *apc,
> if (!rx_oob->buf_va)
> continue;
>
> - dma_unmap_single(dev, rx_oob->sgl[0].address,
> - rx_oob->sgl[0].size, DMA_FROM_DEVICE);
> -
> page = virt_to_head_page(rx_oob->buf_va);
>
> - if (rx_oob->from_pool)
> - page_pool_put_full_page(rxq->page_pool, page, false);
> - else
> - put_page(page);
> + if (rxq->frag_count == 1) {
> + dma_unmap_single(dev, rx_oob->sgl[0].address, rx_oob->sgl[0].size,
> + DMA_FROM_DEVICE);
> +
> + if (rx_oob->from_pool)
> + page_pool_put_full_page(rxq->page_pool, page, false);
> + else
> + put_page(page);
> + } else {
> + page_pool_free_va(rxq->page_pool, rx_oob->buf_va, true);
> + }
>
> rx_oob->buf_va = NULL;
> }
> @@ -2338,16 +2231,11 @@ static void mana_destroy_rxq(struct mana_port_context *apc,
> static int mana_fill_rx_oob(struct mana_recv_buf_oob *rx_oob, u32 mem_key,
> struct mana_rxq *rxq, struct device *dev)
> {
> - struct mana_port_context *mpc = netdev_priv(rxq->ndev);
> bool from_pool = false;
> dma_addr_t da;
> void *va;
>
> - if (mpc->rxbufs_pre)
> - va = mana_get_rxbuf_pre(rxq, &da);
> - else
> - va = mana_get_rxfrag(rxq, dev, &da, &from_pool);
> -
> + va = mana_get_rxfrag(rxq, dev, &da, &from_pool);
> if (!va)
> return -ENOMEM;
>
> @@ -2428,11 +2316,22 @@ static int mana_create_page_pool(struct mana_rxq *rxq, struct gdma_context *gc)
> struct page_pool_params pprm = {};
> int ret;
>
> - pprm.pool_size = mpc->rx_queue_size;
> + pprm.pool_size = mpc->rx_queue_size / rxq->frag_count + 1;
> pprm.nid = gc->numa_node;
> pprm.napi = &rxq->rx_cq.napi;
> pprm.netdev = rxq->ndev;
> pprm.order = get_order(rxq->alloc_size);
> + pprm.queue_idx = rxq->rxq_idx;
> + pprm.dev = gc->dev;
> +
> + /* Let the page pool do the dma map when page sharing with multiple fragments
> + * enabled for rx buffers.
> + */
> + if (rxq->frag_count > 1) {
> + pprm.flags = PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV;
> + pprm.max_len = PAGE_SIZE;
> + pprm.dma_dir = DMA_FROM_DEVICE;
> + }
>
> rxq->page_pool = page_pool_create(&pprm);
>
> @@ -2471,9 +2370,8 @@ static struct mana_rxq *mana_create_rxq(struct mana_port_context *apc,
> rxq->rxq_idx = rxq_idx;
> rxq->rxobj = INVALID_MANA_HANDLE;
>
> - mana_get_rxbuf_cfg(ndev->mtu, &rxq->datasize, &rxq->alloc_size,
> - &rxq->headroom);
> -
> + mana_get_rxbuf_cfg(apc, ndev->mtu, &rxq->datasize, &rxq->alloc_size,
> + &rxq->headroom, &rxq->frag_count);
> /* Create page pool for RX queue */
> err = mana_create_page_pool(rxq, gc);
> if (err) {
> diff --git a/drivers/net/ethernet/microsoft/mana/mana_ethtool.c b/drivers/net/ethernet/microsoft/mana/mana_ethtool.c
> index a1afa75a9463..7ede03c74fb9 100644
> --- a/drivers/net/ethernet/microsoft/mana/mana_ethtool.c
> +++ b/drivers/net/ethernet/microsoft/mana/mana_ethtool.c
> @@ -396,12 +396,6 @@ static int mana_set_channels(struct net_device *ndev,
> unsigned int old_count = apc->num_queues;
> int err;
>
> - err = mana_pre_alloc_rxbufs(apc, ndev->mtu, new_count);
> - if (err) {
> - netdev_err(ndev, "Insufficient memory for new allocations");
> - return err;
> - }
> -
> err = mana_detach(ndev, false);
> if (err) {
> netdev_err(ndev, "mana_detach failed: %d\n", err);
> @@ -416,7 +410,6 @@ static int mana_set_channels(struct net_device *ndev,
> }
>
> out:
> - mana_pre_dealloc_rxbufs(apc);
> return err;
> }
>
> @@ -465,12 +458,7 @@ static int mana_set_ringparam(struct net_device *ndev,
>
> /* pre-allocating new buffers to prevent failures in mana_attach() later */
> apc->rx_queue_size = new_rx;
> - err = mana_pre_alloc_rxbufs(apc, ndev->mtu, apc->num_queues);
> apc->rx_queue_size = old_rx;
> - if (err) {
> - netdev_err(ndev, "Insufficient memory for new allocations\n");
> - return err;
> - }
>
> err = mana_detach(ndev, false);
> if (err) {
> @@ -488,7 +476,6 @@ static int mana_set_ringparam(struct net_device *ndev,
> apc->rx_queue_size = old_rx;
> }
> out:
> - mana_pre_dealloc_rxbufs(apc);
> return err;
> }
>
> diff --git a/include/net/mana/mana.h b/include/net/mana/mana.h
> index e1030a7d2daa..99a3847b0f9d 100644
> --- a/include/net/mana/mana.h
> +++ b/include/net/mana/mana.h
> @@ -65,6 +65,8 @@ enum TRI_STATE {
> #define MANA_STATS_RX_COUNT 5
> #define MANA_STATS_TX_COUNT 11
>
> +#define MANA_RX_FRAG_ALIGNMENT 64
> +
> struct mana_stats_rx {
> u64 packets;
> u64 bytes;
> @@ -328,6 +330,7 @@ struct mana_rxq {
> u32 datasize;
> u32 alloc_size;
> u32 headroom;
> + u32 frag_count;
>
> mana_handle_t rxobj;
>
> @@ -503,14 +506,6 @@ struct mana_port_context {
> /* This points to an array of num_queues of RQ pointers. */
> struct mana_rxq **rxqs;
>
> - /* pre-allocated rx buffer array */
> - void **rxbufs_pre;
> - dma_addr_t *das_pre;
> - int rxbpre_total;
> - u32 rxbpre_datasize;
> - u32 rxbpre_alloc_size;
> - u32 rxbpre_headroom;
> -
> struct bpf_prog *bpf_prog;
>
> /* Create num_queues EQs, SQs, SQ-CQs, RQs and RQ-CQs, respectively. */
> @@ -574,8 +569,6 @@ int mana_query_link_cfg(struct mana_port_context *apc);
> int mana_set_bw_clamp(struct mana_port_context *apc, u32 speed,
> int enable_clamping);
> void mana_query_phy_stats(struct mana_port_context *apc);
> -int mana_pre_alloc_rxbufs(struct mana_port_context *apc, int mtu, int num_queues);
> -void mana_pre_dealloc_rxbufs(struct mana_port_context *apc);
>
> extern const struct ethtool_ops mana_ethtool_ops;
> extern struct dentry *mana_debugfs_root;
> --
> 2.43.0
Rest looks good. After fixing above,
Reviewed-by: Saurabh Sengar <ssengar@linux.microsoft.com>
^ permalink raw reply
* [PATCH 6.12] Drivers: hv: Make the sysfs node size for the ring buffer dynamic
From: Naman Jain @ 2025-07-23 7:02 UTC (permalink / raw)
To: stable, Michael Kelley, Wei Liu, Dexuan Cui
Cc: K . Y . Srinivasan, Haiyang Zhang, Long Li, Greg Kroah-Hartman,
linux, linux-hyperv, linux-kernel
The ring buffer size varies across VMBus channels. The size of sysfs
node for the ring buffer is currently hardcoded to 4 MB. Userspace
clients either use fstat() or hardcode this size for doing mmap().
To address this, make the sysfs node size dynamic to reflect the
actual ring buffer size for each channel. This will ensure that
fstat() on ring sysfs node always returns the correct size of
ring buffer.
This is a backport of the upstream commit
65995e97a1ca ("Drivers: hv: Make the sysfs node size for the ring buffer dynamic")
with modifications, as the original patch has missing dependencies on
kernel v6.12.x. The structure "struct attribute_group" does not have
bin_size field in v6.12.x kernel so the logic of configuring size of
sysfs node for ring buffer has been moved to
vmbus_chan_bin_attr_is_visible().
Original change was not a fix, but it needs to be backported to fix size
related discrepancy caused by the commit mentioned in Fixes tag.
Fixes: bf1299797c3c ("uio_hv_generic: Align ring size to system page")
Cc: <stable@vger.kernel.org> # 6.12.x
Signed-off-by: Naman Jain <namjain@linux.microsoft.com>
---
This change won't apply on older kernels currently due to missing
dependencies. I will take care of them after this goes in.
I did not retain any Reviewed-by or Tested-by tags, since the code has
changed completely, while the functionality remains same.
Requesting Michael, Dexuan, Wei to please review again.
---
drivers/hv/vmbus_drv.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/hv/vmbus_drv.c b/drivers/hv/vmbus_drv.c
index 1f519e925f06..616e63fb2f15 100644
--- a/drivers/hv/vmbus_drv.c
+++ b/drivers/hv/vmbus_drv.c
@@ -1810,7 +1810,6 @@ static struct bin_attribute chan_attr_ring_buffer = {
.name = "ring",
.mode = 0600,
},
- .size = 2 * SZ_2M,
.mmap = hv_mmap_ring_buffer_wrapper,
};
static struct attribute *vmbus_chan_attrs[] = {
@@ -1866,6 +1865,7 @@ static umode_t vmbus_chan_bin_attr_is_visible(struct kobject *kobj,
/* Hide ring attribute if channel's ring_sysfs_visible is set to false */
if (attr == &chan_attr_ring_buffer && !channel->ring_sysfs_visible)
return 0;
+ attr->size = channel->ringbuffer_pagecount << PAGE_SHIFT;
return attr->attr.mode;
}
--
2.34.1
^ permalink raw reply related
* Re: [PATCH v4 1/2] dt-bindings: microsoft: Add vmbus message-connection-id property
From: Krzysztof Kozlowski @ 2025-07-23 6:18 UTC (permalink / raw)
To: Hardik Garg
Cc: apais, conor+dt, decui, devicetree, haiyangz, hargar, krzk+dt,
kys, linux-hyperv, linux-kernel, robh, ssengar, wei.liu, cho
In-Reply-To: <1753240089-29558-1-git-send-email-hargar@linux.microsoft.com>
On 23/07/2025 05:08, Hardik Garg wrote:
>> Host is supposed to have multiple guests, so this feels like you are
>> going to prepare for each guest different DTS with different connection
>> ID. This feels like poor design. DTS is supposed to be relatively static
>> configuration, not runtime choice vmguestid+1.
>
>> The guest cannot access other configuration channels, can it? If it can,
>> it would mean it can eavesdrop on other guests? So obviously it cannot.
>> Therefore from guest point of view this is completely redundant. Guest
>> cannot use any other value, thus guest should not configure it. The
>> guest has only one channel and uses only this one which gets to right
>> place to the host.
>
> Thank you for your feedback. Let me explain the connection ID in more detail:
>
> 1. Message Port Architecture:
> - The connection ID specifies which Hyper-V hypervisor message port (mailbox slot) to use for communication between the host and guest
> - The hypervisor has multiple message ports, but historically VMBus only used one
> - With the introduction of VTL2 (Virtual Trust Level 2), the control plane can now be hosted in VTL2, requiring different message ports for communication
>
> 2. Control Plane Communication:
> - The VMBus control plane on the host needs to communicate with the VMBus driver in the guest
> - When the control plane is hosted in VTL2, it requires a different message port than the standard communication path
> - The connection ID tells the guest whether to use the standard or alternate message port for this control plane communication
>
> 3. Message Processing:
> - Each message is tagged with an ID
> - If the guest uses an incorrect ID, the host won't recognize the message and will drop it
> - This is not about choosing between multiple available channels - it's about using the correct mailbox slot for communication
>
Don't paste me specs in response to questions, but answer specific
questions.
> 4. Security and Isolation:
> - Each guest has its private hypervisor mailbox
> - Multiple guests using the same connection ID cannot interfere with each other
Then all guests can use the same value, 0, making this property redundant.
> - The connection ID is more like a "root ID" that helps enumerate devices on the bus, not a channel selector between guests
>
> 5. Dynamic Nature:
> - The connection ID is specified when the guest starts running
> - The host can change it during VM lifecycle events (reset/reboot)
So not suitable for DT. DT is a static data. You cannot just keep
changing existing DT to match whatever you want runtime.
> - This is why the VMBus driver needs to know the connection ID every time the kernel starts
> - The ID might be different from what it was during the last guest run
Best regards,
Krzysztof
^ permalink raw reply
* Re: [PATCH RESEND] netvsc: transfer lower device max tso size
From: Jason Wang @ 2025-07-23 6:00 UTC (permalink / raw)
To: Jakub Kicinski
Cc: Cindy Lu, K. Y. Srinivasan, Haiyang Zhang, Wei Liu, Dexuan Cui,
Andrew Lunn, David S. Miller, Eric Dumazet, Paolo Abeni,
Simon Horman, Michael Kelley, Shradha Gupta, Kees Cook,
Stanislav Fomichev, Kuniyuki Iwashima, Alexander Lobakin,
Guillaume Nault, Joe Damato, Ahmed Zaki,
open list:Hyper-V/Azure CORE AND DRIVERS,
open list:NETWORKING DRIVERS, open list
In-Reply-To: <20250721181807.752af6a4@kernel.org>
On Tue, Jul 22, 2025 at 9:18 AM Jakub Kicinski <kuba@kernel.org> wrote:
>
> On Tue, 22 Jul 2025 09:04:20 +0800 Jason Wang wrote:
> > On Tue, Jul 22, 2025 at 7:28 AM Jakub Kicinski <kuba@kernel.org> wrote:
> > > On Fri, 18 Jul 2025 14:17:55 +0800 Cindy Lu wrote:
> > > > Subject: [PATCH RESEND] netvsc: transfer lower device max tso size
> > >
> > > You say RESEND but I don't see a link to previous posting anywhere.
>
> Someone should respond to this part, please.
>
> > > I'd rather we didn't extend the magic behavior of hyperv/netvsc any
> > > further.
> >
> > Are you referring to the netdev coupling model of the VF acceleration?
>
> Yes, it tries to apply whole bunch of policy automatically in
> the kernel.
>
> > > We have enough problems with it.
> >
> > But this fixes a real problem, otherwise nested VM performance will be
> > broken due to the GSO software segmentation.
>
> Perhaps, possibly, a migration plan can be devised, away from the
> netvsc model, so we don't have to deal with nuggets of joy like:
> https://lore.kernel.org/all/1752870014-28909-1-git-send-email-haiyangz@linux.microsoft.com/
Btw, if I understand this correctly. This is for future development so
it's not a blocker for this patch?
Thanks
>
^ permalink raw reply
* Re: [PATCH v4 1/2] dt-bindings: microsoft: Add vmbus message-connection-id property
From: Hardik Garg @ 2025-07-23 3:08 UTC (permalink / raw)
To: krzk
Cc: apais, conor+dt, decui, devicetree, haiyangz, hargar, hargar,
krzk+dt, kys, linux-hyperv, linux-kernel, robh, ssengar, wei.liu,
cho
In-Reply-To: <095a1455-c6ac-4a7d-a219-ddfd0a93d8d6@kernel.org>
> Host is supposed to have multiple guests, so this feels like you are
> going to prepare for each guest different DTS with different connection
> ID. This feels like poor design. DTS is supposed to be relatively static
> configuration, not runtime choice vmguestid+1.
> The guest cannot access other configuration channels, can it? If it can,
> it would mean it can eavesdrop on other guests? So obviously it cannot.
> Therefore from guest point of view this is completely redundant. Guest
> cannot use any other value, thus guest should not configure it. The
> guest has only one channel and uses only this one which gets to right
> place to the host.
Thank you for your feedback. Let me explain the connection ID in more detail:
1. Message Port Architecture:
- The connection ID specifies which Hyper-V hypervisor message port (mailbox slot) to use for communication between the host and guest
- The hypervisor has multiple message ports, but historically VMBus only used one
- With the introduction of VTL2 (Virtual Trust Level 2), the control plane can now be hosted in VTL2, requiring different message ports for communication
2. Control Plane Communication:
- The VMBus control plane on the host needs to communicate with the VMBus driver in the guest
- When the control plane is hosted in VTL2, it requires a different message port than the standard communication path
- The connection ID tells the guest whether to use the standard or alternate message port for this control plane communication
3. Message Processing:
- Each message is tagged with an ID
- If the guest uses an incorrect ID, the host won't recognize the message and will drop it
- This is not about choosing between multiple available channels - it's about using the correct mailbox slot for communication
4. Security and Isolation:
- Each guest has its private hypervisor mailbox
- Multiple guests using the same connection ID cannot interfere with each other
- The connection ID is more like a "root ID" that helps enumerate devices on the bus, not a channel selector between guests
5. Dynamic Nature:
- The connection ID is specified when the guest starts running
- The host can change it during VM lifecycle events (reset/reboot)
- This is why the VMBus driver needs to know the connection ID every time the kernel starts
- The ID might be different from what it was during the last guest run
Thanks,
Hardik
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox