Re: [PATCH v2] net: mana: Use page pool fragments for RX buffers instead of full pages to improve memory efficiency.

[Jesper Dangaard Brouer <hawk@kernel.org>]

On 30/07/2025 09.31, Dragos Tatulea wrote:
> On Tue, Jul 29, 2025 at 01:20:07PM -0700, Dipayaan Roy wrote:
>> On Tue, Jul 29, 2025 at 12:15:23PM +0200, Jesper Dangaard Brouer wrote:
>>>
>>>
>>> On 23/07/2025 21.07, Dipayaan Roy wrote:
>>>> This patch enhances RX buffer handling in the mana driver by allocating
>>>> pages from a page pool and slicing them into MTU-sized fragments, rather
>>>> than dedicating a full page per packet. This approach is especially
>>>> beneficial on systems with large page sizes like 64KB.
>>>>
>>>> Key improvements:
>>>>
>>>> - Proper integration of page pool for RX buffer allocations.
>>>> - MTU-sized buffer slicing to improve memory utilization.
>>>> - Reduce overall per Rx queue memory footprint.
>>>> - Automatic fallback to full-page buffers when:
>>>>     * Jumbo frames are enabled (MTU > PAGE_SIZE / 2).
>>>>     * The XDP path is active, to avoid complexities with fragment reuse.
>>>> - Removal of redundant pre-allocated RX buffers used in scenarios like MTU
>>>>    changes, ensuring consistency in RX buffer allocation.
>>>>
>>>> Testing on VMs with 64KB pages shows around 200% throughput improvement.
>>>> Memory efficiency is significantly improved due to reduced wastage in page
>>>> allocations. Example: We are now able to fit 35 rx buffers in a single 64kb
>>>> page for MTU size of 1500, instead of 1 rx buffer per page previously.
>>>>
>>>> Tested:
>>>>
>>>> - iperf3, iperf2, and nttcp benchmarks.
>>>> - Jumbo frames with MTU 9000.
>>>> - Native XDP programs (XDP_PASS, XDP_DROP, XDP_TX, XDP_REDIRECT) for
>>>>    testing the XDP path in driver.
>>>> - Page leak detection (kmemleak).
>>>> - Driver load/unload, reboot, and stress scenarios.
>>>
>>> Chris (Cc) discovered a crash/bug[1] with page pool fragments used
>>> from the mlx5 driver.
>>> He put together a BPF program that reproduces the issue here:
>>> - [2] https://github.com/arges/xdp-redirector
>>>
>>> Can I ask you to test that your driver against this reproducer?
>>>
>>>
>>> [1] https://lore.kernel.org/all/aIEuZy6fUj_4wtQ6@861G6M3/
>>>
>>> --Jesper
>>>
>>
>> Hi Jesper,
>>
>> I was unable to reproduce this issue on mana driver.
>>
> Please note that I had to make a few adjustments to get reprodduction on
> mlx5:
> 
> - Make sure that the veth MACs are recognized by the device. Otherwise
>    traffic might be dropped by the device.
> 
> - Enable GRO on the veth device. Otherwise packets get dropped before
>    they reach the devmap BPF program.
> 
> Try starting the test program with one thread and see if you see packets
> coming through veth1-ns1 end of the veth pair.
> 

Hi Dipayaan,

Enabling GRO on the veth device is quite important for the test to be valid.

I've asked Chris to fix this in the reproducer. He can report back when
he have done this, so you can re-run the test.  It is also good advice
from Dragos that you should check packets are coming through the veth
pair, to make sure the test is working.

The setup.sh script also need to be modified, as it is loading xdp on a
net_device called "ext0" [0], which is specific to our systems (which
default also have GRO enabled for veth).

[0] https://github.com/arges/xdp-redirector/blob/main/setup.sh#L28

--Jesper