Re: [PATCH net-next 00/10] net/mlx5e: Add netdev support for data direct

[Dragos Tatulea <dtatulea@nvidia.com>]

On 10.10.26 01:54, Mina Almasry wrote:
> On Fri, Oct 9, 2026 at 9:42 AM Mina Almasry <almasrymina@google.com> wrote:
>>
>> On Thu, Oct 8, 2026 at 6:28 AM Tariq Toukan <tariqt@nvidia.com> wrote:
>>>
>>> Hi,
>>>
>>> This patch series by Dragos adds data direct support for netdev devices,
>>> allowing a mlx5 netdev to issue DMA commands through multiple paths.
>>> This feature is critical for improving performance and reaching line
>>> rate in certain environments where issuing PCI transactions over one
>>> path may be significantly faster than over another. These differences
>>> can arise from various PCI generations in the system or the specific
>>> system topology.
>>>
>>> Data direct for netdev will work with devmem by returning the data
>>> direct DMA device instead of the DMA device of the NIC. Each PF netdev
>>> registers its own data direct device and support for RX and TX datapath
>>> is added. The feature is selected through a new data direct priv flag.
>>>
>>> The feature is enabled though a ethtool private flag. When this flag
>>> is enabled, the driver will return the data direct device in the
>>> ndo_queue_get_dma_dev op.
>>>
>>> During data_direct device unbind, a re-creation of the channels is
>>> triggered.
>>>
>>> A note about the unbind: it does not does not tear down dmabuf bindings
>>> attached to the data direct device; the recreated channels will still
>>> pick up the binding via rxq->mp_params and post DMA addresses from the
>>> data direct IOMMU domain to the PF, causing IOMMU faults. A devmem
>>> revoke hook is needed to close those bindings before the switch. This is
>>> out of scope for this series.
>>>
>>
>> Elaborate on this please. Are you referring to the fact that on
>> page_pool_scrub we unmap the netmems dma-mapped by the page-pool, but
>> we don't unmap the netmems dma-mapped by the memory provider?
>>
>> What's the impact of leaving this unfixed? Is the machine going to
>> crash if the device goes away but we don't unmap the dmabuf?
> 
IOMMU faults because the dma device is invalid. Either because the
DD dev is still in use and was unbound. Or because the non-DD
one is used and is invalid (due to channel reopen).

> Oh, you're referring to an even different bug than the one I had in
> mind. (I'll see if I can submit a fix for the bug I had in mind).
> 
> But in general I think not addressing the issue you're referring to is
> way too messy. I think we do indeed need to unbind the dma-buf if the
> dma-dev is going away. The invariant should be that the
> binding->attachment->dev should not change at all during the entire
> lifetime of the binding. We already 'revoke' the dma-buf binding if
> the netdev is going away, sorta (there is a bug there I need to fix.
> The uninstall function in the provider doesn't actually unmap the
> dma-buf :sad face:).
> 
> I've worked with my LLM to suggest some changes that (we think) make
> this work, but you may disagree. I'll post the suggestions. But please
> I think this should be handled one way or another.
> 
My idea was to handle the unbind part in a subsequent series because
it probably needs some more back and forth to get this cleanup path
right.

If not acceptable I'll work on adding it to this series.

Thanks,
Dragos