Re: [RFC net-next 1/4] net: Allow non parent devices to be used for ZC DMA

[Dragos Tatulea <dtatulea@nvidia.com>]

On Mon, Jul 07, 2025 at 02:55:11PM -0700, Mina Almasry wrote:
> On Mon, Jul 7, 2025 at 2:35 PM Dragos Tatulea <dtatulea@nvidia.com> wrote:
> >
> > On Mon, Jul 07, 2025 at 11:44:19AM -0700, Mina Almasry wrote:
> > > On Fri, Jul 4, 2025 at 6:11 AM Dragos Tatulea <dtatulea@nvidia.com> wrote:
> > > >
> > > > On Thu, Jul 03, 2025 at 01:58:50PM +0200, Parav Pandit wrote:
> > > > >
> > > > > > From: Jakub Kicinski <kuba@kernel.org>
> > > > > > Sent: 03 July 2025 02:23 AM
> > > > > >
> > > > [...]
> > > > > > Maybe someone with closer understanding can chime in. If the kind of
> > > > > > subfunctions you describe are expected, and there's a generic way of
> > > > > > recognizing them -- automatically going to parent of parent would indeed be
> > > > > > cleaner and less error prone, as you suggest.
> > > > >
> > > > > I am not sure when the parent of parent assumption would fail, but can be
> > > > > a good start.
> > > > >
> > > > > If netdev 8 bytes extension to store dma_dev is concern,
> > > > > probably a netdev IFF_DMA_DEV_PARENT can be elegant to refer parent->parent?
> > > > > So that there is no guess work in devmem layer.
> > > > >
> > > > > That said, my understanding of devmem is limited, so I could be mistaken here.
> > > > >
> > > > > In the long term, the devmem infrastructure likely needs to be
> > > > > modernized to support queue-level DMA mapping.
> > > > > This is useful because drivers like mlx5 already support
> > > > > socket-direct netdev that span across two PCI devices.
> > > > >
> > > > > Currently, devmem is limited to a single PCI device per netdev.
> > > > > While the buffer pool could be per device, the actual DMA
> > > > > mapping might need to be deferred until buffer posting
> > > > > time to support such multi-device scenarios.
> > > > >
> > > > > In an offline discussion, Dragos mentioned that io_uring already
> > > > > operates at the queue level, may be some ideas can be picked up
> > > > > from io_uring?
> > > > The problem for devmem is that the device based API is already set in
> > > > stone so not sure how we can change this. Maybe Mina can chime in.
> > > >
> > >
> > > I think what's being discussed here is pretty straight forward and
> > > doesn't need UAPI changes, right? Or were you referring to another
> > > API?
> > >
> > I was referring to the fact that devmem takes one big buffer, maps it
> > for a single device (in net_devmem_bind_dmabuf()) and then assigns it to
> > queues in net_devmem_bind_dmabuf_to_queue(). As the single buffer is
> > part of the API, I don't see how the mapping could be done in a per
> > queue way.
> >
> 
> Oh, I see. devmem does support mapping a single buffer to multiple
> queues in a single netlink API call, but there is nothing stopping the
> user from mapping N buffers to N queues in N netlink API calls.
Oh, yes, of course. Why didn't I think of that...

Thanks,
Dragos