[GIT PULL] Please pull mlx5 VFIO PCI DMA conversion

[GIT PULL] Please pull mlx5 VFIO PCI DMA conversion

[Leon Romanovsky]

Hi Alex,

Please accept this pull request, which presents subset of new DMA-API
patchset [1] specific for VFIO subsystem.

It is based on Marek's dma-mapping-for-6.16-two-step-api branch, so merging
now will allow us to reduce possible rebase errors in mlx5 vfio code and give
enough time to start to work on second driver conversion. Such conversion will
allow us to generalize the API for VFIO kernel drivers, in similiar way that
was done for RDMA, HMM and block layers.

Thanks

[1] [PATCH v10 00/24] Provide a new two step DMA mapping API
https://lore.kernel.org/all/cover.1745831017.git.leon@kernel.org/

----------------------------------------------------------------
The following changes since commit 3ee7d9496342246f4353716f6bbf64c945ff6e2d:

  docs: core-api: document the IOVA-based API (2025-05-06 08:36:54 +0200)

are available in the Git repository at:

  https://git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git tags/vfio-dma-two-step

for you to fetch changes up to 855c89a25e1756b7699b863afd4d6afcbd0de0d6:

  vfio/mlx5: Enable the DMA link API (2025-05-13 03:58:27 -0400)

----------------------------------------------------------------
Convert mlx5 VFIO PCI driver to new two step DMA API

This PR is based on newly accepted DMA API, which allows us
to avoid building scatter-gather lists just to batch mapping
and unmapping of pages.

VFIO PCI live migration code is building a very large "page list"
for the device. Instead of allocating a scatter list entry per
allocated page it can just allocate an array of 'struct page *',
saving a large amount of memory.

Signed-off-by: Leon Romanovsky <leon@kernel.org>

----------------------------------------------------------------
Leon Romanovsky (3):
      vfio/mlx5: Explicitly use number of pages instead of allocated length
      vfio/mlx5: Rewrite create mkey flow to allow better code reuse
      vfio/mlx5: Enable the DMA link API

 drivers/vfio/pci/mlx5/cmd.c  | 375 +++++++++++++++++++++----------------------
 drivers/vfio/pci/mlx5/cmd.h  |  35 ++--
 drivers/vfio/pci/mlx5/main.c |  87 +++++-----
 3 files changed, 239 insertions(+), 258 deletions(-)

Re: [GIT PULL] Please pull mlx5 VFIO PCI DMA conversion

[Alex Williamson]

On Tue, 13 May 2025 13:48:10 +0300
Leon Romanovsky <leon@kernel.org> wrote:

> Hi Alex,
> 
> Please accept this pull request, which presents subset of new DMA-API
> patchset [1] specific for VFIO subsystem.
> 
> It is based on Marek's dma-mapping-for-6.16-two-step-api branch, so merging
> now will allow us to reduce possible rebase errors in mlx5 vfio code and give
> enough time to start to work on second driver conversion. Such conversion will
> allow us to generalize the API for VFIO kernel drivers, in similiar way that
> was done for RDMA, HMM and block layers.

Hi Leon,

Pull requests are not my typical workflow.  Are these mlx5-vfio-pci
changes intended to enter mainline through the vfio tree or your rdma
tree?  Why do the commits not include a review/ack from Yishai?

Typically I'd expect a patch series for the mlx5-vfio-pci changes that
I would apply, with Yishai's approval, to a shared branch containing the
commits Marek has already accepted.  I'm not sure why we're preempting
that process here.  Thanks,

Alex

> [1] [PATCH v10 00/24] Provide a new two step DMA mapping API
> https://lore.kernel.org/all/cover.1745831017.git.leon@kernel.org/
> 
> ----------------------------------------------------------------
> The following changes since commit 3ee7d9496342246f4353716f6bbf64c945ff6e2d:
> 
>   docs: core-api: document the IOVA-based API (2025-05-06 08:36:54 +0200)
> 
> are available in the Git repository at:
> 
>   https://git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git tags/vfio-dma-two-step
> 
> for you to fetch changes up to 855c89a25e1756b7699b863afd4d6afcbd0de0d6:
> 
>   vfio/mlx5: Enable the DMA link API (2025-05-13 03:58:27 -0400)
> 
> ----------------------------------------------------------------
> Convert mlx5 VFIO PCI driver to new two step DMA API
> 
> This PR is based on newly accepted DMA API, which allows us
> to avoid building scatter-gather lists just to batch mapping
> and unmapping of pages.
> 
> VFIO PCI live migration code is building a very large "page list"
> for the device. Instead of allocating a scatter list entry per
> allocated page it can just allocate an array of 'struct page *',
> saving a large amount of memory.
> 
> Signed-off-by: Leon Romanovsky <leon@kernel.org>
> 
> ----------------------------------------------------------------
> Leon Romanovsky (3):
>       vfio/mlx5: Explicitly use number of pages instead of allocated length
>       vfio/mlx5: Rewrite create mkey flow to allow better code reuse
>       vfio/mlx5: Enable the DMA link API
> 
>  drivers/vfio/pci/mlx5/cmd.c  | 375 +++++++++++++++++++++----------------------
>  drivers/vfio/pci/mlx5/cmd.h  |  35 ++--
>  drivers/vfio/pci/mlx5/main.c |  87 +++++-----
>  3 files changed, 239 insertions(+), 258 deletions(-)
> 

Re: [GIT PULL] Please pull mlx5 VFIO PCI DMA conversion

[Leon Romanovsky]

On Thu, May 15, 2025 at 11:47:15AM -0600, Alex Williamson wrote:
> On Tue, 13 May 2025 13:48:10 +0300
> Leon Romanovsky <leon@kernel.org> wrote:
> 
> > Hi Alex,
> > 
> > Please accept this pull request, which presents subset of new DMA-API
> > patchset [1] specific for VFIO subsystem.
> > 
> > It is based on Marek's dma-mapping-for-6.16-two-step-api branch, so merging
> > now will allow us to reduce possible rebase errors in mlx5 vfio code and give
> > enough time to start to work on second driver conversion. Such conversion will
> > allow us to generalize the API for VFIO kernel drivers, in similiar way that
> > was done for RDMA, HMM and block layers.
> 
> Hi Leon,
> 
> Pull requests are not my typical workflow.  Are these mlx5-vfio-pci
> changes intended to enter mainline through the vfio tree or your rdma
> tree?

VFIO changes will come through your tree. DMA patches are the same as in
Marek's DMA tree and in our RDMA tree.

I prepared PR to save from your hassle of merging dma/dma-mapping-for-6.16-two-step-api
topic from Marek and collecting VFIO patches from ML.

> Why do the commits not include a review/ack from Yishai?

They have Jason's review tags and as far as I know Yishai, he trusts
Jason's judgement.

> 
> Typically I'd expect a patch series for the mlx5-vfio-pci changes that
> I would apply, with Yishai's approval, to a shared branch containing the
> commits Marek has already accepted.  I'm not sure why we're preempting
> that process here.  Thanks,

This is exactly what is in this PR: reviewed VFIO patches which were
posted to the ML on top of Marek's shared branch.

If you prefer, I can repost the VFIO patches.

Thanks

> 
> Alex
> 
> > [1] [PATCH v10 00/24] Provide a new two step DMA mapping API
> > https://lore.kernel.org/all/cover.1745831017.git.leon@kernel.org/
> > 
> > ----------------------------------------------------------------
> > The following changes since commit 3ee7d9496342246f4353716f6bbf64c945ff6e2d:
> > 
> >   docs: core-api: document the IOVA-based API (2025-05-06 08:36:54 +0200)
> > 
> > are available in the Git repository at:
> > 
> >   https://git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git tags/vfio-dma-two-step
> > 
> > for you to fetch changes up to 855c89a25e1756b7699b863afd4d6afcbd0de0d6:
> > 
> >   vfio/mlx5: Enable the DMA link API (2025-05-13 03:58:27 -0400)
> > 
> > ----------------------------------------------------------------
> > Convert mlx5 VFIO PCI driver to new two step DMA API
> > 
> > This PR is based on newly accepted DMA API, which allows us
> > to avoid building scatter-gather lists just to batch mapping
> > and unmapping of pages.
> > 
> > VFIO PCI live migration code is building a very large "page list"
> > for the device. Instead of allocating a scatter list entry per
> > allocated page it can just allocate an array of 'struct page *',
> > saving a large amount of memory.
> > 
> > Signed-off-by: Leon Romanovsky <leon@kernel.org>
> > 
> > ----------------------------------------------------------------
> > Leon Romanovsky (3):
> >       vfio/mlx5: Explicitly use number of pages instead of allocated length
> >       vfio/mlx5: Rewrite create mkey flow to allow better code reuse
> >       vfio/mlx5: Enable the DMA link API
> > 
> >  drivers/vfio/pci/mlx5/cmd.c  | 375 +++++++++++++++++++++----------------------
> >  drivers/vfio/pci/mlx5/cmd.h  |  35 ++--
> >  drivers/vfio/pci/mlx5/main.c |  87 +++++-----
> >  3 files changed, 239 insertions(+), 258 deletions(-)
> > 
> 

Re: [GIT PULL] Please pull mlx5 VFIO PCI DMA conversion

[Alex Williamson]

On Thu, 15 May 2025 22:51:16 +0300
Leon Romanovsky <leon@kernel.org> wrote:

> On Thu, May 15, 2025 at 11:47:15AM -0600, Alex Williamson wrote:
> > On Tue, 13 May 2025 13:48:10 +0300
> > Leon Romanovsky <leon@kernel.org> wrote:
> >   
> > > Hi Alex,
> > > 
> > > Please accept this pull request, which presents subset of new DMA-API
> > > patchset [1] specific for VFIO subsystem.
> > > 
> > > It is based on Marek's dma-mapping-for-6.16-two-step-api branch, so merging
> > > now will allow us to reduce possible rebase errors in mlx5 vfio code and give
> > > enough time to start to work on second driver conversion. Such conversion will
> > > allow us to generalize the API for VFIO kernel drivers, in similiar way that
> > > was done for RDMA, HMM and block layers.  
> > 
> > Hi Leon,
> > 
> > Pull requests are not my typical workflow.  Are these mlx5-vfio-pci
> > changes intended to enter mainline through the vfio tree or your rdma
> > tree?  
> 
> VFIO changes will come through your tree. DMA patches are the same as in
> Marek's DMA tree and in our RDMA tree.

Ok
 
> I prepared PR to save from your hassle of merging dma/dma-mapping-for-6.16-two-step-api
> topic from Marek and collecting VFIO patches from ML.
> 
> > Why do the commits not include a review/ack from Yishai?  
> 
> They have Jason's review tags and as far as I know Yishai, he trusts
> Jason's judgement.

I don't doubt that, but Yishai is listed as the maintainer, so barring
some special circumstances we should at least have an ack.

> > Typically I'd expect a patch series for the mlx5-vfio-pci changes that
> > I would apply, with Yishai's approval, to a shared branch containing the
> > commits Marek has already accepted.  I'm not sure why we're preempting
> > that process here.  Thanks,  
> 
> This is exactly what is in this PR: reviewed VFIO patches which were
> posted to the ML on top of Marek's shared branch.
> 
> If you prefer, I can repost the VFIO patches.

I appreciate the attempt to streamline things and expect this largely
only affects the chain of sign-off approvals, but at the same time
that's an important aspect.  If the code here is identical to v10, we
could use that with Yishai's ack, but it's probably most direct (irony
vs PR noted) to repost just just the mlx5-vfio-pci changes with a
reference to Marek's shared topic branch.  Thanks,

Alex