Conditions for FOLL_LONGTERM mapping in fsdax

Conditions for FOLL_LONGTERM mapping in fsdax

[Usama Arif]

Hi,

We wanted to run a VM with a vfio device assigned to it and with its 
memory-backend-file residing in a persistent memory using fsdax (mounted 
as ext4). It doesnt currently work with the kernel as 
vfio_pin_pages_remote ends up requesting pages with FOLL_LONGTERM which 
is currently not supported. From reading the mailing list, what I 
understood was that this is to do with not having DMA supported on fsdax 
due to issues that come up during truncate/hole-punching. But it was 
solved with [1] by deferring fallocate(), truncate() on a dax mode file 
while any page/block in the file is under active DMA.

If I remove the check which fails the gup opertion with the below diff, 
the VM boots and the vfio device works without any issues. If I try to 
truncate the mem file in fsdax, I can see that the truncate command gets 
deferred (waits in ext4_break_layouts) and the vfio device keeps working 
and sending packets without any issues. Just wanted to check what is 
missing to allow FOLL_LONGTERM gup operations with fsdax? Is it just 
enough to remove the check? Thanks!


diff --git a/mm/gup.c b/mm/gup.c
index eb8d7baf9e4d..f77bb428cf9b 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -1055,9 +1055,6 @@ static int check_vma_flags(struct vm_area_struct 
*vma, unsigned long gup_flags)
         if (gup_flags & FOLL_ANON && !vma_is_anonymous(vma))
                 return -EFAULT;

-       if ((gup_flags & FOLL_LONGTERM) && vma_is_fsdax(vma))
-               return -EOPNOTSUPP;
-
         if (vma_is_secretmem(vma))
                 return -EFAULT;


[1] 
https://lore.kernel.org/all/152669371377.34337.10697370528066177062.stgit@dwillia2-desk3.amr.corp.intel.com/

Regards,
Usama

Conditions for FOLL_LONGTERM mapping in fsdax

[Usama Arif]

Hi,

We wanted to run a VM with a vfio device assigned to it and with its 
memory-backend-file residing in a persistent memory using fsdax (mounted 
as ext4). It doesnt currently work with the kernel as 
vfio_pin_pages_remote ends up requesting pages with FOLL_LONGTERM which 
is currently not supported. From reading the mailing list, what I 
understood was that this is to do with not having DMA supported on fsdax 
due to issues that come up during truncate/hole-punching. But it was 
solved with [1] by deferring fallocate(), truncate() on a dax mode file 
while any page/block in the file is under active DMA.

If I remove the check which fails the gup opertion with the below diff, 
the VM boots and the vfio device works without any issues. If I try to 
truncate the mem file in fsdax, I can see that the truncate command gets 
deferred (waits in ext4_break_layouts) and the vfio device keeps working 
and sending packets without any issues. Just wanted to check what is 
missing to allow FOLL_LONGTERM gup operations with fsdax? Is it just 
enough to remove the check? Thanks!


diff --git a/mm/gup.c b/mm/gup.c
index eb8d7baf9e4d..f77bb428cf9b 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -1055,9 +1055,6 @@ static int check_vma_flags(struct vm_area_struct 
*vma, unsigned long gup_flags)
         if (gup_flags & FOLL_ANON && !vma_is_anonymous(vma))
                 return -EFAULT;

-       if ((gup_flags & FOLL_LONGTERM) && vma_is_fsdax(vma))
-               return -EOPNOTSUPP;
-
         if (vma_is_secretmem(vma))
                 return -EFAULT;


[1] 
https://lore.kernel.org/all/152669371377.34337.10697370528066177062.stgit@dwillia2-desk3.amr.corp.intel.com/

Regards,
Usama

Re: Conditions for FOLL_LONGTERM mapping in fsdax

[Usama Arif]

On 06/11/2023 18:15, Usama Arif wrote:
> Hi,
> 
> We wanted to run a VM with a vfio device assigned to it and with its 
> memory-backend-file residing in a persistent memory using fsdax (mounted 
> as ext4). It doesnt currently work with the kernel as 
> vfio_pin_pages_remote ends up requesting pages with FOLL_LONGTERM which 
> is currently not supported. From reading the mailing list, what I 
> understood was that this is to do with not having DMA supported on fsdax 
> due to issues that come up during truncate/hole-punching. But it was 
> solved with [1] by deferring fallocate(), truncate() on a dax mode file 
> while any page/block in the file is under active DMA.
> 
> If I remove the check which fails the gup opertion with the below diff, 
> the VM boots and the vfio device works without any issues. If I try to 
> truncate the mem file in fsdax, I can see that the truncate command gets 
> deferred (waits in ext4_break_layouts) and the vfio device keeps working 
> and sending packets without any issues. Just wanted to check what is 
> missing to allow FOLL_LONGTERM gup operations with fsdax? Is it just 
> enough to remove the check? Thanks!
> 
> 
> diff --git a/mm/gup.c b/mm/gup.c
> index eb8d7baf9e4d..f77bb428cf9b 100644
> --- a/mm/gup.c
> +++ b/mm/gup.c
> @@ -1055,9 +1055,6 @@ static int check_vma_flags(struct vm_area_struct 
> *vma, unsigned long gup_flags)
>          if (gup_flags & FOLL_ANON && !vma_is_anonymous(vma))
>                  return -EFAULT;
> 
> -       if ((gup_flags & FOLL_LONGTERM) && vma_is_fsdax(vma))
> -               return -EOPNOTSUPP;
> -
>          if (vma_is_secretmem(vma))
>                  return -EFAULT;
> 
> 
> [1] 
> https://lore.kernel.org/all/152669371377.34337.10697370528066177062.stgit@dwillia2-desk3.amr.corp.intel.com/
> 

Hi,

Just wanted to check if there were any comments on this?

Thanks


> Regards,
> Usama

Re: Conditions for FOLL_LONGTERM mapping in fsdax

[Christoph Hellwig]

We don't have any way to recall the LONGTERM mappings, so we can't
support them on DAX for now.

Re: [External] Re: Conditions for FOLL_LONGTERM mapping in fsdax

[Usama Arif]

On 21/11/2023 04:46, Christoph Hellwig wrote:
> We don't have any way to recall the LONGTERM mappings, so we can't
> support them on DAX for now.
> 

By recall do you mean put the LONGTERM pages back? If I removed the 
check in check_vma and allowed the mappings to happen in fsdax, I can 
see that the mappings unmap/unpin in vfio_iommu_type1_unmap_dma later on 
which eventually ends up calling put_pfn.

Thanks

Re: [External] Re: Conditions for FOLL_LONGTERM mapping in fsdax

[Christoph Hellwig]

On Mon, Nov 27, 2023 at 11:52:15AM +0000, Usama Arif wrote:
> By recall do you mean put the LONGTERM pages back? If I removed the check in
> check_vma and allowed the mappings to happen in fsdax, I can see that the
> mappings unmap/unpin in vfio_iommu_type1_unmap_dma later on which eventually
> ends up calling put_pfn.

recall as in stop pinning them on notice from the page owner.