ARM64: Adding write-protect bit for Userfaultfd

ARM64: Adding write-protect bit for Userfaultfd

[Mohamed Husain Noor Mohamed]

Hi,

I am Mohamed Husain. I am a graduate student, working on a research
project using a userfaultfd for distributed shared memory.
We are trying to use Write-Protect mode in ARM64 but based on the
kernel commits we see the support only exists for the x86 kernel.
https://lwn.net/Articles/777258/

I am trying to add the write-protect support, so I am looking for the
unused bits in the PTE. Do you guys have any suggestions on the bits I
could use, or does it require hardware support?

Thanks,
Mohamed

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: ARM64: Adding write-protect bit for Userfaultfd

[Catalin Marinas]

On Fri, Apr 28, 2023 at 01:35:44PM -0400, Mohamed Husain Noor Mohamed wrote:
> I am Mohamed Husain. I am a graduate student, working on a research
> project using a userfaultfd for distributed shared memory.
> We are trying to use Write-Protect mode in ARM64 but based on the
> kernel commits we see the support only exists for the x86 kernel.
> https://lwn.net/Articles/777258/
> 
> I am trying to add the write-protect support, so I am looking for the
> unused bits in the PTE. Do you guys have any suggestions on the bits I
> could use, or does it require hardware support?

Unfortunately, we are pretty short on bits. The architecture only gives
us bits 55 to 58 and they are all used. We could move PTE_PROT_NONE to
another position (e.g. 60) since this is only used when !PTE_VALID and
therefore doesn't affect the actual page attributes and free up a bit.

Alternatively, we could hijack bits 59-62 but there may be out of tree
patches making use of the PBHA imp def feature (AFAICT disabled on the
mainline kernel). Well, I guess one could make the userfaultfd wp
feature conditional.

Cc'ing Anshuman as well, I think he looked at soft-dirty ptes for arm64
before for CRIU live migration and we concluded that userfaultfd was
better but I didn't realise that the write-protect mechanism needs its
own PTE bit as well.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: ARM64: Adding write-protect bit for Userfaultfd

[Mohamed Husain Noor Mohamed]

Hi Catalin,

Thanks for the feedback.

We were able to implement the feature with bits 60 and 61. As you
mentioned we made it a conditional feature.

PTE_UFFD_WP -> 60
PTE_SWP_UFFD_WP -> 61

We want to submit the patch(pull request) since our research requires
this feature in the vanilla kernel. Could you please share the steps
for that?

Thanks,
Mohamed

On Tue, May 2, 2023 at 9:36 AM Catalin Marinas <catalin.marinas@arm.com> wrote:
>
> On Fri, Apr 28, 2023 at 01:35:44PM -0400, Mohamed Husain Noor Mohamed wrote:
> > I am Mohamed Husain. I am a graduate student, working on a research
> > project using a userfaultfd for distributed shared memory.
> > We are trying to use Write-Protect mode in ARM64 but based on the
> > kernel commits we see the support only exists for the x86 kernel.
> > https://lwn.net/Articles/777258/
> >
> > I am trying to add the write-protect support, so I am looking for the
> > unused bits in the PTE. Do you guys have any suggestions on the bits I
> > could use, or does it require hardware support?
>
> Unfortunately, we are pretty short on bits. The architecture only gives
> us bits 55 to 58 and they are all used. We could move PTE_PROT_NONE to
> another position (e.g. 60) since this is only used when !PTE_VALID and
> therefore doesn't affect the actual page attributes and free up a bit.
>
> Alternatively, we could hijack bits 59-62 but there may be out of tree
> patches making use of the PBHA imp def feature (AFAICT disabled on the
> mainline kernel). Well, I guess one could make the userfaultfd wp
> feature conditional.
>
> Cc'ing Anshuman as well, I think he looked at soft-dirty ptes for arm64
> before for CRIU live migration and we concluded that userfaultfd was
> better but I didn't realise that the write-protect mechanism needs its
> own PTE bit as well.
>
> --
> Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: ARM64: Adding write-protect bit for Userfaultfd

[Joey Gouly]

Hi,

On Wed, May 17, 2023 at 11:43:29AM -0400, Mohamed Husain Noor Mohamed wrote:
> Hi Catalin,
> 
> Thanks for the feedback.
> 
> We were able to implement the feature with bits 60 and 61. As you
> mentioned we made it a conditional feature.
> 
> PTE_UFFD_WP -> 60
> PTE_SWP_UFFD_WP -> 61
> 
> We want to submit the patch(pull request) since our research requires
> this feature in the vanilla kernel. Could you please share the steps
> for that?
> 
> Thanks,
> Mohamed
> 

Bits 60-62 will be used by the VMSA 2022 extension called Permission Overlay Extension.
So if this userfaultd wp support is added and uses those bits, it will have to be mutually
exclusive with POE.

Thanks,
Joey

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: ARM64: Adding write-protect bit for Userfaultfd

[Catalin Marinas]

On Wed, May 17, 2023 at 11:43:29AM -0400, Mohamed Husain Noor Mohamed wrote:
> On Tue, May 2, 2023 at 9:36 AM Catalin Marinas <catalin.marinas@arm.com> wrote:
> > On Fri, Apr 28, 2023 at 01:35:44PM -0400, Mohamed Husain Noor Mohamed wrote:
> > > I am Mohamed Husain. I am a graduate student, working on a research
> > > project using a userfaultfd for distributed shared memory.
> > > We are trying to use Write-Protect mode in ARM64 but based on the
> > > kernel commits we see the support only exists for the x86 kernel.
> > > https://lwn.net/Articles/777258/
> > >
> > > I am trying to add the write-protect support, so I am looking for the
> > > unused bits in the PTE. Do you guys have any suggestions on the bits I
> > > could use, or does it require hardware support?
> >
> > Unfortunately, we are pretty short on bits. The architecture only gives
> > us bits 55 to 58 and they are all used. We could move PTE_PROT_NONE to
> > another position (e.g. 60) since this is only used when !PTE_VALID and
> > therefore doesn't affect the actual page attributes and free up a bit.
> >
> > Alternatively, we could hijack bits 59-62 but there may be out of tree
> > patches making use of the PBHA imp def feature (AFAICT disabled on the
> > mainline kernel). Well, I guess one could make the userfaultfd wp
> > feature conditional.
> 
> We were able to implement the feature with bits 60 and 61. As you
> mentioned we made it a conditional feature.
> 
> PTE_UFFD_WP -> 60

I forgot about permission overlays (FEAT_POE in the 2022 architecture
extensions), Joey mentioned this already. We have bit 59 left as
currently we don't use the Attribute Index Extension (part of the same
2022 spec).

> PTE_SWP_UFFD_WP -> 61

Presumably this is only checked on swap ptes. Can you not use the same
bit, 59?

> We want to submit the patch(pull request) since our research requires
> this feature in the vanilla kernel. Could you please share the steps
> for that?

Documentation/process/submitting-patches.rst is a good starting point ;).

The patches would need to be reviewed on the list first (e.g. on
linux-arm-kernel) together with justification since they are using
precious spare bits in the pte. In the past we turned down the use of
a spare bit for soft-dirty ptes (needed by CRIU), though the suggestion
was to rely on UFFD (and it looks like this one may need a bit as well).

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel