Page sizes supported by RISC-V

Page sizes supported by RISC-V

[Florian Weimer]

I'm working on adding a <sys/pagesize.h> header to glibc, so that
programmers can easily determine the range of possible page sizes for a
particular target, for things like pointer tagging and mapped file
alignment requirements.

Linux currently supports 4 KiB pages only.  I couldn't find anything in
the RISC-V specifications that mandates a specific page size, but of
course the way ELF LOAD segments are constructed necessarily imposes a
maximum page size constraint for particular executables.  Today, the
toolchain assumes the maximum supported page size is 4 KiB, and this is
also reflected in arch/riscv/Kconfig in the Linux sources (as far as I
can read Kconfig).

I found a previous proposal for kernel support of 64 KiB pages:

  [RFC PATCH V1 00/11] riscv: Introduce 64K base page
  <https://lore.kernel.org/linux-riscv/20231123065708.91345-1-luxu.kernel@bytedance.com/>

Is this something we should consider today?

I'm leaning towards teaching glibc that the RISC-V page size is always
4 KiB because given the current toolchain defaults, it's necessary
to rebuild the world anyway if supported page sizes ever change.

Thanks,
Florian


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [External] Page sizes supported by RISC-V

[Xu Lu]

Hi Florian,


On Sun, Sep 29, 2024 at 12:36 AM Florian Weimer <fweimer@redhat.com> wrote:
>
> I'm working on adding a <sys/pagesize.h> header to glibc, so that
> programmers can easily determine the range of possible page sizes for a
> particular target, for things like pointer tagging and mapped file
> alignment requirements.
>
> Linux currently supports 4 KiB pages only.  I couldn't find anything in
> the RISC-V specifications that mandates a specific page size, but of
> course the way ELF LOAD segments are constructed necessarily imposes a
> maximum page size constraint for particular executables.  Today, the
> toolchain assumes the maximum supported page size is 4 KiB, and this is
> also reflected in arch/riscv/Kconfig in the Linux sources (as far as I
> can read Kconfig).

Yes, RISC-V kernel supports only 4K page size now. I personally tend
to support more page sizes. We have tested 64K kernel performance on
ARM Ampere Altra platform and found multiple memory-intensive apps'
performance based on 64K page kernel is significantly higher than that
on 4K page kernel.

>
> I found a previous proposal for kernel support of 64 KiB pages:
>
>   [RFC PATCH V1 00/11] riscv: Introduce 64K base page
>   <https://lore.kernel.org/linux-riscv/20231123065708.91345-1-luxu.kernel@bytedance.com/>

This is our first version of 64K page implementation[0], which is
based on Svnapot extension as RISC-V does not support 64K MMU yet. The
main idea of it is allocating/mapping pages in 64K granularity and
always applying Svnapot on ptes. The first version is only a demo and
still has some bugs. We have fixed existing problems and tested its
performance, but found that its performance benefit is not as
significant as we received on ARM 64K. We think the performance
benefit degradation mainly comes from software logic of imitating 64K.
For example, when need to tlb flush a page, instead of using a single
tlb flush instruction, we have to flush all 4K pages of a single 64K
page in a loop logic. And when need to check pte access/dirty bit,
instead of checking a single pte, we have to check all 4K ptes of a
single 64K page in a loop logic. The next version of patches and its
performance data is still being organized and will be published as
soon as possible.

Thus, we are also trying to call for native 64K MMU in RISC-V
community. You can find the discussion in link [1]. We hope more
people can participate in the discussion and provide more data
support.

[0] https://lore.kernel.org/linux-riscv/20231123065708.91345-1-luxu.kernel@bytedance.com/
[1] https://lists.riscv.org/g/tech-privileged/topic/query_about_risc_v_s_support/108641509

Best regards,

Xu Lu

>
> Is this something we should consider today?
>
> I'm leaning towards teaching glibc that the RISC-V page size is always
> 4 KiB because given the current toolchain defaults, it's necessary
> to rebuild the world anyway if supported page sizes ever change.
>
> Thanks,
> Florian
>

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [External] Page sizes supported by RISC-V

[Florian Weimer]

* Xu Lu:

> Thus, we are also trying to call for native 64K MMU in RISC-V
> community. You can find the discussion in link [1]. We hope more
> people can participate in the discussion and provide more data
> support.

If you are seriously considering offering 64K pages as an option, you
need to clarify the specifications and update binutils *today* to
increase the default maximum page size for the binaries the linker
generates.  The transition will only become harder as adoption grows.

Thanks,
Florian


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [External] Page sizes supported by RISC-V

[Xu Lu]

On Sun, Sep 29, 2024 at 1:04 PM Florian Weimer <fweimer@redhat.com> wrote:
>
> * Xu Lu:
>
> > Thus, we are also trying to call for native 64K MMU in RISC-V
> > community. You can find the discussion in link [1]. We hope more
> > people can participate in the discussion and provide more data
> > support.
>
> If you are seriously considering offering 64K pages as an option, you
> need to clarify the specifications and update binutils *today* to
> increase the default maximum page size for the binaries the linker
> generates.  The transition will only become harder as adoption grows.

Thanks. And yes, we have sent related patches to LLVM[0]. The patches
to binutils are on the way. We will appreciate it very much if you can
help to review and provide any suggestions.

[0] https://github.com/llvm/llvm-project/pull/100995

Xu Lu

>
> Thanks,
> Florian
>

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: Page sizes supported by RISC-V

[Jeff Law]

On 9/28/24 10:35 AM, Florian Weimer wrote:
> I'm working on adding a <sys/pagesize.h> header to glibc, so that
> programmers can easily determine the range of possible page sizes for a
> particular target, for things like pointer tagging and mapped file
> alignment requirements.
> 
> Linux currently supports 4 KiB pages only.  I couldn't find anything in
> the RISC-V specifications that mandates a specific page size, but of
> course the way ELF LOAD segments are constructed necessarily imposes a
> maximum page size constraint for particular executables.  Today, the
> toolchain assumes the maximum supported page size is 4 KiB, and this is
> also reflected in arch/riscv/Kconfig in the Linux sources (as far as I
> can read Kconfig).
4k is probably the most common in practice right now.  RISC-V generally 
seems to leave this kind of thing to the implementors to decide, so I 
wouldn't be at all surprised if there's nothing in the privileged spec 
other than specifying a minimum page size.

  Jeff

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: Page sizes supported by RISC-V

[Florian Weimer]

* Jeff Law:

> On 9/28/24 10:35 AM, Florian Weimer wrote:
>> I'm working on adding a <sys/pagesize.h> header to glibc, so that
>> programmers can easily determine the range of possible page sizes for a
>> particular target, for things like pointer tagging and mapped file
>> alignment requirements.
>> Linux currently supports 4 KiB pages only.  I couldn't find anything
>> in
>> the RISC-V specifications that mandates a specific page size, but of
>> course the way ELF LOAD segments are constructed necessarily imposes a
>> maximum page size constraint for particular executables.  Today, the
>> toolchain assumes the maximum supported page size is 4 KiB, and this is
>> also reflected in arch/riscv/Kconfig in the Linux sources (as far as I
>> can read Kconfig).

> 4k is probably the most common in practice right now.  RISC-V
> generally seems to leave this kind of thing to the implementors to
> decide, so I wouldn't be at all surprised if there's nothing in the
> privileged spec other than specifying a minimum page size.

I understand that not making a call in the specifications avoids the
need to resolve these conflicts.  But if you don't define a maximum page
size there, it is effectively specified as 4096 bytes.  This will cause
problems if distributions targeting large systems want a larger page
size because it better aligns with the kind of hardware they target.
It's really bad for cross-distribution binary compatibility, something
that's important outside the embedded space and probably necessary for
mainstreaming.

(Of course you know all this. 8-)

Thanks,
Florian


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: Page sizes supported by RISC-V

[Jeff Law]

On 9/29/24 9:08 AM, Florian Weimer wrote:
> 
> I understand that not making a call in the specifications avoids the
> need to resolve these conflicts.  But if you don't define a maximum page
> size there, it is effectively specified as 4096 bytes.  This will cause
> problems if distributions targeting large systems want a larger page
> size because it better aligns with the kind of hardware they target.
> It's really bad for cross-distribution binary compatibility, something
> that's important outside the embedded space and probably necessary for
> mainstreaming.
> 
> (Of course you know all this. 8-)
Yup.  Once it's baked into one significant distro, it's doing to be 
bloody hard to change.  I'm sure we both remember the PPC pagesize stuff 
from a few years back ;(

I strongly suspect the lack of specification here is mean to give 
degrees of freedom to the implementors, but sometimes those writing the 
specs don't really understand the implication of leaving things like 
this unspecified and how much pain it really causes in the end.

Jeff


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: Page sizes supported by RISC-V

[Maciej W. Rozycki]

On Sun, 29 Sep 2024, Jeff Law wrote:

> I strongly suspect the lack of specification here is mean to give degrees of
> freedom to the implementors, but sometimes those writing the specs don't
> really understand the implication of leaving things like this unspecified and
> how much pain it really causes in the end.

 But 64KiB pages have already been mentioned, so sanctioning this possible 
size in binutils (or LD specifically) would be a reasonable thing to do at 
this time IMHO.

 FWIW out of the plethora sizes (from 1KiB up to 256TiB) possible to have
in hardware (and available on a page-by-page basis) the MIPS/Linux port 
has settled on a fixed for a given kernel configuration page size choice 
among 4KiB, 16KiB, 64KiB, and this has been reflected in binutils, so 
perhaps copying the MIPS bits might be the path of least resistance.

  Maciej

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: Page sizes supported by RISC-V

[Palmer Dabbelt]

On Sun, 29 Sep 2024 08:49:34 PDT (-0700), macro@orcam.me.uk wrote:
> On Sun, 29 Sep 2024, Jeff Law wrote:
>
>> I strongly suspect the lack of specification here is mean to give degrees of
>> freedom to the implementors, but sometimes those writing the specs don't
>> really understand the implication of leaving things like this unspecified and
>> how much pain it really causes in the end.
>
>  But 64KiB pages have already been mentioned, so sanctioning this possible
> size in binutils (or LD specifically) would be a reasonable thing to do at
> this time IMHO.
>
>  FWIW out of the plethora sizes (from 1KiB up to 256TiB) possible to have
> in hardware (and available on a page-by-page basis) the MIPS/Linux port
> has settled on a fixed for a given kernel configuration page size choice
> among 4KiB, 16KiB, 64KiB, and this has been reflected in binutils, so
> perhaps copying the MIPS bits might be the path of least resistance.

This comes up every once in a while and we end generally end up deciding 
we're just stuck with a 4KiB base on RISC-V.  We've got 64KiB fused 
pages in the ISA now, so getting userspace 64KiB aligned whenever 
possible would be a nice to have, but that doesn't get rid of the 4KiB 
base page size.

When someone wants larger base page sizes then they're going to be stuck 
with all the pain that every other port had to go through to make that 
all fit together.  It's a bummer we didn't sort that out ahead of time, 
but we've got a lot of baggage to unwind ;)

>   Maciej
>
> _______________________________________________
> linux-riscv mailing list
> linux-riscv@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-riscv

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: Page sizes supported by RISC-V

[Jeff Law]

On 9/29/24 9:18 AM, Jeff Law wrote:
> 
> 
> On 9/29/24 9:08 AM, Florian Weimer wrote:
>>
>> I understand that not making a call in the specifications avoids the
>> need to resolve these conflicts.  But if you don't define a maximum page
>> size there, it is effectively specified as 4096 bytes.  This will cause
>> problems if distributions targeting large systems want a larger page
>> size because it better aligns with the kind of hardware they target.
>> It's really bad for cross-distribution binary compatibility, something
>> that's important outside the embedded space and probably necessary for
>> mainstreaming.
>>
>> (Of course you know all this. 8-)
> Yup.  Once it's baked into one significant distro, it's doing to be 
> bloody hard to change.  I'm sure we both remember the PPC pagesize stuff 
> from a few years back ;(
> 
> I strongly suspect the lack of specification here is mean to give 
> degrees of freedom to the implementors, but sometimes those writing the 
> specs don't really understand the implication of leaving things like 
> this unspecified and how much pain it really causes in the end.
So resurrecting this discussion.

Is there any realistic chance of bumping the minimum pagesize for risc-v 
at this point?  Xu Lu and his team have some compelling data that 
indicates moving forward to 64k could be highly profitable.  That work 
was done on aarch64, but should be generally applicable to riscv as well.

My biggest worry is if we don't tackle this now, then I suspect the 
window closes forever and I'd hate for the spacemit chip in the BPI to 
mandate something like this going forward.

jeff

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: Page sizes supported by RISC-V

[Maciej W. Rozycki]

On Tue, 14 Jan 2025, Jeff Law wrote:

> >> I understand that not making a call in the specifications avoids the
> >> need to resolve these conflicts.  But if you don't define a maximum page
> >> size there, it is effectively specified as 4096 bytes.  This will cause
> >> problems if distributions targeting large systems want a larger page
> >> size because it better aligns with the kind of hardware they target.
> >> It's really bad for cross-distribution binary compatibility, something
> >> that's important outside the embedded space and probably necessary for
> >> mainstreaming.
> >>
> >> (Of course you know all this. 8-)
> > Yup.  Once it's baked into one significant distro, it's doing to be 
> > bloody hard to change.  I'm sure we both remember the PPC pagesize stuff 
> > from a few years back ;(
> > 
> > I strongly suspect the lack of specification here is mean to give 
> > degrees of freedom to the implementors, but sometimes those writing the 
> > specs don't really understand the implication of leaving things like 
> > this unspecified and how much pain it really causes in the end.
> So resurrecting this discussion.
> 
> Is there any realistic chance of bumping the minimum pagesize for risc-v 
> at this point?  Xu Lu and his team have some compelling data that 
> indicates moving forward to 64k could be highly profitable.  That work 
> was done on aarch64, but should be generally applicable to riscv as well.

 Can't we simply make a reasonable guess/assumption just for the purpose 
of tooling?

 For example MIPS hardware has supported page sizes of up to 256MiB since 
forever and ones of up to 256TiB since ~R3, but we chose to support sizes 
of up to 64KiB only on the software side.  And that was some 20 years ago, 
back when hardware resources were considerably more limited and therefore 
extra VM space consumption from getting things aligned to a larger page 
size was more painful.

 Based on the experience with MIPS/AArch64 shared here there should be no
disagreement as to going for at least 64KiB I suppose.  Then shall we go 
beyond to leave room for some expansion, such as for 256KiB?

  Maciej

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: Page sizes supported by RISC-V

[Florian Weimer]

* Maciej W. Rozycki:

> On Tue, 14 Jan 2025, Jeff Law wrote:
>
>> >> I understand that not making a call in the specifications avoids the
>> >> need to resolve these conflicts.  But if you don't define a maximum page
>> >> size there, it is effectively specified as 4096 bytes.  This will cause
>> >> problems if distributions targeting large systems want a larger page
>> >> size because it better aligns with the kind of hardware they target.
>> >> It's really bad for cross-distribution binary compatibility, something
>> >> that's important outside the embedded space and probably necessary for
>> >> mainstreaming.
>> >>
>> >> (Of course you know all this. 8-)
>> > Yup.  Once it's baked into one significant distro, it's doing to be 
>> > bloody hard to change.  I'm sure we both remember the PPC pagesize stuff 
>> > from a few years back ;(
>> > 
>> > I strongly suspect the lack of specification here is mean to give 
>> > degrees of freedom to the implementors, but sometimes those writing the 
>> > specs don't really understand the implication of leaving things like 
>> > this unspecified and how much pain it really causes in the end.
>> So resurrecting this discussion.
>> 
>> Is there any realistic chance of bumping the minimum pagesize for risc-v 
>> at this point?  Xu Lu and his team have some compelling data that 
>> indicates moving forward to 64k could be highly profitable.  That work 
>> was done on aarch64, but should be generally applicable to riscv as well.
>
> Can't we simply make a reasonable guess/assumption just for the purpose 
> of tooling?

That reasonable default exists today and is 4096.  Some effort is
required to change that.

> Based on the experience with MIPS/AArch64 shared here there should be no
> disagreement as to going for at least 64KiB I suppose.  Then shall we go 
> beyond to leave room for some expansion, such as for 256KiB?

The alternative would be to fix Linux so that it supports multiple page
sizes.  Then from an application perspective (and dynamic linker
perspective), the page size could remain at 4096 bytes.  I understand
this is difficult because once you have multiple page sizes, there are
concerns regarding fragmentation.

Thanks,
Florian


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: Page sizes supported by RISC-V

[Maciej W. Rozycki]

On Wed, 15 Jan 2025, Florian Weimer wrote:

> >> >> I understand that not making a call in the specifications avoids the
> >> >> need to resolve these conflicts.  But if you don't define a maximum page
> >> >> size there, it is effectively specified as 4096 bytes.  This will cause
> >> >> problems if distributions targeting large systems want a larger page
> >> >> size because it better aligns with the kind of hardware they target.
> >> >> It's really bad for cross-distribution binary compatibility, something
> >> >> that's important outside the embedded space and probably necessary for
> >> >> mainstreaming.
> >> >>
> >> >> (Of course you know all this. 8-)
> >> > Yup.  Once it's baked into one significant distro, it's doing to be 
> >> > bloody hard to change.  I'm sure we both remember the PPC pagesize stuff 
> >> > from a few years back ;(
> >> > 
> >> > I strongly suspect the lack of specification here is mean to give 
> >> > degrees of freedom to the implementors, but sometimes those writing the 
> >> > specs don't really understand the implication of leaving things like 
> >> > this unspecified and how much pain it really causes in the end.
> >> So resurrecting this discussion.
> >> 
> >> Is there any realistic chance of bumping the minimum pagesize for risc-v 
> >> at this point?  Xu Lu and his team have some compelling data that 
> >> indicates moving forward to 64k could be highly profitable.  That work 
> >> was done on aarch64, but should be generally applicable to riscv as well.
> >
> > Can't we simply make a reasonable guess/assumption just for the purpose 
> > of tooling?
> 
> That reasonable default exists today and is 4096.  Some effort is
> required to change that.

 I thought we were talking about the maximum rather than the default.  I 
don't think changing the default causes any fuss as long as it's between 
previously established minimum and maximum.

  Maciej

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: Page sizes supported by RISC-V

[Jeff Law]

On 1/15/25 12:51 AM, Florian Weimer wrote:
> * Maciej W. Rozycki:
> 
>> On Tue, 14 Jan 2025, Jeff Law wrote:
>>
>>>>> I understand that not making a call in the specifications avoids the
>>>>> need to resolve these conflicts.  But if you don't define a maximum page
>>>>> size there, it is effectively specified as 4096 bytes.  This will cause
>>>>> problems if distributions targeting large systems want a larger page
>>>>> size because it better aligns with the kind of hardware they target.
>>>>> It's really bad for cross-distribution binary compatibility, something
>>>>> that's important outside the embedded space and probably necessary for
>>>>> mainstreaming.
>>>>>
>>>>> (Of course you know all this. 8-)
>>>> Yup.  Once it's baked into one significant distro, it's doing to be
>>>> bloody hard to change.  I'm sure we both remember the PPC pagesize stuff
>>>> from a few years back ;(
>>>>
>>>> I strongly suspect the lack of specification here is mean to give
>>>> degrees of freedom to the implementors, but sometimes those writing the
>>>> specs don't really understand the implication of leaving things like
>>>> this unspecified and how much pain it really causes in the end.
>>> So resurrecting this discussion.
>>>
>>> Is there any realistic chance of bumping the minimum pagesize for risc-v
>>> at this point?  Xu Lu and his team have some compelling data that
>>> indicates moving forward to 64k could be highly profitable.  That work
>>> was done on aarch64, but should be generally applicable to riscv as well.
>>
>> Can't we simply make a reasonable guess/assumption just for the purpose
>> of tooling?
> 
> That reasonable default exists today and is 4096.  Some effort is
> required to change that.
Right.  Any change in this value is a rebuild-the-world effort with a 
flag day for binary compatibilty.  Plus there's issues for cross distro 
compatibility and even questions about whether or not any existing 
systems could support anything but 4k pagesizes.



> 
>> Based on the experience with MIPS/AArch64 shared here there should be no
>> disagreement as to going for at least 64KiB I suppose.  Then shall we go
>> beyond to leave room for some expansion, such as for 256KiB?
> 
> The alternative would be to fix Linux so that it supports multiple page
> sizes.  Then from an application perspective (and dynamic linker
> perspective), the page size could remain at 4096 bytes.  I understand
> this is difficult because once you have multiple page sizes, there are
> concerns regarding fragmentation.
So essentially break the hard tie between the kernel and usercode and 
allow them to vary independently...  Hmm....

jeff

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv