2GB userspace limitation in ABI N32

2GB userspace limitation in ABI N32

[Ronny Meeus]

Hello

I have a legacy application that we want to port to a MIPS (Cavium)
architecture from a PPC based one.
The board has 4GB memory of which we actually need almost 3GB in
application space. On the PPC this is no issue since the split
user/kernel is 3GB/1GB.
We have to use the N32 ABI Initial tests on MIPS showed me the
user-space limit of 2GB.
We do not want to port the application to a 64bit

Now the question is: are there any workarounds, tricks existing to get
around this limitation?
I found some mailthreads on this subject (n32-big ABI -
http://gcc.gnu.org/ml/gcc/2011-02/msg00278.html,
http://elinux.org/images/1/1f/New-tricks-mips-linux.pdf) but is looks
like this is not accepted by the community. Is there any process
planned or made in this area?

Thanks

---
Ronny

Re: 2GB userspace limitation in ABI N32

[Ralf Baechle]

On Wed, Oct 10, 2012 at 08:32:47AM +0200, Ronny Meeus wrote:

> I have a legacy application that we want to port to a MIPS (Cavium)
> architecture from a PPC based one.
> The board has 4GB memory of which we actually need almost 3GB in
> application space. On the PPC this is no issue since the split
> user/kernel is 3GB/1GB.
> We have to use the N32 ABI Initial tests on MIPS showed me the
> user-space limit of 2GB.
> We do not want to port the application to a 64bit
> 
> Now the question is: are there any workarounds, tricks existing to get
> around this limitation?
> I found some mailthreads on this subject (n32-big ABI -
> http://gcc.gnu.org/ml/gcc/2011-02/msg00278.html,
> http://elinux.org/images/1/1f/New-tricks-mips-linux.pdf) but is looks
> like this is not accepted by the community. Is there any process
> planned or made in this area?

I think limited time and gain killed the propoosed ABI rather than
theoretical issues raised.  Other architectures such as i386 - well,
IIRC any 32-bit ABI with more than 2GB userspace and a signed
ptrdiff_t - are suffering from them as well.

Also there's limited gain and even more limited time to implement things ...

  Ralf

Re: 2GB userspace limitation in ABI N32

[Rich Felker]

On Wed, Oct 10, 2012 at 10:07:56AM +0200, Ralf Baechle wrote:
> On Wed, Oct 10, 2012 at 08:32:47AM +0200, Ronny Meeus wrote:
> 
> > I have a legacy application that we want to port to a MIPS (Cavium)
> > architecture from a PPC based one.
> > The board has 4GB memory of which we actually need almost 3GB in
> > application space. On the PPC this is no issue since the split
> > user/kernel is 3GB/1GB.
> > We have to use the N32 ABI Initial tests on MIPS showed me the
> > user-space limit of 2GB.
> > We do not want to port the application to a 64bit
> > 
> > Now the question is: are there any workarounds, tricks existing to get
> > around this limitation?
> > I found some mailthreads on this subject (n32-big ABI -
> > http://gcc.gnu.org/ml/gcc/2011-02/msg00278.html,
> > http://elinux.org/images/1/1f/New-tricks-mips-linux.pdf) but is looks
> > like this is not accepted by the community. Is there any process
> > planned or made in this area?
> 
> I think limited time and gain killed the propoosed ABI rather than
> theoretical issues raised.  Other architectures such as i386 - well,
> IIRC any 32-bit ABI with more than 2GB userspace and a signed
> ptrdiff_t - are suffering from them as well.

There's no issue with ptrdiff_t being signed 32-bit as long as the
implementation does not allow individual objects larger than 2GB.
Taking differences between pointers into different objects is UB.

Rich

Re: 2GB userspace limitation in ABI N32

[David Daney]

On 10/10/2012 05:57 AM, Rich Felker wrote:
> On Wed, Oct 10, 2012 at 10:07:56AM +0200, Ralf Baechle wrote:
>> On Wed, Oct 10, 2012 at 08:32:47AM +0200, Ronny Meeus wrote:
>>
>>> I have a legacy application that we want to port to a MIPS (Cavium)
>>> architecture from a PPC based one.
>>> The board has 4GB memory of which we actually need almost 3GB in
>>> application space. On the PPC this is no issue since the split
>>> user/kernel is 3GB/1GB.
>>> We have to use the N32 ABI Initial tests on MIPS showed me the
>>> user-space limit of 2GB.
>>> We do not want to port the application to a 64bit
>>>
>>> Now the question is: are there any workarounds, tricks existing to get
>>> around this limitation?
>>> I found some mailthreads on this subject (n32-big ABI -
>>> http://gcc.gnu.org/ml/gcc/2011-02/msg00278.html,
>>> http://elinux.org/images/1/1f/New-tricks-mips-linux.pdf) but is looks
>>> like this is not accepted by the community. Is there any process
>>> planned or made in this area?
>>
>> I think limited time and gain killed the propoosed ABI rather than
>> theoretical issues raised.

Ralf, I and others have put some thought into doing this in the past. 
This is a rough plan for how it would be done:

1) Define a special ELF section/program header similar to GNU_STACK that 
would be used to mark binaries that could use the 4GB n32 extension. 
Modify GNU gas and ld to mark the binaries and properly propagate the 
markers.

2) Add a n32-4GB option to GCC.  In this mode pointers would be zero 
extended when loaded in to registers.  I have a, currently broken, 
prototype of this implemented.

3) Modify the Linux kernel.
3a) Add a thread_info flag to mark threads that use 4GB of address 
space, TASK_SIZE would then depend on this as well as the other TIF_* 
flags that it currently uses.

3b) Fix up the ELF loader to set the 4GB flag based on the program 
header from #1.

3c) Audit n32 system call entry points for places where pointers are 
sign extended.  Change them to zero extend.  There are not many of these.

4) Rebuild all system libraries to support n32-4G.

The only disadvantage of doing this is that the code will be slightly 
larger/slower as it takes three instructions to load a zero extended 
32-bit pointer verses two for n32-2GB.


>>  Other architectures such as i386 - well,
>> IIRC any 32-bit ABI with more than 2GB userspace and a signed
>> ptrdiff_t - are suffering from them as well.
>
> There's no issue with ptrdiff_t being signed 32-bit as long as the
> implementation does not allow individual objects larger than 2GB.
> Taking differences between pointers into different objects is UB.
>

No problem here.  We can just keep loading the VDSO at the 2GB point in 
the address space.  That will break things up so that all possible 
objects are smaller than 2GB.

Re: 2GB userspace limitation in ABI N32

[Maciej W. Rozycki]

On Wed, 10 Oct 2012, David Daney wrote:

> The only disadvantage of doing this is that the code will be slightly
> larger/slower as it takes three instructions to load a zero extended 32-bit
> pointer verses two for n32-2GB.

 And of course such code will only run on 64-bit processors that not only 
support 64-bit data, but 64-bit addressing as well.  That is implement the 
CP0.Status.UX bit rather than CP0.Status.PX only -- the latters are still 
compatible with the true n32 ABI.  See also CP0.Config.AT.

  Maciej

Re: 2GB userspace limitation in ABI N32

[David Daney]

On 10/10/2012 10:10 AM, Maciej W. Rozycki wrote:
> On Wed, 10 Oct 2012, David Daney wrote:
>
>> The only disadvantage of doing this is that the code will be slightly
>> larger/slower as it takes three instructions to load a zero extended 32-bit
>> pointer verses two for n32-2GB.
>
>   And of course such code will only run on 64-bit processors that not only
> support 64-bit data, but 64-bit addressing as well.

That's right.  All of this assumes a fully 64-bit operating system 
kernel (Linux).

It is not really very interesting on 'small' systems that have less than 
about 1GB of RAM.  And obviously impossible if 64-bit addressing is not 
supported.

So the interesting use cases are 'modern' systems with 4GB or more of 
ram installed.  And only then for the subset of applications that need 
more than 2GB of virtual address space but will never need to consider 
more than 4GB.



>  That is implement the
> CP0.Status.UX bit rather than CP0.Status.PX only -- the latters are still
> compatible with the true n32 ABI.  See also CP0.Config.AT.
>
>    Maciej
>
>

Re: 2GB userspace limitation in ABI N32

[Ronny Meeus]

This is exactly the platform we are targeting:
- a Cavium processor
- running 64bit Linux
- 4Gb of ram of which almost 3Gb will be used by 1 process (consisting
of multiple threads)

It would be really great that we could get help from you guys here.
Many thanks for the effort you are putting into this.

On Wed, Oct 10, 2012 at 7:34 PM, David Daney <ddaney.cavm@gmail.com> wrote:
> On 10/10/2012 10:10 AM, Maciej W. Rozycki wrote:
>>
>> On Wed, 10 Oct 2012, David Daney wrote:
>>
>>> The only disadvantage of doing this is that the code will be slightly
>>> larger/slower as it takes three instructions to load a zero extended
>>> 32-bit
>>> pointer verses two for n32-2GB.
>>
>>
>>   And of course such code will only run on 64-bit processors that not only
>> support 64-bit data, but 64-bit addressing as well.
>
>
> That's right.  All of this assumes a fully 64-bit operating system kernel
> (Linux).
>
> It is not really very interesting on 'small' systems that have less than
> about 1GB of RAM.  And obviously impossible if 64-bit addressing is not
> supported.
>
> So the interesting use cases are 'modern' systems with 4GB or more of ram
> installed.  And only then for the subset of applications that need more than
> 2GB of virtual address space but will never need to consider more than 4GB.
>
>
>
>
>>  That is implement the
>> CP0.Status.UX bit rather than CP0.Status.PX only -- the latters are still
>> compatible with the true n32 ABI.  See also CP0.Config.AT.
>>
>>    Maciej
>>
>>
>
>

Re: 2GB userspace limitation in ABI N32

[David Daney]

On 10/10/2012 10:49 AM, Ronny Meeus wrote:
> This is exactly the platform we are targeting:
> - a Cavium processor
> - running 64bit Linux
> - 4Gb of ram of which almost 3Gb will be used by 1 process (consisting
> of multiple threads)
>
> It would be really great that we could get help from you guys here.

As far as I know, we are not actively working on this.  So, as I see it, 
your options are:

A) Use n64.

B) Do all the work yourself.

C) Pay someone to do the work for you.

David Daney


> Many thanks for the effort you are putting into this.
>
> On Wed, Oct 10, 2012 at 7:34 PM, David Daney <ddaney.cavm@gmail.com> wrote:
>> On 10/10/2012 10:10 AM, Maciej W. Rozycki wrote:
>>>
>>> On Wed, 10 Oct 2012, David Daney wrote:
>>>
>>>> The only disadvantage of doing this is that the code will be slightly
>>>> larger/slower as it takes three instructions to load a zero extended
>>>> 32-bit
>>>> pointer verses two for n32-2GB.
>>>
>>>
>>>    And of course such code will only run on 64-bit processors that not only
>>> support 64-bit data, but 64-bit addressing as well.
>>
>>
>> That's right.  All of this assumes a fully 64-bit operating system kernel
>> (Linux).
>>
>> It is not really very interesting on 'small' systems that have less than
>> about 1GB of RAM.  And obviously impossible if 64-bit addressing is not
>> supported.
>>
>> So the interesting use cases are 'modern' systems with 4GB or more of ram
>> installed.  And only then for the subset of applications that need more than
>> 2GB of virtual address space but will never need to consider more than 4GB.
>>
>>
>>
>>
>>>   That is implement the
>>> CP0.Status.UX bit rather than CP0.Status.PX only -- the latters are still
>>> compatible with the true n32 ABI.  See also CP0.Config.AT.
>>>
>>>     Maciej
>>>
>>>
>>
>>
>
>

Re: 2GB userspace limitation in ABI N32

[Ronny Meeus]

Do you have any clue (rough) on the amount of effort this change would cost?

About the limited gain we can discuss: if you have a large application
that has been created assuming 32bit and it needs to be ported to a
64bit architecture, I think the effort can be huge and the risk for
forgetting things is high. It will be very hard to check whether the
system behaves well under all conditions.

---
Ronny

On Wed, Oct 10, 2012 at 10:07 AM, Ralf Baechle <ralf@linux-mips.org> wrote:
> On Wed, Oct 10, 2012 at 08:32:47AM +0200, Ronny Meeus wrote:
>
>> I have a legacy application that we want to port to a MIPS (Cavium)
>> architecture from a PPC based one.
>> The board has 4GB memory of which we actually need almost 3GB in
>> application space. On the PPC this is no issue since the split
>> user/kernel is 3GB/1GB.
>> We have to use the N32 ABI Initial tests on MIPS showed me the
>> user-space limit of 2GB.
>> We do not want to port the application to a 64bit
>>
>> Now the question is: are there any workarounds, tricks existing to get
>> around this limitation?
>> I found some mailthreads on this subject (n32-big ABI -
>> http://gcc.gnu.org/ml/gcc/2011-02/msg00278.html,
>> http://elinux.org/images/1/1f/New-tricks-mips-linux.pdf) but is looks
>> like this is not accepted by the community. Is there any process
>> planned or made in this area?
>
> I think limited time and gain killed the propoosed ABI rather than
> theoretical issues raised.  Other architectures such as i386 - well,
> IIRC any 32-bit ABI with more than 2GB userspace and a signed
> ptrdiff_t - are suffering from them as well.
>
> Also there's limited gain and even more limited time to implement things ...
>
>   Ralf

Re: 2GB userspace limitation in ABI N32

[Ralf Baechle]

On Wed, Oct 10, 2012 at 05:12:16PM +0200, Ronny Meeus wrote:

> Do you have any clue (rough) on the amount of effort this change would cost?

David Daney's reply should give you more information for an estimate.

> About the limited gain we can discuss: if you have a large application
> that has been created assuming 32bit and it needs to be ported to a
> 64bit architecture, I think the effort can be huge and the risk for
> forgetting things is high. It will be very hard to check whether the
> system behaves well under all conditions.

A 64-bit port could start right away without the delay of waiting for
a usable N32-4GB.  Added benefit - with N64 you can grow beyond the 3GB.

Downside, due to larger pointers thus better cache locality 32-bit code
generally performs better.  And I agree that verification of N32-4GB
probably is easier than for a large application that wasn't written
with the intend of 64-bit support.

In my past as a contractor I've dealth with a few customers who were
trying to avoid going 64-bit at all cost.  They had to pay that cost ...

  Ralf