armv7 + lpae broken in 4.1.12+ ?

armv7 + lpae broken in 4.1.12+ ?

[Brad Parker]

I was recently building a 4.1.12 kernel for a generic V7 platform with 
LPAE.   I could not get it to boot and then discovered two things:

- the early printk code in tty/serial call ioremap very early and it 
fails because there's no heap
- the code in proc-v7-3level.S/cpu_v7_set_pte_ext doesn't work

The printk issue is no big deal.  The pte issue is more important.

I found that the pte entries were garbage, which really confused me.  I 
finally discovered that the code in proc-v7-3level.S expects data in r0, 
r2 & r3, but in fact it comes in on r0, r1 & r2.

My guess is that this is some vestige of the change made to 
cpu_v7_switch_mm() some time ago to make the PA a 64 bit value (hence 
using r0,r1).  But this change was not made to cpu_v7_set_pte_ext, so it 
writes garbage.

lol.  I guess I'm the only one who turns on LPAE for v7 cpu's...

The simple fix is to add "mov r3, r2; mov r2,r1" at the begining of 
cpu_v7_set_pte_ext(), but that's a hack (but it does solve the problem 
and the kernel boots).

I suspect the "right thing" is to make the prototype for 
cpu_v7_set_pte_ext() in proc-fns.h use a phys_addr_t but I have not 
tried that yet.

Can someone confirm that what I am seeing is correct?

-brad

armv7 + lpae broken in 4.1.12+ ?

[Russell King - ARM Linux]

On Thu, Nov 26, 2015 at 06:26:38PM -0500, Brad Parker wrote:
> I found that the pte entries were garbage, which really confused me.  I
> finally discovered that the code in proc-v7-3level.S expects data in r0, r2
> & r3, but in fact it comes in on r0, r1 & r2.

This seems wrong.  If you're building LPAE, then you should have AEABI
enabled, which means you're using EABI.  (ARM_LPAE should depend on
AEABI, but it doesn't.)

EABI places 64-bit arguments in an even/odd register pair.  So:

void cpu_set_pte_ext(pte_t *ptep, pte_t pte, unsigned int ext);

ptep is in r0, and if pte_t is 64-bit, then pte is passed in r2 and r3.
Not r1 and r2.  ext is passed on the stack.

-- 
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

armv7 + lpae broken in 4.1.12+ ?

[Will Deacon]

On Thu, Nov 26, 2015 at 06:26:38PM -0500, Brad Parker wrote:
> I was recently building a 4.1.12 kernel for a generic V7 platform with LPAE.
> I could not get it to boot and then discovered two things:
> 
> - the early printk code in tty/serial call ioremap very early and it fails
> because there's no heap
> - the code in proc-v7-3level.S/cpu_v7_set_pte_ext doesn't work
> 
> The printk issue is no big deal.  The pte issue is more important.
> 
> I found that the pte entries were garbage, which really confused me.  I
> finally discovered that the code in proc-v7-3level.S expects data in r0, r2
> & r3, but in fact it comes in on r0, r1 & r2.
> 
> My guess is that this is some vestige of the change made to
> cpu_v7_switch_mm() some time ago to make the PA a 64 bit value (hence using
> r0,r1).  But this change was not made to cpu_v7_set_pte_ext, so it writes
> garbage.
> 
> lol.  I guess I'm the only one who turns on LPAE for v7 cpu's...

FWIW, anybody using virtualisation with a v7 CPU has LPAE enabled. Whilst
that's admittedly not the majority of users, you're certainly not alone.

You are, however, the only person who seems to be hitting this problem.

As Russell says, AEABI requires the 64-bit pte value to be passed in r2
and r3 for cpu_v7_set_pte_ext:

  C.3 If the argument requires double-word alignment (8-byte), the NCRN
      is rounded up to the next even register number.

> The simple fix is to add "mov r3, r2; mov r2,r1" at the begining of
> cpu_v7_set_pte_ext(), but that's a hack (but it does solve the problem and
> the kernel boots).

... so this would actually cause breakage for other people.

Which toolchain are you using?

Will

armv7 + lpae broken in 4.1.12+ ?

[Brad Parker]

On 11/27/15 4:36 AM, Will Deacon wrote:> As Russell says, AEABI requires 
the 64-bit pte value to be passed in > r2 and r3 for cpu_v7_set_pte_ext: 
 > > C.3 If the argument requires double-word alignment (8-byte), the > 
NCRN is rounded up to the next even register number. > >> The simple fix 
is to add "mov r3, r2; mov r2,r1" at the begining >> of 
cpu_v7_set_pte_ext(), but that's a hack (but it does solve the >> 
problem and the kernel boots). > > ... so this would actually cause 
breakage for other people. > > Which toolchain are you using?
[sorry for the re-send;  I keep getting bounced by the email.html 
filter; not sure why]

That all makes sense. I'm using a linaro toolchain (which I'll admit I
grabbed because I was lazy)

gcc-linaro-arm-linux-gnueabihf-4.8-2014.04_linux/bin/arm-linux-gnueabihf-

It seems really odd.  I would have thought "gnueabihf" would be EABI, so
a 64 bit arg should be in r2,r3; but it's not.  It's in r1, r2.

could something in my .config cause this?  (I'd be surprised if that were the case, but...).

I'm happy to try a better toolchain if you can point me to one (binary).

-brad

armv7 + lpae broken in 4.1.12+ ?

[Russell King - ARM Linux]

On Fri, Nov 27, 2015 at 10:55:35AM -0500, Brad Parker wrote:
> That all makes sense. I'm using a linaro toolchain (which I'll admit I
> grabbed because I was lazy)
> 
> gcc-linaro-arm-linux-gnueabihf-4.8-2014.04_linux/bin/arm-linux-gnueabihf-
> 
> It seems really odd.  I would have thought "gnueabihf" would be EABI, so
> a 64 bit arg should be in r2,r3; but it's not.  It's in r1, r2.
> 
> could something in my .config cause this?  (I'd be surprised if that were the case, but...).
> 
> I'm happy to try a better toolchain if you can point me to one (binary).

It doesn't matter what toolchain you have.  We don't rely on how the
toolchain happened to be configured to build the kernel, we explicitly
state to the compiler what we want it to do.

Many people tend to be under the incorrect impression that they need
specific compilers to build the kernel in certain ways.  This is untrue,
and has been untrue for the last 20 years.

If you have CONFIG_AEABI disabled, you will get an OABI kernel, or an
error from the toolchain if the toolchain doesn't support it.  If you
have CONFIG_AEABI enabled, you will get an EABI kernel, or an error
from the toolchain if the toolchain doesn't support it.

As I said in my previous reply, LPAE really ought to depend on AEABI
being enabled.

Please check whether you have CONFIG_AEABI enabled.

-- 
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

armv7 + lpae broken in 4.1.12+ ?

[Brad Parker]

On 11/27/15 11:06 AM, Russell King - ARM Linux wrote:
> Please check whether you have CONFIG_AEABI enabled. 
Yes.  That was it - sorry.  I didn't realize that was a config item (but 
with your explanation and a look at the makefile I see what it's 
doing).  I guess I was blissfully unaware that that option even 
existed.  But not any more :-)

thanks!

That option seem on by default for thumb2 kernels, but not for aarch32.

-brad

armv7 + lpae broken in 4.1.12+ ?

[Russell King - ARM Linux]

On Fri, Nov 27, 2015 at 12:02:30PM -0500, Brad Parker wrote:
> On 11/27/15 11:06 AM, Russell King - ARM Linux wrote:
> >Please check whether you have CONFIG_AEABI enabled.
> Yes.  That was it - sorry.  I didn't realize that was a config item (but
> with your explanation and a look at the makefile I see what it's doing).  I
> guess I was blissfully unaware that that option even existed.  But not any
> more :-)

I'm surprised, because if you try booting an OABI only kernel with an
EABI userspace, it'll hang after getting to userspace.  Ditto the
other way around, unless you have OABI_COMPAT enabled.  However, I
guess you're not getting that far yet...

-- 
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.