Single zImage and A15/LPAE

Single zImage and A15/LPAE

[Stephen Warren]

I'm curious what the single-zImage story is for Cortex A15 CPUs with
LPAE extensions; IIRC, LPAE entails a different page table format and so
isn't going to co-exist in the same zImage as non-LPAE (although doesn't
x86 support that now; I though separate LPAE/non-LPAE kernels went away
there?)

Presumably though, the intent is to be able to disable LPAE in Kconfig,
and create a single zImage with both A9 and A15 support, or to enable
LPAE and only support A15 with LPAE, and which you'd choose to do simply
depends on whether you care about LPAE based on how much RAM the boards
you want to support have (or perhaps also what physical address your
peripherals are at)?

Single zImage and A15/LPAE

[Nicolas Pitre]

On Mon, 8 Oct 2012, Stephen Warren wrote:

> I'm curious what the single-zImage story is for Cortex A15 CPUs with
> LPAE extensions; IIRC, LPAE entails a different page table format and so
> isn't going to co-exist in the same zImage as non-LPAE

LPAE vs non LPAE is an even more invasive change than ARMv6+ vs pre 
ARMv6 support.  So no, I don't think we'll ever support LPAE and non 
LPAE configs in the same kernel binary.

> (although doesn't x86 support that now; I though separate 
> LPAE/non-LPAE kernels went away there?)

I don't think so.  At least Ubuntu apparently still carries a PAE and 
non PAE kernel packages.  Fedora doesn't, probably because they decided 
not to support non PAE capable machines anymore.  We certainly cannot 
make this choice on ARM yet.

> Presumably though, the intent is to be able to disable LPAE in Kconfig,
> and create a single zImage with both A9 and A15 support, or to enable
> LPAE and only support A15 with LPAE, and which you'd choose to do simply
> depends on whether you care about LPAE based on how much RAM the boards
> you want to support have (or perhaps also what physical address your
> peripherals are at)?

Yes.


Nicolas

Single zImage and A15/LPAE

[Arnd Bergmann]

On Monday 08 October 2012 22:18:24 Nicolas Pitre wrote:
> On Mon, 8 Oct 2012, Stephen Warren wrote:
> 
> > I'm curious what the single-zImage story is for Cortex A15 CPUs with
> > LPAE extensions; IIRC, LPAE entails a different page table format and so
> > isn't going to co-exist in the same zImage as non-LPAE
> 
> LPAE vs non LPAE is an even more invasive change than ARMv6+ vs pre 
> ARMv6 support.  So no, I don't think we'll ever support LPAE and non 
> LPAE configs in the same kernel binary.
> 
> > (although doesn't x86 support that now; I though separate 
> > LPAE/non-LPAE kernels went away there?)
> 
> I don't think so.  At least Ubuntu apparently still carries a PAE and 
> non PAE kernel packages.  Fedora doesn't, probably because they decided 
> not to support non PAE capable machines anymore.  We certainly cannot 
> make this choice on ARM yet.

Fedora 18 still has both PAE and non-PAE kernels. I would really hope
they could give up the PAE version in favor of a 64 bit kernel in 
the 32 bit distro, but it seems none of the big distros trust the
compat code enough yet. On x86, the number of 32 bit machines still
running with more than 3GB of RAM installed should be very marginal
now, most people running the PAE kernel actually have 64 bit capable
CPUs and have some legacy 32 bit applications that are easier to
run with a 32 bit user space.

Maybe we get to the same point on ARM in some 10 years, but for
the forseeable future, Cortex-a15 machines with lots of RAM will
be very real and we need to have separate kernels for those.

	Arnd

Single zImage and A15/LPAE

[Dave Martin]

On Tue, Oct 09, 2012 at 07:51:02AM +0000, Arnd Bergmann wrote:
> On Monday 08 October 2012 22:18:24 Nicolas Pitre wrote:
> > On Mon, 8 Oct 2012, Stephen Warren wrote:
> > 
> > > I'm curious what the single-zImage story is for Cortex A15 CPUs with
> > > LPAE extensions; IIRC, LPAE entails a different page table format and so
> > > isn't going to co-exist in the same zImage as non-LPAE
> > 
> > LPAE vs non LPAE is an even more invasive change than ARMv6+ vs pre 
> > ARMv6 support.  So no, I don't think we'll ever support LPAE and non 
> > LPAE configs in the same kernel binary.
> > 
> > > (although doesn't x86 support that now; I though separate 
> > > LPAE/non-LPAE kernels went away there?)
> > 
> > I don't think so.  At least Ubuntu apparently still carries a PAE and 
> > non PAE kernel packages.  Fedora doesn't, probably because they decided 
> > not to support non PAE capable machines anymore.  We certainly cannot 
> > make this choice on ARM yet.
> 
> Fedora 18 still has both PAE and non-PAE kernels. I would really hope
> they could give up the PAE version in favor of a 64 bit kernel in 
> the 32 bit distro, but it seems none of the big distros trust the
> compat code enough yet. On x86, the number of 32 bit machines still
> running with more than 3GB of RAM installed should be very marginal
> now, most people running the PAE kernel actually have 64 bit capable
> CPUs and have some legacy 32 bit applications that are easier to
> run with a 32 bit user space.
> 
> Maybe we get to the same point on ARM in some 10 years, but for
> the forseeable future, Cortex-a15 machines with lots of RAM will
> be very real and we need to have separate kernels for those.

In the medium term we could work around this with a fat kernel in
principle (i.e., bundle a non-LPAE single kernel together with an LPAE
one and choose the right one at boot time).  This can be solved outside
the kernel if necessary.

This is not great, but the general feeling seems to be that combining
LPAE and non-LPAE in the same kernel is just not going to be worth the
pain (either in code or runtime impacts).

For AArch64, we obviously have a different kernel binary anyway.


Cheers
---Dave

Single zImage and A15/LPAE

[Kukjin Kim]

Dave Martin wrote:
> On Tue, Oct 09, 2012 at 07:51:02AM +0000, Arnd Bergmann wrote:
> > On Monday 08 October 2012 22:18:24 Nicolas Pitre wrote:
> > > On Mon, 8 Oct 2012, Stephen Warren wrote:
> > >
> > > > I'm curious what the single-zImage story is for Cortex A15 CPUs with
> > > > LPAE extensions; IIRC, LPAE entails a different page table format
> and so
> > > > isn't going to co-exist in the same zImage as non-LPAE
> > >
> > > LPAE vs non LPAE is an even more invasive change than ARMv6+ vs pre
> > > ARMv6 support.  So no, I don't think we'll ever support LPAE and non
> > > LPAE configs in the same kernel binary.
> > >
> > > > (although doesn't x86 support that now; I though separate
> > > > LPAE/non-LPAE kernels went away there?)
> > >
> > > I don't think so.  At least Ubuntu apparently still carries a PAE and
> > > non PAE kernel packages.  Fedora doesn't, probably because they
> decided
> > > not to support non PAE capable machines anymore.  We certainly cannot
> > > make this choice on ARM yet.
> >
> > Fedora 18 still has both PAE and non-PAE kernels. I would really hope
> > they could give up the PAE version in favor of a 64 bit kernel in
> > the 32 bit distro, but it seems none of the big distros trust the
> > compat code enough yet. On x86, the number of 32 bit machines still
> > running with more than 3GB of RAM installed should be very marginal
> > now, most people running the PAE kernel actually have 64 bit capable
> > CPUs and have some legacy 32 bit applications that are easier to
> > run with a 32 bit user space.
> >
> > Maybe we get to the same point on ARM in some 10 years, but for
> > the forseeable future, Cortex-a15 machines with lots of RAM will
> > be very real and we need to have separate kernels for those.
> 
> In the medium term we could work around this with a fat kernel in
> principle (i.e., bundle a non-LPAE single kernel together with an LPAE
> one and choose the right one at boot time).  This can be solved outside
> the kernel if necessary.
> 
> This is not great, but the general feeling seems to be that combining
> LPAE and non-LPAE in the same kernel is just not going to be worth the
> pain (either in code or runtime impacts).
> 
> For AArch64, we obviously have a different kernel binary anyway.
> 
(+ Olof, Russell)

Just note, I found this in my old mail-box...

So I think, as a result, we need to support separate kernel binary for LPAE
and non-LPAE and you guys don't hold a different view about it.

Current SSDK5440 and SD5v1 boards which are reference boards for EXYNOS5440
has over 4GB memory as a default and that's why I'm writing this e-mail. It
means I should keep separated kernel to support exynos5440 and other exynos
SoCs.

If you have any concerns about that, please let me know.

- Kukjin

Single zImage and A15/LPAE

[Nicolas Pitre]

On Wed, 29 May 2013, Kukjin Kim wrote:

> Dave Martin wrote:
> > On Tue, Oct 09, 2012 at 07:51:02AM +0000, Arnd Bergmann wrote:
> > > On Monday 08 October 2012 22:18:24 Nicolas Pitre wrote:
> > > > On Mon, 8 Oct 2012, Stephen Warren wrote:
> > > >
> > > > > I'm curious what the single-zImage story is for Cortex A15 CPUs with
> > > > > LPAE extensions; IIRC, LPAE entails a different page table format
> > and so
> > > > > isn't going to co-exist in the same zImage as non-LPAE
> > > >
> > > > LPAE vs non LPAE is an even more invasive change than ARMv6+ vs pre
> > > > ARMv6 support.  So no, I don't think we'll ever support LPAE and non
> > > > LPAE configs in the same kernel binary.
> > > >
> > > > > (although doesn't x86 support that now; I though separate
> > > > > LPAE/non-LPAE kernels went away there?)
> > > >
> > > > I don't think so.  At least Ubuntu apparently still carries a PAE and
> > > > non PAE kernel packages.  Fedora doesn't, probably because they
> > decided
> > > > not to support non PAE capable machines anymore.  We certainly cannot
> > > > make this choice on ARM yet.
> > >
> > > Fedora 18 still has both PAE and non-PAE kernels. I would really hope
> > > they could give up the PAE version in favor of a 64 bit kernel in
> > > the 32 bit distro, but it seems none of the big distros trust the
> > > compat code enough yet. On x86, the number of 32 bit machines still
> > > running with more than 3GB of RAM installed should be very marginal
> > > now, most people running the PAE kernel actually have 64 bit capable
> > > CPUs and have some legacy 32 bit applications that are easier to
> > > run with a 32 bit user space.
> > >
> > > Maybe we get to the same point on ARM in some 10 years, but for
> > > the forseeable future, Cortex-a15 machines with lots of RAM will
> > > be very real and we need to have separate kernels for those.
> > 
> > In the medium term we could work around this with a fat kernel in
> > principle (i.e., bundle a non-LPAE single kernel together with an LPAE
> > one and choose the right one at boot time).  This can be solved outside
> > the kernel if necessary.
> > 
> > This is not great, but the general feeling seems to be that combining
> > LPAE and non-LPAE in the same kernel is just not going to be worth the
> > pain (either in code or runtime impacts).
> > 
> > For AArch64, we obviously have a different kernel binary anyway.
> > 
> (+ Olof, Russell)
> 
> Just note, I found this in my old mail-box...
> 
> So I think, as a result, we need to support separate kernel binary for LPAE
> and non-LPAE and you guys don't hold a different view about it.

LPAE and non-LPAE configurations will most likely never be supported 
together in the same kernel binary.  The runtime cost would simply be 
too high to be worth it.

> Current SSDK5440 and SD5v1 boards which are reference boards for EXYNOS5440
> has over 4GB memory as a default and that's why I'm writing this e-mail. It
> means I should keep separated kernel to support exynos5440 and other exynos
> SoCs.

They could all be supported together but with less than 4GB of memory.


Nicolas

Single zImage and A15/LPAE

[Olof Johansson]

On Tue, May 28, 2013 at 6:54 PM, Nicolas Pitre <nicolas.pitre@linaro.org> wrote:
> On Wed, 29 May 2013, Kukjin Kim wrote:
>> (+ Olof, Russell)
>>
>> Just note, I found this in my old mail-box...
>>
>> So I think, as a result, we need to support separate kernel binary for LPAE
>> and non-LPAE and you guys don't hold a different view about it.
>
> LPAE and non-LPAE configurations will most likely never be supported
> together in the same kernel binary.  The runtime cost would simply be
> too high to be worth it.
>
>> Current SSDK5440 and SD5v1 boards which are reference boards for EXYNOS5440
>> has over 4GB memory as a default and that's why I'm writing this e-mail. It
>> means I should keep separated kernel to support exynos5440 and other exynos
>> SoCs.
>
> They could all be supported together but with less than 4GB of memory.

Or the other way around, the other EXYNOS5 CPUs could all enable LPAE
even if they have less than 4GB of memory -- but not EXYNOS4, of
course.

However, instead of forking off and duplicating otherwise
near-identical defconfigs, I suggest just getting your developers used
to passing in CONFIG_LPAE=y on the build command line instead (or in
their local configs for their products, of course). That way there's
less duplication and no need to enable drivers in two places, etc.



-Olof

Single zImage and A15/LPAE

[Olof Johansson]

On Tue, May 28, 2013 at 7:02 PM, Olof Johansson <olof@lixom.net> wrote:
> On Tue, May 28, 2013 at 6:54 PM, Nicolas Pitre <nicolas.pitre@linaro.org> wrote:
>> On Wed, 29 May 2013, Kukjin Kim wrote:
>>> (+ Olof, Russell)
>>>
>>> Just note, I found this in my old mail-box...
>>>
>>> So I think, as a result, we need to support separate kernel binary for LPAE
>>> and non-LPAE and you guys don't hold a different view about it.
>>
>> LPAE and non-LPAE configurations will most likely never be supported
>> together in the same kernel binary.  The runtime cost would simply be
>> too high to be worth it.
>>
>>> Current SSDK5440 and SD5v1 boards which are reference boards for EXYNOS5440
>>> has over 4GB memory as a default and that's why I'm writing this e-mail. It
>>> means I should keep separated kernel to support exynos5440 and other exynos
>>> SoCs.
>>
>> They could all be supported together but with less than 4GB of memory.
>
> Or the other way around, the other EXYNOS5 CPUs could all enable LPAE
> even if they have less than 4GB of memory -- but not EXYNOS4, of
> course.
>
> However, instead of forking off and duplicating otherwise
> near-identical defconfigs, I suggest just getting your developers used
> to passing in CONFIG_LPAE=y on the build command line instead (or in
> their local configs for their products, of course). That way there's
> less duplication and no need to enable drivers in two places, etc.

Nevermind, I'm smoking crack, that doesn't seem to actually work.
(Besides, it's CONFIG_ARM_LPAE).


-Olof

Single zImage and A15/LPAE

[Arnd Bergmann]

On Tuesday 28 May 2013 19:02:48 Olof Johansson wrote:
> On Tue, May 28, 2013 at 6:54 PM, Nicolas Pitre <nicolas.pitre@linaro.org> wrote:
> > On Wed, 29 May 2013, Kukjin Kim wrote:
> >> (+ Olof, Russell)
> >>
> >> Just note, I found this in my old mail-box...
> >>
> >> So I think, as a result, we need to support separate kernel binary for LPAE
> >> and non-LPAE and you guys don't hold a different view about it.
> >
> > LPAE and non-LPAE configurations will most likely never be supported
> > together in the same kernel binary.  The runtime cost would simply be
> > too high to be worth it.
> >
> >> Current SSDK5440 and SD5v1 boards which are reference boards for EXYNOS5440
> >> has over 4GB memory as a default and that's why I'm writing this e-mail. It
> >> means I should keep separated kernel to support exynos5440 and other exynos
> >> SoCs.
> >
> > They could all be supported together but with less than 4GB of memory.
> 
> Or the other way around, the other EXYNOS5 CPUs could all enable LPAE
> even if they have less than 4GB of memory -- but not EXYNOS4, of
> course.

The one bug I see is that it's currently possible to build a kernel
for a Cortex-A5/A8/A9, Scorpion or PJ4 platform with LPAE enabled,
which obviously cannot work.

I think to do this right, we need to treat ARMv7+LPAE as a separate
architecture level, just like we treat ARMv6, ARMv7 as separate
compatible architectures. BTW, is there an official name for that
architecture?

If we introduce a new CONFIG_ARCH_MULTI_V7_LPAE option, that should
be used instead of CONFIG_ARCH_MULTI_V7 for platforms that can be
built as part of CONFIG_ARCH_MULTIPLATFORM and with LPAE support.
The default multi_v7_defconfig should enable all both
ARCH_MULTI_V7 and ARCH_MULTI_V7_LPAE, so we get e.g. OMAP4/OMAP5
adn Exynos4/Exynos5 in one kernel, but when you want to enable
LPAE, you have to disable ARCH_MULTI_V7 and only get OMAP5
and Exynos5 out of those.

	Arnd

Single zImage and A15/LPAE

[Catalin Marinas]

On Wed, May 29, 2013 at 02:01:05PM +0100, Arnd Bergmann wrote:
> On Tuesday 28 May 2013 19:02:48 Olof Johansson wrote:
> > On Tue, May 28, 2013 at 6:54 PM, Nicolas Pitre <nicolas.pitre@linaro.org> wrote:
> > > On Wed, 29 May 2013, Kukjin Kim wrote:
> > >> (+ Olof, Russell)
> > >>
> > >> Just note, I found this in my old mail-box...
> > >>
> > >> So I think, as a result, we need to support separate kernel binary for LPAE
> > >> and non-LPAE and you guys don't hold a different view about it.
> > >
> > > LPAE and non-LPAE configurations will most likely never be supported
> > > together in the same kernel binary.  The runtime cost would simply be
> > > too high to be worth it.
> > >
> > >> Current SSDK5440 and SD5v1 boards which are reference boards for EXYNOS5440
> > >> has over 4GB memory as a default and that's why I'm writing this e-mail. It
> > >> means I should keep separated kernel to support exynos5440 and other exynos
> > >> SoCs.
> > >
> > > They could all be supported together but with less than 4GB of memory.
> > 
> > Or the other way around, the other EXYNOS5 CPUs could all enable LPAE
> > even if they have less than 4GB of memory -- but not EXYNOS4, of
> > course.
> 
> The one bug I see is that it's currently possible to build a kernel
> for a Cortex-A5/A8/A9, Scorpion or PJ4 platform with LPAE enabled,
> which obviously cannot work.
> 
> I think to do this right, we need to treat ARMv7+LPAE as a separate
> architecture level, just like we treat ARMv6, ARMv7 as separate
> compatible architectures. BTW, is there an official name for that
> architecture?

The official is Large Physical Address Extension which is an extension
to ARMv7, so not a new architecture.

-- 
Catalin

Single zImage and A15/LPAE

[Arnd Bergmann]

On Wednesday 29 May 2013, Catalin Marinas wrote:
> > I think to do this right, we need to treat ARMv7+LPAE as a separate
> > architecture level, just like we treat ARMv6, ARMv7 as separate
> > compatible architectures. BTW, is there an official name for that
> > architecture?
> 
> The official is Large Physical Address Extension which is an extension
> to ARMv7, so not a new architecture.

Ah, too bad. I had hoped there were still postfixes like v6K or v4T.
I guess there is no complete list of ARMv7 extensions anywhere, right?

	Arnd

Single zImage and A15/LPAE

[Catalin Marinas]

On Wed, May 29, 2013 at 10:51:40PM +0100, Arnd Bergmann wrote:
> On Wednesday 29 May 2013, Catalin Marinas wrote:
> > > I think to do this right, we need to treat ARMv7+LPAE as a separate
> > > architecture level, just like we treat ARMv6, ARMv7 as separate
> > > compatible architectures. BTW, is there an official name for that
> > > architecture?
> > 
> > The official is Large Physical Address Extension which is an extension
> > to ARMv7, so not a new architecture.
> 
> Ah, too bad. I had hoped there were still postfixes like v6K or v4T.
> I guess there is no complete list of ARMv7 extensions anywhere, right?

It's the ARM ARM, though not a list. With ARMv7 we moved away from
adding suffixes to the architecture number and introduced the CPUID
scheme to check for what's available on a CPU.

-- 
Catalin

Single zImage and A15/LPAE

[Kukjin Kim]

Arnd Bergmann wrote:
> 
> On Wednesday 29 May 2013, Catalin Marinas wrote:
> > > I think to do this right, we need to treat ARMv7+LPAE as a separate
> > > architecture level, just like we treat ARMv6, ARMv7 as separate
> > > compatible architectures. BTW, is there an official name for that
> > > architecture?
> >
> > The official is Large Physical Address Extension which is an extension
> > to ARMv7, so not a new architecture.
> 
> Ah, too bad. I had hoped there were still postfixes like v6K or v4T.
> I guess there is no complete list of ARMv7 extensions anywhere, right?
> 
Just note, EXYNOS4 SoCs and EXYNOS5250 cannot boot when LPAE is enabled so I
think, single zImage should be handled separately for non-LPAE and LPAE. I
need to sort out EXYNOS SoCs for it and prepare registering
exynos5440_defconfig. Of course, EXYNOS5440 can be supported without LPAE
enabling but current bootargs in v3.11-rc1 break the kernel booting.

Thanks,
- Kukjin