Android and compatibility with deprecated armv7 instructions

Android and compatibility with deprecated armv7 instructions

[Colin Cross]

I know this has been discussed before (for example:
http://lists.infradead.org/pipermail/linux-arm-kernel/2013-November/210309.html),
but I thought I'd bring it up again before the Android ABI diverges
from the upstream kernel.

Android armv8 devices expect to have compatibility with existing
native apps, many of which contain deprecated instructions.  For
example, the Unity gaming engine used SWP until Unity 3.5, and many
apps compiled against older versions are still in the app store.
These apps have been running fine with SWP emulation on all recent
Android devices with SMP v7 cpus, so they are obviously fine with the
slowdown of emulation, and they haven't recompiled against a newer
Unity version for 2 years so it is unrealistic to expect them to be
updated.  I've also been told Unity used CP15 barriers.

ffmpeg is widely used in popular Android apps and uses the SETEND
instruction: https://ffmpeg.org/pipermail/ffmpeg-cvslog/2013-August/067252.html

We can carry patches to enable the deprecated armv7 instructions in a
tree shared by all Android vendors, but it seems like a silly reason
to diverge from the upstream kernel.  To maintain a consistent ABI for
all devices we're going to have to test for these instructions in the
Android compatibility test suite, and the upstream kernel wouldn't
pass.

Would you consider taking support for SWP emulation, enabling CP15
barriers (CP15BEN bit only until there's a real device that needs
emulation, also requires clearing COMPAT_PSR_E_BIT in
compat_setup_return) and enabling SETEND, all behind a default-off
CONFIG_DEPRECATED_ARMV7_COMPAT?

Android and compatibility with deprecated armv7 instructions

[Olof Johansson]

On Tue, Jul 1, 2014 at 4:06 PM, Colin Cross <ccross@google.com> wrote:
> I know this has been discussed before (for example:
> http://lists.infradead.org/pipermail/linux-arm-kernel/2013-November/210309.html),
> but I thought I'd bring it up again before the Android ABI diverges
> from the upstream kernel.
>
> Android armv8 devices expect to have compatibility with existing
> native apps, many of which contain deprecated instructions.  For
> example, the Unity gaming engine used SWP until Unity 3.5, and many
> apps compiled against older versions are still in the app store.
> These apps have been running fine with SWP emulation on all recent
> Android devices with SMP v7 cpus, so they are obviously fine with the
> slowdown of emulation, and they haven't recompiled against a newer
> Unity version for 2 years so it is unrealistic to expect them to be
> updated.  I've also been told Unity used CP15 barriers.
>
> ffmpeg is widely used in popular Android apps and uses the SETEND
> instruction: https://ffmpeg.org/pipermail/ffmpeg-cvslog/2013-August/067252.html
>
> We can carry patches to enable the deprecated armv7 instructions in a
> tree shared by all Android vendors, but it seems like a silly reason
> to diverge from the upstream kernel.  To maintain a consistent ABI for
> all devices we're going to have to test for these instructions in the
> Android compatibility test suite, and the upstream kernel wouldn't
> pass.
>
> Would you consider taking support for SWP emulation, enabling CP15
> barriers (CP15BEN bit only until there's a real device that needs
> emulation, also requires clearing COMPAT_PSR_E_BIT in
> compat_setup_return) and enabling SETEND, all behind a default-off
> CONFIG_DEPRECATED_ARMV7_COMPAT?

It sounds really silly to push back against this, since it's actually
needed by so many platforms out there.

One thing that powerpc did was to add a warning when emulated
instructions are used (with ratelimiting to avoid DoS). See
arch/powerpc/kernel/traps.c, ppc_warn_emulated etc. It gives the best
of both worlds: Lets current software run, but also gives a warning
that someone should probably try to get it recompiled for best
performance at some point (for someone who cares about the logs, which
isn't everybody).


-Olof

Android and compatibility with deprecated armv7 instructions

[Mark Brown]

On Tue, Jul 01, 2014 at 04:42:01PM -0700, Olof Johansson wrote:
> On Tue, Jul 1, 2014 at 4:06 PM, Colin Cross <ccross@google.com> wrote:

> > Would you consider taking support for SWP emulation, enabling CP15
> > barriers (CP15BEN bit only until there's a real device that needs
> > emulation, also requires clearing COMPAT_PSR_E_BIT in
> > compat_setup_return) and enabling SETEND, all behind a default-off
> > CONFIG_DEPRECATED_ARMV7_COMPAT?

> It sounds really silly to push back against this, since it's actually
> needed by so many platforms out there.

> One thing that powerpc did was to add a warning when emulated
> instructions are used (with ratelimiting to avoid DoS). See
> arch/powerpc/kernel/traps.c, ppc_warn_emulated etc. It gives the best
> of both worlds: Lets current software run, but also gives a warning
> that someone should probably try to get it recompiled for best
> performance at some point (for someone who cares about the logs, which
> isn't everybody).

Indeed, Alpha used to do a similar thing when it software emulated
unaligned memory access with pretty good results.  It makes it easy
to identify the source of any performance problems that are observed
and even if none are observed the naming and shaming can get things
updated anyway.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20140702/aa5a4311/attachment.sig>

Android and compatibility with deprecated armv7 instructions

[Will Deacon]

On Wed, Jul 02, 2014 at 12:48:00AM +0100, Mark Brown wrote:
> On Tue, Jul 01, 2014 at 04:42:01PM -0700, Olof Johansson wrote:
> > On Tue, Jul 1, 2014 at 4:06 PM, Colin Cross <ccross@google.com> wrote:
> > > Would you consider taking support for SWP emulation, enabling CP15
> > > barriers (CP15BEN bit only until there's a real device that needs
> > > emulation, also requires clearing COMPAT_PSR_E_BIT in
> > > compat_setup_return) and enabling SETEND, all behind a default-off
> > > CONFIG_DEPRECATED_ARMV7_COMPAT?
> 
> > It sounds really silly to push back against this, since it's actually
> > needed by so many platforms out there.

The big problem with emulating instructions that don't even appear in the
hardware anymore is that we end up creating baggage which we can *never*
remove.

I'm against SWP emulation in the kernel for a number of reasons:

  (1) The hardware doesn't have the instruction at all. If we start
      emulating it, then we'll always have to emulate it and it doesn't
      encourage software migration.

  (2) I'm not convinced that it can't be handled in userspace by trapping
      the SIGILL and emulating there (admittedly, this sounds difficult).

  (3) The usual uses of SWP are in homebrew locking implementations and
      are almost certainly a _bug_. For those v7 CPUs that could do SWP,
      it's not even guaranteed to be atomic iirc. Trapping and emulating
      is also bad for performance (although I note that Colin made an
      argument that it was acceptable).

  (4) This only affects legacy binaries. Should we also try to support OABI?
      How about misaligned ldm/stm? We have to draw the line somewhere.

The CP15 barriers are a more interesting case, as the CPUs can *currently*
support those if we flip a bit in the SCTLR. However, I see that as a
slippery slope to emulation if CPUs stop supporting those instructions in
the future (they almost certainly will).

Whilst I appreciate that people are being bitten by this lack of emulation
support, the vast majority of AArch32 code out there is working fine with
the existing compat layer. I think the right way to solve this problem is
to fix the code making use of the missing instructions.

Will

Android and compatibility with deprecated armv7 instructions

[Colin Cross]

On Wed, Jul 2, 2014 at 3:01 AM, Will Deacon <will.deacon@arm.com> wrote:
>
> On Wed, Jul 02, 2014 at 12:48:00AM +0100, Mark Brown wrote:
> > On Tue, Jul 01, 2014 at 04:42:01PM -0700, Olof Johansson wrote:
> > > On Tue, Jul 1, 2014 at 4:06 PM, Colin Cross <ccross@google.com> wrote:
> > > > Would you consider taking support for SWP emulation, enabling CP15
> > > > barriers (CP15BEN bit only until there's a real device that needs
> > > > emulation, also requires clearing COMPAT_PSR_E_BIT in
> > > > compat_setup_return) and enabling SETEND, all behind a default-off
> > > > CONFIG_DEPRECATED_ARMV7_COMPAT?
> >
> > > It sounds really silly to push back against this, since it's actually
> > > needed by so many platforms out there.
>
> The big problem with emulating instructions that don't even appear in the
> hardware anymore is that we end up creating baggage which we can *never*
> remove.
>
> I'm against SWP emulation in the kernel for a number of reasons:
>
>   (1) The hardware doesn't have the instruction at all. If we start
>       emulating it, then we'll always have to emulate it and it doesn't
>       encourage software migration.
>
>   (2) I'm not convinced that it can't be handled in userspace by trapping
>       the SIGILL and emulating there (admittedly, this sounds difficult).
>
>   (3) The usual uses of SWP are in homebrew locking implementations and
>       are almost certainly a _bug_. For those v7 CPUs that could do SWP,
>       it's not even guaranteed to be atomic iirc. Trapping and emulating
>       is also bad for performance (although I note that Colin made an
>       argument that it was acceptable).
>
>   (4) This only affects legacy binaries. Should we also try to support OABI?
>       How about misaligned ldm/stm? We have to draw the line somewhere.

The problem is that we (Android) have to draw the line somewhere else
- there are too many highly visible apps in the app store that still
use these instructions.  When we add them back to our kernels, then we
are no longer ABI compatible with an upstream kernel.

> The CP15 barriers are a more interesting case, as the CPUs can *currently*
> support those if we flip a bit in the SCTLR. However, I see that as a
> slippery slope to emulation if CPUs stop supporting those instructions in
> the future (they almost certainly will).

I agree that this will likely lead to emulation when a CPU
manufacturer eventually decides to leave out hardware support,
although hopefully they won't if they see that the bit is set in SCTLR
on all Android devices.

> Whilst I appreciate that people are being bitten by this lack of emulation
> support, the vast majority of AArch32 code out there is working fine with
> the existing compat layer. I think the right way to solve this problem is
> to fix the code making use of the missing instructions.

A not-insignificant number of apps use these instructions - these
issues have been found by people taking the top 100 or so Android
apps, trying them out, and finding they crash.  Asking them all to
recompile is not feasible.  I view this issue as similar to Linus'
view on kernel ABIs - if somebody uses it, you have to keep it.  As
far as I know, nobody is generating new code with SWP and CP15 barrier
instructions, although ffmpeg is probably still using SETEND.

Android and compatibility with deprecated armv7 instructions

[Will Deacon]

Hi Colin,

On Wed, Jul 02, 2014 at 04:48:07PM +0100, Colin Cross wrote:
> On Wed, Jul 2, 2014 at 3:01 AM, Will Deacon <will.deacon@arm.com> wrote:
> > On Wed, Jul 02, 2014 at 12:48:00AM +0100, Mark Brown wrote:
> > > On Tue, Jul 01, 2014 at 04:42:01PM -0700, Olof Johansson wrote:
> > > > On Tue, Jul 1, 2014 at 4:06 PM, Colin Cross <ccross@google.com> wrote:
> > > > > Would you consider taking support for SWP emulation, enabling CP15
> > > > > barriers (CP15BEN bit only until there's a real device that needs
> > > > > emulation, also requires clearing COMPAT_PSR_E_BIT in
> > > > > compat_setup_return) and enabling SETEND, all behind a default-off
> > > > > CONFIG_DEPRECATED_ARMV7_COMPAT?
> > >
> > > > It sounds really silly to push back against this, since it's actually
> > > > needed by so many platforms out there.
> >
> > The big problem with emulating instructions that don't even appear in the
> > hardware anymore is that we end up creating baggage which we can *never*
> > remove.
> >
> > I'm against SWP emulation in the kernel for a number of reasons:
> >
> >   (1) The hardware doesn't have the instruction at all. If we start
> >       emulating it, then we'll always have to emulate it and it doesn't
> >       encourage software migration.
> >
> >   (2) I'm not convinced that it can't be handled in userspace by trapping
> >       the SIGILL and emulating there (admittedly, this sounds difficult).
> >
> >   (3) The usual uses of SWP are in homebrew locking implementations and
> >       are almost certainly a _bug_. For those v7 CPUs that could do SWP,
> >       it's not even guaranteed to be atomic iirc. Trapping and emulating
> >       is also bad for performance (although I note that Colin made an
> >       argument that it was acceptable).
> >
> >   (4) This only affects legacy binaries. Should we also try to support OABI?
> >       How about misaligned ldm/stm? We have to draw the line somewhere.
> 
> The problem is that we (Android) have to draw the line somewhere else
> - there are too many highly visible apps in the app store that still
> use these instructions.  When we add them back to our kernels, then we
> are no longer ABI compatible with an upstream kernel.

For an ARMv7 kernel, this is still controlled by CONFIG_SWP_EMULATE, so
you would have the exact same issues with kernels where that has been turned
off. You assumedly have a bunch of patches on top of mainline for Android; I
don't understand why this one is any different.

> > The CP15 barriers are a more interesting case, as the CPUs can *currently*
> > support those if we flip a bit in the SCTLR. However, I see that as a
> > slippery slope to emulation if CPUs stop supporting those instructions in
> > the future (they almost certainly will).
> 
> I agree that this will likely lead to emulation when a CPU
> manufacturer eventually decides to leave out hardware support,
> although hopefully they won't if they see that the bit is set in SCTLR
> on all Android devices.

... and what if/when people start building AArch64-only CPUs? Are we going to
emulate the entire AArch32 instruction set in the kernel? Or just the
deprecated/obsolete subset of that ;)

> > Whilst I appreciate that people are being bitten by this lack of emulation
> > support, the vast majority of AArch32 code out there is working fine with
> > the existing compat layer. I think the right way to solve this problem is
> > to fix the code making use of the missing instructions.
> 
> A not-insignificant number of apps use these instructions - these
> issues have been found by people taking the top 100 or so Android
> apps, trying them out, and finding they crash.  Asking them all to
> recompile is not feasible.  I view this issue as similar to Linus'
> view on kernel ABIs - if somebody uses it, you have to keep it.  As
> far as I know, nobody is generating new code with SWP and CP15 barrier
> instructions, although ffmpeg is probably still using SETEND.

I don't actually know *how* you would emulate SETEND. Have you tried? Also,
we haven't ever supported SWP on arm64 compat, so I don't agree with your
comparison to Linus's argument.

Will

Android and compatibility with deprecated armv7 instructions

[Christopher Covington]

Hi Will,

On 07/02/2014 12:16 PM, Will Deacon wrote:
> Hi Colin,
> 
> On Wed, Jul 02, 2014 at 04:48:07PM +0100, Colin Cross wrote:
>> On Wed, Jul 2, 2014 at 3:01 AM, Will Deacon <will.deacon@arm.com> wrote:
>>> On Wed, Jul 02, 2014 at 12:48:00AM +0100, Mark Brown wrote:
>>>> On Tue, Jul 01, 2014 at 04:42:01PM -0700, Olof Johansson wrote:
>>>>> On Tue, Jul 1, 2014 at 4:06 PM, Colin Cross <ccross@google.com> wrote:
>>>>>> Would you consider taking support for SWP emulation, enabling CP15
>>>>>> barriers (CP15BEN bit only until there's a real device that needs
>>>>>> emulation, also requires clearing COMPAT_PSR_E_BIT in
>>>>>> compat_setup_return) and enabling SETEND, all behind a default-off
>>>>>> CONFIG_DEPRECATED_ARMV7_COMPAT?
>>>>
>>>>> It sounds really silly to push back against this, since it's actually
>>>>> needed by so many platforms out there.
>>>
>>> The big problem with emulating instructions that don't even appear in the
>>> hardware anymore is that we end up creating baggage which we can *never*
>>> remove.
>>>
>>> I'm against SWP emulation in the kernel for a number of reasons:
>>>
>>>   (1) The hardware doesn't have the instruction at all. If we start
>>>       emulating it, then we'll always have to emulate it and it doesn't
>>>       encourage software migration.

Why should the Linux kernel have an opinion about software migration?

>>>   (2) I'm not convinced that it can't be handled in userspace by trapping
>>>       the SIGILL and emulating there (admittedly, this sounds difficult).
>>>
>>>   (3) The usual uses of SWP are in homebrew locking implementations and
>>>       are almost certainly a _bug_. For those v7 CPUs that could do SWP,
>>>       it's not even guaranteed to be atomic iirc. Trapping and emulating
>>>       is also bad for performance (although I note that Colin made an
>>>       argument that it was acceptable).
>>>
>>>   (4) This only affects legacy binaries. Should we also try to support OABI?
>>>       How about misaligned ldm/stm? We have to draw the line somewhere.

If someone has the time and interest to implement and maintain such features,
why turn them away, provided there is no performance impact on those who do
not wish to configure in the support? I think supporting everything within our
means is the right place to draw the line.

>> The problem is that we (Android) have to draw the line somewhere else
>> - there are too many highly visible apps in the app store that still
>> use these instructions.  When we add them back to our kernels, then we
>> are no longer ABI compatible with an upstream kernel.
> 
> For an ARMv7 kernel, this is still controlled by CONFIG_SWP_EMULATE, so
> you would have the exact same issues with kernels where that has been turned
> off. You assumedly have a bunch of patches on top of mainline for Android; I
> don't understand why this one is any different.

As for the number of out-of-tree patches, I've been able to run a minimal
Android userspace with just one cgroups patch applied on top of
torvalds/master for at least the past year.

As for the nature of the out-of-tree patches, I don't think this issue is any
different. I think everyone would be better off if the patches or suitable
replacements could be merged and everyone could work off of the same codebase.
My guess is that upstream would get more testing and more contributions and
downstream could work more efficiently.

Christopher

-- 
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by the Linux Foundation.

Android and compatibility with deprecated armv7 instructions

[Will Deacon]

On Wed, Jul 02, 2014 at 07:03:37PM +0100, Christopher Covington wrote:
> Hi Will,

Hello Christopher,

> On 07/02/2014 12:16 PM, Will Deacon wrote:
> > On Wed, Jul 02, 2014 at 04:48:07PM +0100, Colin Cross wrote:
> >> On Wed, Jul 2, 2014 at 3:01 AM, Will Deacon <will.deacon@arm.com> wrote:
> >>> On Wed, Jul 02, 2014 at 12:48:00AM +0100, Mark Brown wrote:
> >>>> On Tue, Jul 01, 2014 at 04:42:01PM -0700, Olof Johansson wrote:
> >>>>> On Tue, Jul 1, 2014 at 4:06 PM, Colin Cross <ccross@google.com> wrote:
> >>>>>> Would you consider taking support for SWP emulation, enabling CP15
> >>>>>> barriers (CP15BEN bit only until there's a real device that needs
> >>>>>> emulation, also requires clearing COMPAT_PSR_E_BIT in
> >>>>>> compat_setup_return) and enabling SETEND, all behind a default-off
> >>>>>> CONFIG_DEPRECATED_ARMV7_COMPAT?
> >>>>
> >>>>> It sounds really silly to push back against this, since it's actually
> >>>>> needed by so many platforms out there.
> >>>
> >>> The big problem with emulating instructions that don't even appear in the
> >>> hardware anymore is that we end up creating baggage which we can *never*
> >>> remove.
> >>>
> >>> I'm against SWP emulation in the kernel for a number of reasons:
> >>>
> >>>   (1) The hardware doesn't have the instruction at all. If we start
> >>>       emulating it, then we'll always have to emulate it and it doesn't
> >>>       encourage software migration.
> 
> Why should the Linux kernel have an opinion about software migration?

The kernel doesn't have an opinion; the community has a bunch of opinions
and we arg^H^H^Hdiscuss them until we reach a conclusion. I have strong
opinions about software migration based on my experiences working for ARM,
hence my feelings here. Others have different opinions based on their
experiences building products out of Linux-based systems. This is the
discussion bit.

To elaborate on this concern, software migration is only a problem for
legacy binaries. As time goes on, the number of legacy binaries will not
increase. However, supporting emulation such as this can easily keep that
number constant and force us to carry code that exists only because of
itself. Furthermore, it's almost always better to migrate to the new
instruction sequence from a performance point of view.

The ARM architecture typically deals with this by first deprecating an
instruction (providing controls to disable it, but still implementing it
in hardware) and then obsoleting it (no hardware support). SWP was
deprecated in ARMv7 and became obsolete in ARMv8.

One proposal could be that we provide options to enable deprecated
instructions (with a message in dmesg), but don't provide emulation when
they're obsolete. This would solve CP15 barriers and SETEND, but not SWP,
where you could argue that the emulation in ARMv7 kernels is the hint that
it's going away.

> >>>   (2) I'm not convinced that it can't be handled in userspace by trapping
> >>>       the SIGILL and emulating there (admittedly, this sounds difficult).
> >>>
> >>>   (3) The usual uses of SWP are in homebrew locking implementations and
> >>>       are almost certainly a _bug_. For those v7 CPUs that could do SWP,
> >>>       it's not even guaranteed to be atomic iirc. Trapping and emulating
> >>>       is also bad for performance (although I note that Colin made an
> >>>       argument that it was acceptable).
> >>>
> >>>   (4) This only affects legacy binaries. Should we also try to support OABI?
> >>>       How about misaligned ldm/stm? We have to draw the line somewhere.
> 
> If someone has the time and interest to implement and maintain such features,
> why turn them away, provided there is no performance impact on those who do
> not wish to configure in the support? I think supporting everything within our
> means is the right place to draw the line.

Partially because we have to maintain the stuff forever, but also because
it sets precedent for merging more (unremoveable) emulation code in the
future, which gets proportionally less testing over time.

> >> The problem is that we (Android) have to draw the line somewhere else
> >> - there are too many highly visible apps in the app store that still
> >> use these instructions.  When we add them back to our kernels, then we
> >> are no longer ABI compatible with an upstream kernel.
> > 
> > For an ARMv7 kernel, this is still controlled by CONFIG_SWP_EMULATE, so
> > you would have the exact same issues with kernels where that has been turned
> > off. You assumedly have a bunch of patches on top of mainline for Android; I
> > don't understand why this one is any different.
> 
> As for the number of out-of-tree patches, I've been able to run a minimal
> Android userspace with just one cgroups patch applied on top of
> torvalds/master for at least the past year.

That's cool, I didn't know that was possible. If this became more common, I
agree that it would be more difficult to push back on the SWP emulation
(*if* it needs a kernel patch at all).

Will

Android and compatibility with deprecated armv7 instructions

[Mark Brown]

On Wed, Jul 02, 2014 at 07:25:00PM +0100, Will Deacon wrote:

> Partially because we have to maintain the stuff forever, but also because
> it sets precedent for merging more (unremoveable) emulation code in the
> future, which gets proportionally less testing over time.

If testing is a concern would a testsuite help address that?  I'd expect
we could get it run regularly on mainline by someone like Linaro or
Google.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20140702/56c009c1/attachment.sig>

Android and compatibility with deprecated armv7 instructions

[Rob Herring]

On Wed, Jul 2, 2014 at 11:16 AM, Will Deacon <will.deacon@arm.com> wrote:
> Hi Colin,
>
> On Wed, Jul 02, 2014 at 04:48:07PM +0100, Colin Cross wrote:
>> On Wed, Jul 2, 2014 at 3:01 AM, Will Deacon <will.deacon@arm.com> wrote:
>> > On Wed, Jul 02, 2014 at 12:48:00AM +0100, Mark Brown wrote:
>> > > On Tue, Jul 01, 2014 at 04:42:01PM -0700, Olof Johansson wrote:
>> > > > On Tue, Jul 1, 2014 at 4:06 PM, Colin Cross <ccross@google.com> wrote:
>> > > > > Would you consider taking support for SWP emulation, enabling CP15
>> > > > > barriers (CP15BEN bit only until there's a real device that needs
>> > > > > emulation, also requires clearing COMPAT_PSR_E_BIT in
>> > > > > compat_setup_return) and enabling SETEND, all behind a default-off
>> > > > > CONFIG_DEPRECATED_ARMV7_COMPAT?
>> > >
>> > > > It sounds really silly to push back against this, since it's actually
>> > > > needed by so many platforms out there.
>> >
>> > The big problem with emulating instructions that don't even appear in the
>> > hardware anymore is that we end up creating baggage which we can *never*
>> > remove.
>> >
>> > I'm against SWP emulation in the kernel for a number of reasons:
>> >
>> >   (1) The hardware doesn't have the instruction at all. If we start
>> >       emulating it, then we'll always have to emulate it and it doesn't
>> >       encourage software migration.
>> >
>> >   (2) I'm not convinced that it can't be handled in userspace by trapping
>> >       the SIGILL and emulating there (admittedly, this sounds difficult).
>> >
>> >   (3) The usual uses of SWP are in homebrew locking implementations and
>> >       are almost certainly a _bug_. For those v7 CPUs that could do SWP,
>> >       it's not even guaranteed to be atomic iirc. Trapping and emulating
>> >       is also bad for performance (although I note that Colin made an
>> >       argument that it was acceptable).
>> >
>> >   (4) This only affects legacy binaries. Should we also try to support OABI?
>> >       How about misaligned ldm/stm? We have to draw the line somewhere.
>>
>> The problem is that we (Android) have to draw the line somewhere else
>> - there are too many highly visible apps in the app store that still
>> use these instructions.  When we add them back to our kernels, then we
>> are no longer ABI compatible with an upstream kernel.
>
> For an ARMv7 kernel, this is still controlled by CONFIG_SWP_EMULATE, so
> you would have the exact same issues with kernels where that has been turned
> off. You assumedly have a bunch of patches on top of mainline for Android; I
> don't understand why this one is any different.

So we don't want any vendor patches either since they already have
other patches in their kernels? Avoiding more patches for Android is a
good thing. And don't worry, we will find something else for John S to
do.

>> > The CP15 barriers are a more interesting case, as the CPUs can *currently*
>> > support those if we flip a bit in the SCTLR. However, I see that as a
>> > slippery slope to emulation if CPUs stop supporting those instructions in
>> > the future (they almost certainly will).
>>
>> I agree that this will likely lead to emulation when a CPU
>> manufacturer eventually decides to leave out hardware support,
>> although hopefully they won't if they see that the bit is set in SCTLR
>> on all Android devices.
>
> ... and what if/when people start building AArch64-only CPUs? Are we going to
> emulate the entire AArch32 instruction set in the kernel? Or just the
> deprecated/obsolete subset of that ;)

That is requirement on Si vendors, not application developers. They
are different orders of magnitude. I imagine if this was a matter of
fixing 10 apps, then this would not be being discussed.

Rob

Android and compatibility with deprecated armv7 instructions

[Mark Brown]

On Wed, Jul 02, 2014 at 11:01:33AM +0100, Will Deacon wrote:

>   (1) The hardware doesn't have the instruction at all. If we start
>       emulating it, then we'll always have to emulate it and it doesn't
>       encourage software migration.

That's what the logging other architectures did in similar situations
was intended to help address - even if people don't care about the
performance impact having loud complaints in the logs tends to cause
some useful subset of users to report it as a problem.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20140702/e4f13bf2/attachment.sig>

Android and compatibility with deprecated armv7 instructions

[Will Deacon]

On Wed, Jul 02, 2014 at 05:39:23PM +0100, Mark Brown wrote:
> On Wed, Jul 02, 2014 at 11:01:33AM +0100, Will Deacon wrote:
> 
> >   (1) The hardware doesn't have the instruction at all. If we start
> >       emulating it, then we'll always have to emulate it and it doesn't
> >       encourage software migration.
> 
> That's what the logging other architectures did in similar situations
> was intended to help address - even if people don't care about the
> performance impact having loud complaints in the logs tends to cause
> some useful subset of users to report it as a problem.

I don't think logging something to dmesg is going to make a blind bit of
difference on an Android device. Maybe we could get SWP emulation to request
access to the users contacts and send everybody a text message each time
they issue the instruction?

Seriously though, nothing causes loud complaints like a SIGILL. This thread
is evidence of that.

Will

Android and compatibility with deprecated armv7 instructions

[Mark Brown]

On Wed, Jul 02, 2014 at 06:01:41PM +0100, Will Deacon wrote:
> On Wed, Jul 02, 2014 at 05:39:23PM +0100, Mark Brown wrote:
> > On Wed, Jul 02, 2014 at 11:01:33AM +0100, Will Deacon wrote:

> > >   (1) The hardware doesn't have the instruction at all. If we start
> > >       emulating it, then we'll always have to emulate it and it doesn't
> > >       encourage software migration.

> > That's what the logging other architectures did in similar situations
> > was intended to help address - even if people don't care about the
> > performance impact having loud complaints in the logs tends to cause
> > some useful subset of users to report it as a problem.

> I don't think logging something to dmesg is going to make a blind bit of
> difference on an Android device. Maybe we could get SWP emulation to request

The same is true for the use cases this was being done for - you don't
need *every* user to notice, you need enough users to notice.

> Seriously though, nothing causes loud complaints like a SIGILL. This thread
> is evidence of that.

The trick is to make the complaints go to the application rather than
system vendor.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20140702/83e5b8df/attachment.sig>

Android and compatibility with deprecated armv7 instructions

[Grant Likely]

On Wed, Jul 2, 2014 at 6:01 PM, Will Deacon <will.deacon@arm.com> wrote:
> On Wed, Jul 02, 2014 at 05:39:23PM +0100, Mark Brown wrote:
>> On Wed, Jul 02, 2014 at 11:01:33AM +0100, Will Deacon wrote:
>>
>> >   (1) The hardware doesn't have the instruction at all. If we start
>> >       emulating it, then we'll always have to emulate it and it doesn't
>> >       encourage software migration.
>>
>> That's what the logging other architectures did in similar situations
>> was intended to help address - even if people don't care about the
>> performance impact having loud complaints in the logs tends to cause
>> some useful subset of users to report it as a problem.
>
> I don't think logging something to dmesg is going to make a blind bit of
> difference on an Android device. Maybe we could get SWP emulation to request
> access to the users contacts and send everybody a text message each time
> they issue the instruction?
>
> Seriously though, nothing causes loud complaints like a SIGILL. This thread
> is evidence of that.

I think that looks at the problem entirely the wrong way around. We
don't break userspace. That's been a rule for a very long time.
Android is not the embedded world where we could get away with a whole
lot. There is a *freaking huge* installed base of applications.
Breaking them is not an option, and I think Colin's question makes it
clear that Android is going make sure that doesn't happen regardless
of what the mainline kernel does... and they are right to do so.

g.

Android and compatibility with deprecated armv7 instructions

[Catalin Marinas]

(I've been away for a day and missed all the fun ;))

On Wed, Jul 02, 2014 at 11:14:47PM +0100, Grant Likely wrote:
> On Wed, Jul 2, 2014 at 6:01 PM, Will Deacon <will.deacon@arm.com> wrote:
> > On Wed, Jul 02, 2014 at 05:39:23PM +0100, Mark Brown wrote:
> >> On Wed, Jul 02, 2014 at 11:01:33AM +0100, Will Deacon wrote:
> >>
> >> >   (1) The hardware doesn't have the instruction at all. If we start
> >> >       emulating it, then we'll always have to emulate it and it doesn't
> >> >       encourage software migration.
> >>
> >> That's what the logging other architectures did in similar situations
> >> was intended to help address - even if people don't care about the
> >> performance impact having loud complaints in the logs tends to cause
> >> some useful subset of users to report it as a problem.
> >
> > I don't think logging something to dmesg is going to make a blind bit of
> > difference on an Android device. Maybe we could get SWP emulation to request
> > access to the users contacts and send everybody a text message each time
> > they issue the instruction?
> >
> > Seriously though, nothing causes loud complaints like a SIGILL. This thread
> > is evidence of that.
> 
> I think that looks at the problem entirely the wrong way around. We
> don't break userspace. That's been a rule for a very long time.

In my view, we don't break user space. It's rather a choice of how much
legacy we support on ARMv8 hardware with an arm64 kernel. We don't even
provide HWCAP_SWP for compat (not that anyone bothers checking that).
It's a similar reason we choose not to support/emulate NWFPE or OABI
even though there are existing binaries out there. If we choose to
support SWP in the arm64 kernel, then we won't be able to break user
space and have to carry it for a long time (unless you figure out a way
to remove such features from the kernel because HWCAP certainly doesn't
work, nor printk).

The ARM feature deprecation/removal cycle allows for two architecture
revisions before being entirely removed. SWP has been deprecated in
ARMv6, made optional in ARMv7 (with the possibility of disabling it if
still available) and removed in ARMv8. The Thumb-1/2 instruction sets
never had the SWP instruction. Even when SWP was available in certain
ARMv7 implementations, it was no longer guaranteed to be atomic.

> Android is not the embedded world where we could get away with a whole
> lot. There is a *freaking huge* installed base of applications.
> Breaking them is not an option, and I think Colin's question makes it
> clear that Android is going make sure that doesn't happen regardless
> of what the mainline kernel does... and they are right to do so.

I don't know how huge this installed base of applications is. AFAIK,
it's limited to a (maybe significant) number of Android games all based
on certain library which no longer uses SWP in its recent releases. I
may be wrong but the information I have so far is that this huge base of
applications does not go beyond Android. Furthermore, people getting a
new Android phone with ARMv8 will have to re-download applications
anyway, so the currently installed base does _not_ matter. What matters
is what is provided in the Android _app store_.

Note that I don't say Google should break those applications but they
can carry a patch in their Android kernel while reaching out to
developers to sort their code (can the Android app store be scanned?).
What I don't want is to be in a situation 10 years from now when we
still carry SWP emulation code that no-one uses but we can't remove
because it would break the user space features we agreed upon.

I have limited knowledge of Android user space but I think SWP emulation
could also be implemented in user space via a SIGILL handler in the
zygote thread and inherited by forked apps (performance doesn't really
matter here). A similar example for Android is the binder driver
user-kernel ABI. AFAIK, Google decided not to provide a compat ABI for
the 64-bit compilation of this driver but update the AArch32 user-space
library to use the new 64-bit ABI. That's perfectly fine by me, they
chose not to provide such ABI in the kernel but solve it entirely in
user space and could do the same with SWP.

-- 
Catalin

Android and compatibility with deprecated armv7 instructions

[Arnd Bergmann]

On Thursday 03 July 2014 11:41:35 Catalin Marinas wrote:
> 
> In my view, we don't break user space. It's rather a choice of how much
> legacy we support on ARMv8 hardware with an arm64 kernel. We don't even
> provide HWCAP_SWP for compat (not that anyone bothers checking that).
> It's a similar reason we choose not to support/emulate NWFPE or OABI
> even though there are existing binaries out there. If we choose to
> support SWP in the arm64 kernel, then we won't be able to break user
> space and have to carry it for a long time (unless you figure out a way
> to remove such features from the kernel because HWCAP certainly doesn't
> work, nor printk).
> 
> The ARM feature deprecation/removal cycle allows for two architecture
> revisions before being entirely removed. SWP has been deprecated in
> ARMv6, made optional in ARMv7 (with the possibility of disabling it if
> still available) and removed in ARMv8. The Thumb-1/2 instruction sets
> never had the SWP instruction. Even when SWP was available in certain
> ARMv7 implementations, it was no longer guaranteed to be atomic.

The problem really is that the ARM architecture deprecation method is
not compatible with the one we use in the kernel. I don't think there
is a good answer to that.

Note that other architectures have a different way of handling this
by never removing instructions. Obviously knowing that does not help
deal with the problem on ARM.

> > Android is not the embedded world where we could get away with a whole
> > lot. There is a *freaking huge* installed base of applications.
> > Breaking them is not an option, and I think Colin's question makes it
> > clear that Android is going make sure that doesn't happen regardless
> > of what the mainline kernel does... and they are right to do so.
> 
> I don't know how huge this installed base of applications is. AFAIK,
> it's limited to a (maybe significant) number of Android games all based
> on certain library which no longer uses SWP in its recent releases. I
> may be wrong but the information I have so far is that this huge base of
> applications does not go beyond Android. Furthermore, people getting a
> new Android phone with ARMv8 will have to re-download applications
> anyway, so the currently installed base does _not_ matter. What matters
> is what is provided in the Android _app store_.
>
> Note that I don't say Google should break those applications but they
> can carry a patch in their Android kernel while reaching out to
> developers to sort their code (can the Android app store be scanned?).
> What I don't want is to be in a situation 10 years from now when we
> still carry SWP emulation code that no-one uses but we can't remove
> because it would break the user space features we agreed upon.

The rule is still that we don't break stuff, it's not that we can't
make interface changes over a long period of time.

The way the ARM architecture feature deprecation works, we are pretty
much forced to make incompatible changes eventually, but we can
decide when to make them. Going from a 32-bit kernel to a 64-bit
kernel is a reasonable point for breaking compatibility with some
old features, but it's not the only option.

For someone shipping a device, it's pretty obvious that they want to
be as compatible as possible with existing applications, and they
definitely will patch their kernel to do this.

For someone shipping an Android SDK OTOH, it would be best to enforce
that no features that are deprecated in any architecture level are used,
and it would be good to provide a way for the kernel to allow that.
This has clearly not been the case in the past, and I find it hard
to blame the Android developers for that given the language in the
Kconfig help text ("if unsure, say Y", default y if SMP).\
It would be nice however to see that done now, to ensure that
all newly built applications are actually good in this regard.

There are two related questions that we have to ask ourselves:

a) I believe there are other instructions that are marked deprecated
   in ARMv8 and that will likely be removed at a later point. Can
   we add an option to the kernel to trap them all and either abort
   the program or emit a warning so we don't get another surprise
   like this when ARMv9 eventually comes out?

b) how do we deal with obsolete instructions in a the 32-bit kernel?
   Running a 32-bit kernel in a virtual machine on ARMv8 hardware
   should give a similar behavior to running them on a 64-bit kernel,
   so we probably have to deal with 32-bit ARMv8 environments as well.

> I have limited knowledge of Android user space but I think SWP emulation
> could also be implemented in user space via a SIGILL handler in the
> zygote thread and inherited by forked apps (performance doesn't really
> matter here). A similar example for Android is the binder driver
> user-kernel ABI. AFAIK, Google decided not to provide a compat ABI for
> the 64-bit compilation of this driver but update the AArch32 user-space
> library to use the new 64-bit ABI. That's perfectly fine by me, they
> chose not to provide such ABI in the kernel but solve it entirely in
> user space and could do the same with SWP.

I believe a difference here is that installable Android apps are not
expected to call the binder ioctls directly, so as long as the library
interface is unchanged, nothing is broken.

	Arnd

Android and compatibility with deprecated armv7 instructions

[Russell King - ARM Linux]

On Thu, Jul 03, 2014 at 04:28:38PM +0200, Arnd Bergmann wrote:
> On Thursday 03 July 2014 11:41:35 Catalin Marinas wrote:
> > 
> > In my view, we don't break user space. It's rather a choice of how much
> > legacy we support on ARMv8 hardware with an arm64 kernel. We don't even
> > provide HWCAP_SWP for compat (not that anyone bothers checking that).
> > It's a similar reason we choose not to support/emulate NWFPE or OABI
> > even though there are existing binaries out there. If we choose to
> > support SWP in the arm64 kernel, then we won't be able to break user
> > space and have to carry it for a long time (unless you figure out a way
> > to remove such features from the kernel because HWCAP certainly doesn't
> > work, nor printk).
> > 
> > The ARM feature deprecation/removal cycle allows for two architecture
> > revisions before being entirely removed. SWP has been deprecated in
> > ARMv6, made optional in ARMv7 (with the possibility of disabling it if
> > still available) and removed in ARMv8. The Thumb-1/2 instruction sets
> > never had the SWP instruction. Even when SWP was available in certain
> > ARMv7 implementations, it was no longer guaranteed to be atomic.
> 
> The problem really is that the ARM architecture deprecation method is
> not compatible with the one we use in the kernel. I don't think there
> is a good answer to that.
> 
> Note that other architectures have a different way of handling this
> by never removing instructions. Obviously knowing that does not help
> deal with the problem on ARM.

Actually, there's not much difference.

Let's take a look at what x86 does.  As the instruction set there
develops, they too "retire" old instructions.  Rather than deprecating
them and then removing them entirely, instructions which were once
fast become slower because the core is no longer optimised to execute
that instruction.

This encourages those who wish to have high performance to move to the
new solution - but most importantly, existing software continues to
work.

We can have that on ARM, but only for userspace.  The difference is
that the kernel has to do the backwards compatibility rather than the
(nonexistent) CPU microcode - and we do have that capability for SWP.

We may not find it desirable to do it for ARMv8, but _if_ we wish to
allow users to migrate... if we wish to allow the existing set of
userspace programs (some of which may be statically linked to their
C library) to continue to work, there's only one option here.

While I can understand ARM Ltd not wanting to pollute their nice clean
ARM64 kernel with ARM32 "legacy junk", the fact of the matter is that
even the CPU is already polluted with the legacy ARM32 stuff - it
supports ARM32 binaries.

As ARM Ltd is moving towards server type systems, they should really
start demonstrating that they understand this point, otherwise people
are going to run away from ARM.  ISA compatibility - and a standardised
platform is what has made x86 become the universal platform that it
has.

The problem here is the embedded thinking, which is "we can make any
change, we just need to update and rebuild all the software which runs
on the CPU."  That really doesn't work when people want to provide a
single binary which works everywhere.

Going back to x86, you can still install DOS on PCs, and it still
works...

Now, for SWP in userspace, the only thing that we need SWP to do is
to provide the atomicity that userspace locking demands.  We already
have that for ARMv7, and the file is 280 lines.  That isn't much to
ask for.

However, we /do/ need the CP15 barrier instructions as well.  We,
as kernel developers, have omitted to provide a way to tell userspace
whether the CP15 barrier instructions are available, *and* whether
the new barrier instructions are there too.  So, like it or not, it
is /our/ failing as kernel developers that userspace is running into
these issues, and it is /our/ responsibility to make sure that
userspace does not break, irrespective of what path is chosed by
the raw CPU support.

Yes, we can talk about what they should be doing.  For SWP, they
/should/ be checking the HWCAPs for HWCAP_SWP, and not using it if
it isn't there.  Not everyone will do that, and while you can argue
that's a bug, to the end user it's a program which crashes and is
unusable.  That's a pretty poor user experience.

All these ABI changes concern me greatly.  The OABI to EABI change
concerned me.  The change to alignment behaviour of LDR/STR concerned
me.  SWP concerns me.  Each time we just say "well, people can just
rebuild their programs".  This trivialising of ABI change really has
to stop.

I think I can accurately judge what Linus' reaction would be on this
point, and it would involve a long email stating his disgust with the
ARM architecture as a whole and why he can't take the ARM architecture
seriously.  Of course, we could Cc him to find out... but the result
will not be pleasant.

-- 
FTTC broadband for 0.8mile line: now at 9.7Mbps down 460kbps up... slowly
improving, and getting towards what was expected from it.

Android and compatibility with deprecated armv7 instructions

[Grant Likely]

On Thu, Jul 3, 2014 at 4:00 PM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Thu, Jul 03, 2014 at 04:28:38PM +0200, Arnd Bergmann wrote:
>> On Thursday 03 July 2014 11:41:35 Catalin Marinas wrote:
>> >
>> > In my view, we don't break user space. It's rather a choice of how much
>> > legacy we support on ARMv8 hardware with an arm64 kernel. We don't even
>> > provide HWCAP_SWP for compat (not that anyone bothers checking that).
>> > It's a similar reason we choose not to support/emulate NWFPE or OABI
>> > even though there are existing binaries out there. If we choose to
>> > support SWP in the arm64 kernel, then we won't be able to break user
>> > space and have to carry it for a long time (unless you figure out a way
>> > to remove such features from the kernel because HWCAP certainly doesn't
>> > work, nor printk).
>> >
>> > The ARM feature deprecation/removal cycle allows for two architecture
>> > revisions before being entirely removed. SWP has been deprecated in
>> > ARMv6, made optional in ARMv7 (with the possibility of disabling it if
>> > still available) and removed in ARMv8. The Thumb-1/2 instruction sets
>> > never had the SWP instruction. Even when SWP was available in certain
>> > ARMv7 implementations, it was no longer guaranteed to be atomic.
>>
>> The problem really is that the ARM architecture deprecation method is
>> not compatible with the one we use in the kernel. I don't think there
>> is a good answer to that.
>>
>> Note that other architectures have a different way of handling this
>> by never removing instructions. Obviously knowing that does not help
>> deal with the problem on ARM.
>
> Actually, there's not much difference.
>
> Let's take a look at what x86 does.  As the instruction set there
> develops, they too "retire" old instructions.  Rather than deprecating
> them and then removing them entirely, instructions which were once
> fast become slower because the core is no longer optimised to execute
> that instruction.
>
> This encourages those who wish to have high performance to move to the
> new solution - but most importantly, existing software continues to
> work.
>
> We can have that on ARM, but only for userspace.  The difference is
> that the kernel has to do the backwards compatibility rather than the
> (nonexistent) CPU microcode - and we do have that capability for SWP.
>
> We may not find it desirable to do it for ARMv8, but _if_ we wish to
> allow users to migrate... if we wish to allow the existing set of
> userspace programs (some of which may be statically linked to their
> C library) to continue to work, there's only one option here.
>
> While I can understand ARM Ltd not wanting to pollute their nice clean
> ARM64 kernel with ARM32 "legacy junk", the fact of the matter is that
> even the CPU is already polluted with the legacy ARM32 stuff - it
> supports ARM32 binaries.
>
> As ARM Ltd is moving towards server type systems, they should really
> start demonstrating that they understand this point, otherwise people
> are going to run away from ARM.  ISA compatibility - and a standardised
> platform is what has made x86 become the universal platform that it
> has.
>
> The problem here is the embedded thinking, which is "we can make any
> change, we just need to update and rebuild all the software which runs
> on the CPU."  That really doesn't work when people want to provide a
> single binary which works everywhere.

Well said. Exactly my point. I was ready to write a long reply, but
this email pretty much covers it.

g.

Android and compatibility with deprecated armv7 instructions

[Catalin Marinas]

On Thu, Jul 03, 2014 at 04:00:08PM +0100, Russell King - ARM Linux wrote:
> On Thu, Jul 03, 2014 at 04:28:38PM +0200, Arnd Bergmann wrote:
> > On Thursday 03 July 2014 11:41:35 Catalin Marinas wrote:
> > > In my view, we don't break user space. It's rather a choice of how much
> > > legacy we support on ARMv8 hardware with an arm64 kernel. We don't even
> > > provide HWCAP_SWP for compat (not that anyone bothers checking that).
> > > It's a similar reason we choose not to support/emulate NWFPE or OABI
> > > even though there are existing binaries out there. If we choose to
> > > support SWP in the arm64 kernel, then we won't be able to break user
> > > space and have to carry it for a long time (unless you figure out a way
> > > to remove such features from the kernel because HWCAP certainly doesn't
> > > work, nor printk).
> > > 
> > > The ARM feature deprecation/removal cycle allows for two architecture
> > > revisions before being entirely removed. SWP has been deprecated in
> > > ARMv6, made optional in ARMv7 (with the possibility of disabling it if
> > > still available) and removed in ARMv8. The Thumb-1/2 instruction sets
> > > never had the SWP instruction. Even when SWP was available in certain
> > > ARMv7 implementations, it was no longer guaranteed to be atomic.
> > 
> > The problem really is that the ARM architecture deprecation method is
> > not compatible with the one we use in the kernel. I don't think there
> > is a good answer to that.
> > 
> > Note that other architectures have a different way of handling this
> > by never removing instructions. Obviously knowing that does not help
> > deal with the problem on ARM.
> 
> Actually, there's not much difference.
> 
> Let's take a look at what x86 does.  As the instruction set there
> develops, they too "retire" old instructions.  Rather than deprecating
> them and then removing them entirely, instructions which were once
> fast become slower because the core is no longer optimised to execute
> that instruction.
> 
> This encourages those who wish to have high performance to move to the
> new solution - but most importantly, existing software continues to
> work.

For many reasons, that's not always possible on ARM. Maybe the
architects learnt from this but it's not just ARM Ltd here. SWP wasn't
removed because ARM Ltd found something better but because on certain
hardware its atomicity semantics could no longer be provided (bus
locking), hence the default y if SMP for the ARMv7 SWP emulation.

> We can have that on ARM, but only for userspace.  The difference is
> that the kernel has to do the backwards compatibility rather than the
> (nonexistent) CPU microcode - and we do have that capability for SWP.
> 
> We may not find it desirable to do it for ARMv8, but _if_ we wish to
> allow users to migrate... if we wish to allow the existing set of
> userspace programs (some of which may be statically linked to their
> C library) to continue to work, there's only one option here.
> 
> While I can understand ARM Ltd

Please don't generalise to ARM Ltd here, even though Will and myself are
employed by ARM. We try hard to separate the kernel maintenance
decisions from the ARM (management) ones (in this instance, they may
actually differ).

> not wanting to pollute their nice clean
> ARM64 kernel with ARM32 "legacy junk", the fact of the matter is that
> even the CPU is already polluted with the legacy ARM32 stuff - it
> supports ARM32 binaries.

It's not about polluting the kernel (I've seen at least one incarnation
of such patch and, as you said, it's small), but rather setting some
limit on how much "legacy" we add to the arm64 kernel and commit to
maintain long term. Our stance has always been that only non-deprecated
ARMv7 features are supported by arm64 compat (that's why I don't buy the
breaking user-space argument). Let's say we allow non-deprecated ARMv5
features as well (SWP), how far back to we go? I doubt anyone thinks
ARMv4 and OABI is still needed on an arm64 kernel.

> However, we /do/ need the CP15 barrier instructions as well.  We,
> as kernel developers, have omitted to provide a way to tell userspace
> whether the CP15 barrier instructions are available, *and* whether
> the new barrier instructions are there too.  So, like it or not, it
> is /our/ failing as kernel developers that userspace is running into
> these issues, and it is /our/ responsibility to make sure that
> userspace does not break, irrespective of what path is chosed by
> the raw CPU support.

Such barriers currently only require an SCTLR_EL1 bit to be set. But
what do we do in future version of the architecture if they are no
longer present? I guess we start emulating them (or NAK architectural
attempts at removing them, though that's not always successful and on
some occasions for good reasons).

The main argument on this thread is about how far back in the
architecture roadmap should the arm64 kernel support/emulate. Do we need
to have a plan in place for eventually removing such emulation (like
maybe even making the emulation slower and slower)?

-- 
Catalin

Android and compatibility with deprecated armv7 instructions

[Russell King - ARM Linux]

On Thu, Jul 03, 2014 at 06:13:56PM +0100, Catalin Marinas wrote:
> On Thu, Jul 03, 2014 at 04:00:08PM +0100, Russell King - ARM Linux wrote:
> > On Thu, Jul 03, 2014 at 04:28:38PM +0200, Arnd Bergmann wrote:
> > > On Thursday 03 July 2014 11:41:35 Catalin Marinas wrote:
> > > > In my view, we don't break user space. It's rather a choice of how much
> > > > legacy we support on ARMv8 hardware with an arm64 kernel. We don't even
> > > > provide HWCAP_SWP for compat (not that anyone bothers checking that).
> > > > It's a similar reason we choose not to support/emulate NWFPE or OABI
> > > > even though there are existing binaries out there. If we choose to
> > > > support SWP in the arm64 kernel, then we won't be able to break user
> > > > space and have to carry it for a long time (unless you figure out a way
> > > > to remove such features from the kernel because HWCAP certainly doesn't
> > > > work, nor printk).
> > > > 
> > > > The ARM feature deprecation/removal cycle allows for two architecture
> > > > revisions before being entirely removed. SWP has been deprecated in
> > > > ARMv6, made optional in ARMv7 (with the possibility of disabling it if
> > > > still available) and removed in ARMv8. The Thumb-1/2 instruction sets
> > > > never had the SWP instruction. Even when SWP was available in certain
> > > > ARMv7 implementations, it was no longer guaranteed to be atomic.
> > > 
> > > The problem really is that the ARM architecture deprecation method is
> > > not compatible with the one we use in the kernel. I don't think there
> > > is a good answer to that.
> > > 
> > > Note that other architectures have a different way of handling this
> > > by never removing instructions. Obviously knowing that does not help
> > > deal with the problem on ARM.
> > 
> > Actually, there's not much difference.
> > 
> > Let's take a look at what x86 does.  As the instruction set there
> > develops, they too "retire" old instructions.  Rather than deprecating
> > them and then removing them entirely, instructions which were once
> > fast become slower because the core is no longer optimised to execute
> > that instruction.
> > 
> > This encourages those who wish to have high performance to move to the
> > new solution - but most importantly, existing software continues to
> > work.
> 
> For many reasons, that's not always possible on ARM. Maybe the
> architects learnt from this but it's not just ARM Ltd here. SWP wasn't
> removed because ARM Ltd found something better but because on certain
> hardware its atomicity semantics could no longer be provided (bus
> locking), hence the default y if SMP for the ARMv7 SWP emulation.

Yes, absolutely true.  SWP is a /very/ heavy weight instruction on
hardware which implements it - it goes right down to the bus level,
sometimes to the peripheral buses as well.

StrongARM with a DC21285 did this - the StrongARM has a LOCK signal
which instructs the DC21285 to assert the PCI LOCK signal across
the read and the write.

> The main argument on this thread is about how far back in the
> architecture roadmap should the arm64 kernel support/emulate. Do we need
> to have a plan in place for eventually removing such emulation (like
> maybe even making the emulation slower and slower)?

That is something that we, as kernel developers, are not in a
position to dictate.

That is more a userland thing, and, just like when we try and deprecate
a kernel feature, if someone notices its gone missing, that's a bug
that needs to be resolved.

In other words, it's always been the case that you /can/ remove things
that no one is using, but if someone is using it, then it can't be
removed.

The second point here is that we as kernel developers have failed big
time to deal with this.

We have failed to provide the information that userspace needs to be
able to decide which instructions to use.  Yes, we have sort of done
it with HWCAPs, but HWCAPs are not entirely sufficient to determine
whether a particular instruction should be used.

For instance, looking just at HWCAPs, can you tell me at what point
the SWP instruction should no longer be used?

Remember that HWCAP_SWP is set for ARMv6 with SMP.  It's also set
for ARMv7 SMP as well:

processor       : 0
model name      : ARMv7 Processor rev 10 (v7l)
Features        : swp half thumb fastmult vfp edsp neon vfpv3 tls vfpd32 

processor       : 1
model name      : ARMv7 Processor rev 10 (v7l)
Features        : swp half thumb fastmult vfp edsp neon vfpv3 tls vfpd32 

So, the presence of HWCAP_SWP doesn't mean "you can use SWP".  In
fact, because of our failign, HWCAP_SWP is absolutely meaningless to
userland.

Moreover, since we _have_ been advertising its presence at ARMv7
despite it being deprecated, that's another bug on us as kernel
developers.

So when we go and say "you can no longer use this on ARMv8 because
the architecture deprecated it"... err, the architecture may have,
but we've *not* been telling userland about that.

-- 
FTTC broadband for 0.8mile line: now at 9.7Mbps down 460kbps up... slowly
improving, and getting towards what was expected from it.

Android and compatibility with deprecated armv7 instructions

[Arnd Bergmann]

On Thursday 03 July 2014 18:13:56 Catalin Marinas wrote:
> > not wanting to pollute their nice clean
> > ARM64 kernel with ARM32 "legacy junk", the fact of the matter is that
> > even the CPU is already polluted with the legacy ARM32 stuff - it
> > supports ARM32 binaries.
> 
> It's not about polluting the kernel (I've seen at least one incarnation
> of such patch and, as you said, it's small), but rather setting some
> limit on how much "legacy" we add to the arm64 kernel and commit to
> maintain long term. Our stance has always been that only non-deprecated
> ARMv7 features are supported by arm64 compat (that's why I don't buy the
> breaking user-space argument). Let's say we allow non-deprecated ARMv5
> features as well (SWP), how far back to we go? I doubt anyone thinks
> ARMv4 and OABI is still needed on an arm64 kernel.

Again, the rule is that we don't break things. The request came from
Android because they have binaries that go back to ARMv5. They don't
have binaries going back to ARMv4, and they never had OABI, so it would
be pointless to go back that far.

For any of the regular non-embedded distros, they seem to be fairly
eager to break backwards compatibility on a binary level already,
so it's also very likely that they'd be interested in ARMv4 or OABI
compatibility.

If a real use case for those came up, we'd probably have to talk
about the trade-offs. I'd assume even if actual users were hurt by
the lack of OABI support on ARMv8 (which I assume they are not),
I would likely still argue that supporting it is too hard. The
security implications of supporting another (rarely used) ABI
should be enough for this.

> > However, we /do/ need the CP15 barrier instructions as well.  We,
> > as kernel developers, have omitted to provide a way to tell userspace
> > whether the CP15 barrier instructions are available, *and* whether
> > the new barrier instructions are there too.  So, like it or not, it
> > is /our/ failing as kernel developers that userspace is running into
> > these issues, and it is /our/ responsibility to make sure that
> > userspace does not break, irrespective of what path is chosed by
> > the raw CPU support.
> 
> Such barriers currently only require an SCTLR_EL1 bit to be set. But
> what do we do in future version of the architecture if they are no
> longer present? I guess we start emulating them (or NAK architectural
> attempts at removing them, though that's not always successful and on
> some occasions for good reasons).

I'd vote for having a kernel option to choose between either leaving the
bit (a) enabled, or (b) leaving it disabled and putting a nasty warning
(e.g. WARN_ON_ONCE()) out on the console or (c) printing that warning
and also aborting the task. There are use cases for all of them.

It might be good to have a sysctl control, at least a global one that
turns all instruction emulation off.

> The main argument on this thread is about how far back in the
> architecture roadmap should the arm64 kernel support/emulate. Do we need
> to have a plan in place for eventually removing such emulation (like
> maybe even making the emulation slower and slower)?

The default answer is "as long as anybody is using it". I had the
same idea with making it gradually slower but in reality that isn't
going to help. I think the best option is to make it easy to find
and debug any application that is using deprecated instructions so
people stop relying on them, and encourage distros to turn the emulation
code off as soon as they are ready.

	Arnd

Android and compatibility with deprecated armv7 instructions

[Russell King - ARM Linux]

On Thu, Jul 03, 2014 at 08:15:32PM +0200, Arnd Bergmann wrote:
> The default answer is "as long as anybody is using it". I had the
> same idea with making it gradually slower but in reality that isn't
> going to help. I think the best option is to make it easy to find
> and debug any application that is using deprecated instructions so
> people stop relying on them, and encourage distros to turn the emulation
> code off as soon as they are ready.

If you read the musl thread from last week, you'll understand why
there's a problem there.

Yes, SWP is deprecated in ARMv6 and ARMv7.  However, it is required to
implement atomics in ARMv5 - there is _no_ other way.

So, let's say that you want to build an application which will run on
Android from ARMv5 through to ARMv8.

You may need to make the decision about when to use SWP and when to
use the new exclusives.  One possible way is to hook SIGILL, and
try executing a LDREX.  If it succeeds, great, they can be used.

Now, consider this.  You build your application for ARMv5.  The toolchain
in this instance will *not* warn you that SWP is deprecated.  You've been
running it on Android platforms from ARMv5 upwards.  You have never had
a report of failure.

An ARMv8 Android device comes to market, and you now start getting lots
of bug reports...

Nothing before that point told you that there was a problem.

To prove this, I just ran this on iMX6DL:

#include <stdio.h>

int val = 5;

extern int swap(int v, int *a);

asm(""
"       .arm\n"
"swap:  swp     r2, r0, [r1]\n"
"       mov     r0, r2\n"
"       bx      lr\n"
"       .type   swap, #function\n"
"       .size   swap, . - swap\n"
"       .thumb");

int main()
{
        int v = swap(1, &val);

        printf("%u\n", v);
        return 0;
}

which produces the correct answer of 5.  SWP_EMULATE is off.  HWCAP_SWP
is set (as it always has been).  There's no warning in the kernel
message log.

Now look at the situation on ARMv8.  Executing the above suddenly fails
with SIGILL.

This means that from the developer and user point of view, there has
been *no* visible deprecation of this instruction.  There's only
"it used to work on ARMv7 but doesn't on ARMv8".

Yes, SWP_EMULATE was introduced.  Great, that has the power to issue
warnings, except it doesn't because they're at DEBUG level, which
normally gets compiled out.  So that doesn't help warn people.

Even with SWP_EMULATE enabled and with the debug problem fixed... does
it help warn people?  Only if you're running on a CPU with virtualisation
extensions, because it silently continues to work on CPUs without.

-- 
FTTC broadband for 0.8mile line: now at 9.7Mbps down 460kbps up... slowly
improving, and getting towards what was expected from it.

Android and compatibility with deprecated armv7 instructions

[Will Deacon]

On Thu, Jul 03, 2014 at 07:30:55PM +0100, Russell King - ARM Linux wrote:
> which produces the correct answer of 5.  SWP_EMULATE is off.  HWCAP_SWP
> is set (as it always has been).  There's no warning in the kernel
> message log.

Even more worrying, that swp you just executed probably wasn't atomic. Try
it with a multithreaded program...

> Now look at the situation on ARMv8.  Executing the above suddenly fails
> with SIGILL.
> 
> This means that from the developer and user point of view, there has
> been *no* visible deprecation of this instruction.  There's only
> "it used to work on ARMv7 but doesn't on ARMv8".
> 
> Yes, SWP_EMULATE was introduced.  Great, that has the power to issue
> warnings, except it doesn't because they're at DEBUG level, which
> normally gets compiled out.  So that doesn't help warn people.
> 
> Even with SWP_EMULATE enabled and with the debug problem fixed... does
> it help warn people?  Only if you're running on a CPU with virtualisation
> extensions, because it silently continues to work on CPUs without.

Not *quite* true; SWP_EMULATE=y disables the h/w SWP instruction by
configuring the sctlr accordingly. So even without virtualisation
extensions (more specifically, on cores with MP extensions), you would
still see the warning if it was fixed.

Will

Android and compatibility with deprecated armv7 instructions

[Russell King - ARM Linux]

On Thu, Jul 03, 2014 at 07:37:39PM +0100, Will Deacon wrote:
> On Thu, Jul 03, 2014 at 07:30:55PM +0100, Russell King - ARM Linux wrote:
> > which produces the correct answer of 5.  SWP_EMULATE is off.  HWCAP_SWP
> > is set (as it always has been).  There's no warning in the kernel
> > message log.
> 
> Even more worrying, that swp you just executed probably wasn't atomic. Try
> it with a multithreaded program...
> 
> > Now look at the situation on ARMv8.  Executing the above suddenly fails
> > with SIGILL.
> > 
> > This means that from the developer and user point of view, there has
> > been *no* visible deprecation of this instruction.  There's only
> > "it used to work on ARMv7 but doesn't on ARMv8".
> > 
> > Yes, SWP_EMULATE was introduced.  Great, that has the power to issue
> > warnings, except it doesn't because they're at DEBUG level, which
> > normally gets compiled out.  So that doesn't help warn people.
> > 
> > Even with SWP_EMULATE enabled and with the debug problem fixed... does
> > it help warn people?  Only if you're running on a CPU with virtualisation
> > extensions, because it silently continues to work on CPUs without.
> 
> Not *quite* true; SWP_EMULATE=y disables the h/w SWP instruction by
> configuring the sctlr accordingly. So even without virtualisation
> extensions (more specifically, on cores with MP extensions), you would
> still see the warning if it was fixed.

Well, does anyone have SWP_EMULATE enabled?

arch/arm/configs/spear13xx_defconfig:# CONFIG_SWP_EMULATE is not set
arch/arm/configs/mvebu_v7_defconfig:# CONFIG_SWP_EMULATE is not set
arch/arm/configs/prima2_defconfig:# CONFIG_SWP_EMULATE is not set
arch/arm/configs/marzen_defconfig:# CONFIG_SWP_EMULATE is not set
arch/arm/configs/shmobile_defconfig:# CONFIG_SWP_EMULATE is not set
arch/arm/configs/koelsch_defconfig:# CONFIG_SWP_EMULATE is not set
arch/arm/configs/vexpress_defconfig:# CONFIG_SWP_EMULATE is not set

I think not.  Even with SWP_EMULATE set, HWCAP_SWP is still
advertised, which you must admit is a very contradictory situation.

-- 
FTTC broadband for 0.8mile line: now at 9.7Mbps down 460kbps up... slowly
improving, and getting towards what was expected from it.

Android and compatibility with deprecated armv7 instructions

[Will Deacon]

On Thu, Jul 03, 2014 at 07:52:43PM +0100, Russell King - ARM Linux wrote:
> On Thu, Jul 03, 2014 at 07:37:39PM +0100, Will Deacon wrote:
> > On Thu, Jul 03, 2014 at 07:30:55PM +0100, Russell King - ARM Linux wrote:
> > > which produces the correct answer of 5.  SWP_EMULATE is off.  HWCAP_SWP
> > > is set (as it always has been).  There's no warning in the kernel
> > > message log.
> > 
> > Even more worrying, that swp you just executed probably wasn't atomic. Try
> > it with a multithreaded program...
> > 
> > > Now look at the situation on ARMv8.  Executing the above suddenly fails
> > > with SIGILL.
> > > 
> > > This means that from the developer and user point of view, there has
> > > been *no* visible deprecation of this instruction.  There's only
> > > "it used to work on ARMv7 but doesn't on ARMv8".
> > > 
> > > Yes, SWP_EMULATE was introduced.  Great, that has the power to issue
> > > warnings, except it doesn't because they're at DEBUG level, which
> > > normally gets compiled out.  So that doesn't help warn people.
> > > 
> > > Even with SWP_EMULATE enabled and with the debug problem fixed... does
> > > it help warn people?  Only if you're running on a CPU with virtualisation
> > > extensions, because it silently continues to work on CPUs without.
> > 
> > Not *quite* true; SWP_EMULATE=y disables the h/w SWP instruction by
> > configuring the sctlr accordingly. So even without virtualisation
> > extensions (more specifically, on cores with MP extensions), you would
> > still see the warning if it was fixed.
> 
> Well, does anyone have SWP_EMULATE enabled?
> 
> arch/arm/configs/spear13xx_defconfig:# CONFIG_SWP_EMULATE is not set
> arch/arm/configs/mvebu_v7_defconfig:# CONFIG_SWP_EMULATE is not set
> arch/arm/configs/prima2_defconfig:# CONFIG_SWP_EMULATE is not set
> arch/arm/configs/marzen_defconfig:# CONFIG_SWP_EMULATE is not set
> arch/arm/configs/shmobile_defconfig:# CONFIG_SWP_EMULATE is not set
> arch/arm/configs/koelsch_defconfig:# CONFIG_SWP_EMULATE is not set
> arch/arm/configs/vexpress_defconfig:# CONFIG_SWP_EMULATE is not set
> 
> I think not.  Even with SWP_EMULATE set, HWCAP_SWP is still
> advertised, which you must admit is a very contradictory situation.

Yup, it's probably worth trying to address these inconsistencies regardless
of the conclusion for the 64-bit kernel. Removing the hwcap and screaming
when emulation happens both sound very sensible to me.

Will

Android and compatibility with deprecated armv7 instructions

[Catalin Marinas]

On Thu, Jul 03, 2014 at 07:30:55PM +0100, Russell King - ARM Linux wrote:
> On Thu, Jul 03, 2014 at 08:15:32PM +0200, Arnd Bergmann wrote:
> > The default answer is "as long as anybody is using it". I had the
> > same idea with making it gradually slower but in reality that isn't
> > going to help. I think the best option is to make it easy to find
> > and debug any application that is using deprecated instructions so
> > people stop relying on them, and encourage distros to turn the emulation
> > code off as soon as they are ready.
> 
> Yes, SWP is deprecated in ARMv6 and ARMv7.  However, it is required to
> implement atomics in ARMv5 - there is _no_ other way.
> 
> So, let's say that you want to build an application which will run on
> Android from ARMv5 through to ARMv8.
> 
> You may need to make the decision about when to use SWP and when to
> use the new exclusives.  One possible way is to hook SIGILL, and
> try executing a LDREX.  If it succeeds, great, they can be used.
> 
> Now, consider this.  You build your application for ARMv5.  The toolchain
> in this instance will *not* warn you that SWP is deprecated.  You've been
> running it on Android platforms from ARMv5 upwards.  You have never had
> a report of failure.

With EABI, we tried to address the differences between architectures by
providing the kuser helpers. Most EABI distros moved to using them and
no longer executing SWP in user space (remember glibc having a cmpxchg
implementation based on SWP that "mostly" worked). Over the years the
hope was that it would completely disappear but I agree that we failed
to address this properly at the kernel level (e.g. more aggressive
warnings, remove HWCAP_SWP).

On arm64 we took a radical stance - SIGILL (and if you are still
worried, emulate it in user space).

> An ARMv8 Android device comes to market, and you now start getting lots
> of bug reports...

We should have started addressing these in ARMv7 but unfortunately we
didn't have a good plan for getting rid of emulated instructions.

> Even with SWP_EMULATE enabled and with the debug problem fixed... does
> it help warn people?  Only if you're running on a CPU with virtualisation
> extensions, because it silently continues to work on CPUs without.

Some clarification here. The virtualisation extensions made SWP
_optional_ (i.e. it may not be present at all). The ARMv7
multiprocessing extensions introduced the possibility to disable/trap
SWP via the SCTLR.SW bit because it was no longer atomic. So basically
native SWP only works (as expected, atomically) on ARMv7 UP and earlier.

-- 
Catalin

Android and compatibility with deprecated armv7 instructions

[Russell King - ARM Linux]

On Fri, Jul 04, 2014 at 09:57:45AM +0100, Catalin Marinas wrote:
> On Thu, Jul 03, 2014 at 07:30:55PM +0100, Russell King - ARM Linux wrote:
> > Even with SWP_EMULATE enabled and with the debug problem fixed... does
> > it help warn people?  Only if you're running on a CPU with virtualisation
> > extensions, because it silently continues to work on CPUs without.
> 
> Some clarification here. The virtualisation extensions made SWP
> _optional_ (i.e. it may not be present at all). The ARMv7
> multiprocessing extensions introduced the possibility to disable/trap
> SWP via the SCTLR.SW bit because it was no longer atomic. So basically
> native SWP only works (as expected, atomically) on ARMv7 UP and earlier.

Yes, but we've cocked this up - on ARMv7 SMP, we don't force SWP to be
enabled.  So, you may have SWP in hardware, which may not be atomic,
but as everyone seems to have SWP_EMULATE /disabled/ we don't know
whether the instruction even exists in any programs or libraries.

It may be that it's been completely eliminated, but we don't know that,
because we've never had the trapping enabled for SMP systems.

On the flip side (as I mentioned to Will) I don't think the situation is
quite as serious as Grant makes it out to be for Android.

There, the user base is already used to apps which don't work with new
Android devices.  For example, despite Android scaling the graphics to
the screen size, there are apps that don't work merely because the screen
is bigger, and the play store knows this and doesn't offer them.  However,
it /is/ still possible to install them (and some people have) by
downloading the .apk file and putting that on SD card.  In this case,
the app (a game) worked perfectly except that the game controls were at
an absolute screen position rather than a scaled position, which is why
the game was no longer offered for the later devices.

Moreover, when you download from the play store, you are only presented
with the version which is appropriate for your device - when you buy a
new device, and you re-fetch your apps, you don't get the same version
that you've had on previous device if there's one more appropriate for
your new device.

So the Linux userspace policy doesn't really apply all that much to the
Android space - I'm not saying we shouldn't try, I'm saying that users
are already used to their apps not working on new devices for a wide
variety of reasons.

The "this app starts unexpectedly crashing (due to swp) when I install
it on a new device" shouldn't happen because they shouldn't be presented
with the app in the first place.

At least this is what I've been told by an experienced Android user and
support person (who does do things like install from .apk's...)

-- 
FTTC broadband for 0.8mile line: now at 9.7Mbps down 460kbps up... slowly
improving, and getting towards what was expected from it.

Android and compatibility with deprecated armv7 instructions

[Arnd Bergmann]

On Friday 04 July 2014 10:25:19 Russell King - ARM Linux wrote:
> Moreover, when you download from the play store, you are only presented
> with the version which is appropriate for your device - when you buy a
> new device, and you re-fetch your apps, you don't get the same version
> that you've had on previous device if there's one more appropriate for
> your new device.

This is certainly true, but at the same time the device makers have
a valid interest in making as many apps as possible be enabled without
requiring a recompile. If it becomes known that device A supports 90%
of the popular apps while device B only supports 70% of the same apps
(until they get recompiled), the decision whether to put emulation in
or not is more about maximizing revenue than about theoretical
correctness.

	Arnd

Android and compatibility with deprecated armv7 instructions

[Dr. David Alan Gilbert]

* Arnd Bergmann (arnd at arndb.de) wrote:
> On Friday 04 July 2014 10:25:19 Russell King - ARM Linux wrote:
> > Moreover, when you download from the play store, you are only presented
> > with the version which is appropriate for your device - when you buy a
> > new device, and you re-fetch your apps, you don't get the same version
> > that you've had on previous device if there's one more appropriate for
> > your new device.
> 
> This is certainly true, but at the same time the device makers have
> a valid interest in making as many apps as possible be enabled without
> requiring a recompile. If it becomes known that device A supports 90%
> of the popular apps while device B only supports 70% of the same apps
> (until they get recompiled), the decision whether to put emulation in
> or not is more about maximizing revenue than about theoretical
> correctness.

I've got a case of mislabelled/misdelivered CPU type in the play store,
where it's delivering my ARMv6t phone a library that has v6t2 instructions
in (cbz/cbnz) - for Google's own Google+ app.

Dave

-- 
 -----Open up your eyes, open up your mind, open up your code -------   
/ Dr. David Alan Gilbert    |       Running GNU/Linux       | Happy  \ 
\ gro.gilbert @ treblig.org |                               | In Hex /
 \ _________________________|_____ http://www.treblig.org   |_______/

Android and compatibility with deprecated armv7 instructions

[Grant Likely]

On Fri, 4 Jul 2014 10:25:19 +0100, Russell King - ARM Linux <linux@arm.linux.org.uk> wrote:
> On Fri, Jul 04, 2014 at 09:57:45AM +0100, Catalin Marinas wrote:
> > On Thu, Jul 03, 2014 at 07:30:55PM +0100, Russell King - ARM Linux wrote:
> > > Even with SWP_EMULATE enabled and with the debug problem fixed... does
> > > it help warn people?  Only if you're running on a CPU with virtualisation
> > > extensions, because it silently continues to work on CPUs without.
> > 
> > Some clarification here. The virtualisation extensions made SWP
> > _optional_ (i.e. it may not be present at all). The ARMv7
> > multiprocessing extensions introduced the possibility to disable/trap
> > SWP via the SCTLR.SW bit because it was no longer atomic. So basically
> > native SWP only works (as expected, atomically) on ARMv7 UP and earlier.
> 
> Yes, but we've cocked this up - on ARMv7 SMP, we don't force SWP to be
> enabled.  So, you may have SWP in hardware, which may not be atomic,
> but as everyone seems to have SWP_EMULATE /disabled/ we don't know
> whether the instruction even exists in any programs or libraries.
> 
> It may be that it's been completely eliminated, but we don't know that,
> because we've never had the trapping enabled for SMP systems.
> 
> On the flip side (as I mentioned to Will) I don't think the situation is
> quite as serious as Grant makes it out to be for Android.
> 
> There, the user base is already used to apps which don't work with new
> Android devices.  For example, despite Android scaling the graphics to
> the screen size, there are apps that don't work merely because the screen
> is bigger, and the play store knows this and doesn't offer them.  However,
> it /is/ still possible to install them (and some people have) by
> downloading the .apk file and putting that on SD card.  In this case,
> the app (a game) worked perfectly except that the game controls were at
> an absolute screen position rather than a scaled position, which is why
> the game was no longer offered for the later devices.

I don't think I'm wrong about how serious the issue is. Read through
Colin's email again. He is talking about exactly the situation that I
described.

Nor do I think that "the user base is already used to app which don't
work" is even remotely a good justification for allowing things to
continue to suck. Android has matured a lot since the early days, and
the compatibility problems that plagued the early devices don't happen
so much anymore.

g.

Android and compatibility with deprecated armv7 instructions

[Russell King - ARM Linux]

On Fri, Jul 04, 2014 at 01:56:15PM +0100, Grant Likely wrote:
> On Fri, 4 Jul 2014 10:25:19 +0100, Russell King - ARM Linux <linux@arm.linux.org.uk> wrote:
> > On Fri, Jul 04, 2014 at 09:57:45AM +0100, Catalin Marinas wrote:
> > > On Thu, Jul 03, 2014 at 07:30:55PM +0100, Russell King - ARM Linux wrote:
> > > > Even with SWP_EMULATE enabled and with the debug problem fixed... does
> > > > it help warn people?  Only if you're running on a CPU with virtualisation
> > > > extensions, because it silently continues to work on CPUs without.
> > > 
> > > Some clarification here. The virtualisation extensions made SWP
> > > _optional_ (i.e. it may not be present at all). The ARMv7
> > > multiprocessing extensions introduced the possibility to disable/trap
> > > SWP via the SCTLR.SW bit because it was no longer atomic. So basically
> > > native SWP only works (as expected, atomically) on ARMv7 UP and earlier.
> > 
> > Yes, but we've cocked this up - on ARMv7 SMP, we don't force SWP to be
> > enabled.  So, you may have SWP in hardware, which may not be atomic,
> > but as everyone seems to have SWP_EMULATE /disabled/ we don't know
> > whether the instruction even exists in any programs or libraries.
> > 
> > It may be that it's been completely eliminated, but we don't know that,
> > because we've never had the trapping enabled for SMP systems.
> > 
> > On the flip side (as I mentioned to Will) I don't think the situation is
> > quite as serious as Grant makes it out to be for Android.
> > 
> > There, the user base is already used to apps which don't work with new
> > Android devices.  For example, despite Android scaling the graphics to
> > the screen size, there are apps that don't work merely because the screen
> > is bigger, and the play store knows this and doesn't offer them.  However,
> > it /is/ still possible to install them (and some people have) by
> > downloading the .apk file and putting that on SD card.  In this case,
> > the app (a game) worked perfectly except that the game controls were at
> > an absolute screen position rather than a scaled position, which is why
> > the game was no longer offered for the later devices.
> 
> I don't think I'm wrong about how serious the issue is. Read through
> Colin's email again. He is talking about exactly the situation that I
> described.
> 
> Nor do I think that "the user base is already used to app which don't
> work" is even remotely a good justification for allowing things to
> continue to suck. Android has matured a lot since the early days, and
> the compatibility problems that plagued the early devices don't happen
> so much anymore.

My points are based upon a lengthy discussion last night with a friend who
is on the front line support of a major retailer of mobiles and tablets,
whom I've known for a number of years.

There's lots of cases where apps are not presented for later devices,
and this is the standard model there.

While we can say that it shouldn't happen, that's not reality.  As I
pointed out, apps can become incompatible just because your device has
a different sized screen.  Would you really argue that the screen is
part of the ABI and thus should never change because it might break
some Android app?

What I'm pointing out is that the Linux philosophy is a good one (and
don't get me wrong, it's something that we /should/ strive for -
especially as we haven't done enough to warn people about SWP going
away), it doesn't appear to translate to the Android space.

I'm pretty sure that there will be a whole truck load of issues which
will stop apps working on an ARMv8 phone, and the lack of SWP is
going to be soo drowned out in the noise of everything else.

Also, as we haven't forced SWP_EMULATE to be enabled on SMP, if there
/are/ users of the instruction on SMP kernels, they're already broken
(though, the instruction may still execute, it won't execute
atomically there, and so is useless for locking or atomic exchanging
purposes.)

I'm about to push a number of changes out today though:

1. save where the last kernel misaligned fault happened (we have a
   64-bit one one in nsm_init_private) and report via /proc/cpu/alignment.

2. ensure that SWP_EMULATE is forced on for SMP configurations.

3. log and save SWP emulation use via kernel log and /proc/cpu/swp_emulate.

4. kill HWCAP_SWP when we don't want people using the SWP instruction.

(4) is pretty worthless anyway, because we haven't done that from the
start - which means userspace can't make decisions based upon that
hwcap whether they can or can't use this instruction.

-- 
FTTC broadband for 0.8mile line: now at 9.7Mbps down 460kbps up... slowly
improving, and getting towards what was expected from it.

Android and compatibility with deprecated armv7 instructions

[Will Deacon]

On Thu, Jul 03, 2014 at 07:15:32PM +0100, Arnd Bergmann wrote:
> On Thursday 03 July 2014 18:13:56 Catalin Marinas wrote:
> > > However, we /do/ need the CP15 barrier instructions as well.  We,
> > > as kernel developers, have omitted to provide a way to tell userspace
> > > whether the CP15 barrier instructions are available, *and* whether
> > > the new barrier instructions are there too.  So, like it or not, it
> > > is /our/ failing as kernel developers that userspace is running into
> > > these issues, and it is /our/ responsibility to make sure that
> > > userspace does not break, irrespective of what path is chosed by
> > > the raw CPU support.
> > 
> > Such barriers currently only require an SCTLR_EL1 bit to be set. But
> > what do we do in future version of the architecture if they are no
> > longer present? I guess we start emulating them (or NAK architectural
> > attempts at removing them, though that's not always successful and on
> > some occasions for good reasons).
> 
> I'd vote for having a kernel option to choose between either leaving the
> bit (a) enabled, or (b) leaving it disabled and putting a nasty warning
> (e.g. WARN_ON_ONCE()) out on the console or (c) printing that warning
> and also aborting the task. There are use cases for all of them.

Great, so now we can argue about what the default behaviour should be. I
vote for (c) :)

> It might be good to have a sysctl control, at least a global one that
> turns all instruction emulation off.

Yeah, and a bunch of stats in sysfs or something.

> > The main argument on this thread is about how far back in the
> > architecture roadmap should the arm64 kernel support/emulate. Do we need
> > to have a plan in place for eventually removing such emulation (like
> > maybe even making the emulation slower and slower)?
> 
> The default answer is "as long as anybody is using it". I had the
> same idea with making it gradually slower but in reality that isn't
> going to help. I think the best option is to make it easy to find
> and debug any application that is using deprecated instructions so
> people stop relying on them, and encourage distros to turn the emulation
> code off as soon as they are ready.

My initial objections when this was posted were primarily driven by not
knowing when we can safely remove a feature. The default position with the
arm64 kernel was not to emulate anything, then we would see whether we ever
had valid users of these features and could consider them on a case-by-case
basis. I still think that was the right thing to do.

However, saying "as long as anybody is using it" is really saying "as long
as anybody is complaining about it being missing". Once we've added
emulation, nobody will ever complain about it being missing, because it
won't be missing. Do you have any ideas about phasing out emulation code
when you don't know if it's used or not?

Will

Android and compatibility with deprecated armv7 instructions

[Arnd Bergmann]

On Thursday 03 July 2014 19:46:26 Will Deacon wrote:
> 
> My initial objections when this was posted were primarily driven by not
> knowing when we can safely remove a feature. The default position with the
> arm64 kernel was not to emulate anything, then we would see whether we ever
> had valid users of these features and could consider them on a case-by-case
> basis. I still think that was the right thing to do.
> 
> However, saying "as long as anybody is using it" is really saying "as long
> as anybody is complaining about it being missing". Once we've added
> emulation, nobody will ever complain about it being missing, because it
> won't be missing. Do you have any ideas about phasing out emulation code
> when you don't know if it's used or not?

Not a lot of good ideas. One thing that can be done is to move the Kconfig
option for enabling the emulation under 'CONFIG_EXPERT' in a couple of years,
possibly renaming it a few times every other year so people that blindly
reuse their .config files will have it disabled again. If we do that, there
should of course be a message printed that references a description for
how to turn it back on and report that you are still using it.

It's certainly a very lengthy process. Just look at how long NWFPE has
been in the kernel after it has become obsolete.

	Arnd

Android and compatibility with deprecated armv7 instructions

[Russell King - ARM Linux]

On Thu, Jul 03, 2014 at 08:53:06PM +0200, Arnd Bergmann wrote:
> It's certainly a very lengthy process. Just look at how long NWFPE has
> been in the kernel after it has become obsolete.

May I remind you that I still run OABI... and still use NWFPE here.
The machine which runs this mutt uses NWFPE with OABI.  The machine
that exim passes the message on to runs NWFPE with OABI.  The machine
which runs screen and irssi runs NWFPE with OABI.  The machine here
which operates as a print server runs NWFPE with OABI.  That's four
machines.

I did state at the time that I wanted there to be a way to migrate
easily from OABI to EABI - we sort of got that with the kernel's OABI
compat stuff, but it was ignored for userland, so there's no way that
I've been able to upgrade these machines (eg by adding an EABI glibc,
and recompiling the libraries/applications through gradual replacement),
and I don't have EABI platforms that can replace them.

What kernels do they run?

Linux pandora 3.12.6+ #28 Sun Dec 29 11:53:29 GMT 2013 armv5tel armv5tel armv5tel GNU/Linux
Linux n2100 3.1.8+ #15 Tue Nov 13 14:06:47 GMT 2012 armv5tel armv5tel armv5tel GNU/Linux
Linux flint 3.12.6+ #86 Tue Apr 1 12:25:44 BST 2014 armv4l armv4l armv4l GNU/Linux
Linux lists 3.9.0 #61 Tue May 14 19:35:42 BST 2013 armv4tl armv4tl armv4tl GNU/Linux

One of these is responsible for discovering the memory leak in the
raid code which no one else could find...

-- 
FTTC broadband for 0.8mile line: now at 9.7Mbps down 460kbps up... slowly
improving, and getting towards what was expected from it.

Android and compatibility with deprecated armv7 instructions

[Arnd Bergmann]

On Thursday 03 July 2014 20:07:42 Russell King - ARM Linux wrote:
> On Thu, Jul 03, 2014 at 08:53:06PM +0200, Arnd Bergmann wrote:
> > It's certainly a very lengthy process. Just look at how long NWFPE has
> > been in the kernel after it has become obsolete.
> 
> May I remind you that I still run OABI... and still use NWFPE here.
> The machine which runs this mutt uses NWFPE with OABI.  The machine
> that exim passes the message on to runs NWFPE with OABI.  The machine
> which runs screen and irssi runs NWFPE with OABI.  The machine here
> which operates as a print server runs NWFPE with OABI.  That's four
> machines.

But the real question is whether you'd want to upgrade your user
space if you get to replace some of these four machines with 64-bit
hardware.

	Arnd

Android and compatibility with deprecated armv7 instructions

[Grant Likely]

On Thu, 3 Jul 2014 18:13:56 +0100, Catalin Marinas <catalin.marinas@arm.com> wrote:
> On Thu, Jul 03, 2014 at 04:00:08PM +0100, Russell King - ARM Linux wrote:
> > We can have that on ARM, but only for userspace.  The difference is
> > that the kernel has to do the backwards compatibility rather than the
> > (nonexistent) CPU microcode - and we do have that capability for SWP.
> > 
> > We may not find it desirable to do it for ARMv8, but _if_ we wish to
> > allow users to migrate... if we wish to allow the existing set of
> > userspace programs (some of which may be statically linked to their
> > C library) to continue to work, there's only one option here.
> > 
> > While I can understand ARM Ltd
> 
> Please don't generalise to ARM Ltd here, even though Will and myself are
> employed by ARM. We try hard to separate the kernel maintenance
> decisions from the ARM (management) ones (in this instance, they may
> actually differ).

In case it gets lost in the heat of this debate, I have certainly
noticed that you do so and I truly appreciate it. Well done.

g.

Android and compatibility with deprecated armv7 instructions

[Grant Likely]

On Thu, Jul 3, 2014 at 11:41 AM, Catalin Marinas
<catalin.marinas@arm.com> wrote:
> (I've been away for a day and missed all the fun ;))
>
> On Wed, Jul 02, 2014 at 11:14:47PM +0100, Grant Likely wrote:
>> Android is not the embedded world where we could get away with a whole
>> lot. There is a *freaking huge* installed base of applications.
>> Breaking them is not an option, and I think Colin's question makes it
>> clear that Android is going make sure that doesn't happen regardless
>> of what the mainline kernel does... and they are right to do so.
>
> I don't know how huge this installed base of applications is. AFAIK,
> it's limited to a (maybe significant) number of Android games all based
> on certain library which no longer uses SWP in its recent releases. I
> may be wrong but the information I have so far is that this huge base of
> applications does not go beyond Android. Furthermore, people getting a
> new Android phone with ARMv8 will have to re-download applications
> anyway, so the currently installed base does _not_ matter. What matters
> is what is provided in the Android _app store_.

Okay, I have to bite on this one....

Ah, no. The installed base *does* matter. Breaking things under the
assumption that they can be fixed with an update is a horrible reason
for breaking stuff. That creates hell for developers.

The first indication they'll have that something is wrong is they'll
start getting negative reviews in the Play store ("This software is
****, it crashes immediately"). Then they've got to figure out why
things have gone wrong. It won't necessarily be obvious that the
complaining users have v8 hardware. Obtain a v8 device, and then
figure out how to update all of their affected apps. During that
entire time their app is broken and they are getting blamed by users
for putting out crap software. And that assumes that the app is
actively maintained. Multiply this by every affected developer.

Then there are the apps that work just fine, but don't have any active
maintenance, either because it is an old project that doesn't have a
team on it anymore or the vendor has disappeared. Do you really think
it is okay to break working apps that are probably not going to get
updated?

Third, there are side loaded apps. An update may not be readily or
easily available. The Play store is not the only game in town. There
are also kioskified Android devices that are basically stock, but have
a custom application installed on it. There is also the Amazon app
store.

Finally, stating that "the developer can just fix it" is a huge
assumption. You're basically saying every app development shop has
extra resources to go back and fix things in their older apps.

So, no. I completely reject any notion that breaking existing apps is
okay. If we're going to say that v8 still supports 32-bit apps, then
it has to be all of v7, not just the 'good' bits. Nor do I think
saying "it's just a bunch of games" justifies anything. We're kernel
engineers. Applications are applications and we don't break userspace.
Period.

> Note that I don't say Google should break those applications but they
> can carry a patch in their Android kernel while reaching out to
> developers to sort their code (can the Android app store be scanned?).
> What I don't want is to be in a situation 10 years from now when we
> still carry SWP emulation code that no-one uses but we can't remove
> because it would break the user space features we agreed upon.

Welcome to system programming. This is what we do. It is a *good*
thing that an x86 userspace from 1995 can still be booted.

> I have limited knowledge of Android user space but I think SWP emulation
> could also be implemented in user space via a SIGILL handler in the
> zygote thread and inherited by forked apps (performance doesn't really
> matter here). A similar example for Android is the binder driver
> user-kernel ABI. AFAIK, Google decided not to provide a compat ABI for
> the 64-bit compilation of this driver but update the AArch32 user-space
> library to use the new 64-bit ABI. That's perfectly fine by me, they
> chose not to provide such ABI in the kernel but solve it entirely in
> user space and could do the same with SWP.

Really? Why would we even want that? We're far better positioned in
the kernel to present the correct behaviour that any trapping from a
userspace application.

g.

Android and compatibility with deprecated armv7 instructions

[Russell King - ARM Linux]

On Thu, Jul 03, 2014 at 05:22:30PM +0100, Grant Likely wrote:
> On Thu, Jul 3, 2014 at 11:41 AM, Catalin Marinas
> <catalin.marinas@arm.com> wrote:
> > (I've been away for a day and missed all the fun ;))
> >
> > On Wed, Jul 02, 2014 at 11:14:47PM +0100, Grant Likely wrote:
> >> Android is not the embedded world where we could get away with a whole
> >> lot. There is a *freaking huge* installed base of applications.
> >> Breaking them is not an option, and I think Colin's question makes it
> >> clear that Android is going make sure that doesn't happen regardless
> >> of what the mainline kernel does... and they are right to do so.
> >
> > I don't know how huge this installed base of applications is. AFAIK,
> > it's limited to a (maybe significant) number of Android games all based
> > on certain library which no longer uses SWP in its recent releases. I
> > may be wrong but the information I have so far is that this huge base of
> > applications does not go beyond Android. Furthermore, people getting a
> > new Android phone with ARMv8 will have to re-download applications
> > anyway, so the currently installed base does _not_ matter. What matters
> > is what is provided in the Android _app store_.
> 
> Okay, I have to bite on this one....
> 
> Ah, no. The installed base *does* matter. Breaking things under the
> assumption that they can be fixed with an update is a horrible reason
> for breaking stuff. That creates hell for developers.
> 
> The first indication they'll have that something is wrong is they'll
> start getting negative reviews in the Play store ("This software is
> ****, it crashes immediately"). Then they've got to figure out why
> things have gone wrong. It won't necessarily be obvious that the
> complaining users have v8 hardware. Obtain a v8 device, and then
> figure out how to update all of their affected apps. During that
> entire time their app is broken and they are getting blamed by users
> for putting out crap software. And that assumes that the app is
> actively maintained. Multiply this by every affected developer.
> 
> Then there are the apps that work just fine, but don't have any active
> maintenance, either because it is an old project that doesn't have a
> team on it anymore or the vendor has disappeared. Do you really think
> it is okay to break working apps that are probably not going to get
> updated?
> 
> Third, there are side loaded apps. An update may not be readily or
> easily available. The Play store is not the only game in town. There
> are also kioskified Android devices that are basically stock, but have
> a custom application installed on it. There is also the Amazon app
> store.
> 
> Finally, stating that "the developer can just fix it" is a huge
> assumption. You're basically saying every app development shop has
> extra resources to go back and fix things in their older apps.
> 
> So, no. I completely reject any notion that breaking existing apps is
> okay. If we're going to say that v8 still supports 32-bit apps, then
> it has to be all of v7, not just the 'good' bits. Nor do I think
> saying "it's just a bunch of games" justifies anything. We're kernel
> engineers. Applications are applications and we don't break userspace.
> Period.

+1 on all points above.  I'd go further - if we're going to say that v8
still supports 32-bit apps, that covers at least v6 *as well*.

For reference, the full story on SWP is:

- present in all ARMv4, ARMv5 CPUs
- deprecated but still present in ARMv6, ARMv7 CPUs
- optionally present in ARMv7VE CPUs

The ARM ARM doesn't positively identify what an ARMv7VE is, but my
guess would be ARMv7 with the virtualisation extensions.

-- 
FTTC broadband for 0.8mile line: now at 9.7Mbps down 460kbps up... slowly
improving, and getting towards what was expected from it.

Android and compatibility with deprecated armv7 instructions

[Will Deacon]

On Thu, Jul 03, 2014 at 06:05:58PM +0100, Russell King - ARM Linux wrote:
> On Thu, Jul 03, 2014 at 05:22:30PM +0100, Grant Likely wrote:
> > So, no. I completely reject any notion that breaking existing apps is
> > okay. If we're going to say that v8 still supports 32-bit apps, then
> > it has to be all of v7, not just the 'good' bits. Nor do I think
> > saying "it's just a bunch of games" justifies anything. We're kernel
> > engineers. Applications are applications and we don't break userspace.
> > Period.
> 
> +1 on all points above.  I'd go further - if we're going to say that v8
> still supports 32-bit apps, that covers at least v6 *as well*.

We've never pretended to support anything other than ARMv8 in the compat
layer. uname even reports this in the machine name.

If people are suddenly so concerned about *full* compatibility with an ARMv7
kernel, that needs a lot more than just SWP emulation:

  - Alignment fixups for ldm/stm
  - SETEND
  - CP15 barriers
  - SWI breakpoints + branch through zero syscalls
  (- SWP)

By the arguments presented so far, I can't see why we wouldn't also need
OABI too. In other words, where do we draw the line? If we're not completely
compatible, then the compatibility argument suddenly becomes subjective.

It seems that people really want us to implement the subset of the ABI which
is needed by the Google Play store and are trying to dress that up as the
ARMv7 kernel ABI. The latter is a lot more work and conflating the two isn't
especially helpful.

Will

Android and compatibility with deprecated armv7 instructions

[Arnd Bergmann]

On Thursday 03 July 2014 18:32:26 Will Deacon wrote:
> On Thu, Jul 03, 2014 at 06:05:58PM +0100, Russell King - ARM Linux wrote:
> > On Thu, Jul 03, 2014 at 05:22:30PM +0100, Grant Likely wrote:
> > > So, no. I completely reject any notion that breaking existing apps is
> > > okay. If we're going to say that v8 still supports 32-bit apps, then
> > > it has to be all of v7, not just the 'good' bits. Nor do I think
> > > saying "it's just a bunch of games" justifies anything. We're kernel
> > > engineers. Applications are applications and we don't break userspace.
> > > Period.
> > 
> > +1 on all points above.  I'd go further - if we're going to say that v8
> > still supports 32-bit apps, that covers at least v6 *as well*.
> 
> We've never pretended to support anything other than ARMv8 in the compat
> layer. uname even reports this in the machine name.
> 
> If people are suddenly so concerned about *full* compatibility with an ARMv7
> kernel, that needs a lot more than just SWP emulation:
> 
>   - Alignment fixups for ldm/stm
>   - SETEND
>   - CP15 barriers
>   - SWI breakpoints + branch through zero syscalls
>   (- SWP)

Thanks for the list. Of course we would only have to support the ones
that anybody is using on ARMv8. We know about SWP and I suppose SETEND
as well, cp15 barriers are likely to be needed by someone, and I have
no idea about the others.

Do you know if it's actually technically possible to emulate all of
them? E.g. does ARMv8 allow implementations that cannot switch endianess
at all?

> By the arguments presented so far, I can't see why we wouldn't also need
> OABI too. In other words, where do we draw the line? If we're not completely
> compatible, then the compatibility argument suddenly becomes subjective.

As I just said in the other thread, OABI is pretty clearly on the other
side of the line. Same for NWFPE and BINFMT_AOUT (you removed the latter
on ARM32 already).
 
> It seems that people really want us to implement the subset of the ABI which
> is needed by the Google Play store and are trying to dress that up as the
> ARMv7 kernel ABI. The latter is a lot more work and conflating the two isn't
> especially helpful.

Right. This should be about keeping users from getting mad at us (or at
people using our kernel, who then get mad at us), not about strict adherence
an ABI if nobody cares about it.

	Arnd

Android and compatibility with deprecated armv7 instructions

[Peter Maydell]

On 3 July 2014 19:23, Arnd Bergmann <arnd@arndb.de> wrote:
> E.g. does ARMv8 allow implementations that cannot
> switch endianess at all?

Yes, the architecture permits single-endianness implementations
(in which case SCTLR.SED is RES1, and the various other
endianness controls are RES0 or RES1 depending on whether
the CPU is always-little-endian or always-big-endian). It also
permits "EL0 is mixed-endianness but EL1/2/3 are always
big/little endian". (See the "Mixed-endian support" section
in the v8 ARM ARM.)

-- PMM

Android and compatibility with deprecated armv7 instructions

[Will Deacon]

On Thu, Jul 03, 2014 at 07:23:41PM +0100, Arnd Bergmann wrote:
> On Thursday 03 July 2014 18:32:26 Will Deacon wrote:
> > On Thu, Jul 03, 2014 at 06:05:58PM +0100, Russell King - ARM Linux wrote:
> > > On Thu, Jul 03, 2014 at 05:22:30PM +0100, Grant Likely wrote:
> > > > So, no. I completely reject any notion that breaking existing apps is
> > > > okay. If we're going to say that v8 still supports 32-bit apps, then
> > > > it has to be all of v7, not just the 'good' bits. Nor do I think
> > > > saying "it's just a bunch of games" justifies anything. We're kernel
> > > > engineers. Applications are applications and we don't break userspace.
> > > > Period.
> > > 
> > > +1 on all points above.  I'd go further - if we're going to say that v8
> > > still supports 32-bit apps, that covers at least v6 *as well*.
> > 
> > We've never pretended to support anything other than ARMv8 in the compat
> > layer. uname even reports this in the machine name.
> > 
> > If people are suddenly so concerned about *full* compatibility with an ARMv7
> > kernel, that needs a lot more than just SWP emulation:
> > 
> >   - Alignment fixups for ldm/stm
> >   - SETEND
> >   - CP15 barriers
> >   - SWI breakpoints + branch through zero syscalls
> >   (- SWP)
> 
> Thanks for the list. Of course we would only have to support the ones
> that anybody is using on ARMv8. We know about SWP and I suppose SETEND
> as well, cp15 barriers are likely to be needed by someone, and I have
> no idea about the others.
> 
> Do you know if it's actually technically possible to emulate all of
> them? E.g. does ARMv8 allow implementations that cannot switch endianess
> at all?

You can set SCTLR.E0E on exception return to change the endianness of
userspace, but there could be some `gotchas' with SETEND and the CPSR. I
need to think more about it.

> > By the arguments presented so far, I can't see why we wouldn't also need
> > OABI too. In other words, where do we draw the line? If we're not completely
> > compatible, then the compatibility argument suddenly becomes subjective.
> 
> As I just said in the other thread, OABI is pretty clearly on the other
> side of the line. Same for NWFPE and BINFMT_AOUT (you removed the latter
> on ARM32 already).
>  
> > It seems that people really want us to implement the subset of the ABI which
> > is needed by the Google Play store and are trying to dress that up as the
> > ARMv7 kernel ABI. The latter is a lot more work and conflating the two isn't
> > especially helpful.
> 
> Right. This should be about keeping users from getting mad at us (or at
> people using our kernel, who then get mad at us), not about strict adherence
> an ABI if nobody cares about it.

Agreed. I just got a bit annoyed that people were trying to use the `never
break ABI' argument. We've already chosen to break the ABI in a bunch of
places, and I think that at least some of those decisions are reasonable.

Will

Android and compatibility with deprecated armv7 instructions

[Mark Brown]

On Thu, Jul 03, 2014 at 06:32:26PM +0100, Will Deacon wrote:
> On Thu, Jul 03, 2014 at 06:05:58PM +0100, Russell King - ARM Linux wrote:

> > +1 on all points above.  I'd go further - if we're going to say that v8
> > still supports 32-bit apps, that covers at least v6 *as well*.

> We've never pretended to support anything other than ARMv8 in the compat
> layer. uname even reports this in the machine name.

That may have been the intention but sadly I rather suspect you'll find
that this doesn't match what users have been understanding.

> By the arguments presented so far, I can't see why we wouldn't also need
> OABI too. In other words, where do we draw the line? If we're not completely
> compatible, then the compatibility argument suddenly becomes subjective.

> It seems that people really want us to implement the subset of the ABI which
> is needed by the Google Play store and are trying to dress that up as the
> ARMv7 kernel ABI. The latter is a lot more work and conflating the two isn't
> especially helpful.

Right, or at least that's the set where there is the combination of
caring and time/ability to implement required to do the work to provide
the compatibility which is probably the real metric.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20140703/0afe6d31/attachment.sig>

Android and compatibility with deprecated armv7 instructions

[Måns Rullgård]

Will Deacon <will.deacon@arm.com> writes:

> On Thu, Jul 03, 2014 at 06:05:58PM +0100, Russell King - ARM Linux wrote:
>> On Thu, Jul 03, 2014 at 05:22:30PM +0100, Grant Likely wrote:
>> > So, no. I completely reject any notion that breaking existing apps is
>> > okay. If we're going to say that v8 still supports 32-bit apps, then
>> > it has to be all of v7, not just the 'good' bits. Nor do I think
>> > saying "it's just a bunch of games" justifies anything. We're kernel
>> > engineers. Applications are applications and we don't break userspace.
>> > Period.
>> 
>> +1 on all points above.  I'd go further - if we're going to say that v8
>> still supports 32-bit apps, that covers at least v6 *as well*.
>
> We've never pretended to support anything other than ARMv8 in the compat
> layer. uname even reports this in the machine name.
>
> If people are suddenly so concerned about *full* compatibility with an ARMv7
> kernel, that needs a lot more than just SWP emulation:
>
>   - Alignment fixups for ldm/stm

No ARM variant ever supported unaligned ldm/stm.

-- 
M?ns Rullg?rd
mans at mansr.com

Android and compatibility with deprecated armv7 instructions

[Russell King - ARM Linux]

On Thu, Jul 03, 2014 at 11:16:16PM +0100, M?ns Rullg?rd wrote:
> Will Deacon <will.deacon@arm.com> writes:
> 
> > On Thu, Jul 03, 2014 at 06:05:58PM +0100, Russell King - ARM Linux wrote:
> >> On Thu, Jul 03, 2014 at 05:22:30PM +0100, Grant Likely wrote:
> >> > So, no. I completely reject any notion that breaking existing apps is
> >> > okay. If we're going to say that v8 still supports 32-bit apps, then
> >> > it has to be all of v7, not just the 'good' bits. Nor do I think
> >> > saying "it's just a bunch of games" justifies anything. We're kernel
> >> > engineers. Applications are applications and we don't break userspace.
> >> > Period.
> >> 
> >> +1 on all points above.  I'd go further - if we're going to say that v8
> >> still supports 32-bit apps, that covers at least v6 *as well*.
> >
> > We've never pretended to support anything other than ARMv8 in the compat
> > layer. uname even reports this in the machine name.
> >
> > If people are suddenly so concerned about *full* compatibility with an ARMv7
> > kernel, that needs a lot more than just SWP emulation:
> >
> >   - Alignment fixups for ldm/stm
> 
> No ARM variant ever supported unaligned ldm/stm.

Quite right but that's not the point being discussed.  Please note that
the sentence says "with an ARMv7 *kernel*" - we are not talking about
the architecture there.

So, what's more to the point is that on 32-bit ARM userspace under Linux,
we _have_ supported it since early 2000 up to present.  It's not currently
supported on 64-bit ARM running Linux, even when running a 32-bit binary
in userspace.

Ergo, it's a user visible ABI change, one which we don't know whether it
matters.  In all probability, it doesn't because (hopefully) no one ever
does unaligned LDMs/STMs - I think it would require hand-crafted assembly,
at which point you're talking about optimising something, and you'd be
silly to do it as it would invoke the alignment fault handling which
would be slow.

-- 
FTTC broadband for 0.8mile line: now at 9.7Mbps down 460kbps up... slowly
improving, and getting towards what was expected from it.

Android and compatibility with deprecated armv7 instructions

[Ard Biesheuvel]

On 4 July 2014 00:47, Russell King - ARM Linux <linux@arm.linux.org.uk> wrote:
> On Thu, Jul 03, 2014 at 11:16:16PM +0100, M?ns Rullg?rd wrote:
>> Will Deacon <will.deacon@arm.com> writes:
>>
>> > On Thu, Jul 03, 2014 at 06:05:58PM +0100, Russell King - ARM Linux wrote:
>> >> On Thu, Jul 03, 2014 at 05:22:30PM +0100, Grant Likely wrote:
>> >> > So, no. I completely reject any notion that breaking existing apps is
>> >> > okay. If we're going to say that v8 still supports 32-bit apps, then
>> >> > it has to be all of v7, not just the 'good' bits. Nor do I think
>> >> > saying "it's just a bunch of games" justifies anything. We're kernel
>> >> > engineers. Applications are applications and we don't break userspace.
>> >> > Period.
>> >>
>> >> +1 on all points above.  I'd go further - if we're going to say that v8
>> >> still supports 32-bit apps, that covers at least v6 *as well*.
>> >
>> > We've never pretended to support anything other than ARMv8 in the compat
>> > layer. uname even reports this in the machine name.
>> >
>> > If people are suddenly so concerned about *full* compatibility with an ARMv7
>> > kernel, that needs a lot more than just SWP emulation:
>> >
>> >   - Alignment fixups for ldm/stm
>>
>> No ARM variant ever supported unaligned ldm/stm.
>
> Quite right but that's not the point being discussed.  Please note that
> the sentence says "with an ARMv7 *kernel*" - we are not talking about
> the architecture there.
>
> So, what's more to the point is that on 32-bit ARM userspace under Linux,
> we _have_ supported it since early 2000 up to present.  It's not currently
> supported on 64-bit ARM running Linux, even when running a 32-bit binary
> in userspace.
>
> Ergo, it's a user visible ABI change, one which we don't know whether it
> matters.  In all probability, it doesn't because (hopefully) no one ever
> does unaligned LDMs/STMs - I think it would require hand-crafted assembly,
> at which point you're talking about optimising something, and you'd be
> silly to do it as it would invoke the alignment fault handling which
> would be slow.
>

Well, if something like this

struct bar {
    long l[4];
};

void foo(struct bar *dst, struct bar const *src)
{
    *dst = *src;
}

produces this:

foo:
@ args = 0, pretend = 0, frame = 0
@ frame_needed = 0, uses_anonymous_args = 0
@ link register save eliminated.
    mov ip, r0
    ldmia r1, {r0, r1, r2, r3}
    stmia ip, {r0, r1, r2, r3}
    bx lr

won't it take just a single cast from some unaligned char* to struct
bar* to trigger this?

-- 
Ard.

Android and compatibility with deprecated armv7 instructions

[Catalin Marinas]

On Fri, Jul 04, 2014 at 08:08:05AM +0100, Ard Biesheuvel wrote:
> On 4 July 2014 00:47, Russell King - ARM Linux <linux@arm.linux.org.uk> wrote:
> > On Thu, Jul 03, 2014 at 11:16:16PM +0100, M?ns Rullg?rd wrote:
> >> Will Deacon <will.deacon@arm.com> writes:
> >>
> >> > On Thu, Jul 03, 2014 at 06:05:58PM +0100, Russell King - ARM Linux wrote:
> >> >> On Thu, Jul 03, 2014 at 05:22:30PM +0100, Grant Likely wrote:
> >> >> > So, no. I completely reject any notion that breaking existing apps is
> >> >> > okay. If we're going to say that v8 still supports 32-bit apps, then
> >> >> > it has to be all of v7, not just the 'good' bits. Nor do I think
> >> >> > saying "it's just a bunch of games" justifies anything. We're kernel
> >> >> > engineers. Applications are applications and we don't break userspace.
> >> >> > Period.
> >> >>
> >> >> +1 on all points above.  I'd go further - if we're going to say that v8
> >> >> still supports 32-bit apps, that covers at least v6 *as well*.
> >> >
> >> > We've never pretended to support anything other than ARMv8 in the compat
> >> > layer. uname even reports this in the machine name.
> >> >
> >> > If people are suddenly so concerned about *full* compatibility with an ARMv7
> >> > kernel, that needs a lot more than just SWP emulation:
> >> >
> >> >   - Alignment fixups for ldm/stm
> >>
> >> No ARM variant ever supported unaligned ldm/stm.
> >
> > Quite right but that's not the point being discussed.  Please note that
> > the sentence says "with an ARMv7 *kernel*" - we are not talking about
> > the architecture there.
> >
> > So, what's more to the point is that on 32-bit ARM userspace under Linux,
> > we _have_ supported it since early 2000 up to present.  It's not currently
> > supported on 64-bit ARM running Linux, even when running a 32-bit binary
> > in userspace.
> >
> > Ergo, it's a user visible ABI change, one which we don't know whether it
> > matters.  In all probability, it doesn't because (hopefully) no one ever
> > does unaligned LDMs/STMs - I think it would require hand-crafted assembly,
> > at which point you're talking about optimising something, and you'd be
> > silly to do it as it would invoke the alignment fault handling which
> > would be slow.
> >
> 
> Well, if something like this
> 
> struct bar {
>     long l[4];
> };
> 
> void foo(struct bar *dst, struct bar const *src)
> {
>     *dst = *src;
> }
> 
> produces this:
> 
> foo:
> @ args = 0, pretend = 0, frame = 0
> @ frame_needed = 0, uses_anonymous_args = 0
> @ link register save eliminated.
>     mov ip, r0
>     ldmia r1, {r0, r1, r2, r3}
>     stmia ip, {r0, r1, r2, r3}
>     bx lr
> 
> won't it take just a single cast from some unaligned char* to struct
> bar* to trigger this?

Is this even allowed by the C ABI? The compiler generates the LDMs
because function foo() gets a struct pointer which is guaranteed to be
aligned.

-- 
Catalin

Android and compatibility with deprecated armv7 instructions

[Ard Biesheuvel]

On 4 July 2014 10:24, Catalin Marinas <catalin.marinas@arm.com> wrote:
> On Fri, Jul 04, 2014 at 08:08:05AM +0100, Ard Biesheuvel wrote:
>> On 4 July 2014 00:47, Russell King - ARM Linux <linux@arm.linux.org.uk> wrote:
>> > On Thu, Jul 03, 2014 at 11:16:16PM +0100, M?ns Rullg?rd wrote:
>> >> Will Deacon <will.deacon@arm.com> writes:
>> >>
>> >> > On Thu, Jul 03, 2014 at 06:05:58PM +0100, Russell King - ARM Linux wrote:
>> >> >> On Thu, Jul 03, 2014 at 05:22:30PM +0100, Grant Likely wrote:
>> >> >> > So, no. I completely reject any notion that breaking existing apps is
>> >> >> > okay. If we're going to say that v8 still supports 32-bit apps, then
>> >> >> > it has to be all of v7, not just the 'good' bits. Nor do I think
>> >> >> > saying "it's just a bunch of games" justifies anything. We're kernel
>> >> >> > engineers. Applications are applications and we don't break userspace.
>> >> >> > Period.
>> >> >>
>> >> >> +1 on all points above.  I'd go further - if we're going to say that v8
>> >> >> still supports 32-bit apps, that covers at least v6 *as well*.
>> >> >
>> >> > We've never pretended to support anything other than ARMv8 in the compat
>> >> > layer. uname even reports this in the machine name.
>> >> >
>> >> > If people are suddenly so concerned about *full* compatibility with an ARMv7
>> >> > kernel, that needs a lot more than just SWP emulation:
>> >> >
>> >> >   - Alignment fixups for ldm/stm
>> >>
>> >> No ARM variant ever supported unaligned ldm/stm.
>> >
>> > Quite right but that's not the point being discussed.  Please note that
>> > the sentence says "with an ARMv7 *kernel*" - we are not talking about
>> > the architecture there.
>> >
>> > So, what's more to the point is that on 32-bit ARM userspace under Linux,
>> > we _have_ supported it since early 2000 up to present.  It's not currently
>> > supported on 64-bit ARM running Linux, even when running a 32-bit binary
>> > in userspace.
>> >
>> > Ergo, it's a user visible ABI change, one which we don't know whether it
>> > matters.  In all probability, it doesn't because (hopefully) no one ever
>> > does unaligned LDMs/STMs - I think it would require hand-crafted assembly,
>> > at which point you're talking about optimising something, and you'd be
>> > silly to do it as it would invoke the alignment fault handling which
>> > would be slow.
>> >
>>
>> Well, if something like this
>>
>> struct bar {
>>     long l[4];
>> };
>>
>> void foo(struct bar *dst, struct bar const *src)
>> {
>>     *dst = *src;
>> }
>>
>> produces this:
>>
>> foo:
>> @ args = 0, pretend = 0, frame = 0
>> @ frame_needed = 0, uses_anonymous_args = 0
>> @ link register save eliminated.
>>     mov ip, r0
>>     ldmia r1, {r0, r1, r2, r3}
>>     stmia ip, {r0, r1, r2, r3}
>>     bx lr
>>
>> won't it take just a single cast from some unaligned char* to struct
>> bar* to trigger this?
>
> Is this even allowed by the C ABI? The compiler generates the LDMs
> because function foo() gets a struct pointer which is guaranteed to be
> aligned.
>

Well, I was merely responding to Russell's assertion that it would
require hand-crafted assembly to trigger alignment faults in ldm/stm
instructions.

The point is not whether the C ABI allows it, the point is whether
a) the current 32-bit ARM kernel allows it, and
b) how likely it is to appear in existing code

-- 
Ard.

Android and compatibility with deprecated armv7 instructions

[Måns Rullgård]

Ard Biesheuvel <ard.biesheuvel@linaro.org> writes:

> On 4 July 2014 10:24, Catalin Marinas <catalin.marinas@arm.com> wrote:
>> On Fri, Jul 04, 2014 at 08:08:05AM +0100, Ard Biesheuvel wrote:
>>> On 4 July 2014 00:47, Russell King - ARM Linux <linux@arm.linux.org.uk> wrote:
>>> > On Thu, Jul 03, 2014 at 11:16:16PM +0100, M?ns Rullg?rd wrote:
>>> >> Will Deacon <will.deacon@arm.com> writes:
>>> >>
>>> >> > On Thu, Jul 03, 2014 at 06:05:58PM +0100, Russell King - ARM Linux wrote:
>>> >> >> On Thu, Jul 03, 2014 at 05:22:30PM +0100, Grant Likely wrote:
>>> >> >> > So, no. I completely reject any notion that breaking existing apps is
>>> >> >> > okay. If we're going to say that v8 still supports 32-bit apps, then
>>> >> >> > it has to be all of v7, not just the 'good' bits. Nor do I think
>>> >> >> > saying "it's just a bunch of games" justifies anything. We're kernel
>>> >> >> > engineers. Applications are applications and we don't break userspace.
>>> >> >> > Period.
>>> >> >>
>>> >> >> +1 on all points above.  I'd go further - if we're going to say that v8
>>> >> >> still supports 32-bit apps, that covers at least v6 *as well*.
>>> >> >
>>> >> > We've never pretended to support anything other than ARMv8 in the compat
>>> >> > layer. uname even reports this in the machine name.
>>> >> >
>>> >> > If people are suddenly so concerned about *full* compatibility with an ARMv7
>>> >> > kernel, that needs a lot more than just SWP emulation:
>>> >> >
>>> >> >   - Alignment fixups for ldm/stm
>>> >>
>>> >> No ARM variant ever supported unaligned ldm/stm.
>>> >
>>> > Quite right but that's not the point being discussed.  Please note that
>>> > the sentence says "with an ARMv7 *kernel*" - we are not talking about
>>> > the architecture there.
>>> >
>>> > So, what's more to the point is that on 32-bit ARM userspace under Linux,
>>> > we _have_ supported it since early 2000 up to present.  It's not currently
>>> > supported on 64-bit ARM running Linux, even when running a 32-bit binary
>>> > in userspace.
>>> >
>>> > Ergo, it's a user visible ABI change, one which we don't know whether it
>>> > matters.  In all probability, it doesn't because (hopefully) no one ever
>>> > does unaligned LDMs/STMs - I think it would require hand-crafted assembly,
>>> > at which point you're talking about optimising something, and you'd be
>>> > silly to do it as it would invoke the alignment fault handling which
>>> > would be slow.
>>> >
>>>
>>> Well, if something like this
>>>
>>> struct bar {
>>>     long l[4];
>>> };
>>>
>>> void foo(struct bar *dst, struct bar const *src)
>>> {
>>>     *dst = *src;
>>> }
>>>
>>> produces this:
>>>
>>> foo:
>>> @ args = 0, pretend = 0, frame = 0
>>> @ frame_needed = 0, uses_anonymous_args = 0
>>> @ link register save eliminated.
>>>     mov ip, r0
>>>     ldmia r1, {r0, r1, r2, r3}
>>>     stmia ip, {r0, r1, r2, r3}
>>>     bx lr
>>>
>>> won't it take just a single cast from some unaligned char* to struct
>>> bar* to trigger this?
>>
>> Is this even allowed by the C ABI?

No.

>> The compiler generates the LDMs because function foo() gets a struct
>> pointer which is guaranteed to be aligned.
>>
>
> Well, I was merely responding to Russell's assertion that it would
> require hand-crafted assembly to trigger alignment faults in ldm/stm
> instructions.
>
> The point is not whether the C ABI allows it, the point is whether
> a) the current 32-bit ARM kernel allows it, and

Unfortunately, it does.  In my opinion, adding the unaligned emulation
code was a mistake, and turning it on by default was a bigger mistake.

> b) how likely it is to appear in existing code

Quite unlikely, but there's probably something out there doing it.

Regardless of the points above, unlike SWP, unaligned LDM/STM has never
been correct code.  This makes the argument for keeping the emulation
much weaker.

-- 
M?ns Rullg?rd
mans at mansr.com

Android and compatibility with deprecated armv7 instructions

[Russell King - ARM Linux]

On Fri, Jul 04, 2014 at 10:21:54AM +0100, M?ns Rullg?rd wrote:
> Ard Biesheuvel <ard.biesheuvel@linaro.org> writes:
> >>> foo:
> >>> @ args = 0, pretend = 0, frame = 0
> >>> @ frame_needed = 0, uses_anonymous_args = 0
> >>> @ link register save eliminated.
> >>>     mov ip, r0
> >>>     ldmia r1, {r0, r1, r2, r3}
> >>>     stmia ip, {r0, r1, r2, r3}
> >>>     bx lr
> >>>
> >>> won't it take just a single cast from some unaligned char* to struct
> >>> bar* to trigger this?
> >>
> >> Is this even allowed by the C ABI?
> 
> No.

Sort of.  It isn't that legal, but we /do/ have bits of kernel code
(particularly networking) which /do/ make use of this.  Whether it's
legal or not really doesn't come into it.

It falls into the same class as the additional padding of structures
which caused soo much pain in the early days, where many people used
structures to define the layout of data passed between different
systems, expecting (eg) a three char struct to have a size of 3 rather
than 4.

The general rule for "portable" userspace has traditionally been "if it
works on x86, then the code is correct."  Unfortunately, you can do lots
of things on x86 which aren't legal C.

> > The point is not whether the C ABI allows it, the point is whether
> > a) the current 32-bit ARM kernel allows it, and
> 
> Unfortunately, it does.  In my opinion, adding the unaligned emulation
> code was a mistake, and turning it on by default was a bigger mistake.

Different people have differing opinions, and the decision was made
more than 14 years ago.  Hindsight is also a wonderful tool that we
could all use, but the problem is that's non-causal, so unless you
have a tardis or a time machine...

-- 
FTTC broadband for 0.8mile line: now at 9.7Mbps down 460kbps up... slowly
improving, and getting towards what was expected from it.

Android and compatibility with deprecated armv7 instructions

[Måns Rullgård]

Russell King - ARM Linux <linux@arm.linux.org.uk> writes:

> On Fri, Jul 04, 2014 at 10:21:54AM +0100, M?ns Rullg?rd wrote:
>> Ard Biesheuvel <ard.biesheuvel@linaro.org> writes:
>> >>> foo:
>> >>> @ args = 0, pretend = 0, frame = 0
>> >>> @ frame_needed = 0, uses_anonymous_args = 0
>> >>> @ link register save eliminated.
>> >>>     mov ip, r0
>> >>>     ldmia r1, {r0, r1, r2, r3}
>> >>>     stmia ip, {r0, r1, r2, r3}
>> >>>     bx lr
>> >>>
>> >>> won't it take just a single cast from some unaligned char* to struct
>> >>> bar* to trigger this?
>> >>
>> >> Is this even allowed by the C ABI?
>> 
>> No.
>
> Sort of.  It isn't that legal, but we /do/ have bits of kernel code
> (particularly networking) which /do/ make use of this.  Whether it's
> legal or not really doesn't come into it.

You keep saying this, and the netdev guys keep saying that isn't true,
and if it does happen, it's a bug they want to know about.

> It falls into the same class as the additional padding of structures
> which caused soo much pain in the early days, where many people used
> structures to define the layout of data passed between different
> systems, expecting (eg) a three char struct to have a size of 3 rather
> than 4.

It's quite different from that.  Even if the struct layout isn't the
expected, there's nothing _invalid_ about the code; it just happens to
do the wrong thing.  Code using unaligned pointers is invalid at a
fundamental level and can't be expected to work (whatever that means)
anywhere.

We don't attempt to "fix" code relying on race conditions not coming out
the wrong way, even if some old version of the scheduler always worked
in its favour.

> The general rule for "portable" userspace has traditionally been "if it
> works on x86, then the code is correct."  Unfortunately, you can do lots
> of things on x86 which aren't legal C.

Now that even compilers generate aligned SSE loads and stores, this has
changed to some extent.

>> > The point is not whether the C ABI allows it, the point is whether
>> > a) the current 32-bit ARM kernel allows it, and
>> 
>> Unfortunately, it does.  In my opinion, adding the unaligned emulation
>> code was a mistake, and turning it on by default was a bigger mistake.
>
> Different people have differing opinions, and the decision was made
> more than 14 years ago.  Hindsight is also a wonderful tool that we
> could all use, but the problem is that's non-causal, so unless you
> have a tardis or a time machine...

That shouldn't stop us learning from past mistakes.

-- 
M?ns Rullg?rd
mans at mansr.com

Android and compatibility with deprecated armv7 instructions

[Russell King - ARM Linux]

On Fri, Jul 04, 2014 at 11:21:08AM +0100, M?ns Rullg?rd wrote:
> Russell King - ARM Linux <linux@arm.linux.org.uk> writes:
> > Sort of.  It isn't that legal, but we /do/ have bits of kernel code
> > (particularly networking) which /do/ make use of this.  Whether it's
> > legal or not really doesn't come into it.
> 
> You keep saying this, and the netdev guys keep saying that isn't true,
> and if it does happen, it's a bug they want to know about.

Take a look at the code which parses the ethernet header.  This uses
struct ethhdr, which is passed a pointer which is typically not word
aligned (intentionally so, to ensure that the IP header *is* word
aligned.)

They may say that it isn't true, but I've had the discussion with them
about adding the packed attribute to ethhdr, and it's something they
are completely against.

> > It falls into the same class as the additional padding of structures
> > which caused soo much pain in the early days, where many people used
> > structures to define the layout of data passed between different
> > systems, expecting (eg) a three char struct to have a size of 3 rather
> > than 4.
> 
> It's quite different from that.  Even if the struct layout isn't the
> expected, there's nothing _invalid_ about the code; it just happens to
> do the wrong thing.  Code using unaligned pointers is invalid at a
> fundamental level and can't be expected to work (whatever that means)
> anywhere.

That's your opinion (again) which is different from other people's
opinion.  You haven't been the one to discover these things, and
raise them with the authors.

The normal response has been "ARM is broken, ARM needs to be fixed
so it behaves like x86 does" even when the code is not legal C for
whatever reason.

Thankfully, since then things have moved on a bit and some of the
issues did get resolved.  However, the one you refer to above
concerning unaligned pointers... that still is very much alive and
well in the kernel networking code even today.

It is _standard_ practise to receive a network packet into a buffer
which is offset by two bytes, and then cast the start address to
a struct ethhdr.  To prove this to yourself (because it's obvious
that you don't believe whatever I say) you need to read and understand
rather a lot of code - the network drivers, the socket buffer code,
and eth_type_trans() (which is where the implicit cast to struct ethhdr
occurs.)

-- 
FTTC broadband for 0.8mile line: now at 9.7Mbps down 460kbps up... slowly
improving, and getting towards what was expected from it.

Android and compatibility with deprecated armv7 instructions

[Ard Biesheuvel]

On 4 July 2014 12:33, Russell King - ARM Linux <linux@arm.linux.org.uk> wrote:
> On Fri, Jul 04, 2014 at 11:21:08AM +0100, M?ns Rullg?rd wrote:
>> Russell King - ARM Linux <linux@arm.linux.org.uk> writes:
>> > Sort of.  It isn't that legal, but we /do/ have bits of kernel code
>> > (particularly networking) which /do/ make use of this.  Whether it's
>> > legal or not really doesn't come into it.
>>
>> You keep saying this, and the netdev guys keep saying that isn't true,
>> and if it does happen, it's a bug they want to know about.
>
> Take a look at the code which parses the ethernet header.  This uses
> struct ethhdr, which is passed a pointer which is typically not word
> aligned (intentionally so, to ensure that the IP header *is* word
> aligned.)
>
> They may say that it isn't true, but I've had the discussion with them
> about adding the packed attribute to ethhdr, and it's something they
> are completely against.
>
>> > It falls into the same class as the additional padding of structures
>> > which caused soo much pain in the early days, where many people used
>> > structures to define the layout of data passed between different
>> > systems, expecting (eg) a three char struct to have a size of 3 rather
>> > than 4.
>>
>> It's quite different from that.  Even if the struct layout isn't the
>> expected, there's nothing _invalid_ about the code; it just happens to
>> do the wrong thing.  Code using unaligned pointers is invalid at a
>> fundamental level and can't be expected to work (whatever that means)
>> anywhere.
>
> That's your opinion (again) which is different from other people's
> opinion.  You haven't been the one to discover these things, and
> raise them with the authors.
>
> The normal response has been "ARM is broken, ARM needs to be fixed
> so it behaves like x86 does" even when the code is not legal C for
> whatever reason.
>
> Thankfully, since then things have moved on a bit and some of the
> issues did get resolved.  However, the one you refer to above
> concerning unaligned pointers... that still is very much alive and
> well in the kernel networking code even today.
>
> It is _standard_ practise to receive a network packet into a buffer
> which is offset by two bytes, and then cast the start address to
> a struct ethhdr.  To prove this to yourself (because it's obvious
> that you don't believe whatever I say) you need to read and understand
> rather a lot of code - the network drivers, the socket buffer code,
> and eth_type_trans() (which is where the implicit cast to struct ethhdr
> occurs.)
>

I won't contradict anything you say here, but the question is really
how likely it is whether this happens in userspace.

My assessment would be that it is not as unlikely as some of us think:
I know from prior experience (TomTom) that it's not just games (if
that even matters) that ship with a fairly large native [non-Java]
component based on a legacy, proprietary code base, in this case doing
decryption and decompression of a proprietary map format. I don't know
for a fact that this particular issue exists in this case, but it is
code that follows a similar pattern as the case Russell describes,
i.e., overlaying some structure on a stream of data.

-- 
Ard.

Android and compatibility with deprecated armv7 instructions

[Nicolas Pitre]

On Fri, 4 Jul 2014, Ard Biesheuvel wrote:

> On 4 July 2014 12:33, Russell King - ARM Linux <linux@arm.linux.org.uk> wrote:
> > It is _standard_ practise to receive a network packet into a buffer
> > which is offset by two bytes, and then cast the start address to
> > a struct ethhdr.  To prove this to yourself (because it's obvious
> > that you don't believe whatever I say) you need to read and understand
> > rather a lot of code - the network drivers, the socket buffer code,
> > and eth_type_trans() (which is where the implicit cast to struct ethhdr
> > occurs.)
> >
> 
> I won't contradict anything you say here, but the question is really
> how likely it is whether this happens in userspace.
> 
> My assessment would be that it is not as unlikely as some of us think:
> I know from prior experience (TomTom) that it's not just games (if
> that even matters) that ship with a fairly large native [non-Java]
> component based on a legacy, proprietary code base, in this case doing
> decryption and decompression of a proprietary map format. I don't know
> for a fact that this particular issue exists in this case, but it is
> code that follows a similar pattern as the case Russell describes,
> i.e., overlaying some structure on a stream of data.

We know this isn't legal C to do so, but people do and did it 
nevertheless.  Especially in proprietary code.

And in the Android case we're talking about binary executables and many 
of them are unlikely to ever be recompiled.  Puristic comments about 
language definition conformance won't fix that.

So... how likely are fixed-up unaligned accesses in user space?  At 
least if they happen the kernel may tell you via /proc/cpu/alignment 
where different fixups are counted. Documentation available in 
Documentation/arm/mem_alignment.

So someone at Google should really think about adding statistics 
capturing from /proc/cpu/alignment to Android and gather results from 
the largest user base possible.  Maybe do it with Chrome OS as well.  
There shouldn't be any privacy concerns tied to those statistics.

And if the half/dword/multi counts are ever growing from zero then 
pragmatism must prevail i.e. those misaligned accesses must be emulated 
on ARM64 as well (although for ARM32 user space apps only).


Nicolas

Android and compatibility with deprecated armv7 instructions

[Catalin Marinas]

On Thu, Jul 03, 2014 at 05:22:30PM +0100, Grant Likely wrote:
> On Thu, Jul 3, 2014 at 11:41 AM, Catalin Marinas
> <catalin.marinas@arm.com> wrote:
> > (I've been away for a day and missed all the fun ;))
> >
> > On Wed, Jul 02, 2014 at 11:14:47PM +0100, Grant Likely wrote:
> >> Android is not the embedded world where we could get away with a whole
> >> lot. There is a *freaking huge* installed base of applications.
> >> Breaking them is not an option, and I think Colin's question makes it
> >> clear that Android is going make sure that doesn't happen regardless
> >> of what the mainline kernel does... and they are right to do so.
> >
> > I don't know how huge this installed base of applications is. AFAIK,
> > it's limited to a (maybe significant) number of Android games all based
> > on certain library which no longer uses SWP in its recent releases. I
> > may be wrong but the information I have so far is that this huge base of
> > applications does not go beyond Android. Furthermore, people getting a
> > new Android phone with ARMv8 will have to re-download applications
> > anyway, so the currently installed base does _not_ matter. What matters
> > is what is provided in the Android _app store_.
> 
> Okay, I have to bite on this one....
> 
> Ah, no. The installed base *does* matter. Breaking things under the
> assumption that they can be fixed with an update is a horrible reason
> for breaking stuff. That creates hell for developers.

My point is that you *cannot* upgrade your phone from ARMv7 to ARMv8
without re-downloading the apps. You don't carry over your filesystem
(installed base) on an SD card from the old phone to the new one. That's
where Google has some control I didn't say that Google should break
those apps but actively try to update them (in the meantime, they could
provide some form of SWP emulation).

> So, no. I completely reject any notion that breaking existing apps is
> okay. If we're going to say that v8 still supports 32-bit apps, then
> it has to be all of v7, not just the 'good' bits. Nor do I think
> saying "it's just a bunch of games" justifies anything. We're kernel
> engineers. Applications are applications and we don't break userspace.
> Period.

Here you need to define user-space. OABI?

> > Note that I don't say Google should break those applications but they
> > can carry a patch in their Android kernel while reaching out to
> > developers to sort their code (can the Android app store be scanned?).
> > What I don't want is to be in a situation 10 years from now when we
> > still carry SWP emulation code that no-one uses but we can't remove
> > because it would break the user space features we agreed upon.
> 
> Welcome to system programming. This is what we do. It is a *good*
> thing that an x86 userspace from 1995 can still be booted.

I'm not sure how much of it is just the merit of Linux but rather the
hardware backwards compatibility. As you can see with SWP, we don't
always have this on ARM (and don't blame the kernel maintainers for this
;)).

The same question again - shall we support OABI or we just add ABI
features based on who shouts louder?

> > I have limited knowledge of Android user space but I think SWP emulation
> > could also be implemented in user space via a SIGILL handler in the
> > zygote thread and inherited by forked apps (performance doesn't really
> > matter here). A similar example for Android is the binder driver
> > user-kernel ABI. AFAIK, Google decided not to provide a compat ABI for
> > the 64-bit compilation of this driver but update the AArch32 user-space
> > library to use the new 64-bit ABI. That's perfectly fine by me, they
> > chose not to provide such ABI in the kernel but solve it entirely in
> > user space and could do the same with SWP.
> 
> Really? Why would we even want that? We're far better positioned in
> the kernel to present the correct behaviour that any trapping from a
> userspace application.

Because by adding it to the kernel we declare it ABI (rather than just
an Android issue on ARMv8 hardware; I'm not currently aware of other
ARMv7 Linux distros affected).

-- 
Catalin

Android and compatibility with deprecated armv7 instructions

[Grant Likely]

On Thu, 3 Jul 2014 18:43:18 +0100, Catalin Marinas <catalin.marinas@arm.com> wrote:
> On Thu, Jul 03, 2014 at 05:22:30PM +0100, Grant Likely wrote:
> > On Thu, Jul 3, 2014 at 11:41 AM, Catalin Marinas
> > <catalin.marinas@arm.com> wrote:
> > > (I've been away for a day and missed all the fun ;))
> > >
> > > On Wed, Jul 02, 2014 at 11:14:47PM +0100, Grant Likely wrote:
> > >> Android is not the embedded world where we could get away with a whole
> > >> lot. There is a *freaking huge* installed base of applications.
> > >> Breaking them is not an option, and I think Colin's question makes it
> > >> clear that Android is going make sure that doesn't happen regardless
> > >> of what the mainline kernel does... and they are right to do so.
> > >
> > > I don't know how huge this installed base of applications is. AFAIK,
> > > it's limited to a (maybe significant) number of Android games all based
> > > on certain library which no longer uses SWP in its recent releases. I
> > > may be wrong but the information I have so far is that this huge base of
> > > applications does not go beyond Android. Furthermore, people getting a
> > > new Android phone with ARMv8 will have to re-download applications
> > > anyway, so the currently installed base does _not_ matter. What matters
> > > is what is provided in the Android _app store_.
> > 
> > Okay, I have to bite on this one....
> > 
> > Ah, no. The installed base *does* matter. Breaking things under the
> > assumption that they can be fixed with an update is a horrible reason
> > for breaking stuff. That creates hell for developers.
> 
> My point is that you *cannot* upgrade your phone from ARMv7 to ARMv8
> without re-downloading the apps. You don't carry over your filesystem
> (installed base) on an SD card from the old phone to the new one. That's
> where Google has some control I didn't say that Google should break
> those apps but actively try to update them (in the meantime, they could
> provide some form of SWP emulation).

Look at Colin's original email. There are apps in the app store that
work fine now, but have seen zero updates in the last 2 years. It is
very unlikely that they will be upgraded. They will break on a v8 phone.
This is a software ecosystem issue. Yes of course Google should be
encouraging those apps to be updated, but the reality is there will be a
non-trivial percentage of them which never will be. Breaking those
applications is hostile to the platform.

I'm concerned that this whole debate shows contempt for any kind of
platform stability. For the first time *ever* (and, arguably, despite
our best efforts) we have a huge ISV ecosystem for Linux. Even more
amazingly, it's built around ARM, not x86! Woo! For years we hoped this
would happen. So then we choose to dump all over it by deciding old binaries
don't matter? For shame.

> > So, no. I completely reject any notion that breaking existing apps is
> > okay. If we're going to say that v8 still supports 32-bit apps, then
> > it has to be all of v7, not just the 'good' bits. Nor do I think
> > saying "it's just a bunch of games" justifies anything. We're kernel
> > engineers. Applications are applications and we don't break userspace.
> > Period.
> 
> Here you need to define user-space. OABI?

This is where we can be pragmatic. Listen to our users. Colin was very
clear about what some apps need in order to keep working. He's not even
asking for old kernels or system libraries to be supported because he
has control over those components and they can be fixed. The important
ABI is the ABI used by Android applications.

We can drop ABIs that nobody uses. If nobody complains, is it really
broken?

> > > Note that I don't say Google should break those applications but they
> > > can carry a patch in their Android kernel while reaching out to
> > > developers to sort their code (can the Android app store be scanned?).
> > > What I don't want is to be in a situation 10 years from now when we
> > > still carry SWP emulation code that no-one uses but we can't remove
> > > because it would break the user space features we agreed upon.
> > 
> > Welcome to system programming. This is what we do. It is a *good*
> > thing that an x86 userspace from 1995 can still be booted.
> 
> I'm not sure how much of it is just the merit of Linux but rather the
> hardware backwards compatibility. As you can see with SWP, we don't
> always have this on ARM (and don't blame the kernel maintainers for this
> ;)).
> 
> The same question again - shall we support OABI or we just add ABI
> features based on who shouts louder?

Yes. Listen to the people who shout. When they shout acknowledge it as a
bug and fix it. I'll defer to rmk on the OABI question. I don't think there
is a OABI ecosystem that needs to be carried over to v8, but I'm not
informed enough to take a position.

> > > I have limited knowledge of Android user space but I think SWP emulation
> > > could also be implemented in user space via a SIGILL handler in the
> > > zygote thread and inherited by forked apps (performance doesn't really
> > > matter here). A similar example for Android is the binder driver
> > > user-kernel ABI. AFAIK, Google decided not to provide a compat ABI for
> > > the 64-bit compilation of this driver but update the AArch32 user-space
> > > library to use the new 64-bit ABI. That's perfectly fine by me, they
> > > chose not to provide such ABI in the kernel but solve it entirely in
> > > user space and could do the same with SWP.
> > 
> > Really? Why would we even want that? We're far better positioned in
> > the kernel to present the correct behaviour that any trapping from a
> > userspace application.
> 
> Because by adding it to the kernel we declare it ABI (rather than just
> an Android issue on ARMv8 hardware; I'm not currently aware of other
> ARMv7 Linux distros affected).

It's already an ABI. That's the point. Cat, bag, out.

Arguing that other distros don't have the problem is bogus because it
presumes all distros have an ISV ecosystem. Despite trying, none of them
have been able to achieve that yet.

g.

Android and compatibility with deprecated armv7 instructions

[Mark Brown]

On Fri, Jul 04, 2014 at 02:22:50PM +0100, Grant Likely wrote:
> On Thu, 3 Jul 2014 18:43:18 +0100, Catalin Marinas <catalin.marinas@arm.com> wrote:
> > On Thu, Jul 03, 2014 at 05:22:30PM +0100, Grant Likely wrote:

> > > Welcome to system programming. This is what we do. It is a *good*
> > > thing that an x86 userspace from 1995 can still be booted.

> > I'm not sure how much of it is just the merit of Linux but rather the
> > hardware backwards compatibility. As you can see with SWP, we don't
> > always have this on ARM (and don't blame the kernel maintainers for this
> > ;)).

> > The same question again - shall we support OABI or we just add ABI
> > features based on who shouts louder?

> Yes. Listen to the people who shout. When they shout acknowledge it as a
> bug and fix it. I'll defer to rmk on the OABI question. I don't think there

Often it's also going to be reasonable to ask those complaining to
contribute code towards fixing it (or find someone who can).

> > > > library to use the new 64-bit ABI. That's perfectly fine by me, they
> > > > chose not to provide such ABI in the kernel but solve it entirely in
> > > > user space and could do the same with SWP.

> > > Really? Why would we even want that? We're far better positioned in
> > > the kernel to present the correct behaviour that any trapping from a
> > > userspace application.

> > Because by adding it to the kernel we declare it ABI (rather than just
> > an Android issue on ARMv8 hardware; I'm not currently aware of other
> > ARMv7 Linux distros affected).

> It's already an ABI. That's the point. Cat, bag, out.

> Arguing that other distros don't have the problem is bogus because it
> presumes all distros have an ISV ecosystem. Despite trying, none of them
> have been able to achieve that yet.

With distro applications it's not just ISVs you need to worry about,
it's also locally built software.  Very few people run only distro
software so either they're buying things in or they're building and
developing software themselves.  Of course in an ideal world locally
built software will be easy to rebuild and redeploy but the world isn't
always ideal.

It's probably also worth noting that the original discussion of this
that Colin referenced was started by Ming Lei from Canonical referencing
a closed application presumably running on Ubuntu so there's *some*
binary only ISV market for ARM outside of Android that's affected.

Another question I think it's worth asking is what the use case for 32
bit mode is other than preexisting binaries - I guess there's a case for
smaller binaries giving better cache utilisation?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20140704/65e009f5/attachment.sig>

Android and compatibility with deprecated armv7 instructions

[Arnd Bergmann]

On Friday 04 July 2014 20:24:05 Mark Brown wrote:
> 
> Another question I think it's worth asking is what the use case for 32
> bit mode is other than preexisting binaries - I guess there's a case for
> smaller binaries giving better cache utilisation?

If you wanted that, you'd probably run x32-mode (or whatever it gets called
on ARM64), which I'd expect to also be more efficient ;-)

Seriously, backwards-compatibility is the only reason I can think of
really, but that doesn't /have/ to mean existing binaries; there is a large
amount of source code that can readily be compiled to any CPU as long as
it's little-endian 32-bit.

	Arnd

Android and compatibility with deprecated armv7 instructions

[Måns Rullgård]

Arnd Bergmann <arnd@arndb.de> writes:

> On Friday 04 July 2014 20:24:05 Mark Brown wrote:
>> 
>> Another question I think it's worth asking is what the use case for 32
>> bit mode is other than preexisting binaries - I guess there's a case for
>> smaller binaries giving better cache utilisation?
>
> If you wanted that, you'd probably run x32-mode (or whatever it gets called
> on ARM64), which I'd expect to also be more efficient ;-)
>
> Seriously, backwards-compatibility is the only reason I can think of
> really, but that doesn't /have/ to mean existing binaries; there is a large
> amount of source code that can readily be compiled to any CPU as long as
> it's little-endian 32-bit.

There's also a lot of code that has been heavily optimised for 32-bit
ARM but doesn't (yet) have the same level of optimisation for ARM64, so
even if it builds and works for ARM64, it won't be nearly as fast.

-- 
M?ns Rullg?rd
mans at mansr.com

Android and compatibility with deprecated armv7 instructions

[Mark Brown]

On Fri, Jul 04, 2014 at 09:33:24PM +0200, Arnd Bergmann wrote:
> On Friday 04 July 2014 20:24:05 Mark Brown wrote:

> > Another question I think it's worth asking is what the use case for 32
> > bit mode is other than preexisting binaries - I guess there's a case for
> > smaller binaries giving better cache utilisation?

> If you wanted that, you'd probably run x32-mode (or whatever it gets called
> on ARM64), which I'd expect to also be more efficient ;-)

Well, indeed.

> Seriously, backwards-compatibility is the only reason I can think of
> really, but that doesn't /have/ to mean existing binaries; there is a large
> amount of source code that can readily be compiled to any CPU as long as
> it's little-endian 32-bit.

Sure, and even some more that only runs if it's big endian 32 bit.
Definitely relatively esoteric compared to Android though, at least for
ARM for the time being.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20140704/661eb3b1/attachment.sig>

Android and compatibility with deprecated armv7 instructions

[Catalin Marinas]

On Fri, Jul 04, 2014 at 02:22:50PM +0100, Grant Likely wrote:
> On Thu, 3 Jul 2014 18:43:18 +0100, Catalin Marinas <catalin.marinas@arm.com> wrote:
> > On Thu, Jul 03, 2014 at 05:22:30PM +0100, Grant Likely wrote:
> > > So, no. I completely reject any notion that breaking existing apps is
> > > okay. If we're going to say that v8 still supports 32-bit apps, then
> > > it has to be all of v7, not just the 'good' bits. Nor do I think
> > > saying "it's just a bunch of games" justifies anything. We're kernel
> > > engineers. Applications are applications and we don't break userspace.
> > > Period.
> > 
> > Here you need to define user-space. OABI?
> 
> This is where we can be pragmatic. Listen to our users. Colin was very
> clear about what some apps need in order to keep working. He's not even
> asking for old kernels or system libraries to be supported because he
> has control over those components and they can be fixed. The important
> ABI is the ABI used by Android applications.
> 
> We can drop ABIs that nobody uses. If nobody complains, is it really
> broken?

OK, so I think we now agree that being asked not to break the ABI ever
is not feasible (and not in the community's interest either; think about
carrying over NWFPE for ARMv8 just because someone wants to run an old
distro when there are alternatives already, such requests would need
proper justification).

Another aspect is that users don't always know what's good (or bad) for
them. EABI provided means via kuser helpers to use arch-independent
barrier code and cmpxchg. Glibc and various threads libraries moved to
using them but as we can see, not all user space. As Russell pointed
out, we failed to raise the SWP issue properly in the ARMv7 kernel (more
warnings, always emulation) and I see this as a valid reason to allow
for a _transition_ period in ARMv8.

But we have to agree on a way to handle deprecated/removed architecture
features (as I said, some of them are removed for good reasons). And
that's also for user's benefit, performance wise they should rather use
native than emulated features.

The timeline I propose would be:

1. Architecture feature deprecation (still present, no way to disable)
   - In Linux we need to find ways and push for alternatives to be
     adopted by user space (like we did with kuser helpers)
2. Architecture feature disabling (still present but needs to be enabled
   explicitly via SCTLR bit)
   - Linux disables the feature by default and provides emulation
     (enabled by default), clear warnings
   - In certain cases, there may be justification to enable the hardware
     feature but certainly not in defconfig nor distro Image builds.
     Otherwise they can't complain that users are still using such
     feature at point 3 below
3. Architecture feature removed
   - Default Linux behaviour changes to SIGILL
   - Emulation could be still available in the kernel but maybe under
     a CONFIG_EXPERT option
4. Eventually remove kernel emulation support entirely (or don't carry
   it over to a new port, though I don't foresee any arm128 ;))

With SWP, we are currently at 3/4 with the counter arguments that 2
wasn't entirely clear to users, so the transition needs to be
carried over for ARMv8 (but I'm against 2 for arm64, 3 is as far as I
would go). Arguably, with CP15 barriers we are at point 2, in which case
if we add them to the arm64 kernel, they should be emulated with clear
warnings (ratelimited).

-- 
Catalin

Android and compatibility with deprecated armv7 instructions

[Russell King - ARM Linux]

On Sat, Jul 05, 2014 at 12:14:07PM +0100, Catalin Marinas wrote:
> The timeline I propose would be:
> 
> 1. Architecture feature deprecation (still present, no way to disable)
>    - In Linux we need to find ways and push for alternatives to be
>      adopted by user space (like we did with kuser helpers)
> 2. Architecture feature disabling (still present but needs to be enabled
>    explicitly via SCTLR bit)
>    - Linux disables the feature by default and provides emulation
>      (enabled by default), clear warnings
>    - In certain cases, there may be justification to enable the hardware
>      feature but certainly not in defconfig nor distro Image builds.
>      Otherwise they can't complain that users are still using such
>      feature at point 3 below
> 3. Architecture feature removed
>    - Default Linux behaviour changes to SIGILL
>    - Emulation could be still available in the kernel but maybe under
>      a CONFIG_EXPERT option
> 4. Eventually remove kernel emulation support entirely (or don't carry
>    it over to a new port, though I don't foresee any arm128 ;))
> 
> With SWP, we are currently at 3/4 with the counter arguments that 2
> wasn't entirely clear to users, so the transition needs to be
> carried over for ARMv8 (but I'm against 2 for arm64, 3 is as far as I
> would go).

That's reasonable - the patches I sent out turn off hardware SWP
where possible when SMP is enabled, switching it to emulation mode.
Plus we disable HWCAP_SWP where the exclusives are available,
whether or not the hardware has support for SWP.  This includes
ARMv6* (which needs careful review.)

However, don't forget that even when we disable HWCAP_SWP, that's of
limited use to userspace because we have soo many kernels which don't.

We still don't log SWP usage - they're still at debug level (which
means compiled out in standard kernel builds), neither do we log the
location of SWPs in the user binary (like we do with alignment faults.)
Both of these still need addressing.

-- 
FTTC broadband for 0.8mile line: now at 9.7Mbps down 460kbps up... slowly
improving, and getting towards what was expected from it.

Android and compatibility with deprecated armv7 instructions

[Mark Brown]

On Sat, Jul 05, 2014 at 12:14:07PM +0100, Catalin Marinas wrote:

> The timeline I propose would be:
> 
> 1. Architecture feature deprecation (still present, no way to disable)
>    - In Linux we need to find ways and push for alternatives to be
>      adopted by user space (like we did with kuser helpers)

Is it worth adding a non-default option to either disable and emulate or
just completely disable during this phase rather than at step 2 (perhaps
with a sysctl)?  That would help people who want to test that what they
are doing is going to work going forwards.  The work is going to have to
happen anyway.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20140705/ea9f54d8/attachment.sig>

Android and compatibility with deprecated armv7 instructions

[Catalin Marinas]

On 5 Jul 2014, at 17:43, Mark Brown <broonie@kernel.org> wrote:
> On Sat, Jul 05, 2014 at 12:14:07PM +0100, Catalin Marinas wrote:
> 
>> The timeline I propose would be:
>> 
>> 1. Architecture feature deprecation (still present, no way to disable)
>>   - In Linux we need to find ways and push for alternatives to be
>>     adopted by user space (like we did with kuser helpers)
> 
> Is it worth adding a non-default option to either disable and emulate or
> just completely disable during this phase rather than at step 2 (perhaps
> with a sysctl)?  That would help people who want to test that what they
> are doing is going to work going forwards.  The work is going to have to
> happen anyway.

The problem is that when the feature is deprecated, we may not always
get the option to disable it (it?s more of an advance warning to think
about it, that?s why I added emulation at step 2). For example SWP has
been deprecated in ARMv6 but no way to disable it until ARMv7 MP. If we
get the option (and we should probably recommend this to the
architects), I agree, you can start the emulation work at step 1 but
defaulting to native deprecated feature rather than emulation.

Catalin

Android and compatibility with deprecated armv7 instructions

[Arnd Bergmann]

On Saturday 05 July 2014 18:06:29 Catalin Marinas wrote:
> On 5 Jul 2014, at 17:43, Mark Brown <broonie@kernel.org> wrote:
> > On Sat, Jul 05, 2014 at 12:14:07PM +0100, Catalin Marinas wrote:
> > 
> >> The timeline I propose would be:
> >> 
> >> 1. Architecture feature deprecation (still present, no way to disable)
> >>   - In Linux we need to find ways and push for alternatives to be
> >>     adopted by user space (like we did with kuser helpers)
> > 
> > Is it worth adding a non-default option to either disable and emulate or
> > just completely disable during this phase rather than at step 2 (perhaps
> > with a sysctl)?  That would help people who want to test that what they
> > are doing is going to work going forwards.  The work is going to have to
> > happen anyway.
> 
> The problem is that when the feature is deprecated, we may not always
> get the option to disable it (it?s more of an advance warning to think
> about it, that?s why I added emulation at step 2). For example SWP has
> been deprecated in ARMv6 but no way to disable it until ARMv7 MP. If we
> get the option (and we should probably recommend this to the
> architects), I agree, you can start the emulation work at step 1 but
> defaulting to native deprecated feature rather than emulation.

I think if a there is no way to turn off a feature, we can't really
treat it as "deprecated", since we are lacking the most important tool
to help users migrate away from it. Do you know why the architecture
folks believe it makes sense to have distinct steps 1 and 2?

Another problem that I see with the way that features are phased out
on ARM is how the hypervisor architecture makes it really hard to
run a guest in an older architecture version. For instance on s390,
you can basically emulate any prior machine from the past 50 years
by selectively trapping some of the instructions from the guest
into the hypervisor, at least for any instruction that has had
a different behavior in the past.

	Arnd

Android and compatibility with deprecated armv7 instructions

[Catalin Marinas]

On 5 Jul 2014, at 19:43, Arnd Bergmann <arnd@arndb.de> wrote:
> On Saturday 05 July 2014 18:06:29 Catalin Marinas wrote:
>> On 5 Jul 2014, at 17:43, Mark Brown <broonie@kernel.org> wrote:
>>> On Sat, Jul 05, 2014 at 12:14:07PM +0100, Catalin Marinas wrote:
>>> 
>>>> The timeline I propose would be:
>>>> 
>>>> 1. Architecture feature deprecation (still present, no way to disable)
>>>>  - In Linux we need to find ways and push for alternatives to be
>>>>    adopted by user space (like we did with kuser helpers)
>>> 
>>> Is it worth adding a non-default option to either disable and emulate or
>>> just completely disable during this phase rather than at step 2 (perhaps
>>> with a sysctl)?  That would help people who want to test that what they
>>> are doing is going to work going forwards.  The work is going to have to
>>> happen anyway.
>> 
>> The problem is that when the feature is deprecated, we may not always
>> get the option to disable it (it?s more of an advance warning to think
>> about it, that?s why I added emulation at step 2). For example SWP has
>> been deprecated in ARMv6 but no way to disable it until ARMv7 MP. If we
>> get the option (and we should probably recommend this to the
>> architects), I agree, you can start the emulation work at step 1 but
>> defaulting to native deprecated feature rather than emulation.
> 
> I think if a there is no way to turn off a feature, we can't really
> treat it as "deprecated", since we are lacking the most important tool
> to help users migrate away from it. Do you know why the architecture
> folks believe it makes sense to have distinct steps 1 and 2?

During this phase you usually only get compiler warnings which are
clearly not enough for already built apps.

But it looks like this is changing with ARMv8. SETEND is deprecated and
disable bit available. IT instruction also deprecated with a disable bit
(basically only allowing for one subsequent conditional instruction,
though I don?t think we can easily get rid of this one in user space).

> Another problem that I see with the way that features are phased out
> on ARM is how the hypervisor architecture makes it really hard to
> run a guest in an older architecture version. For instance on s390,
> you can basically emulate any prior machine from the past 50 years
> by selectively trapping some of the instructions from the guest
> into the hypervisor, at least for any instruction that has had
> a different behavior in the past.

That?s indeed not possible (basically a SWP at EL1 would trap as undef
at EL1 rather than EL2). But is there much value in this? Do we have a
large base of pre-built OS kernels? We are trying hard to get to single
Image, let alone using old builds of a kernel.

Catalin

Android and compatibility with deprecated armv7 instructions

[Arnd Bergmann]

On Saturday 05 July 2014, Catalin Marinas wrote:
> On 5 Jul 2014, at 19:43, Arnd Bergmann <arnd@arndb.de> wrote:
>
> > I think if a there is no way to turn off a feature, we can't really
> > treat it as "deprecated", since we are lacking the most important tool
> > to help users migrate away from it. Do you know why the architecture
> > folks believe it makes sense to have distinct steps 1 and 2?
> 
> During this phase you usually only get compiler warnings which are
> clearly not enough for already built apps.
> 
> But it looks like this is changing with ARMv8. SETEND is deprecated and
> disable bit available. IT instruction also deprecated with a disable bit
> (basically only allowing for one subsequent conditional instruction,
> though I don?t think we can easily get rid of this one in user space).

Ok. I can also see ways to emulate SETEND from kernel side by modifying
the user pt_regs, but I don't see how we could emulate IT without anything
short of a full interpretation of user instructions from the kernel.

> > Another problem that I see with the way that features are phased out
> > on ARM is how the hypervisor architecture makes it really hard to
> > run a guest in an older architecture version. For instance on s390,
> > you can basically emulate any prior machine from the past 50 years
> > by selectively trapping some of the instructions from the guest
> > into the hypervisor, at least for any instruction that has had
> > a different behavior in the past.
> 
> That?s indeed not possible (basically a SWP at EL1 would trap as undef
> at EL1 rather than EL2). But is there much value in this? Do we have a
> large base of pre-built OS kernels? We are trying hard to get to single
> Image, let alone using old builds of a kernel.

There are countless reasons why you'd want this actually, including:

- running an old arm7tdmi rtos build that you lost the source code for
  but that would be cheaper to run on a new cortex-a7 emulating the
  peripherals than to rewrite and revalidate

- running OABI binaries in a 32-bit guest on an armv8 (or future version)

- testing armv4 kernel builds in a kvm guest using qemu models

- running Windows CE binaries in a virtual machine

- running an ARMv8 SBSA based OS on ARMv9 hardware

- ...

The single image approach can by definition only address the hardware
we already know, while one of the main uses of virtualization is to
run existing software on hardware that was unknown at the time the
binaries were compiled.

	Arnd

Android and compatibility with deprecated armv7 instructions

[Janne Grunau]

Arnd Bergmann <arnd <at> arndb.de> writes:
> 
> On Saturday 05 July 2014, Catalin Marinas wrote:
> > On 5 Jul 2014, at 19:43, Arnd Bergmann <arnd <at> arndb.de> wrote:
> >
> > > I think if a there is no way to turn off a feature, we can't really
> > > treat it as "deprecated", since we are lacking the most important tool
> > > to help users migrate away from it. Do you know why the architecture
> > > folks believe it makes sense to have distinct steps 1 and 2?
> > 
> > During this phase you usually only get compiler warnings which are
> > clearly not enough for already built apps.
> > 
> > But it looks like this is changing with ARMv8. SETEND is deprecated and
> > disable bit available. IT instruction also deprecated with a disable bit
> > (basically only allowing for one subsequent conditional instruction,
> > though I don?t think we can easily get rid of this one in user space).
> 
> Ok. I can also see ways to emulate SETEND from kernel side by modifying
> the user pt_regs,

The single use of SETEND in FFmpeg/libav is not important and could easily
be disabled on ARMv8. Is there a better way than parsing AT_PLATFORM for
runtime detection of the CPU arch? A HWCAP for the missing SETEND would be
nicer. That of course doesn't fix existing binaries.

> but I don't see how we could emulate IT without anything
> short of a full interpretation of user instructions from the kernel.

Avoiding deprecated forms of IT would be harder. It's used more often in
libav (~50 times) so it should be only disabled if it's emulated or not
available using an explicit HWCAP. Not using Thumb for those functions would
be another option for libav. Again this doesn't fix existing binaries and it
doesn't fix compiled code.

Janne

-- 
libav arm/aarch64 developer

Android and compatibility with deprecated armv7 instructions

[Catalin Marinas]

On Mon, Jul 07, 2014 at 01:59:54PM +0000, Janne Grunau wrote:
> Arnd Bergmann <arnd <at> arndb.de> writes:
> > but I don't see how we could emulate IT without anything
> > short of a full interpretation of user instructions from the kernel.
> 
> Avoiding deprecated forms of IT would be harder. It's used more often in
> libav (~50 times)

Is this explicitly written IT instructions or in generated code (gas,
gcc)?

> so it should be only disabled if it's emulated or not
> available using an explicit HWCAP. Not using Thumb for those functions would
> be another option for libav. Again this doesn't fix existing binaries and it
> doesn't fix compiled code.

As I said, I don't see this going away (easily). I've heard the
toolchain guys adding warnings but they don't help unless you rebuild
your code. Given the amount of IT instructions, I don't even propose
adding emulation/warnings to the kernel for the time being (that would
be step 1 in my "instruction deprecation" proposal, we have to think
about it; some trapping would be handy to to assess the impact but
still defaulting to native).

-- 
Catalin

Android and compatibility with deprecated armv7 instructions

[Janne Grunau]

On 2014-07-07 15:52:08 +0100, Catalin Marinas wrote:
> On Mon, Jul 07, 2014 at 01:59:54PM +0000, Janne Grunau wrote:
> > Arnd Bergmann <arnd <at> arndb.de> writes:
> > > but I don't see how we could emulate IT without anything
> > > short of a full interpretation of user instructions from the kernel.
> > 
> > Avoiding deprecated forms of IT would be harder. It's used more often in
> > libav (~50 times)
> 
> Is this explicitly written IT instructions or in generated code (gas,
> gcc)?

Those 50 times are explicitly written IT instructions with more than one 
conditional instruction in assembly. At least on linux none of the 
functions should be used without runtime cpu/feature detection.

gcc 4.8.3 itself generates tons of IT with more than one conditional 
instruction (~1500 for a 90k SLOC C library).

> > so it should be only disabled if it's emulated or not
> > available using an explicit HWCAP. Not using Thumb for those functions would
> > be another option for libav. Again this doesn't fix existing binaries and it
> > doesn't fix compiled code.
> 
> As I said, I don't see this going away (easily). I've heard the
> toolchain guys adding warnings but they don't help unless you rebuild
> your code. Given the amount of IT instructions, I don't even propose
> adding emulation/warnings to the kernel for the time being (that would
> be step 1 in my "instruction deprecation" proposal, we have to think
> about it; some trapping would be handy to to assess the impact but
> still defaulting to native).

I guess that would just detect thumb2 binaries not compiled with 
'-mrestrict-it' i.e. all legacy thumb2 binaries.

Janne

Android and compatibility with deprecated armv7 instructions

[Peter Maydell]

On 7 July 2014 14:59, Janne Grunau <j@jannau.net> wrote:
> Avoiding deprecated forms of IT would be harder. It's used more often in
> libav (~50 times) so it should be only disabled if it's emulated or not
> available using an explicit HWCAP. Not using Thumb for those functions would
> be another option for libav.

It would probably be worth benchmarking an implementation which
avoided the deprecated IT forms on current v7/v8 hardware against
the current code; these things often get deprecated because they're
hard to implement efficiently, so it's entirely possible you'd find that
they're better avoided even on some cores where they're still present
(either now or in future).

thanks
-- PMM

Android and compatibility with deprecated armv7 instructions

[Måns Rullgård]

Janne Grunau <j@jannau.net> writes:

> Arnd Bergmann <arnd <at> arndb.de> writes:
>> 
>> On Saturday 05 July 2014, Catalin Marinas wrote:
>> > On 5 Jul 2014, at 19:43, Arnd Bergmann <arnd <at> arndb.de> wrote:
>> >
>> > > I think if a there is no way to turn off a feature, we can't really
>> > > treat it as "deprecated", since we are lacking the most important tool
>> > > to help users migrate away from it. Do you know why the architecture
>> > > folks believe it makes sense to have distinct steps 1 and 2?
>> > 
>> > During this phase you usually only get compiler warnings which are
>> > clearly not enough for already built apps.
>> > 
>> > But it looks like this is changing with ARMv8. SETEND is deprecated and
>> > disable bit available. IT instruction also deprecated with a disable bit
>> > (basically only allowing for one subsequent conditional instruction,
>> > though I don?t think we can easily get rid of this one in user space).
>> 
>> Ok. I can also see ways to emulate SETEND from kernel side by modifying
>> the user pt_regs,
>
> The single use of SETEND in FFmpeg/libav is not important and could easily
> be disabled on ARMv8. Is there a better way than parsing AT_PLATFORM for
> runtime detection of the CPU arch? A HWCAP for the missing SETEND would be
> nicer. That of course doesn't fix existing binaries.

SETEND is serialising on some v7 cores as well, so it really shouldn't
be used at all.

>> but I don't see how we could emulate IT without anything
>> short of a full interpretation of user instructions from the kernel.
>
> Avoiding deprecated forms of IT would be harder. It's used more often in
> libav (~50 times) so it should be only disabled if it's emulated or not
> available using an explicit HWCAP. Not using Thumb for those functions would
> be another option for libav. Again this doesn't fix existing binaries and it
> doesn't fix compiled code.

IT covering more than one conditional instruction is trivial to fix by
either a) adding IT before each instruction, b) replacing it with a
branch, or in some cases c) making all but the last instruction
conditional because the rest don't actually need to be.  Long runs of
conditional instructions have been frowned upon for quite some time
anyway.

-- 
M?ns Rullg?rd
mans at mansr.com

Android and compatibility with deprecated armv7 instructions

[Catalin Marinas]

On Sun, Jul 06, 2014 at 04:39:21PM +0100, Arnd Bergmann wrote:
> On Saturday 05 July 2014, Catalin Marinas wrote:
> > On 5 Jul 2014, at 19:43, Arnd Bergmann <arnd@arndb.de> wrote:
> >
> > > I think if a there is no way to turn off a feature, we can't really
> > > treat it as "deprecated", since we are lacking the most important tool
> > > to help users migrate away from it. Do you know why the architecture
> > > folks believe it makes sense to have distinct steps 1 and 2?
> > 
> > During this phase you usually only get compiler warnings which are
> > clearly not enough for already built apps.
> > 
> > But it looks like this is changing with ARMv8. SETEND is deprecated and
> > disable bit available. IT instruction also deprecated with a disable bit
> > (basically only allowing for one subsequent conditional instruction,
> > though I don?t think we can easily get rid of this one in user space).
> 
> Ok. I can also see ways to emulate SETEND from kernel side by modifying
> the user pt_regs, but I don't see how we could emulate IT without anything
> short of a full interpretation of user instructions from the kernel.

Single-step ;).

I need to get clarification but hopefully even if IT undefs, the SPSR IT
bits are still taken into account. So the emulation/warning handler
needs to decode IT and set SPSR in pt_regs accordingly before returning
to the next instruction.

But I agree, we can't remove this instruction entirely from the
architecture until we are completely sure there are no users (which is
nearly impossible).

> > > Another problem that I see with the way that features are phased out
> > > on ARM is how the hypervisor architecture makes it really hard to
> > > run a guest in an older architecture version. For instance on s390,
> > > you can basically emulate any prior machine from the past 50 years
> > > by selectively trapping some of the instructions from the guest
> > > into the hypervisor, at least for any instruction that has had
> > > a different behavior in the past.
> > 
> > That?s indeed not possible (basically a SWP at EL1 would trap as undef
> > at EL1 rather than EL2). But is there much value in this? Do we have a
> > large base of pre-built OS kernels? We are trying hard to get to single
> > Image, let alone using old builds of a kernel.
> 
> There are countless reasons why you'd want this actually, including:
> 
> - running an old arm7tdmi rtos build that you lost the source code for
>   but that would be cheaper to run on a new cortex-a7 emulating the
>   peripherals than to rewrite and revalidate

That's a too rare case to justify the additional CPU gates.

> - running OABI binaries in a 32-bit guest on an armv8 (or future version)

But you can already run an ARMv7 kernel now with OABI enabled on an
ARMv8 (either native or guest).

> - testing armv4 kernel builds in a kvm guest using qemu models

Again, this would require pre-ARMv6 MMU to be carried over for little
benefit (well, just to developers but hard to justify ARMv4 hardware
compatibility to marketing).

> - running Windows CE binaries in a virtual machine

It could be but a certain company never mentioned it.

> - running an ARMv8 SBSA based OS on ARMv9 hardware

I think back to ARMv7 we kind of have compatibility. What's missing is
some obsolete/undefined instructions to be trapped at EL2. This could
probably be considered if ARM decides to deprecated new instructions in
the future (though I think the current non-deprecated features are
stable enough).

-- 
Catalin

Android and compatibility with deprecated armv7 instructions

[Arnd Bergmann]

On Monday 07 July 2014, Catalin Marinas wrote:
> > - running an old arm7tdmi rtos build that you lost the source code for
> >   but that would be cheaper to run on a new cortex-a7 emulating the
> >   peripherals than to rewrite and revalidate
> 
> That's a too rare case to justify the additional CPU gates.

For trapping just a few extra instructions? That doesn't sound too
expensive.

> > - running OABI binaries in a 32-bit guest on an armv8 (or future version)
> 
> But you can already run an ARMv7 kernel now with OABI enabled on an
> ARMv8 (either native or guest).
> 
> > - testing armv4 kernel builds in a kvm guest using qemu models
> 
> Again, this would require pre-ARMv6 MMU to be carried over for little
> benefit (well, just to developers but hard to justify ARMv4 hardware
> compatibility to marketing).

I don't think anybody was talking about emulating the MMU in hardware.
It doesn't have to be fast, so you'd just do shadow page tables.



	Arnd

Android and compatibility with deprecated armv7 instructions

[Grant Likely]

On Sat, 5 Jul 2014 12:14:07 +0100, Catalin Marinas <catalin.marinas@arm.com> wrote:
> On Fri, Jul 04, 2014 at 02:22:50PM +0100, Grant Likely wrote:
> > On Thu, 3 Jul 2014 18:43:18 +0100, Catalin Marinas <catalin.marinas@arm.com> wrote:
> > > On Thu, Jul 03, 2014 at 05:22:30PM +0100, Grant Likely wrote:
> > > > So, no. I completely reject any notion that breaking existing apps is
> > > > okay. If we're going to say that v8 still supports 32-bit apps, then
> > > > it has to be all of v7, not just the 'good' bits. Nor do I think
> > > > saying "it's just a bunch of games" justifies anything. We're kernel
> > > > engineers. Applications are applications and we don't break userspace.
> > > > Period.
> > > 
> > > Here you need to define user-space. OABI?
> > 
> > This is where we can be pragmatic. Listen to our users. Colin was very
> > clear about what some apps need in order to keep working. He's not even
> > asking for old kernels or system libraries to be supported because he
> > has control over those components and they can be fixed. The important
> > ABI is the ABI used by Android applications.
> > 
> > We can drop ABIs that nobody uses. If nobody complains, is it really
> > broken?
> 
> OK, so I think we now agree that being asked not to break the ABI ever
> is not feasible (and not in the community's interest either; think about
> carrying over NWFPE for ARMv8 just because someone wants to run an old
> distro when there are alternatives already, such requests would need
> proper justification).
> 
> Another aspect is that users don't always know what's good (or bad) for
> them. EABI provided means via kuser helpers to use arch-independent
> barrier code and cmpxchg. Glibc and various threads libraries moved to
> using them but as we can see, not all user space. As Russell pointed
> out, we failed to raise the SWP issue properly in the ARMv7 kernel (more
> warnings, always emulation) and I see this as a valid reason to allow
> for a _transition_ period in ARMv8.
> 
> But we have to agree on a way to handle deprecated/removed architecture
> features (as I said, some of them are removed for good reasons). And
> that's also for user's benefit, performance wise they should rather use
> native than emulated features.
> 
> The timeline I propose would be:
> 
> 1. Architecture feature deprecation (still present, no way to disable)
>    - In Linux we need to find ways and push for alternatives to be
>      adopted by user space (like we did with kuser helpers)
> 2. Architecture feature disabling (still present but needs to be enabled
>    explicitly via SCTLR bit)
>    - Linux disables the feature by default and provides emulation
>      (enabled by default), clear warnings
>    - In certain cases, there may be justification to enable the hardware
>      feature but certainly not in defconfig nor distro Image builds.
>      Otherwise they can't complain that users are still using such
>      feature at point 3 below
> 3. Architecture feature removed
>    - Default Linux behaviour changes to SIGILL
>    - Emulation could be still available in the kernel but maybe under
>      a CONFIG_EXPERT option
> 4. Eventually remove kernel emulation support entirely (or don't carry
>    it over to a new port, though I don't foresee any arm128 ;))
> 
> With SWP, we are currently at 3/4 with the counter arguments that 2
> wasn't entirely clear to users, so the transition needs to be
> carried over for ARMv8 (but I'm against 2 for arm64, 3 is as far as I
> would go). Arguably, with CP15 barriers we are at point 2, in which case
> if we add them to the arm64 kernel, they should be emulated with clear
> warnings (ratelimited).

I could quibble on details, but I'm fine with that approach.

It would also be good to have ongoing feedback about if the emulation is
getting used in the field. Colin, if the kernel emits warnings on
emulation, is that something that Google gathers statistics on for
devices in the field?

g.

Android and compatibility with deprecated armv7 instructions

[Colin Cross]

On Mon, Jul 7, 2014 at 5:28 AM, Grant Likely <grant.likely@secretlab.ca> wrote:
> On Sat, 5 Jul 2014 12:14:07 +0100, Catalin Marinas <catalin.marinas@arm.com> wrote:
>> On Fri, Jul 04, 2014 at 02:22:50PM +0100, Grant Likely wrote:
>> > On Thu, 3 Jul 2014 18:43:18 +0100, Catalin Marinas <catalin.marinas@arm.com> wrote:
>> > > On Thu, Jul 03, 2014 at 05:22:30PM +0100, Grant Likely wrote:
>> > > > So, no. I completely reject any notion that breaking existing apps is
>> > > > okay. If we're going to say that v8 still supports 32-bit apps, then
>> > > > it has to be all of v7, not just the 'good' bits. Nor do I think
>> > > > saying "it's just a bunch of games" justifies anything. We're kernel
>> > > > engineers. Applications are applications and we don't break userspace.
>> > > > Period.
>> > >
>> > > Here you need to define user-space. OABI?
>> >
>> > This is where we can be pragmatic. Listen to our users. Colin was very
>> > clear about what some apps need in order to keep working. He's not even
>> > asking for old kernels or system libraries to be supported because he
>> > has control over those components and they can be fixed. The important
>> > ABI is the ABI used by Android applications.
>> >
>> > We can drop ABIs that nobody uses. If nobody complains, is it really
>> > broken?
>>
>> OK, so I think we now agree that being asked not to break the ABI ever
>> is not feasible (and not in the community's interest either; think about
>> carrying over NWFPE for ARMv8 just because someone wants to run an old
>> distro when there are alternatives already, such requests would need
>> proper justification).
>>
>> Another aspect is that users don't always know what's good (or bad) for
>> them. EABI provided means via kuser helpers to use arch-independent
>> barrier code and cmpxchg. Glibc and various threads libraries moved to
>> using them but as we can see, not all user space. As Russell pointed
>> out, we failed to raise the SWP issue properly in the ARMv7 kernel (more
>> warnings, always emulation) and I see this as a valid reason to allow
>> for a _transition_ period in ARMv8.
>>
>> But we have to agree on a way to handle deprecated/removed architecture
>> features (as I said, some of them are removed for good reasons). And
>> that's also for user's benefit, performance wise they should rather use
>> native than emulated features.
>>
>> The timeline I propose would be:
>>
>> 1. Architecture feature deprecation (still present, no way to disable)
>>    - In Linux we need to find ways and push for alternatives to be
>>      adopted by user space (like we did with kuser helpers)
>> 2. Architecture feature disabling (still present but needs to be enabled
>>    explicitly via SCTLR bit)
>>    - Linux disables the feature by default and provides emulation
>>      (enabled by default), clear warnings
>>    - In certain cases, there may be justification to enable the hardware
>>      feature but certainly not in defconfig nor distro Image builds.
>>      Otherwise they can't complain that users are still using such
>>      feature at point 3 below
>> 3. Architecture feature removed
>>    - Default Linux behaviour changes to SIGILL
>>    - Emulation could be still available in the kernel but maybe under
>>      a CONFIG_EXPERT option
>> 4. Eventually remove kernel emulation support entirely (or don't carry
>>    it over to a new port, though I don't foresee any arm128 ;))
>>
>> With SWP, we are currently at 3/4 with the counter arguments that 2
>> wasn't entirely clear to users, so the transition needs to be
>> carried over for ARMv8 (but I'm against 2 for arm64, 3 is as far as I
>> would go). Arguably, with CP15 barriers we are at point 2, in which case
>> if we add them to the arm64 kernel, they should be emulated with clear
>> warnings (ratelimited).
>
> I could quibble on details, but I'm fine with that approach.
>
> It would also be good to have ongoing feedback about if the emulation is
> getting used in the field. Colin, if the kernel emits warnings on
> emulation, is that something that Google gathers statistics on for
> devices in the field?

We do capture snapshots of the kernel log on dogfooding and
experimental devices, but it's not a great source for statistical data
because it only gets the most recent 64kB.  We do capture per-process
cpu stats, if we could get it into a
/proc/<pid>/deprecated_instructions file that would provide the most
accurate statistics.  I can also talk to our developer relations teams
about approaching it from the Play Store side.