VFP available on a single CPU in a dual-CPU complex

VFP available on a single CPU in a dual-CPU complex

[Florian Fainelli]

Hello,

On Broadcom's BCM63138 SoC, the second Cortex A9 CPU does not have
VFP, which is a problem as one might imagine because we currently
assume it is available for all CPUs within the complex.

I started to patch vfp_support_entry to test for CPU1 and branch to a
different location, but that raises a bunch of question, in particular
what to do if NEON/VFP came from the kernel, how can we migrate that
execution to CPU0?

If the answer is don't use VFP, I guess that's fine, and I can have a
runtime check in vfp_init() that checks for BCM63138 and set
vfp_vector to vfp_null_entry even though we are running on a multi-v7
kernel with VFP enabled.

Thanks!
-- 
Florian

VFP available on a single CPU in a dual-CPU complex

[Arnd Bergmann]

On Friday 15 August 2014, Florian Fainelli wrote:
> Hello,
> 
> On Broadcom's BCM63138 SoC, the second Cortex A9 CPU does not have
> VFP, which is a problem as one might imagine because we currently
> assume it is available for all CPUs within the complex.

Wow, that is pretty crazy.

> I started to patch vfp_support_entry to test for CPU1 and branch to a
> different location, but that raises a bunch of question, in particular
> what to do if NEON/VFP came from the kernel, how can we migrate that
> execution to CPU0?
>
> If the answer is don't use VFP, I guess that's fine, and I can have a
> runtime check in vfp_init() that checks for BCM63138 and set
> vfp_vector to vfp_null_entry even though we are running on a multi-v7
> kernel with VFP enabled.

I think adding hooks to the scheduler for this case is problematic, so
the easiest way would of course be to give the user a choice between
VFP or SMP on this hardware. For anything beyond that, you could do
a prototype patch (possibly for integration into OpenWRT) so we can
see how ugly it gets to support this setup.

One potentially workable approach may be to just forbid in-kernel
VFP (there is very little of that already, and we might not be
handling it right in a multiplatform kernel, I haven't checked)
if not all CPUs support it. For user space, you could mandate that
user space sets the right affinity mask for any process using VFP.
A general-purpose distro that relies on VFP wouldn't work on this
machine anyway, so we can reasonably assume that there is custom
user space involved if there is any VFP.

	Arnd

VFP available on a single CPU in a dual-CPU complex

[Russell King - ARM Linux]

On Thu, Aug 14, 2014 at 09:45:09PM -0700, Florian Fainelli wrote:
> On Broadcom's BCM63138 SoC, the second Cortex A9 CPU does not have
> VFP, which is a problem as one might imagine because we currently
> assume it is available for all CPUs within the complex.

I really don't want to hear this.  This is not a symmetric system.
It's pretty broken.

Given that modern userspace assumes that VFP will always be available
as hardware instructions, this is not going to be nice.  It's probably
easier to ignore that non-VFP CPU1 and just treat it as a uniprocessor
system.

-- 
FTTC broadband for 0.8mile line: currently at 9.5Mbps down 400kbps up
according to speedtest.net.

VFP available on a single CPU in a dual-CPU complex

[Will Deacon]

On Fri, Aug 15, 2014 at 10:03:01AM +0100, Arnd Bergmann wrote:
> On Friday 15 August 2014, Florian Fainelli wrote:
> > Hello,
> > 
> > On Broadcom's BCM63138 SoC, the second Cortex A9 CPU does not have
> > VFP, which is a problem as one might imagine because we currently
> > assume it is available for all CPUs within the complex.
> 
> Wow, that is pretty crazy.
> 
> > I started to patch vfp_support_entry to test for CPU1 and branch to a
> > different location, but that raises a bunch of question, in particular
> > what to do if NEON/VFP came from the kernel, how can we migrate that
> > execution to CPU0?
> >
> > If the answer is don't use VFP, I guess that's fine, and I can have a
> > runtime check in vfp_init() that checks for BCM63138 and set
> > vfp_vector to vfp_null_entry even though we are running on a multi-v7
> > kernel with VFP enabled.
> 
> I think adding hooks to the scheduler for this case is problematic, so
> the easiest way would of course be to give the user a choice between
> VFP or SMP on this hardware. For anything beyond that, you could do
> a prototype patch (possibly for integration into OpenWRT) so we can
> see how ugly it gets to support this setup.

I agree. We have something similar on the Realview-PBX (dual A9, only one
CPU with NEON) and the answer is not to enable NEON if you want SMP.

Will

VFP available on a single CPU in a dual-CPU complex

[Arnd Bergmann]

On Friday 15 August 2014 10:14:45 Will Deacon wrote:
> On Fri, Aug 15, 2014 at 10:03:01AM +0100, Arnd Bergmann wrote:
> > On Friday 15 August 2014, Florian Fainelli wrote:
> > > Hello,
> > > 
> > > On Broadcom's BCM63138 SoC, the second Cortex A9 CPU does not have
> > > VFP, which is a problem as one might imagine because we currently
> > > assume it is available for all CPUs within the complex.
> > 
> > Wow, that is pretty crazy.
> > 
> > > I started to patch vfp_support_entry to test for CPU1 and branch to a
> > > different location, but that raises a bunch of question, in particular
> > > what to do if NEON/VFP came from the kernel, how can we migrate that
> > > execution to CPU0?
> > >
> > > If the answer is don't use VFP, I guess that's fine, and I can have a
> > > runtime check in vfp_init() that checks for BCM63138 and set
> > > vfp_vector to vfp_null_entry even though we are running on a multi-v7
> > > kernel with VFP enabled.
> > 
> > I think adding hooks to the scheduler for this case is problematic, so
> > the easiest way would of course be to give the user a choice between
> > VFP or SMP on this hardware. For anything beyond that, you could do
> > a prototype patch (possibly for integration into OpenWRT) so we can
> > see how ugly it gets to support this setup.
> 
> I agree. We have something similar on the Realview-PBX (dual A9, only one
> CPU with NEON) and the answer is not to enable NEON if you want SMP.

Does that work with a multiplatform kernel?

I have patches to enable realview for ARCH_MULTIPLATFORM, and I wonder
if that gets in the way. Do we have code to detect this setup at runtime
and disable NEON/VFP if only a subset of the enabled CPUs have it?

	Arnd

VFP available on a single CPU in a dual-CPU complex

[Florian Fainelli]

On 08/15/2014 11:54 AM, Arnd Bergmann wrote:
> On Friday 15 August 2014 10:14:45 Will Deacon wrote:
>> On Fri, Aug 15, 2014 at 10:03:01AM +0100, Arnd Bergmann wrote:
>>> On Friday 15 August 2014, Florian Fainelli wrote:
>>>> Hello,
>>>>
>>>> On Broadcom's BCM63138 SoC, the second Cortex A9 CPU does not have
>>>> VFP, which is a problem as one might imagine because we currently
>>>> assume it is available for all CPUs within the complex.
>>>
>>> Wow, that is pretty crazy.
>>>
>>>> I started to patch vfp_support_entry to test for CPU1 and branch to a
>>>> different location, but that raises a bunch of question, in particular
>>>> what to do if NEON/VFP came from the kernel, how can we migrate that
>>>> execution to CPU0?
>>>>
>>>> If the answer is don't use VFP, I guess that's fine, and I can have a
>>>> runtime check in vfp_init() that checks for BCM63138 and set
>>>> vfp_vector to vfp_null_entry even though we are running on a multi-v7
>>>> kernel with VFP enabled.
>>>
>>> I think adding hooks to the scheduler for this case is problematic, so
>>> the easiest way would of course be to give the user a choice between
>>> VFP or SMP on this hardware. For anything beyond that, you could do
>>> a prototype patch (possibly for integration into OpenWRT) so we can
>>> see how ugly it gets to support this setup.
>>
>> I agree. We have something similar on the Realview-PBX (dual A9, only one
>> CPU with NEON) and the answer is not to enable NEON if you want SMP.
> 
> Does that work with a multiplatform kernel?
> 
> I have patches to enable realview for ARCH_MULTIPLATFORM, and I wonder
> if that gets in the way. Do we have code to detect this setup at runtime
> and disable NEON/VFP if only a subset of the enabled CPUs have it?

I was wondering about that too, here are the two options I see for these
pathological cases:

- if SMP is enabled, and CONFIG_VFP is also enabled, refuse bringing up
cores that do not support VFP/NEON, thus becoming a SMP on UP, and do
that for the platform/SoC-specific SMP support code

- or, in the VFP initialization, completely disable VFP (overriding
VFP_arch in SoC specific code?) because at least one of CPU does not
support it

I would favor enabling SMP over enabling VFP, because that's probably
what most people would expect.
--
Florian

VFP available on a single CPU in a dual-CPU complex

[Måns Rullgård]

Florian Fainelli <f.fainelli@gmail.com> writes:

> On 08/15/2014 11:54 AM, Arnd Bergmann wrote:
>> On Friday 15 August 2014 10:14:45 Will Deacon wrote:
>>> On Fri, Aug 15, 2014 at 10:03:01AM +0100, Arnd Bergmann wrote:
>>>> On Friday 15 August 2014, Florian Fainelli wrote:
>>>>> Hello,
>>>>>
>>>>> On Broadcom's BCM63138 SoC, the second Cortex A9 CPU does not have
>>>>> VFP, which is a problem as one might imagine because we currently
>>>>> assume it is available for all CPUs within the complex.
>>>>
>>>> Wow, that is pretty crazy.
>>>>
>>>>> I started to patch vfp_support_entry to test for CPU1 and branch to a
>>>>> different location, but that raises a bunch of question, in particular
>>>>> what to do if NEON/VFP came from the kernel, how can we migrate that
>>>>> execution to CPU0?
>>>>>
>>>>> If the answer is don't use VFP, I guess that's fine, and I can have a
>>>>> runtime check in vfp_init() that checks for BCM63138 and set
>>>>> vfp_vector to vfp_null_entry even though we are running on a multi-v7
>>>>> kernel with VFP enabled.
>>>>
>>>> I think adding hooks to the scheduler for this case is problematic, so
>>>> the easiest way would of course be to give the user a choice between
>>>> VFP or SMP on this hardware. For anything beyond that, you could do
>>>> a prototype patch (possibly for integration into OpenWRT) so we can
>>>> see how ugly it gets to support this setup.
>>>
>>> I agree. We have something similar on the Realview-PBX (dual A9, only one
>>> CPU with NEON) and the answer is not to enable NEON if you want SMP.
>> 
>> Does that work with a multiplatform kernel?
>> 
>> I have patches to enable realview for ARCH_MULTIPLATFORM, and I wonder
>> if that gets in the way. Do we have code to detect this setup at runtime
>> and disable NEON/VFP if only a subset of the enabled CPUs have it?
>
> I was wondering about that too, here are the two options I see for these
> pathological cases:
>
> - if SMP is enabled, and CONFIG_VFP is also enabled, refuse bringing up
> cores that do not support VFP/NEON, thus becoming a SMP on UP, and do
> that for the platform/SoC-specific SMP support code
>
> - or, in the VFP initialization, completely disable VFP (overriding
> VFP_arch in SoC specific code?) because at least one of CPU does not
> support it
>
> I would favor enabling SMP over enabling VFP, because that's probably
> what most people would expect.

The Realview system has VFP on both cores but NEON on only one IIRC.

For pathological cases like these, and these are the only two I've heard
of, disallowing VFP/NEON in kernel code is probably the sanest choice.
For userspace, it makes sense to enable it as usual and rely on CPU
affinity to keep things working.  If these systems were common,
extending the per-task VFP enabling to also limit the CPU affinity would
probably be something to consider.  As such configurations seem to be
quite rare, it's probably just as well to leave this to the system
builder.  Whatever the kernel does, a regular softfloat system will work
as expected.  Leaving the option for those who know what they're doing
to run VFP/NEON code with the appropriate CPU affinity set lets such
people take full advantage of the hardware without unduly complicating
the kernel for these rare situation.

-- 
M?ns Rullg?rd
mans at mansr.com

VFP available on a single CPU in a dual-CPU complex

[Catalin Marinas]

On Fri, Aug 15, 2014 at 10:24:39PM +0100, M?ns Rullg?rd wrote:
> Florian Fainelli <f.fainelli@gmail.com> writes:
> 
> > On 08/15/2014 11:54 AM, Arnd Bergmann wrote:
> >> On Friday 15 August 2014 10:14:45 Will Deacon wrote:
> >>> On Fri, Aug 15, 2014 at 10:03:01AM +0100, Arnd Bergmann wrote:
> >>>> On Friday 15 August 2014, Florian Fainelli wrote:
> >>>>> Hello,
> >>>>>
> >>>>> On Broadcom's BCM63138 SoC, the second Cortex A9 CPU does not have
> >>>>> VFP, which is a problem as one might imagine because we currently
> >>>>> assume it is available for all CPUs within the complex.
> >>>>
> >>>> Wow, that is pretty crazy.
> >>>>
> >>>>> I started to patch vfp_support_entry to test for CPU1 and branch to a
> >>>>> different location, but that raises a bunch of question, in particular
> >>>>> what to do if NEON/VFP came from the kernel, how can we migrate that
> >>>>> execution to CPU0?
> >>>>>
> >>>>> If the answer is don't use VFP, I guess that's fine, and I can have a
> >>>>> runtime check in vfp_init() that checks for BCM63138 and set
> >>>>> vfp_vector to vfp_null_entry even though we are running on a multi-v7
> >>>>> kernel with VFP enabled.
> >>>>
> >>>> I think adding hooks to the scheduler for this case is problematic, so
> >>>> the easiest way would of course be to give the user a choice between
> >>>> VFP or SMP on this hardware. For anything beyond that, you could do
> >>>> a prototype patch (possibly for integration into OpenWRT) so we can
> >>>> see how ugly it gets to support this setup.
> >>>
> >>> I agree. We have something similar on the Realview-PBX (dual A9, only one
> >>> CPU with NEON) and the answer is not to enable NEON if you want SMP.
> >> 
> >> Does that work with a multiplatform kernel?
> >> 
> >> I have patches to enable realview for ARCH_MULTIPLATFORM, and I wonder
> >> if that gets in the way. Do we have code to detect this setup at runtime
> >> and disable NEON/VFP if only a subset of the enabled CPUs have it?
> >
> > I was wondering about that too, here are the two options I see for these
> > pathological cases:
> >
> > - if SMP is enabled, and CONFIG_VFP is also enabled, refuse bringing up
> > cores that do not support VFP/NEON, thus becoming a SMP on UP, and do
> > that for the platform/SoC-specific SMP support code
> >
> > - or, in the VFP initialization, completely disable VFP (overriding
> > VFP_arch in SoC specific code?) because at least one of CPU does not
> > support it
> >
> > I would favor enabling SMP over enabling VFP, because that's probably
> > what most people would expect.
> 
> The Realview system has VFP on both cores but NEON on only one IIRC.

That's the PBX-A9 IIRC.

> For pathological cases like these, and these are the only two I've heard
> of, disallowing VFP/NEON in kernel code is probably the sanest choice.

I agree.

> For userspace, it makes sense to enable it as usual and rely on CPU
> affinity to keep things working. 

If we rely on user-space to set affinity, we also need per-CPU hwcap or
some method to inform user which CPUs have VFP/Neon. But I really don't
like user space parsing /proc/cpuinfo, which leaves us with disabling
VFP altogether.

> If these systems were common,
> extending the per-task VFP enabling to also limit the CPU affinity would
> probably be something to consider.

We tried this before in ARM and it was kind of working. The only problem
was that because glibc was setting FPCSR, all processes ended up on the
same CPU. So we had to add a timer to clear the affinity for a task
after a while. But I would definitely not recommend such patches for
mainline.

I don't have any preference for SMP+!VFP vs UP+VFP in this case. I guess
Broadcom should know better what the target market is and what kind of
user space is supposed to run on such system.

-- 
Catalin