[Xenomai] xenomai/ipipe arm64 port

[Xenomai] xenomai/ipipe arm64 port

[Don Mahurin]

Hi all,

We would like to submit our current work on the arm64 port of
ipipe/xenomai. We hope that this contribution will encourage further
development of arm64 support in ipipe/xenomai.

This port was largely developed on ipipe-3.10 as a base, so it is likely to
be the most well tested there. Though recently, we have also ported these
changes to ipipe-3.14.44. ( We noticed that 3.10 is missing from rc7. Is
this intended, or will 3.10 support return? )

At the end of this message are the relevant git repos and branches for the
arm64 port. Also included is a build script to build a minimal busybox
linux system which may be ran using qemu (arm64 virt).


Development status:

The 3.14 port is missing the fast-syscall changes (in entry.S) from armv7,
and instead contains the 3.10 entry.S changes forward ported to 3.14.

FPU support is incomplete.

Smokey tests:

3.10/rc6 smokey tests all pass
3.14/rc6 smokey tests all pass
3.14/rc7 smokey tests sched-quota and sched-tp fail


git repos:

https://gitlab.mperpetuo.com/it/xenomai-3.git
https://gitlab.mperpetuo.com/it/ipipe.git
https://gitlab.mperpetuo.com/it/build-scripts.git

ipipe/xenomai arm64 branches:

ipipe-core-3.10.32-arm-10-arm64-2c, v3.0-rc6-arm64-1c
ipipe-core-3.14.44-arm-11-arm64-2, v3.0-rc6-arm64-1c
ipipe-core-3.14.44-arm-12-arm64-2, v3.0-rc7-arm64-1c


The system is built with the following commands.
git clone https://gitlab.mperpetuo.com/it/build-scripts.git
mkdir -p build
cd build
../build-scripts/build_linux_emulator-arm64.sh
or
../build-scripts/build_linux_emulator-arm64.sh --314-rc6
or
../build-scripts/build_linux_emulator-arm64.sh --314-rc7

To run the resulting build:
sudo apt-get install qemu-system-arm
( cd out/target/product/xenomai_arm64; ./run.sh )


We look forward to any feedback or questions. Please let us know how we may
continue to contribute to move this arm64 port forward.

-Don Mahurin

Re: [Xenomai] xenomai/ipipe arm64 port

[Philippe Gerum]

On 08/25/2015 02:13 AM, Don Mahurin wrote:
> Hi all,
> 
> We would like to submit our current work on the arm64 port of
> ipipe/xenomai. We hope that this contribution will encourage further
> development of arm64 support in ipipe/xenomai.
>

arm64 support is definitely a high priority item. Thanks for tackling this.

> This port was largely developed on ipipe-3.10 as a base, so it is likely to
> be the most well tested there. Though recently, we have also ported these
> changes to ipipe-3.14.44. ( We noticed that 3.10 is missing from rc7. Is
> this intended, or will 3.10 support return? )

ipipe-core-3.10.32-arm-10.patch is still part of the distribution in
-rc7. This said, the absence of a patch targeting a particular kernel
release in a given Xenomai version does not mean the latter can't run
over the former. This rather means that we did not check for it. Xenomai
3 over Cobalt requires a kernel release >= 3.10 though.

> 
> At the end of this message are the relevant git repos and branches for the
> arm64 port. Also included is a build script to build a minimal busybox
> linux system which may be ran using qemu (arm64 virt).
> 
> 
> Development status:
> 
> The 3.14 port is missing the fast-syscall changes (in entry.S) from armv7,
> and instead contains the 3.10 entry.S changes forward ported to 3.14.
> 
> FPU support is incomplete.
> 
> Smokey tests:
> 
> 3.10/rc6 smokey tests all pass
> 3.14/rc6 smokey tests all pass
> 3.14/rc7 smokey tests sched-quota and sched-tp fail
> 

This is likely because -rc7 has added checks to sched-* tests which have
time-dependent results, so they may not match the expected value in a
qemu-based execution.

> 
> git repos:
> 
> https://gitlab.mperpetuo.com/it/xenomai-3.git
> https://gitlab.mperpetuo.com/it/ipipe.git
> https://gitlab.mperpetuo.com/it/build-scripts.git
>

It looks like curl does not like the cypher used by this server. http is
fine though.

> ipipe/xenomai arm64 branches:
> 
> ipipe-core-3.10.32-arm-10-arm64-2c, v3.0-rc6-arm64-1c
> ipipe-core-3.14.44-arm-11-arm64-2, v3.0-rc6-arm64-1c
> ipipe-core-3.14.44-arm-12-arm64-2, v3.0-rc7-arm64-1c
> 
> 
> The system is built with the following commands.
> git clone https://gitlab.mperpetuo.com/it/build-scripts.git
> mkdir -p build
> cd build
> ../build-scripts/build_linux_emulator-arm64.sh
> or
> ../build-scripts/build_linux_emulator-arm64.sh --314-rc6
> or
> ../build-scripts/build_linux_emulator-arm64.sh --314-rc7
> 

This went almost fine except the rsync step which assumes a native arm64
host for populating the rootfs, I have no such beast. But that's not a
major issue, I can take another path for building it.

> To run the resulting build:
> sudo apt-get install qemu-system-arm
> ( cd out/target/product/xenomai_arm64; ./run.sh )
> 
> 
> We look forward to any feedback or questions. Please let us know how we may
> continue to contribute to move this arm64 port forward.
> 

I only had a quick look to the code so far, but as a preliminary step,
you may want to split cobalt/arch/arm (and the related include areas)
from cobalt/arch/arm64 in the Xenomai tree, following the convention
used in the mainline kernel. Having a moderate amount of duplicate code
between arm and arm64 is acceptable, compared to dealing with
cross-architecture headers sprinkled with #ifdefery. arm and arm64 are
distinct enough.

I believe we should target 3.14 for the first implementation, since the
I-pipe bits in 3.10 are lagging slightly behind already, 3.14 contains
the latest changes to the interrupt pipeline though.

-- 
Philippe.

Re: [Xenomai] xenomai/ipipe arm64 port

[Jorge Ramirez Ortiz]

On 08/25/2015 10:08 AM, Philippe Gerum wrote:
> On 08/25/2015 02:13 AM, Don Mahurin wrote:
>> Hi all,
>>
>> We would like to submit our current work on the arm64 port of
>> ipipe/xenomai. We hope that this contribution will encourage further
>> development of arm64 support in ipipe/xenomai.
>>
> 
> arm64 support is definitely a high priority item. Thanks for tackling this.

There are a numbers of cheap (<100USD) and well documented aarch64 boards [1]
that could be used as a reference platform (different SoC vendors)
I'd suggest we use Qualcomm's Dragon 410c.


[1] https://www.96boards.org/

Re: [Xenomai] xenomai/ipipe arm64 port

[Jan Kiszka]

On 2015-08-25 17:20, Jorge Ramirez Ortiz wrote:
> On 08/25/2015 10:08 AM, Philippe Gerum wrote:
>> On 08/25/2015 02:13 AM, Don Mahurin wrote:
>>> Hi all,
>>>
>>> We would like to submit our current work on the arm64 port of
>>> ipipe/xenomai. We hope that this contribution will encourage further
>>> development of arm64 support in ipipe/xenomai.
>>>
>>
>> arm64 support is definitely a high priority item. Thanks for tackling this.
> 
> There are a numbers of cheap (<100USD) and well documented aarch64 boards [1]
> that could be used as a reference platform (different SoC vendors)
> I'd suggest we use Qualcomm's Dragon 410c.

The Qualcomm thing is pretty ugly beast, using a proprietary interrupt
controller instead of the standard GIC as strongly recommended by ARM.
As an ARM kernel developer recently put it when I asked about this
board: "I prefer the Raspberry Pi2 over that one." And the Pi2 is
already broken in its design.

The HiKey looks better in this regard, but one has to check which of the
mentioned boards are already upstream (or very close to this). Same for
uboot support. No one should hack on vendor trees any more.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] xenomai/ipipe arm64 port

[Jorge Ramirez Ortiz]

On 08/25/2015 12:13 PM, Jan Kiszka wrote:
> On 2015-08-25 17:20, Jorge Ramirez Ortiz wrote:
>> On 08/25/2015 10:08 AM, Philippe Gerum wrote:
>>> On 08/25/2015 02:13 AM, Don Mahurin wrote:
>>>> Hi all,
>>>>
>>>> We would like to submit our current work on the arm64 port of
>>>> ipipe/xenomai. We hope that this contribution will encourage further
>>>> development of arm64 support in ipipe/xenomai.
>>>>
>>>
>>> arm64 support is definitely a high priority item. Thanks for tackling this.
>>
>> There are a numbers of cheap (<100USD) and well documented aarch64 boards [1]
>> that could be used as a reference platform (different SoC vendors)
>> I'd suggest we use Qualcomm's Dragon 410c.
> 
> The Qualcomm thing is pretty ugly beast, using a proprietary interrupt
> controller instead of the standard GIC as strongly recommended by ARM.
> As an ARM kernel developer recently put it when I asked about this
> board: "I prefer the Raspberry Pi2 over that one." And the Pi2 is
> already broken in its design.

do you know the name of the developer?

Incidentally I am working on both boards - I gate keep the HiKey kernel [0] but
do some level of support on all 96 boards.
It seems to me that Qualcomm might be putting more resources on the the level of
support and streaming efforts (hence my recommendation).

Having said that, HiKey has support for OP-TEE (always interesting [1] ) and in
the next release we will ship UEFI which we are currently validating.

The 410c ubuntu kernel is here [2] - the Android kernel is also available on
another repo.

[0] https://github.com/96boards/linux
[1] https://github.com/OP-TEE/optee_os
[2]
https://git.linaro.org/landing-teams/working/qualcomm/kernel.git/shortlog/refs/heads/release/qcomlt-4.0

> 
> The HiKey looks better in this regard, but one has to check which of the
> mentioned boards are already upstream (or very close to this). Same for
> uboot support. No one should hack on vendor trees any more.

Upstream support is ongoing for both boards (we upstreamed the HiKey basic
platform support a couple of months ago but that is not to say that it is in a
better shape with respect to drivers and peripherals - graphics are - currently,
might change in the near future-  better on the 410c for instance).

in any case, I think we should pick one SoC.

Re: [Xenomai] xenomai/ipipe arm64 port

[Jorge Ramirez Ortiz]

On 08/25/2015 01:07 PM, Jorge Ramirez Ortiz wrote:
> On 08/25/2015 12:13 PM, Jan Kiszka wrote:
>> On 2015-08-25 17:20, Jorge Ramirez Ortiz wrote:
>>> On 08/25/2015 10:08 AM, Philippe Gerum wrote:
>>>> On 08/25/2015 02:13 AM, Don Mahurin wrote:
>>>>> Hi all,
>>>>>
>>>>> We would like to submit our current work on the arm64 port of
>>>>> ipipe/xenomai. We hope that this contribution will encourage further
>>>>> development of arm64 support in ipipe/xenomai.
>>>>>
>>>>
>>>> arm64 support is definitely a high priority item. Thanks for tackling this.
>>>
>>> There are a numbers of cheap (<100USD) and well documented aarch64 boards [1]
>>> that could be used as a reference platform (different SoC vendors)
>>> I'd suggest we use Qualcomm's Dragon 410c.
>>
>> The Qualcomm thing is pretty ugly beast, using a proprietary interrupt
>> controller instead of the standard GIC as strongly recommended by ARM.


Looking at the device tree setting it seems to me it is using GICv2

drivers/irqchip/irq-gic.c

Re: [Xenomai] xenomai/ipipe arm64 port

[Jan Kiszka]

On 2015-08-25 19:34, Jorge Ramirez Ortiz wrote:
> On 08/25/2015 01:07 PM, Jorge Ramirez Ortiz wrote:
>> On 08/25/2015 12:13 PM, Jan Kiszka wrote:
>>> On 2015-08-25 17:20, Jorge Ramirez Ortiz wrote:
>>>> On 08/25/2015 10:08 AM, Philippe Gerum wrote:
>>>>> On 08/25/2015 02:13 AM, Don Mahurin wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> We would like to submit our current work on the arm64 port of
>>>>>> ipipe/xenomai. We hope that this contribution will encourage further
>>>>>> development of arm64 support in ipipe/xenomai.
>>>>>>
>>>>>
>>>>> arm64 support is definitely a high priority item. Thanks for tackling this.
>>>>
>>>> There are a numbers of cheap (<100USD) and well documented aarch64 boards [1]
>>>> that could be used as a reference platform (different SoC vendors)
>>>> I'd suggest we use Qualcomm's Dragon 410c.
>>>
>>> The Qualcomm thing is pretty ugly beast, using a proprietary interrupt
>>> controller instead of the standard GIC as strongly recommended by ARM.
> 
> 
> Looking at the device tree setting it seems to me it is using GICv2
> 
> drivers/irqchip/irq-gic.c
> 

Really? I'm looking at upstream right now, and there is no "arm,gic-*"
compatible string - in contrast to the hi6220.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] xenomai/ipipe arm64 port

[Jorge Ramirez Ortiz]

On 08/25/2015 01:36 PM, Jan Kiszka wrote:
> On 2015-08-25 19:34, Jorge Ramirez Ortiz wrote:
>> On 08/25/2015 01:07 PM, Jorge Ramirez Ortiz wrote:
>>> On 08/25/2015 12:13 PM, Jan Kiszka wrote:
>>>> On 2015-08-25 17:20, Jorge Ramirez Ortiz wrote:
>>>>> On 08/25/2015 10:08 AM, Philippe Gerum wrote:
>>>>>> On 08/25/2015 02:13 AM, Don Mahurin wrote:
>>>>>>> Hi all,
>>>>>>>
>>>>>>> We would like to submit our current work on the arm64 port of
>>>>>>> ipipe/xenomai. We hope that this contribution will encourage further
>>>>>>> development of arm64 support in ipipe/xenomai.
>>>>>>>
>>>>>>
>>>>>> arm64 support is definitely a high priority item. Thanks for tackling this.
>>>>>
>>>>> There are a numbers of cheap (<100USD) and well documented aarch64 boards [1]
>>>>> that could be used as a reference platform (different SoC vendors)
>>>>> I'd suggest we use Qualcomm's Dragon 410c.
>>>>
>>>> The Qualcomm thing is pretty ugly beast, using a proprietary interrupt
>>>> controller instead of the standard GIC as strongly recommended by ARM.
>>
>>
>> Looking at the device tree setting it seems to me it is using GICv2
>>
>> drivers/irqchip/irq-gic.c
>>
> 
> Really? I'm looking at upstream right now, and there is no "arm,gic-*"
> compatible string - in contrast to the hi6220.


right, the APQ8016 uses compatible = "qcom,msm-qgic2" handled in
drivers/irqchip/irq-gic.c

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.2-rc8#n135

Re: [Xenomai] xenomai/ipipe arm64 port

[Jan Kiszka]

On 2015-08-25 19:54, Jorge Ramirez Ortiz wrote:
> On 08/25/2015 01:36 PM, Jan Kiszka wrote:
>> On 2015-08-25 19:34, Jorge Ramirez Ortiz wrote:
>>> On 08/25/2015 01:07 PM, Jorge Ramirez Ortiz wrote:
>>>> On 08/25/2015 12:13 PM, Jan Kiszka wrote:
>>>>> On 2015-08-25 17:20, Jorge Ramirez Ortiz wrote:
>>>>>> On 08/25/2015 10:08 AM, Philippe Gerum wrote:
>>>>>>> On 08/25/2015 02:13 AM, Don Mahurin wrote:
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> We would like to submit our current work on the arm64 port of
>>>>>>>> ipipe/xenomai. We hope that this contribution will encourage further
>>>>>>>> development of arm64 support in ipipe/xenomai.
>>>>>>>>
>>>>>>>
>>>>>>> arm64 support is definitely a high priority item. Thanks for tackling this.
>>>>>>
>>>>>> There are a numbers of cheap (<100USD) and well documented aarch64 boards [1]
>>>>>> that could be used as a reference platform (different SoC vendors)
>>>>>> I'd suggest we use Qualcomm's Dragon 410c.
>>>>>
>>>>> The Qualcomm thing is pretty ugly beast, using a proprietary interrupt
>>>>> controller instead of the standard GIC as strongly recommended by ARM.
>>>
>>>
>>> Looking at the device tree setting it seems to me it is using GICv2
>>>
>>> drivers/irqchip/irq-gic.c
>>>
>>
>> Really? I'm looking at upstream right now, and there is no "arm,gic-*"
>> compatible string - in contrast to the hi6220.
> 
> 
> right, the APQ8016 uses compatible = "qcom,msm-qgic2" handled in
> drivers/irqchip/irq-gic.c
> 
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/boot/dts/qcom/msm8916.dtsi?id=refs/tags/v4.2-rc8#n135
> 

Interesting. That /may/ work out smoothly for I-pipe, and it just
doesn't work for virtualization because the "qgic2" lacks GICH and GICV
regions, thus virtualization extensions.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] xenomai/ipipe arm64 port

[Jan Kiszka]

On 2015-08-25 19:07, Jorge Ramirez Ortiz wrote:
> On 08/25/2015 12:13 PM, Jan Kiszka wrote:
>> On 2015-08-25 17:20, Jorge Ramirez Ortiz wrote:
>>> On 08/25/2015 10:08 AM, Philippe Gerum wrote:
>>>> On 08/25/2015 02:13 AM, Don Mahurin wrote:
>>>>> Hi all,
>>>>>
>>>>> We would like to submit our current work on the arm64 port of
>>>>> ipipe/xenomai. We hope that this contribution will encourage further
>>>>> development of arm64 support in ipipe/xenomai.
>>>>>
>>>>
>>>> arm64 support is definitely a high priority item. Thanks for tackling this.
>>>
>>> There are a numbers of cheap (<100USD) and well documented aarch64 boards [1]
>>> that could be used as a reference platform (different SoC vendors)
>>> I'd suggest we use Qualcomm's Dragon 410c.
>>
>> The Qualcomm thing is pretty ugly beast, using a proprietary interrupt
>> controller instead of the standard GIC as strongly recommended by ARM.
>> As an ARM kernel developer recently put it when I asked about this
>> board: "I prefer the Raspberry Pi2 over that one." And the Pi2 is
>> already broken in its design.
> 
> do you know the name of the developer?

Will mail you privately.

> 
> Incidentally I am working on both boards - I gate keep the HiKey kernel [0] but
> do some level of support on all 96 boards.
> It seems to me that Qualcomm might be putting more resources on the the level of
> support and streaming efforts (hence my recommendation).

I didn't look into that aspect yet, but this is surely valuable as well.

> 
> Having said that, HiKey has support for OP-TEE (always interesting [1] ) and in
> the next release we will ship UEFI which we are currently validating.

Is UEFI an advantage? Will we be able to hack on it to fix lacking
features and broken bits (that was always required so far with uboot for
the boards I tried with Cortex-A7/15)?

> 
> The 410c ubuntu kernel is here [2] - the Android kernel is also available on
> another repo.
> 
> [0] https://github.com/96boards/linux
> [1] https://github.com/OP-TEE/optee_os
> [2]
> https://git.linaro.org/landing-teams/working/qualcomm/kernel.git/shortlog/refs/heads/release/qcomlt-4.0
> 
>>
>> The HiKey looks better in this regard, but one has to check which of the
>> mentioned boards are already upstream (or very close to this). Same for
>> uboot support. No one should hack on vendor trees any more.
> 
> Upstream support is ongoing for both boards (we upstreamed the HiKey basic
> platform support a couple of months ago but that is not to say that it is in a
> better shape with respect to drivers and peripherals - graphics are - currently,
> might change in the near future-  better on the 410c for instance).
> 
> in any case, I think we should pick one SoC.
> 

Probably. The HiKey seems to have run out of stock right now or is late
with shipping.

The DragonBoard I saw live last week, but when I asked about
virtualization support, that nasty detail came up. Given that
Xenomai/I-pipe touches a lot the interrupt handling, having a
pre-existing well-known controller in place (you can simply use the
arm32 bits) seemed useful to me. I'm not sure what other things are
problematic because I stopped digging deeper when that board was
classified unsuitable for virtualization.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] xenomai/ipipe arm64 port

[Jorge Ramirez Ortiz]

On 08/25/2015 01:34 PM, Jan Kiszka wrote:
> On 2015-08-25 19:07, Jorge Ramirez Ortiz wrote:
>> On 08/25/2015 12:13 PM, Jan Kiszka wrote:
>>> On 2015-08-25 17:20, Jorge Ramirez Ortiz wrote:
>>>> On 08/25/2015 10:08 AM, Philippe Gerum wrote:
>>>>> On 08/25/2015 02:13 AM, Don Mahurin wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> We would like to submit our current work on the arm64 port of
>>>>>> ipipe/xenomai. We hope that this contribution will encourage further
>>>>>> development of arm64 support in ipipe/xenomai.
>>>>>>
>>>>>
>>>>> arm64 support is definitely a high priority item. Thanks for tackling this.
>>>>
>>>> There are a numbers of cheap (<100USD) and well documented aarch64 boards [1]
>>>> that could be used as a reference platform (different SoC vendors)
>>>> I'd suggest we use Qualcomm's Dragon 410c.
>>>
>>> The Qualcomm thing is pretty ugly beast, using a proprietary interrupt
>>> controller instead of the standard GIC as strongly recommended by ARM.
>>> As an ARM kernel developer recently put it when I asked about this
>>> board: "I prefer the Raspberry Pi2 over that one." And the Pi2 is
>>> already broken in its design.
>>
>> do you know the name of the developer?
> 
> Will mail you privately.
> 
>>
>> Incidentally I am working on both boards - I gate keep the HiKey kernel [0] but
>> do some level of support on all 96 boards.
>> It seems to me that Qualcomm might be putting more resources on the the level of
>> support and streaming efforts (hence my recommendation).
> 
> I didn't look into that aspect yet, but this is surely valuable as well.
> 
>>
>> Having said that, HiKey has support for OP-TEE (always interesting [1] ) and in
>> the next release we will ship UEFI which we are currently validating.
> 
> Is UEFI an advantage? Will we be able to hack on it to fix lacking
> features and broken bits (that was always required so far with uboot for
> the boards I tried with Cortex-A7/15)?

It is in this case (the previous bootloader that Hikey was using was proprietary
and not open)
Hikey's UEFI [1] is available just as the ATF [2]

last time I checked though, UEFI only brings up 1 of the 8 cores but this might
be fixed in the next release.

[1] https://github.com/96boards/edk2
[2] https://github.com/96boards/arm-trusted-firmware

uboot support is also on its way (there is also support for JTAG which I dont
think it is present in the Dragon board)


> 
>>
>> The 410c ubuntu kernel is here [2] - the Android kernel is also available on
>> another repo.
>>
>> [0] https://github.com/96boards/linux
>> [1] https://github.com/OP-TEE/optee_os
>> [2]
>> https://git.linaro.org/landing-teams/working/qualcomm/kernel.git/shortlog/refs/heads/release/qcomlt-4.0
>>
>>>
>>> The HiKey looks better in this regard, but one has to check which of the
>>> mentioned boards are already upstream (or very close to this). Same for
>>> uboot support. No one should hack on vendor trees any more.
>>
>> Upstream support is ongoing for both boards (we upstreamed the HiKey basic
>> platform support a couple of months ago but that is not to say that it is in a
>> better shape with respect to drivers and peripherals - graphics are - currently,
>> might change in the near future-  better on the 410c for instance).
>>
>> in any case, I think we should pick one SoC.
>>
> 
> Probably. The HiKey seems to have run out of stock right now or is late
> with shipping.
> 
> The DragonBoard I saw live last week, but when I asked about
> virtualization support, that nasty detail came up. Given that
> Xenomai/I-pipe touches a lot the interrupt handling, having a
> pre-existing well-known controller in place (you can simply use the
> arm32 bits) seemed useful to me. I'm not sure what other things are
> problematic because I stopped digging deeper when that board was
> classified unsuitable for virtualization.

this is good feedback. I will pass it down to the Qualcomm folks.

Re: [Xenomai] xenomai/ipipe arm64 port

[Don Mahurin]

On Tue, Aug 25, 2015 at 7:08 AM, Philippe Gerum <rpm@xenomai.org> wrote:

> On 08/25/2015 02:13 AM, Don Mahurin wrote:
> > Hi all,
> >
> > We would like to submit our current work on the arm64 port of
> > ipipe/xenomai. We hope that this contribution will encourage further
> > development of arm64 support in ipipe/xenomai.
> >
>
> arm64 support is definitely a high priority item. Thanks for tackling this.


We look forward to getting the changes into upstream xenomai/ipipe as well.



> > This port was largely developed on ipipe-3.10 as a base, so it is likely
> to
> > be the most well tested there. Though recently, we have also ported these
> > changes to ipipe-3.14.44. ( We noticed that 3.10 is missing from rc7. Is
> > this intended, or will 3.10 support return? )
>
> ipipe-core-3.10.32-arm-10.patch is still part of the distribution in
> -rc7. This said, the absence of a patch targeting a particular kernel
> release in a given Xenomai version does not mean the latter can't run
> over the former. This rather means that we did not check for it. Xenomai
> 3 over Cobalt requires a kernel release >= 3.10 though.


Thanks for confirming this.


>
>
> > At the end of this message are the relevant git repos and branches for
> the
> > arm64 port. Also included is a build script to build a minimal busybox
> > linux system which may be ran using qemu (arm64 virt).
> >
> >
> > Development status:
> >
> > The 3.14 port is missing the fast-syscall changes (in entry.S) from
> armv7,
> > and instead contains the 3.10 entry.S changes forward ported to 3.14.
> >
> > FPU support is incomplete.
> >
> > Smokey tests:
> >
> > 3.10/rc6 smokey tests all pass
> > 3.14/rc6 smokey tests all pass
> > 3.14/rc7 smokey tests sched-quota and sched-tp fail
> >
>
> This is likely because -rc7 has added checks to sched-* tests which have
> time-dependent results, so they may not match the expected value in a
> qemu-based execution.
>

Would it be worthwhile to optionally disable the time-dependent nature of
the tests, just for use in quick/automated emulator testing?


>
> >
> > git repos:
> >
> > https://gitlab.mperpetuo.com/it/xenomai-3.git
> > https://gitlab.mperpetuo.com/it/ipipe.git
> > https://gitlab.mperpetuo.com/it/build-scripts.git
> >
>
> It looks like curl does not like the cypher used by this server. http is
> fine though.


hmm,  should work. will look at this.


>
> >
> > The system is built with the following commands.
> > git clone https://gitlab.mperpetuo.com/it/build-scripts.git
> > mkdir -p build
> > cd build
> > ../build-scripts/build_linux_emulator-arm64.sh
> >
>
> This went almost fine except the rsync step which assumes a native arm64
> host for populating the rootfs, I have no such beast. But that's not a
> major issue, I can take another path for building it.


Yes, to build the system, we copy the arm64 shared libraries from the host
system (used by cross compiler).  Using Ubuntu 14.10. May need some
adjustment for other Linux host systems.


>

>
> > We look forward to any feedback or questions. Please let us know how we
> may
> > continue to contribute to move this arm64 port forward.
> >
>
> I only had a quick look to the code so far, but as a preliminary step,
> you may want to split cobalt/arch/arm (and the related include areas)
> from cobalt/arch/arm64 in the Xenomai tree, following the convention
> used in the mainline kernel. Having a moderate amount of duplicate code
> between arm and arm64 is acceptable, compared to dealing with
> cross-architecture headers sprinkled with #ifdefery. arm and arm64 are
> distinct enough.
>

It is mostly separate with kernel/cobalt/arch/arm and
kernel/cobalt/arch/arm64 already.

We did question if that was desired, as powerpc 32 and 64 bit share the
same architecture as do x86 and x86_64.
But I guess that also follows the convention in the mainline kernel.

I think there was still one case where kernel/cobalt/arch/arm was used by a
arm64 build.  We will look at fixing this.

In general, how should we proceed with arm64 changes?  We can continue to
make updates in the separate repos.  But is there anything we need to do to
get a thorough review of the changes by the community and get this
incorporated into upstream xenomai/ipipe?

-Don

Re: [Xenomai] xenomai/ipipe arm64 port

[Jorge Ramirez Ortiz]

On 08/25/2015 02:05 PM, Don Mahurin wrote:
>>> > >
>>> > > The system is built with the following commands.
>>> > > git clone https://gitlab.mperpetuo.com/it/build-scripts.git
>>> > > mkdir -p build
>>> > > cd build
>>> > > ../build-scripts/build_linux_emulator-arm64.sh
>>> > >
>> >
>> > This went almost fine except the rsync step which assumes a native arm64
>> > host for populating the rootfs, I have no such beast. But that's not a
>> > major issue, I can take another path for building it.
> 
> Yes, to build the system, we copy the arm64 shared libraries from the host
> system (used by cross compiler).  Using Ubuntu 14.10. May need some
> adjustment for other Linux host systems.
> 
> 

if you need some aarch64 libraries, you can use a one of the rootfs below to
pull them

https://builds.96boards.org/releases/hikey/linaro/debian/15.06/

Download one of the images (hikey-jessie_alip_20150701-323.emmc.img.gz for instance)

$ simg2img hikey-jessie_alip_20150701-323.emmc.img raw.img
$ sudo mount raw.img <mnt_point>



-- 
jro

Re: [Xenomai] xenomai/ipipe arm64 port

[Jan Kiszka]

On 2015-08-25 20:05, Don Mahurin wrote:
>>> Smokey tests:
>>>
>>> 3.10/rc6 smokey tests all pass
>>> 3.14/rc6 smokey tests all pass
>>> 3.14/rc7 smokey tests sched-quota and sched-tp fail
>>>
>>
>> This is likely because -rc7 has added checks to sched-* tests which have
>> time-dependent results, so they may not match the expected value in a
>> qemu-based execution.
>>
> 
> Would it be worthwhile to optionally disable the time-dependent nature of
> the tests, just for use in quick/automated emulator testing?
> 

I was wondering the same recently because tests also break in KVM on
x86, my preferred target for ad-hoc tests.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] xenomai/ipipe arm64 port

[Philippe Gerum]

On 08/25/2015 08:05 PM, Don Mahurin wrote:
> On Tue, Aug 25, 2015 at 7:08 AM, Philippe Gerum <rpm@xenomai.org
> <mailto:rpm@xenomai.org>> wrote:
> 
>     This is likely because -rc7 has added checks to sched-* tests which have
>     time-dependent results, so they may not match the expected value in a
>     qemu-based execution.
> 
> 
> Would it be worthwhile to optionally disable the time-dependent nature
> of the tests, just for use in quick/automated emulator testing?
>  

I believe so. We should probably have smokey accept a --vm option or
something along, to tell it about a virtual execution environment, which
may be freely interpreted by any test code concerned, e.g. to figure out
whether timings might be inaccurate due to a significant slowdown. We
could likely figure this out automatically, but I'd rather leave this
decision to the user. I'll have a look at this asap.

> 
> In general, how should we proceed with arm64 changes?  We can continue
> to make updates in the separate repos.  But is there anything we need to
> do to get a thorough review of the changes by the community and get this
> incorporated into upstream xenomai/ipipe? 
> 

I will open dedicated arm64 branches tomorrow into the project
repositories for both the ipipe and xenomai, that could be merged to the
corresponding mainline branches when ready for prime time. The target
would be Xenomai 3.1, it's too late for bringing this in 3.0-rc.

So the process should be fairly common and straightforward: a pull
request from your end for code present in your repo would trigger the
review, bits getting merged gradually from your repo to the dedicated
arm64 branches, along with other contributions, until we can all declare
these branches ready for mainline.

-- 
Philippe.

Re: [Xenomai] xenomai/ipipe arm64 port

[Gilles Chanteperdrix]

On Tue, Aug 25, 2015 at 04:08:07PM +0200, Philippe Gerum wrote:
> On 08/25/2015 02:13 AM, Don Mahurin wrote:
> > Hi all,
> > 
> > We would like to submit our current work on the arm64 port of
> > ipipe/xenomai. We hope that this contribution will encourage further
> > development of arm64 support in ipipe/xenomai.
> >
> 
> arm64 support is definitely a high priority item. Thanks for tackling this.
> 
> > This port was largely developed on ipipe-3.10 as a base, so it is likely to
> > be the most well tested there. Though recently, we have also ported these
> > changes to ipipe-3.14.44. ( We noticed that 3.10 is missing from rc7. Is
> > this intended, or will 3.10 support return? )
> 
> ipipe-core-3.10.32-arm-10.patch is still part of the distribution in
> -rc7. This said, the absence of a patch targeting a particular kernel
> release in a given Xenomai version does not mean the latter can't run
> over the former. This rather means that we did not check for it. Xenomai
> 3 over Cobalt requires a kernel release >= 3.10 though.
> 
> > 
> > At the end of this message are the relevant git repos and branches for the
> > arm64 port. Also included is a build script to build a minimal busybox
> > linux system which may be ran using qemu (arm64 virt).
> > 
> > 
> > Development status:
> > 
> > The 3.14 port is missing the fast-syscall changes (in entry.S) from armv7,
> > and instead contains the 3.10 entry.S changes forward ported to 3.14.
> > 
> > FPU support is incomplete.
> > 
> > Smokey tests:
> > 
> > 3.10/rc6 smokey tests all pass
> > 3.14/rc6 smokey tests all pass
> > 3.14/rc7 smokey tests sched-quota and sched-tp fail
> > 
> 
> This is likely because -rc7 has added checks to sched-* tests which have
> time-dependent results, so they may not match the expected value in a
> qemu-based execution.
> 
> > 
> > git repos:
> > 
> > https://gitlab.mperpetuo.com/it/xenomai-3.git
> > https://gitlab.mperpetuo.com/it/ipipe.git
> > https://gitlab.mperpetuo.com/it/build-scripts.git
> >
> 
> It looks like curl does not like the cypher used by this server. http is
> fine though.

I do not know if any configuration was changed, but git clone did
not complai here for the first URL.

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] xenomai/ipipe arm64 port

[Jorge Ramirez Ortiz]

On 08/24/2015 08:13 PM, Don Mahurin wrote:
> The system is built with the following commands.
> git clone https://gitlab.mperpetuo.com/it/build-scripts.git
> mkdir -p build
> cd build
> ../build-scripts/build_linux_emulator-arm64.sh
> or
> ../build-scripts/build_linux_emulator-arm64.sh --314-rc6
> or
> ../build-scripts/build_linux_emulator-arm64.sh --314-rc7


what compiler version are you using?


-- 
jro

Re: [Xenomai] xenomai/ipipe arm64 port

[Don Mahurin]

dpkg -l gcc-aarch64-linux-gnu

gcc-aarch64-linux-gnu             4:4.9.1-1             amd64
  The GNU C compiler for arm64 architecture

On Wed, Aug 26, 2015 at 7:40 AM, Jorge Ramirez Ortiz <jro@xenomai.org>
wrote:

> On 08/24/2015 08:13 PM, Don Mahurin wrote:
> > The system is built with the following commands.
> > git clone https://gitlab.mperpetuo.com/it/build-scripts.git
> > mkdir -p build
> > cd build
> > ../build-scripts/build_linux_emulator-arm64.sh
> > or
> > ../build-scripts/build_linux_emulator-arm64.sh --314-rc6
> > or
> > ../build-scripts/build_linux_emulator-arm64.sh --314-rc7
>
>
> what compiler version are you using?
>
>
> --
> jro
>

Re: [Xenomai] xenomai/ipipe arm64 port

[Jorge Ramirez Ortiz]

On 08/24/2015 08:13 PM, Don Mahurin wrote:
> Hi all,
> 
> We would like to submit our current work on the arm64 port of
> ipipe/xenomai. We hope that this contribution will encourage further
> development of arm64 support in ipipe/xenomai.
> 
> This port was largely developed on ipipe-3.10 as a base, so it is likely to
> be the most well tested there. Though recently, we have also ported these
> changes to ipipe-3.14.44. ( We noticed that 3.10 is missing from rc7. Is
> this intended, or will 3.10 support return? )
> 
> At the end of this message are the relevant git repos and branches for the
> arm64 port. Also included is a build script to build a minimal busybox
> linux system which may be ran using qemu (arm64 virt).
> 
> 
> Development status:
> 
> The 3.14 port is missing the fast-syscall changes (in entry.S) from armv7,
> and instead contains the 3.10 entry.S changes forward ported to 3.14.


is the IPIPE tracing supported?

Re: [Xenomai] xenomai/ipipe arm64 port

[Don Mahurin]

On Thu, Aug 27, 2015 at 10:07 AM, Jorge Ramirez Ortiz <jro@xenomai.org>
wrote:

> On 08/24/2015 08:13 PM, Don Mahurin wrote:
> > Hi all,
> >
> > We would like to submit our current work on the arm64 port of
> > ipipe/xenomai. We hope that this contribution will encourage further
> > development of arm64 support in ipipe/xenomai.
> >
> > This port was largely developed on ipipe-3.10 as a base, so it is likely
> to
> > be the most well tested there. Though recently, we have also ported these
> > changes to ipipe-3.14.44. ( We noticed that 3.10 is missing from rc7. Is
> > this intended, or will 3.10 support return? )
> >
> > At the end of this message are the relevant git repos and branches for
> the
> > arm64 port. Also included is a build script to build a minimal busybox
> > linux system which may be ran using qemu (arm64 virt).
> >
> >
> > Development status:
> >
> > The 3.14 port is missing the fast-syscall changes (in entry.S) from
> armv7,
> > and instead contains the 3.10 entry.S changes forward ported to 3.14.
>
>
> is the IPIPE tracing supported?
>
>
IPIPE tracing is not complete in that set of changes.

We have additional changes that enable ipipe tracing, which we will submit
once the initial changes are in a branch.

-Don

Re: [Xenomai] xenomai/ipipe arm64 port

[Jorge Ramirez Ortiz]

On 08/24/2015 08:13 PM, Don Mahurin wrote:
> Hi all,
> 
> We would like to submit our current work on the arm64 port of
> ipipe/xenomai. We hope that this contribution will encourage further
> development of arm64 support in ipipe/xenomai.
> 
> This port was largely developed on ipipe-3.10 as a base, so it is likely to
> be the most well tested there. Though recently, we have also ported these
> changes to ipipe-3.14.44. ( We noticed that 3.10 is missing from rc7. Is
> this intended, or will 3.10 support return? )
> 
> At the end of this message are the relevant git repos and branches for the
> arm64 port. Also included is a build script to build a minimal busybox
> linux system which may be ran using qemu (arm64 virt).
> 
> 
> Development status:
> 
> The 3.14 port is missing the fast-syscall changes (in entry.S) from armv7,
> and instead contains the 3.10 entry.S changes forward ported to 3.14.
> 
> FPU support is incomplete.
> 
> Smokey tests:
> 
> 3.10/rc6 smokey tests all pass
> 3.14/rc6 smokey tests all pass
> 3.14/rc7 smokey tests sched-quota and sched-tp fail
> 
> 
> git repos:
> 
> https://gitlab.mperpetuo.com/it/xenomai-3.git
> https://gitlab.mperpetuo.com/it/ipipe.git
> https://gitlab.mperpetuo.com/it/build-scripts.git
> 
> ipipe/xenomai arm64 branches:
> 
> ipipe-core-3.10.32-arm-10-arm64-2c, v3.0-rc6-arm64-1c
> ipipe-core-3.14.44-arm-11-arm64-2, v3.0-rc6-arm64-1c
> ipipe-core-3.14.44-arm-12-arm64-2, v3.0-rc7-arm64-1c
> 
> 

Don,

I picked up your trees (branches below):

- ipipe branch: ipipe-core-3.14.44-arm-12-arm64-2
- xenomai branch: v3.0-rc7-arm64-1c).

I then merged them on HiKey's octacore A53 [0] vendor tree [1] previously
patched with ipipe:
- ipipe -core-3.18.12-arm-4.patch

root@linaro-developer:/usr/xenomai/bin# cat /proc/ipipe/version
12

root@linaro-developer:/usr/xenomai/bin# cat /proc/xenomai/version


3.0-rc7

root@linaro-developer:/usr/xenomai/bin# cat /proc/cpuinfo
Processor       : AArch64 Processor rev 3 (aarch64)
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x41
CPU architecture: AArch64
CPU variant     : 0x0
CPU part        : 0xd03
CPU revision    : 3

Hardware        : HiKey Development Board

With these changes in, I was able to execute the xeno-test suppported suite [non
SMP] (results below).

During the latency tests + dohell however I was getting a number of these events.
Have you happen to notice them during your runs?


[  210.661546] ls[1638]: unhandled level 2 translation fault (11) at 0x00000000,
esr 0x92000006
[  210.789815] pgd = ffffffc034fab000
[  210.853570] [00000000] *pgd=0000000037f8c003, *pud=0000000037f8c003,
*pmd=0000000000000000
[  210.923645]
[  211.029789] CPU: 0 PID: 1638 Comm: ls Not tainted 3.18.0-ipipe+ #3
[  211.135697] task: ffffffc034eb2ec0 ti: ffffffc035228000 task.ti: ffffffc035228000
[  211.239840] PC is at 0x7fb2d65b90
[  211.335692] LR is at 0x7fb2d58e60
[  211.343190] pc : [<0000007fb2d65b90>] lr : [<0000007fb2d58e60>] pstate: 40000000
[  211.503681] sp : 0000007fc4bfeda0
[  211.605713] x29: 0000007fc4bfeda0 x28: 0000000000000000
[  211.615535] x27: 000000002ae23100 x26: 0000000000000000
[  211.805734] x25: 000000002ae23130 x24: 000000002ae2ccb0
[  211.815575] x23: 0000007fb2e977f0 x22: 0000000000429000
[  212.005751] x21: 0000000000000000 x20: 0000000000000000
[  212.085690] x19: 0000007fb2e2e178 x18: 000000000000000d
[  212.095575] x17: 0000007fb2d58e44 x16: 0000000000429318
[  212.269880] x15: 0000007fb2e2f588 x14: 0000000000000000
[  212.359708] x13: 0000000001000000 x12: 0000000000000002
[  212.449749] x11: 0101010101010101 x10: 0000000000000000
[  212.579720] x9 : ff717373602e3231 x8 : 000000002ae25590
[  212.676101] x7 : 0000000000000000 x6 : 0000000000000000
[  212.779470] x5 : 0000000000000005 x4 : 0000000000000000
[  212.880130] x3 : 0000000000000000 x2 : 0000000000000000
[  212.959890] x1 : 0000000000000000 x0 : 0000000000000000




root@linaro-developer:/usr/xenomai/bin# ./xeno-test
Started child 1574: /bin/bash /usr/xenomai/bin/xeno-test-run-wrapper ./xeno-test
++ echo 0
++ testdir=/usr/xenomai/bin
++ /usr/xenomai/bin/smokey --run
arith OK
bufp skipped (no kernel support)
iddp skipped (no kernel support)
leaks OK
posix_clock OK
posix_cond OK
posix_fork OK
mutex_trylock not supported
posix_mutex OK
posix_select OK
rtdm skipped (no kernel support)
sched_quota skipped (no kernel support)
sched_tp skipped (no kernel support)
sigdebug skipped (no kernel support)
timerfd OK
vdso_access OK
xddp skipped (no kernel support)
++ /usr/xenomai/bin/clocktest -D -T 30 -C CLOCK_HOST_REALTIME
XNVDSO_FEAT_HOST_REALTIME not available
== Testing built-in CLOCK_HOST_REALTIME (32)
CPU      ToD offset [us] ToD drift [us/s]      warps max delta [us]
--- -------------------- ---------------- ---------- --------------
  0                  0.0            0.000          0            0.0
clock_gettime failed for clock id 32
XNVDSO_FEAT_HOST_REALTIME not available
++ /usr/xenomai/bin/clocktest -T 30
== Testing built-in CLOCK_REALTIME (0)
CPU      ToD offset [us] ToD drift [us/s]      warps max delta [us]
--- -------------------- ---------------- ---------- --------------
  0  -1404413153131434.2            0.028          0            0.0
++ /usr/xenomai/bin/switchtest -T 30
== Testing FPU check routines...
== FPU check routines: unimplemented, skipping FPU switches tests.
== Threads: sleeper-0 rtk-1 rtk-2 rtup-3 rtup-4 rtus-5 rtus-6 rtuo-7 rtuo-8
RTT|  00:00:01
RTH|ctx switches|-------total
RTD|        8208|        8208
RTD|        8208|       16416
RTD|        8210|       24626
RTD|        8208|       32834
RTD|        8208|       41042
RTD|        7961|       49003
RTD|        8208|       57211
RTD|        8210|       65421
RTD|        8210|       73631
RTD|        8231|       81862
RTD|        8208|       90070
RTD|        8185|       98255
RTD|        8208|      106463
RTD|        8154|      114617
RTD|        8213|      122830
RTD|        8208|      131038
RTD|        8208|      139246
RTD|        8210|      147456
RTD|        8208|      155664
RTD|        8208|      163872
RTD|        8210|      172082
RTT|  00:00:22
RTH|ctx switches|-------total
RTD|        8249|      180331
RTD|        8210|      188541
RTD|        8208|      196749
RTD|        8208|      204957
RTD|        8210|      213167
RTD|        8231|      221398
RTD|        8208|      229606
RTD|        8210|      237816
RTD|        7992|      245808
++ start_load
++ echo start_load
++ check_alive /usr/xenomai/bin/latency
++ echo check_alive /usr/xenomai/bin/latency
++ wait_load
++ read rc
Started child 1630: dohell 900
Started child 1635: /usr/xenomai/bin/latency
== Sampling period: 1000 us
== Test mode: periodic user-mode task
== All results in microseconds
warming up...
RTT|  00:00:01  (periodic user-mode task, 1000 us period, priority 99)
RTH|----lat min|----lat avg|----lat max|-overrun|---msw|---lat best|--lat worst
RTD|      4.167|      7.106|     16.667|       0|     0|      4.167|     16.667
RTD|      4.167|      7.116|     18.334|       0|     0|      4.167|     18.334
RTD|      4.167|      6.843|     12.500|       0|     0|      4.167|     18.334
RTD|      4.167|      6.939|     13.334|       0|     0|      4.167|     18.334
RTD|      4.167|      6.937|     17.500|       0|     0|      4.167|     18.334
...
...
...
RTT|  00:06:19  (periodic user-mode task, 1000 us period, priority 99)
RTH|----lat min|----lat avg|----lat max|-overrun|---msw|---lat best|--lat worst
RTD|      4.167|      6.143|     11.667|       0|     0|      3.334|     60.000
RTD|      4.167|      6.214|     13.334|       0|     0|      3.334|     60.000
RTD|      4.167|      6.213|     11.667|       0|     0|      3.334|     60.000
RTD|      4.167|      6.241|     12.500|       0|     0|      3.334|     60.000
RTD|      4.167|      6.126|     11.667|       0|     0|      3.334|     60.000
RTD|      4.167|      6.206|     12.500|       0|     0|      3.334|     60.000
RTD|      4.167|      5.984|     12.500|       0|     0|      3.334|     60.000
RTD|      4.167|      6.132|     13.334|       0|     0|      3.334|     60.000



[0] https://www.96boards.org/products/ce/hikey/
[1] https://github.com/96boards/linux

Re: [Xenomai] xenomai/ipipe arm64 port

[Jorge Ramirez Ortiz]

On 09/01/2015 02:27 PM, Dmitriy Cherkasov wrote:
> We have seen this before, but so far have not found a way to consistently
> reproduce it.
> 
> There are some caveats with the current way context switching is done in Xenomai.
> 
> On armv7, there was a completely separate implementation of xnarch_switch_to(),
> as well as separate fpu and TLS switching. At this stage for armv8 however, we
> are doing a direct call to the kernel's __switch_to(), which then performs the
> relevant register switches.
> 
> Another possibility is that this may be lurking in the mm switch. During calls 
> to check_and_switch_context(), the interrupt state may not always
> match what is expected according to the comments in the original
> version of that function. 
> 
> Overall, this mechanism could definitely benefit from further review.
> 


Thanks Dimitry.
I'll also try to look into this for this specific platform (however I am hopping
that either Gilles or Philippe can spend some time on this soon)

Just for completeness, as things stand this is pretty much that value I am
seeing over consecutive runs.

RTD|      4.167|      6.099|     11.667|       0|     0|      3.333|     66.667
RTD|      4.167|      6.217|     13.333|       0|     0|      3.333|     66.667
RTD|      4.167|      6.259|     11.667|       0|     0|      3.333|     66.667
RTD|      4.167|      6.226|     12.500|       0|     0|      3.333|     66.667
RTD|      4.167|      6.105|     11.667|       0|     0|      3.333|     66.667
RTD|      4.167|      6.181|     12.500|       0|     0|      3.333|     66.667
RTD|      4.167|      6.232|     12.500|       0|     0|      3.333|     66.667
RTD|      4.167|      6.483|     20.000|       0|     0|      3.333|     66.667
RTD|      4.167|      6.409|      9.167|       0|     0|      3.333|     66.667
RTD|      4.167|      5.707|     12.500|       0|     0|      3.333|     66.667
RTD|      4.167|      5.026|      6.667|       0|     0|      3.333|     66.667
RTT|  00:15:04  (periodic user-mode task, 1000 us period, priority 99)
RTH|----lat min|----lat avg|----lat max|-overrun|---msw|---lat best|--lat worst
RTD|      4.167|      5.025|      6.667|       0|     0|      3.333|     66.667
Load script terminated, terminating checked scripts
---|-----------|-----------|-----------|--------|------|-------------------------
RTS|      3.333|      6.540|     66.667|       0|     0|    00:15:04/00:15:04
pipe_in: /tmp/xeno-test-in-9797

-- 
jro

Re: [Xenomai] xenomai/ipipe arm64 port

[Philippe Gerum]

On 09/01/2015 09:11 PM, Jorge Ramirez Ortiz wrote:
> On 09/01/2015 02:27 PM, Dmitriy Cherkasov wrote:
>> We have seen this before, but so far have not found a way to consistently
>> reproduce it.
>>
>> There are some caveats with the current way context switching is done in Xenomai.
>>
>> On armv7, there was a completely separate implementation of xnarch_switch_to(),
>> as well as separate fpu and TLS switching. At this stage for armv8 however, we
>> are doing a direct call to the kernel's __switch_to(), which then performs the
>> relevant register switches.
>>
>> Another possibility is that this may be lurking in the mm switch. During calls 
>> to check_and_switch_context(), the interrupt state may not always
>> match what is expected according to the comments in the original
>> version of that function. 
>>
>> Overall, this mechanism could definitely benefit from further review.
>>
> 
> 
> Thanks Dimitry.
> I'll also try to look into this for this specific platform (however I am hopping
> that either Gilles or Philippe can spend some time on this soon)

No arm64 hw here, so unless this is easily reproducible on a vm, I can
only help with code review. Did you enable
CONFIG_XENO_ARCH_UNLOCKED_SWITCH? If so, any change when turning it off?

> 
> Just for completeness, as things stand this is pretty much that value I am
> seeing over consecutive runs.
> 
> RTD|      4.167|      6.099|     11.667|       0|     0|      3.333|     66.667

This latency seems high for such class of hardware, especially with no
load. We'll need the tracer to dig this too.

-- 
Philippe.

Re: [Xenomai] xenomai/ipipe arm64 port

[Jorge Ramirez Ortiz]

>>
>> Thanks Dimitry.
>> I'll also try to look into this for this specific platform (however I am hopping
>> that either Gilles or Philippe can spend some time on this soon)
> 
> No arm64 hw here, so unless this is easily reproducible on a vm, I can
> only help with code review. Did you enable
> CONFIG_XENO_ARCH_UNLOCKED_SWITCH? If so, any change when turning it off?

I'll try to send you one of the boards from Cambridge.
I could give you access to my lab here but disabling power management will end
up overheating the CPU and with the early kernel release I am not sure how safe
it would be just relying on the thermal monitor (I have seen it kicking in so it
should work but I'd rather not trust it too much yet)

It shouldn't take more than a few days anyway

CONFIG_XENO_ARCH_UNLOCKED_SWITCH id not set

# Machine/platform-specific options



#



# CONFIG_XENO_ARCH_UNLOCKED_SWITCH is not set



# CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH is not set

> 
>>
>> Just for completeness, as things stand this is pretty much that value I am
>> seeing over consecutive runs.
>>
>> RTD|      4.167|      6.099|     11.667|       0|     0|      3.333|     66.667
> 
> This latency seems high for such class of hardware, especially with no
> load. We'll need the tracer to dig this too.
> 

Don mentioned that the trace is not ready yet.


-- 
jro

Re: [Xenomai] xenomai/ipipe arm64 port

[Hongfei Cheng]

On Tue, Sep 1, 2015 at 1:14 PM, Jorge Ramirez Ortiz <jro@xenomai.org> wrote:
>
> Don mentioned that the trace is not ready yet.

We do have the ipipe tracer patch for arm64. We've been running it on
a vendor-specific platform. Will send out the patch after testing it
on QEMU.

Hongfei

Re: [Xenomai] xenomai/ipipe arm64 port

[Don Mahurin]

On Tue, Sep 1, 2015 at 1:14 PM, Jorge Ramirez Ortiz <jro@xenomai.org> wrote:

>
>
> > This latency seems high for such class of hardware, especially with no
> > load. We'll need the tracer to dig this too.
> >
>
> Don mentioned that the trace is not ready yet.
>

I updated the branches on our server:
xenomai - arm64
ipipe - ipipe-3.14.44-arm64

These branches are base on the branches of the same names created on
xenomai.org (Thanks Philippe). Added are a few changes required for ipipe
tracing.

For function tracing, a couple changes were required to be cherry picked
from 3.18.

I can submit our changes as pull requests over the next few days. In the
mean time, the changes are available for review.

-Don

Re: [Xenomai] xenomai/ipipe arm64 port

[Philippe Gerum]

On 09/02/2015 02:43 AM, Don Mahurin wrote:
> ipipe - ipipe-3.14.44-arm64
> 

This branch is available as is, as devel/ipipe-3.14.44-arm64 in the
mainline tree.

Thanks,

-- 
Philippe.

Re: [Xenomai] xenomai/ipipe arm64 port

[Dmitriy Cherkasov]

Hello,

I've added some basic FPU support. It does not do lazy switching, since 
this is not supported in the arm64 kernel yet. I've tested it 
successfully with switchtest. Below is a pull request if you guys are 
interested in picking this up.

-Dmitriy

The following changes since commit 17095784c6d3d44dc7f1512ffab9bb957e298466:

   cobalt/arm64: leave mm tracking to the pipeline (2015-09-17 15:08:34 
+0200)

are available in the git repository at:

   http://gitlab.mperpetuo.com/it/xenomai-3.git arm64-fp

for you to fetch changes up to 4825d1c170b773e82f90dd15409dde489447301e:

   cobalt/arm64: add basic FPU support (2015-09-24 12:04:42 -0700)

----------------------------------------------------------------
Dmitriy Cherkasov (1):
       cobalt/arm64: add basic FPU support

  kernel/cobalt/arch/arm64/Kconfig                   |   2 +-
  .../cobalt/arch/arm64/include/asm/xenomai/fptest.h |  10 +-
  .../cobalt/arch/arm64/include/asm/xenomai/thread.h |  17 +-
  .../arch/arm64/include/asm/xenomai/uapi/fptest.h   |  80 +++--
  kernel/cobalt/arch/arm64/thread.c                  | 329 
++++++---------------
  lib/cobalt/arch/arm64/features.c                   |  22 +-
  6 files changed, 146 insertions(+), 314 deletions(-)

Re: [Xenomai] xenomai/ipipe arm64 port

[Gilles Chanteperdrix]

On Thu, Sep 24, 2015 at 12:39:38PM -0700, Dmitriy Cherkasov wrote:
> Hello,
> 
> I've added some basic FPU support. It does not do lazy switching, since 
> this is not supported in the arm64 kernel yet.

Xenomai only does lazy switching for the first use of FPU by a task,
but then switches eagerly: a user-space task is always created
without the XNFPU bit. Its first use of FPU causes a fault, which
then switches FPU context, and set the XNFPU bit. Starting from
there, every context switch to this tasks switches the FPU context.

> I've tested it 
> successfully with switchtest. Below is a pull request if you guys are 
> interested in picking this up.
> 
> -Dmitriy
> 
> The following changes since commit 17095784c6d3d44dc7f1512ffab9bb957e298466:
> 
>    cobalt/arm64: leave mm tracking to the pipeline (2015-09-17 15:08:34 
> +0200)
> 
> are available in the git repository at:
> 
>    http://gitlab.mperpetuo.com/it/xenomai-3.git arm64-fp
> 
> for you to fetch changes up to 4825d1c170b773e82f90dd15409dde489447301e:
> 
>    cobalt/arm64: add basic FPU support (2015-09-24 12:04:42 -0700)

Just had a look. It looks fine, except that:

- you should not set the XNFPU bit in xnarch_init_shadow_tcb,
this has been done by the upper layers
(kernel/cobalt/posix/thread.c, function pthread_create) for user
threads, and some kernel threads do not set the XNFPU bit when they
do not use the FPU

- you do not need to set it in xnarch_switch_fpu either:
xnarch_switch_fpu is only called for tasks which have the XNFPU bit
set (kernel/cobalt/thread.c function xnthread_switch_fpu).

- the context switch should set or clear a hardware bit somewhere to
cause any use of the FPU to cause a trap: some threads do not have
the XNFPU bit (some kernel threads), and you want any FPU use by
these threads to cause a trap rather than clobbering the fpu
context. Note that switchtest does not allow to test this case. This
bit should be flipped when loading the FPU context for a thread
which has the XNFPU bit.

Note that the last point requires modifying the I-pipe for arm64 to
emit I-pipe events when entering a fault caused by any FPU use. But
this modification is mandatory anyway, you also need it to handle
gracefully the FPU exceptions (div by zero, etc...).

And once you have that modification done, modifying the code to do
lazy switching for the first fault is not hard.

If you agree, I can propose patches for I-pipe and Xenomai to do
that, and let you test if it works.

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] xenomai/ipipe arm64 port

[Dmitriy Cherkasov]

Hi Gilles,

Thank you for the feedback.

Yes, I saw that the armv7 design had trapped FPU usage and switched 
context then. While this is good for performance of threads that never 
use the FPU, it may introduce a one-time delay for threads that do, 
which may not be desirable in a real-time system.

I haven't had a chance to measure the specific delay, so I am not sure 
what the real impact is.

Also, since Linux does not currently support lazy switching, we would 
have to make sure the FPU is enabled (meaning FPU access traps are 
disabled) every time we switch back to the kernel. As far as I can tell, 
the FPU in Linux is always on for arm64. This is in contrast to armv7, 
where it was okay to return to the kernel with FPU traps enabled.

-Dmitriy

On 09/25/2015 08:02 AM, Gilles Chanteperdrix wrote:
> On Thu, Sep 24, 2015 at 12:39:38PM -0700, Dmitriy Cherkasov wrote:
>> Hello,
>>
>> I've added some basic FPU support. It does not do lazy switching, since
>> this is not supported in the arm64 kernel yet.
> Xenomai only does lazy switching for the first use of FPU by a task,
> but then switches eagerly: a user-space task is always created
> without the XNFPU bit. Its first use of FPU causes a fault, which
> then switches FPU context, and set the XNFPU bit. Starting from
> there, every context switch to this tasks switches the FPU context.
>
>> I've tested it
>> successfully with switchtest. Below is a pull request if you guys are
>> interested in picking this up.
>>
>> -Dmitriy
>>
>> The following changes since commit 17095784c6d3d44dc7f1512ffab9bb957e298466:
>>
>>     cobalt/arm64: leave mm tracking to the pipeline (2015-09-17 15:08:34
>> +0200)
>>
>> are available in the git repository at:
>>
>>     http://gitlab.mperpetuo.com/it/xenomai-3.git arm64-fp
>>
>> for you to fetch changes up to 4825d1c170b773e82f90dd15409dde489447301e:
>>
>>     cobalt/arm64: add basic FPU support (2015-09-24 12:04:42 -0700)
> Just had a look. It looks fine, except that:
>
> - you should not set the XNFPU bit in xnarch_init_shadow_tcb,
> this has been done by the upper layers
> (kernel/cobalt/posix/thread.c, function pthread_create) for user
> threads, and some kernel threads do not set the XNFPU bit when they
> do not use the FPU
>
> - you do not need to set it in xnarch_switch_fpu either:
> xnarch_switch_fpu is only called for tasks which have the XNFPU bit
> set (kernel/cobalt/thread.c function xnthread_switch_fpu).
>
> - the context switch should set or clear a hardware bit somewhere to
> cause any use of the FPU to cause a trap: some threads do not have
> the XNFPU bit (some kernel threads), and you want any FPU use by
> these threads to cause a trap rather than clobbering the fpu
> context. Note that switchtest does not allow to test this case. This
> bit should be flipped when loading the FPU context for a thread
> which has the XNFPU bit.
>
> Note that the last point requires modifying the I-pipe for arm64 to
> emit I-pipe events when entering a fault caused by any FPU use. But
> this modification is mandatory anyway, you also need it to handle
> gracefully the FPU exceptions (div by zero, etc...).
>
> And once you have that modification done, modifying the code to do
> lazy switching for the first fault is not hard.
>
> If you agree, I can propose patches for I-pipe and Xenomai to do
> that, and let you test if it works.
>

Re: [Xenomai] xenomai/ipipe arm64 port

[Gilles Chanteperdrix]

On Fri, Sep 25, 2015 at 10:14:14AM -0700, Dmitriy Cherkasov wrote:
> Hi Gilles,
> 
> Thank you for the feedback.
> 
> Yes, I saw that the armv7 design had trapped FPU usage and switched 
> context then. While this is good for performance of threads that never 
> use the FPU, it may introduce a one-time delay for threads that do, 
> which may not be desirable in a real-time system.
> 
> I haven't had a chance to measure the specific delay, so I am not sure 
> what the real impact is.

You can avoid that and only switch FPU for threads which have XNFPU,
that is what we did in xenomai 2.x). The cost of your solution is an
FPU switch for every context switch, whereas some threads, and
especially kernel-space threads, will never use the FPU.

> 
> Also, since Linux does not currently support lazy switching, we would 
> have to make sure the FPU is enabled (meaning FPU access traps are 
> disabled) every time we switch back to the kernel. As far as I can tell, 
> the FPU in Linux is always on for arm64. This is in contrast to armv7, 
> where it was okay to return to the kernel with FPU traps enabled.

That does not look like a problem.

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] xenomai/ipipe arm64 port

[Gilles Chanteperdrix]

On Fri, Sep 25, 2015 at 08:01:37PM +0200, Gilles Chanteperdrix wrote:
> On Fri, Sep 25, 2015 at 10:14:14AM -0700, Dmitriy Cherkasov wrote:
> > Hi Gilles,
> > 
> > Thank you for the feedback.
> > 
> > Yes, I saw that the armv7 design had trapped FPU usage and switched 
> > context then. While this is good for performance of threads that never 
> > use the FPU, it may introduce a one-time delay for threads that do, 
> > which may not be desirable in a real-time system.
> > 
> > I haven't had a chance to measure the specific delay, so I am not sure 
> > what the real impact is.
> 
> You can avoid that and only switch FPU for threads which have XNFPU,
> that is what we did in xenomai 2.x). The cost of your solution is an
> FPU switch for every context switch, whereas some threads, and
> especially kernel-space threads, will never use the FPU.
> 
> > 
> > Also, since Linux does not currently support lazy switching, we would 
> > have to make sure the FPU is enabled (meaning FPU access traps are 
> > disabled) every time we switch back to the kernel. As far as I can tell, 
> > the FPU in Linux is always on for arm64. This is in contrast to armv7, 
> > where it was okay to return to the kernel with FPU traps enabled.
> 
> That does not look like a problem.

Ok, I feel I have not been completely clear. There are three
questions: 

- whether every context switch should switch the fpu context: your
answer is yes, my answer is no: simply remove the line forcibly
setting the XNFPU bit in xnarch_init_shadow_tcb and your port will
only switch the FPU context for those tasks that have the XNFPU bit
set. All user-space tasks have the XNFPU bit set, and some
kernel-space tasks. And if you do that it would be nice to forbid
using the FPU when entering a task which does not have the FPU bit
set.

- whether xnarch_init_shadow_tcb should forcibly clear the XNFPU
bit, and the XNFPU bit be enabled upon first use of the FPU, your
answer is no, my answer is yes. But in fact, the x86 and arm 32 bits
could be the default, and the tasks which do not want to pay the
price of a trap upon first use of the FPU could trigger an
initialization of the FPU with pthread_set_mode_np for instance.

- whether the FPU faults should emit I-pipe events: here there is no
other answer possible than yes, because otherwise the port is broken
in case of FPU exception (division by zero, etc...).

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] xenomai/ipipe arm64 port

[Dmitriy Cherkasov]

On Sat, Sep 26, 2015 at 4:24 AM, Gilles Chanteperdrix <
gilles.chanteperdrix@xenomai.org> wrote:

>
> Ok, I feel I have not been completely clear. There are three
> questions:
>
> - whether every context switch should switch the fpu context: your
> answer is yes, my answer is no: simply remove the line forcibly
> setting the XNFPU bit in xnarch_init_shadow_tcb and your port will
> only switch the FPU context for those tasks that have the XNFPU bit
> set. All user-space tasks have the XNFPU bit set, and some
> kernel-space tasks. And if you do that it would be nice to forbid
> using the FPU when entering a task which does not have the FPU bit
> set.


> - whether xnarch_init_shadow_tcb should forcibly clear the XNFPU
> bit, and the XNFPU bit be enabled upon first use of the FPU, your
> answer is no, my answer is yes. But in fact, the x86 and arm 32 bits
> could be the default, and the tasks which do not want to pay the
> price of a trap upon first use of the FPU could trigger an
> initialization of the FPU with pthread_set_mode_np for instance.
>

To be accurate, my answer is not "no", but more of a "not yet".

My original goal is to get it working without lazy switching, then optimize
later if needed.


>
> - whether the FPU faults should emit I-pipe events: here there is no
> other answer possible than yes, because otherwise the port is broken
> in case of FPU exception (division by zero, etc...).
>
>
Yes, this looks correct, and thank you for the patch. I've applied it to
ipipe and made the relevant changes to Xenomai. I will push this to our
gitlab shortly.

However, armv8 does not trap divide by zero or overflows, and does not seem
to trap other things like sqrt(-1) either (at least on A53). So currently
I've been unable to exercise this code.

-Dmitriy

Re: [Xenomai] xenomai/ipipe arm64 port

[Gilles Chanteperdrix]

On Mon, Sep 28, 2015 at 04:57:28PM -0700, Dmitriy Cherkasov wrote:
> On Sat, Sep 26, 2015 at 4:24 AM, Gilles Chanteperdrix <
> gilles.chanteperdrix@xenomai.org> wrote:
> 
> >
> > Ok, I feel I have not been completely clear. There are three
> > questions:
> >
> > - whether every context switch should switch the fpu context: your
> > answer is yes, my answer is no: simply remove the line forcibly
> > setting the XNFPU bit in xnarch_init_shadow_tcb and your port will
> > only switch the FPU context for those tasks that have the XNFPU bit
> > set. All user-space tasks have the XNFPU bit set, and some
> > kernel-space tasks. And if you do that it would be nice to forbid
> > using the FPU when entering a task which does not have the FPU bit
> > set.
> 
> 
> > - whether xnarch_init_shadow_tcb should forcibly clear the XNFPU
> > bit, and the XNFPU bit be enabled upon first use of the FPU, your
> > answer is no, my answer is yes. But in fact, the x86 and arm 32 bits
> > could be the default, and the tasks which do not want to pay the
> > price of a trap upon first use of the FPU could trigger an
> > initialization of the FPU with pthread_set_mode_np for instance.
> >
> 
> To be accurate, my answer is not "no", but more of a "not yet".
> 
> My original goal is to get it working without lazy switching, then optimize
> later if needed.
> 
> 
> >
> > - whether the FPU faults should emit I-pipe events: here there is no
> > other answer possible than yes, because otherwise the port is broken
> > in case of FPU exception (division by zero, etc...).
> >
> >
> Yes, this looks correct, and thank you for the patch. I've applied it to
> ipipe and made the relevant changes to Xenomai. I will push this to our
> gitlab shortly.
> 
> However, armv8 does not trap divide by zero or overflows, and does not seem
> to trap other things like sqrt(-1) either (at least on A53). So currently
> I've been unable to exercise this code.

Ok, this page:
http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.den0024a/CJHECGIH.html

seems to indicate that armv8 may or may not trap upon
exception. Or maybe the exceptions can be enabled with the FPCR
register? Or the glibc functions defined in the fenv.h header?

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] xenomai/ipipe arm64 port

[Jorge Ramirez Ortiz]

On 09/28/2015 08:12 PM, Gilles Chanteperdrix wrote:
> On Mon, Sep 28, 2015 at 04:57:28PM -0700, Dmitriy Cherkasov wrote:
>> On Sat, Sep 26, 2015 at 4:24 AM, Gilles Chanteperdrix <
>> gilles.chanteperdrix@xenomai.org> wrote:
>>
>>>
>>> Ok, I feel I have not been completely clear. There are three
>>> questions:
>>>
>>> - whether every context switch should switch the fpu context: your
>>> answer is yes, my answer is no: simply remove the line forcibly
>>> setting the XNFPU bit in xnarch_init_shadow_tcb and your port will
>>> only switch the FPU context for those tasks that have the XNFPU bit
>>> set. All user-space tasks have the XNFPU bit set, and some
>>> kernel-space tasks. And if you do that it would be nice to forbid
>>> using the FPU when entering a task which does not have the FPU bit
>>> set.
>>
>>
>>> - whether xnarch_init_shadow_tcb should forcibly clear the XNFPU
>>> bit, and the XNFPU bit be enabled upon first use of the FPU, your
>>> answer is no, my answer is yes. But in fact, the x86 and arm 32 bits
>>> could be the default, and the tasks which do not want to pay the
>>> price of a trap upon first use of the FPU could trigger an
>>> initialization of the FPU with pthread_set_mode_np for instance.
>>>
>>
>> To be accurate, my answer is not "no", but more of a "not yet".
>>
>> My original goal is to get it working without lazy switching, then optimize
>> later if needed.
>>
>>
>>>
>>> - whether the FPU faults should emit I-pipe events: here there is no
>>> other answer possible than yes, because otherwise the port is broken
>>> in case of FPU exception (division by zero, etc...).
>>>
>>>
>> Yes, this looks correct, and thank you for the patch. I've applied it to
>> ipipe and made the relevant changes to Xenomai. I will push this to our
>> gitlab shortly.
>>
>> However, armv8 does not trap divide by zero or overflows, and does not seem
>> to trap other things like sqrt(-1) either (at least on A53). So currently
>> I've been unable to exercise this code.
> 
> Ok, this page:
> http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.den0024a/CJHECGIH.html
> 
> seems to indicate that armv8 may or may not trap upon
> exception. Or maybe the exceptions can be enabled with the FPCR
> register? Or the glibc functions defined in the fenv.h header?
> 

the way it is worded it appears to be a decision left to the SoC vendor.
I'll post the question to support@arm.com for confirmation.

-- 
jro

Re: [Xenomai] xenomai/ipipe arm64 port

[Dmitriy Cherkasov]

On 09/29/2015 05:54 AM, Jorge Ramirez Ortiz wrote:
> On 09/28/2015 08:12 PM, Gilles Chanteperdrix wrote:
>> On Mon, Sep 28, 2015 at 04:57:28PM -0700, Dmitriy Cherkasov wrote:
>> Ok, this page:
>> http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.den0024a/CJHECGIH.html
>>
>> seems to indicate that armv8 may or may not trap upon
>> exception. Or maybe the exceptions can be enabled with the FPCR
>> register? Or the glibc functions defined in the fenv.h header?
>>
> the way it is worded it appears to be a decision left to the SoC vendor.
> I'll post the question to support@arm.com for confirmation.
>
 From some earlier docs, it looks like it was possible to toggle 
trapping with FPCR, but as Lennart mentioned those bits appear to now be 
reserved. On our hardware, setting them has no effect.


I also enjoyed this part:

Both floating-point and NEON are required in all standard ARMv8 
implementations. However, implementations targeting specialized markets 
may support the following combinations:

  *

    No NEON or floating-point

Re: [Xenomai] xenomai/ipipe arm64 port

[Gilles Chanteperdrix]

On Tue, Sep 29, 2015 at 10:31:29AM -0700, Dmitriy Cherkasov wrote:
> 
> 
> On 09/29/2015 05:54 AM, Jorge Ramirez Ortiz wrote:
> > On 09/28/2015 08:12 PM, Gilles Chanteperdrix wrote:
> >> On Mon, Sep 28, 2015 at 04:57:28PM -0700, Dmitriy Cherkasov wrote:
> >> Ok, this page:
> >> http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.den0024a/CJHECGIH.html
> >>
> >> seems to indicate that armv8 may or may not trap upon
> >> exception. Or maybe the exceptions can be enabled with the FPCR
> >> register? Or the glibc functions defined in the fenv.h header?
> >>
> > the way it is worded it appears to be a decision left to the SoC vendor.
> > I'll post the question to support@arm.com for confirmation.
> >
>  From some earlier docs, it looks like it was possible to toggle 
> trapping with FPCR, but as Lennart mentioned those bits appear to now be 
> reserved. On our hardware, setting them has no effect.
> 
> 
> I also enjoyed this part:
> 
> Both floating-point and NEON are required in all standard ARMv8 
> implementations. However, implementations targeting specialized markets 
> may support the following combinations:
> 
>   *
> 
>     No NEON or floating-point

Well, I guess there is a difference between "standard" and
"implementation targeting specialized markets".

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] xenomai/ipipe arm64 port

[Jorge Ramirez Ortiz]

On 09/29/2015 01:47 PM, Gilles Chanteperdrix wrote:
> On Tue, Sep 29, 2015 at 10:31:29AM -0700, Dmitriy Cherkasov wrote:
>>
>>
>> On 09/29/2015 05:54 AM, Jorge Ramirez Ortiz wrote:
>>> On 09/28/2015 08:12 PM, Gilles Chanteperdrix wrote:
>>>> On Mon, Sep 28, 2015 at 04:57:28PM -0700, Dmitriy Cherkasov wrote:
>>>> Ok, this page:
>>>> http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.den0024a/CJHECGIH.html
>>>>
>>>> seems to indicate that armv8 may or may not trap upon
>>>> exception. Or maybe the exceptions can be enabled with the FPCR
>>>> register? Or the glibc functions defined in the fenv.h header?
>>>>
>>> the way it is worded it appears to be a decision left to the SoC vendor.
>>> I'll post the question to support@arm.com for confirmation.
>>>
>>  From some earlier docs, it looks like it was possible to toggle 
>> trapping with FPCR, but as Lennart mentioned those bits appear to now be 
>> reserved. On our hardware, setting them has no effect.
>>
>>
>> I also enjoyed this part:
>>
>> Both floating-point and NEON are required in all standard ARMv8 
>> implementations. However, implementations targeting specialized markets 
>> may support the following combinations:
>>
>>   *
>>
>>     No NEON or floating-point
> 
> Well, I guess there is a difference between "standard" and
> "implementation targeting specialized markets".
> 

the answer from ARM with respect to exception trapping and FPU :

"Only if the SoC vendor has an ARM architectural license and it was one of their
custom processors, FPU and exception trapping might or might not be present.
All current ARMv8-A processors designed by ARM support floating-point and SIMD
support without exception trapping. "



-- 
jro

Re: [Xenomai] xenomai/ipipe arm64 port

[Don Mahurin]

On Mon, Sep 28, 2015 at 5:12 PM, Gilles Chanteperdrix <
gilles.chanteperdrix@xenomai.org> wrote:

> On Mon, Sep 28, 2015 at 04:57:28PM -0700, Dmitriy Cherkasov wrote:
> > On Sat, Sep 26, 2015 at 4:24 AM, Gilles Chanteperdrix <
> > gilles.chanteperdrix@xenomai.org> wrote:
> >
> > >
> > > Ok, I feel I have not been completely clear. There are three
> > > questions:
> > >
> > > - whether every context switch should switch the fpu context: your
> > > answer is yes, my answer is no: simply remove the line forcibly
> > > setting the XNFPU bit in xnarch_init_shadow_tcb and your port will
> > > only switch the FPU context for those tasks that have the XNFPU bit
> > > set. All user-space tasks have the XNFPU bit set, and some
> > > kernel-space tasks. And if you do that it would be nice to forbid
> > > using the FPU when entering a task which does not have the FPU bit
> > > set.
> >
> >
> > > - whether xnarch_init_shadow_tcb should forcibly clear the XNFPU
> > > bit, and the XNFPU bit be enabled upon first use of the FPU, your
> > > answer is no, my answer is yes. But in fact, the x86 and arm 32 bits
> > > could be the default, and the tasks which do not want to pay the
> > > price of a trap upon first use of the FPU could trigger an
> > > initialization of the FPU with pthread_set_mode_np for instance.
> > >
> >
> > To be accurate, my answer is not "no", but more of a "not yet".
> >
> > My original goal is to get it working without lazy switching, then
> optimize
> > later if needed.
> >
> >
> > >
> > > - whether the FPU faults should emit I-pipe events: here there is no
> > > other answer possible than yes, because otherwise the port is broken
> > > in case of FPU exception (division by zero, etc...).
> > >
> > >
> > Yes, this looks correct, and thank you for the patch. I've applied it to
> > ipipe and made the relevant changes to Xenomai. I will push this to our
> > gitlab shortly.
> >
> > However, armv8 does not trap divide by zero or overflows, and does not
> seem
> > to trap other things like sqrt(-1) either (at least on A53). So currently
> > I've been unable to exercise this code.
>
> Ok, this page:
>
> http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.den0024a/CJHECGIH.html
>
> seems to indicate that armv8 may or may not trap upon
> exception. Or maybe the exceptions can be enabled with the FPCR
> register? Or the glibc functions defined in the fenv.h header?


calling feenableexcept seemed to have no effect.

Re: [Xenomai] xenomai/ipipe arm64 port

[Lennart Sorensen]

On Mon, Sep 28, 2015 at 04:57:28PM -0700, Dmitriy Cherkasov wrote:
> To be accurate, my answer is not "no", but more of a "not yet".
> 
> My original goal is to get it working without lazy switching, then optimize
> later if needed.
> 
> Yes, this looks correct, and thank you for the patch. I've applied it to
> ipipe and made the relevant changes to Xenomai. I will push this to our
> gitlab shortly.
> 
> However, armv8 does not trap divide by zero or overflows, and does not seem
> to trap other things like sqrt(-1) either (at least on A53). So currently
> I've been unable to exercise this code.

Wow, they dropped the exceptions from armv8 and just made those bits
reserved?  Why would they do that?

-- 
Len Sorensen

Re: [Xenomai] xenomai/ipipe arm64 port

[Gilles Chanteperdrix]

On Tue, Sep 29, 2015 at 10:14:39AM -0400, Lennart Sorensen wrote:
> On Mon, Sep 28, 2015 at 04:57:28PM -0700, Dmitriy Cherkasov wrote:
> > To be accurate, my answer is not "no", but more of a "not yet".
> > 
> > My original goal is to get it working without lazy switching, then optimize
> > later if needed.
> > 
> > Yes, this looks correct, and thank you for the patch. I've applied it to
> > ipipe and made the relevant changes to Xenomai. I will push this to our
> > gitlab shortly.
> > 
> > However, armv8 does not trap divide by zero or overflows, and does not seem
> > to trap other things like sqrt(-1) either (at least on A53). So currently
> > I've been unable to exercise this code.
> 
> Wow, they dropped the exceptions from armv8 and just made those bits
> reserved?  Why would they do that?

Well, now that I think about it, not a long time ago, I tried
generating FPU exceptions traps on omap3, omap4 and maybe
at91sama5d3 and discovered these processors could not generate such
traps either.

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] xenomai/ipipe arm64 port

[Dmitriy Cherkasov]

The following changes since commit 17095784c6d3d44dc7f1512ffab9bb957e298466:

   cobalt/arm64: leave mm tracking to the pipeline (2015-09-17 15:08:34 
+0200)

are available in the git repository at:

   http://gitlab.mperpetuo.com/it/xenomai-3.git arm64-fp

for you to fetch changes up to ef3ddd26d4ceb2ceec22e0df996e1215ea9935d5:

   cobalt/arm64: add lazy FPU switching (2015-10-01 16:42:56 -0700)

----------------------------------------------------------------
Dmitriy Cherkasov (3):
       cobalt/arm64: add basic FPU support
       cobalt/arm64: fix fpu exception names
       cobalt/arm64: add lazy FPU switching

  kernel/cobalt/arch/arm64/Kconfig                           |   2 +-
  kernel/cobalt/arch/arm64/include/asm/xenomai/fptest.h      |  11 ++-----
  kernel/cobalt/arch/arm64/include/asm/xenomai/thread.h      |  17 
+++--------
  kernel/cobalt/arch/arm64/include/asm/xenomai/uapi/fptest.h |  83 
+++++++++++++++++++++++++++++++++++----------------
  kernel/cobalt/arch/arm64/machine.c                         |   4 +--
  kernel/cobalt/arch/arm64/thread.c                          | 344 
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---------------------------------------------------------------------------------------------------------------------------------------------
  lib/cobalt/arch/arm64/features.c                           |   4 +--
  7 files changed, 183 insertions(+), 282 deletions(-)

Re: [Xenomai] xenomai/ipipe arm64 port

[Gilles Chanteperdrix]

On Thu, Oct 01, 2015 at 04:51:51PM -0700, Dmitriy Cherkasov wrote:
> The following changes since commit 17095784c6d3d44dc7f1512ffab9bb957e298466:
> 
>    cobalt/arm64: leave mm tracking to the pipeline (2015-09-17 15:08:34 
> +0200)
> 
> are available in the git repository at:
> 
>    http://gitlab.mperpetuo.com/it/xenomai-3.git arm64-fp

Ok, I retrieved your code, some comments inlined:

> /*
> static void enable_fpsimd(void) {
> 	__asm__ __volatile__("mrs x1, cpacr_el1\n\
> 			orr x1, x1, #(0x3 << 20)\n\
> 			msr cpacr_el1, x1\n\
> 			isb" : : : "x1", "memory", "cc");
> }

> static void disable_fpsimd(void) {
> 	__asm__ __volatile__("mrs x1, cpacr_el1\n\
> 				and x1, x1, #~(0x3 << 20)\n\
> 				msr cpacr_el1, x1\n\
> 				isb" : : : "x1", "memory", "cc");
> }

Coding style: braces at the beginning of line, and you probably want to
make these functions inline.

> int xnarch_fault_fpu_p(struct ipipe_trap_data *d)
> {
> 	/* check if this is an FPU access trap to be handled by Xenomai */
> 	if(d->exception == IPIPE_TRAP_FPU_ACC){
> 		return 1;
> 	}
> 	/* FPU already enabled, propagate fault to kernel */
> 	return 0;
> }

return d->exception == IPIPE_TAP_FPU_ACC ?


> static inline struct fpsimd_state *get_fpu_owner(struct xnarchtcb *tcb) {
> 	return &(tcb->core.tsp->fpsimd_state);
> }

coding style, brace again

> void xnarch_leave_root(struct xnthread *root)
> {
> 	struct xnarchtcb *rootcb = xnthread_archtcb(root);
> 	rootcb->fpup = get_fpu_owner(rootcb);
> 	disable_fpsimd();
> }

This is not the place to disable fpu. The context switch is the place.


> void xnarch_save_fpu(struct xnthread *thread)
> {
> 	struct xnarchtcb *tcb = &(thread->tcb);
> 	if (xnarch_fpu_ptr(tcb))
> 		fpsimd_save_state(tcb->fpup);
> }

> void xnarch_switch_fpu(struct xnthread *from, struct xnthread *to)
> {
> 	struct fpsimd_state *const from_fpup = from ? from->tcb.fpup : NULL;
> 	struct fpsimd_state *const to_fpup = to->tcb.fpup;

> 	/*
> 	 * This only gets called if XNFPU flag is set, or if migrating to Linux.
> 	 * In both cases, this means turn on FPU and switch.
> 	 */
> 	enable_fpsimd();

> 	if (xnthread_test_state(to, XNROOT) == 0) {
> 		if (from_fpup == to_fpup)
> 			return;

> 		if (from_fpup)
> 			fpsimd_save_state(from_fpup);

> 		fpsimd_load_state(to_fpup);
> 	}
> 	else {
> 		/* Going to Linux. */
> 		if (from_fpup)
> 			fpsimd_save_state(from_fpup);

> 		fpsimd_load_state(to_fpup);
> 	}

When returning to root, you can also skip reloading the fpu if
from_fpu == to_fpu. If they are equal, the real-time threads have
not touched the fpu context, thus it is still valid and what needs
to be done is to simply reenable the fpu.


> int xnarch_handle_fpu_fault(struct xnthread *from, 
> 			struct xnthread *to, struct ipipe_trap_data *d)
> {
> 	spl_t s;

> 	if (xnthread_test_state(to, XNFPU))
> 		/* FPU is already enabled, probably an exception */
> 	   return 0;

You want to BUG() here, since the fpu exception is unequivocally
related to access to disabled fpu, if this happens, it means there
is a bug somewhere.


> 	xnlock_get_irqsave(&nklock, s);
> 	xnthread_set_state(to, XNFPU);
> 	xnlock_put_irqrestore(&nklock, s);

> 	xnarch_switch_fpu(from, to);

> 	return 1;

> }

> void xnarch_init_shadow_tcb(struct xnthread *thread)
> {
> 	spl_t s;
> 	struct xnarchtcb *tcb = xnthread_archtcb(thread);

> 	tcb->fpup = &(tcb->core.host_task->thread.fpsimd_state);

> 	xnlock_get_irqsave(&nklock, s);
> 	xnthread_clear_state(thread, XNFPU);
> 	xnlock_put_irqrestore(&nklock, s);

> }
> #endif /* CONFIG_XENO_ARCH_FPU */

I am not sure you need taking the nklock here, the thread is in its
initialization process, it is not known by the scheduler yet, so
nothing should be able to change its state concurrently.

> void xnarch_switch_to(struct xnthread *out, struct xnthread *in)
> {
> 	struct xnarchtcb *out_tcb = &out->tcb, *in_tcb = &in->tcb;
> 	struct mm_struct *prev_mm, *next_mm;
> 	struct task_struct *next;

> 	next = in_tcb->core.host_task;
> 	prev_mm = out_tcb->core.active_mm;

> 	next_mm = in_tcb->core.mm;
> 	if (next_mm == NULL) {
> 		in_tcb->core.active_mm = prev_mm;
> 		enter_lazy_tlb(prev_mm, next);
> 	} else {
> 		ipipe_switch_mm_head(prev_mm, next_mm, next);
> 		/*
> 		 * We might be switching back to the root thread,
> 		 * which we preempted earlier, shortly after "current"
> 		 * dropped its mm context in the do_exit() path
> 		 * (next->mm == NULL). In that particular case, the
> 		 * kernel expects a lazy TLB state for leaving the mm.
> 		 */
> 		if (next->mm == NULL)
> 			enter_lazy_tlb(prev_mm, next);
> 	}

> 	xnarch_tls_thread_switch(in_tcb->core.tip->task);
> 	xnarch_contextidr_thread_switch(in_tcb->core.tip->task);

> 	/* check if we need to switch FPU on return to Linux */
> 	if (xnthread_test_state(in, XNROOT) == 1)
> 		xnarch_switch_fpu(out, in);

No, doing this here is wrong. xnarch_switch_fpu is called by the
cobalt core at the right moment.

*
> 	/*
> 	 * Complete any pending TLB or cache maintenance on this CPU in case
> 	 * the thread migrates to a different CPU.
> 	 */
> 	dsb(ish);

> 	cpu_switch_to(out_tcb->core.tip->task, in_tcb->core.tip->task);
> }

You want to disable the fpu before calling cpu_switch_to. If
switching to a thread with XNFPU, xnarch_switch_fpu will be called,
which enables the fpu, and if switching to a thread without XNFPU,
we want the FPU to be disabled so that the task will take a fault
upon its first use of fpu.

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] xenomai/ipipe arm64 port

[Gilles Chanteperdrix]

On Fri, Oct 02, 2015 at 12:01:42PM +0200, Gilles Chanteperdrix wrote:
> On Thu, Oct 01, 2015 at 04:51:51PM -0700, Dmitriy Cherkasov wrote:
> > The following changes since commit 17095784c6d3d44dc7f1512ffab9bb957e298466:
> > 
> >    cobalt/arm64: leave mm tracking to the pipeline (2015-09-17 15:08:34 
> > +0200)
> > 
> > are available in the git repository at:
> > 
> >    http://gitlab.mperpetuo.com/it/xenomai-3.git arm64-fp
> 
> Ok, I retrieved your code, some comments inlined:
> 
> > /*
> > static void enable_fpsimd(void) {
> > 	__asm__ __volatile__("mrs x1, cpacr_el1\n\
> > 			orr x1, x1, #(0x3 << 20)\n\
> > 			msr cpacr_el1, x1\n\
> > 			isb" : : : "x1", "memory", "cc");
> > }
> 
> > static void disable_fpsimd(void) {
> > 	__asm__ __volatile__("mrs x1, cpacr_el1\n\
> > 				and x1, x1, #~(0x3 << 20)\n\
> > 				msr cpacr_el1, x1\n\
> > 				isb" : : : "x1", "memory", "cc");
> > }
> 
> Coding style: braces at the beginning of line, and you probably want to
> make these functions inline.

Also, hardcoding a register may not be a good idea. Especially x1,
since if I understand correctly, it is used as function argument, so
by using x1, you may be forcing the compiler to do some useless
register saves and load. And also, the only instructions which
requires inline assembley are msr, mrs, and isb.

So, I would do something like:

static inline long get_cpacr(void)
{
	long result;
	__asm__ ("mrs %0, cpacr_el1": "=r"(result) : /* */ : "memory", "cc");
	return result;
}

static inline set_cpacr(long val)
{
	__asm__ __volatile__ (
		"msr cpacr_el1, %0\n\"
		"isb" : /* */ : "r"(val) : "memory", "cc");
}

then define enable_fpsimd/disable_fpsimd in plain C, using these
building blocks. Also, I do not see why these instructions require
clobbering memory and cc, but here you know better than I do.

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] xenomai/ipipe arm64 port

[Dmitriy Cherkasov]

The following changes since commit 17095784c6d3d44dc7f1512ffab9bb957e298466:

   cobalt/arm64: leave mm tracking to the pipeline (2015-09-17 15:08:34 
+0200)

are available in the git repository at:

   http://gitlab.mperpetuo.com/it/xenomai-3.git arm64-fp

for you to fetch changes up to 37d9698b827c664027180a07f151346d0979e7d6:

   cobalt/arm64: FPU code cleanup (2015-10-02 13:03:38 -0700)

----------------------------------------------------------------
Dmitriy Cherkasov (4):
       cobalt/arm64: add basic FPU support
       cobalt/arm64: fix fpu exception names
       cobalt/arm64: add lazy FPU switching
       cobalt/arm64: FPU code cleanup

  kernel/cobalt/arch/arm64/Kconfig                           |   2 +-
  kernel/cobalt/arch/arm64/include/asm/xenomai/fptest.h      |  11 +--
  kernel/cobalt/arch/arm64/include/asm/xenomai/thread.h      |  17 +---
  kernel/cobalt/arch/arm64/include/asm/xenomai/uapi/fptest.h |  83 
++++++++++++------
  kernel/cobalt/arch/arm64/machine.c                         |   4 +-
  kernel/cobalt/arch/arm64/thread.c                          | 335 
++++++++++++++++++++++-------------------------------------------------
  lib/cobalt/arch/arm64/features.c                           |   4 +-
  7 files changed, 173 insertions(+), 283 deletions(-)

Re: [Xenomai] xenomai/ipipe arm64 port

[Philippe Gerum]

Hi Dmitriy,

On 10/02/2015 10:18 PM, Dmitriy Cherkasov wrote:
> The following changes since commit
> 17095784c6d3d44dc7f1512ffab9bb957e298466:
> 
>   cobalt/arm64: leave mm tracking to the pipeline (2015-09-17 15:08:34
> +0200)
> 
> are available in the git repository at:
> 
>   http://gitlab.mperpetuo.com/it/xenomai-3.git arm64-fp
> 
> for you to fetch changes up to 37d9698b827c664027180a07f151346d0979e7d6:
> 
>   cobalt/arm64: FPU code cleanup (2015-10-02 13:03:38 -0700)
> 

Which branch is this work based on?

-- 
Philippe.

Re: [Xenomai] xenomai/ipipe arm64 port

[Philippe Gerum]

On 10/03/2015 11:53 AM, Philippe Gerum wrote:
> 
> Hi Dmitriy,
> 
> On 10/02/2015 10:18 PM, Dmitriy Cherkasov wrote:
>> The following changes since commit
>> 17095784c6d3d44dc7f1512ffab9bb957e298466:
>>
>>   cobalt/arm64: leave mm tracking to the pipeline (2015-09-17 15:08:34
>> +0200)
>>
>> are available in the git repository at:
>>
>>   http://gitlab.mperpetuo.com/it/xenomai-3.git arm64-fp
>>
>> for you to fetch changes up to 37d9698b827c664027180a07f151346d0979e7d6:
>>
>>   cobalt/arm64: FPU code cleanup (2015-10-02 13:03:38 -0700)
>>
> 
> Which branch is this work based on?
> 

Ok found out. You seem to be tracking the master branch from xenomai-3,
please rebase over the -next branch.

-- 
Philippe.

Re: [Xenomai] xenomai/ipipe arm64 port

[Philippe Gerum]

On 10/03/2015 12:01 PM, Philippe Gerum wrote:
> On 10/03/2015 11:53 AM, Philippe Gerum wrote:
>>
>> Hi Dmitriy,
>>
>> On 10/02/2015 10:18 PM, Dmitriy Cherkasov wrote:
>>> The following changes since commit
>>> 17095784c6d3d44dc7f1512ffab9bb957e298466:
>>>
>>>   cobalt/arm64: leave mm tracking to the pipeline (2015-09-17 15:08:34
>>> +0200)
>>>
>>> are available in the git repository at:
>>>
>>>   http://gitlab.mperpetuo.com/it/xenomai-3.git arm64-fp
>>>
>>> for you to fetch changes up to 37d9698b827c664027180a07f151346d0979e7d6:
>>>
>>>   cobalt/arm64: FPU code cleanup (2015-10-02 13:03:38 -0700)
>>>
>>
>> Which branch is this work based on?
>>
> 
> Ok found out. You seem to be tracking the master branch from xenomai-3,
> please rebase over the -next branch.
> 

I rebased the arm64 branch containing my recent updates to this port
over -next and pushed back. You may want to rebase over that arm64
branch directly. Thanks,

-- 
Philippe.

Re: [Xenomai] xenomai/ipipe arm64 port

[Philippe Gerum]

On 09/01/2015 09:11 PM, Jorge Ramirez Ortiz wrote:
> On 09/01/2015 02:27 PM, Dmitriy Cherkasov wrote:
>> We have seen this before, but so far have not found a way to consistently
>> reproduce it.
>>
>> There are some caveats with the current way context switching is done in Xenomai.
>>
>> On armv7, there was a completely separate implementation of xnarch_switch_to(),
>> as well as separate fpu and TLS switching. At this stage for armv8 however, we
>> are doing a direct call to the kernel's __switch_to(), which then performs the
>> relevant register switches.
>>
>> Another possibility is that this may be lurking in the mm switch. During calls 
>> to check_and_switch_context(), the interrupt state may not always
>> match what is expected according to the comments in the original
>> version of that function. 
>>
>> Overall, this mechanism could definitely benefit from further review.
>>
> 
> 
> Thanks Dimitry.
> I'll also try to look into this for this specific platform (however I am hopping
> that either Gilles or Philippe can spend some time on this soon)
> 
> Just for completeness, as things stand this is pretty much that value I am
> seeing over consecutive runs.
> 
> RTD|      4.167|      6.099|     11.667|       0|     0|      3.333|     66.667
> RTD|      4.167|      6.217|     13.333|       0|     0|      3.333|     66.667
> RTD|      4.167|      6.259|     11.667|       0|     0|      3.333|     66.667
> RTD|      4.167|      6.226|     12.500|       0|     0|      3.333|     66.667
> RTD|      4.167|      6.105|     11.667|       0|     0|      3.333|     66.667
> RTD|      4.167|      6.181|     12.500|       0|     0|      3.333|     66.667
> RTD|      4.167|      6.232|     12.500|       0|     0|      3.333|     66.667
> RTD|      4.167|      6.483|     20.000|       0|     0|      3.333|     66.667
> RTD|      4.167|      6.409|      9.167|       0|     0|      3.333|     66.667
> RTD|      4.167|      5.707|     12.500|       0|     0|      3.333|     66.667
> RTD|      4.167|      5.026|      6.667|       0|     0|      3.333|     66.667
> RTT|  00:15:04  (periodic user-mode task, 1000 us period, priority 99)
> RTH|----lat min|----lat avg|----lat max|-overrun|---msw|---lat best|--lat worst
> RTD|      4.167|      5.025|      6.667|       0|     0|      3.333|     66.667
> Load script terminated, terminating checked scripts
> ---|-----------|-----------|-----------|--------|------|-------------------------
> RTS|      3.333|      6.540|     66.667|       0|     0|    00:15:04/00:15:04
> pipe_in: /tmp/xeno-test-in-9797
> 

This won't address the current issue, but the notification of the head
domain on memory aborts needs fixing:

diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
index f363368..e59da5e 100644
--- a/arch/arm64/mm/fault.c
+++ b/arch/arm64/mm/fault.c
@@ -517,10 +517,10 @@ asmlinkage void __exception do_mem_abort(unsigned
long addr, unsigned int esr,
 	if (!inf->fn(addr, esr, regs))
 		return;

-	irqflags = ipipe_fault_entry();
-
 	if (__ipipe_report_trap(IPIPE_TRAP_UNKNOWN, regs))
-		goto out;
+		return;
+
+	irqflags = ipipe_fault_entry();

 	pr_alert("Unhandled fault: %s (0x%08x) at 0x%016lx\n",
 		 inf->name, esr, addr);
@@ -530,7 +530,7 @@ asmlinkage void __exception do_mem_abort(unsigned
long addr, unsigned int esr,
 	info.si_code  = inf->code;
 	info.si_addr  = (void __user *)addr;
 	arm64_notify_die("", regs, &info, esr);
-out:
+
 	ipipe_fault_exit(irqflags);
 }

@@ -544,17 +544,17 @@ asmlinkage void __exception
do_sp_pc_abort(unsigned long addr,
 	struct siginfo info;
 	unsigned long irqflags;

-	irqflags = ipipe_fault_entry();
-
 	if (__ipipe_report_trap(IPIPE_TRAP_ALIGNMENT, regs))
-		goto out;
+		return;
+
+	irqflags = ipipe_fault_entry();

 	info.si_signo = SIGBUS;
 	info.si_errno = 0;
 	info.si_code  = BUS_ADRALN;
 	info.si_addr  = (void __user *)addr;
 	arm64_notify_die("", regs, &info, esr);
-out:
+
 	ipipe_fault_exit(irqflags);
 }
-- 
Philippe.

Re: [Xenomai] xenomai/ipipe arm64 port

[Jorge Ramirez Ortiz]

On 09/01/2015 03:30 PM, Philippe Gerum wrote:
> On 09/01/2015 09:11 PM, Jorge Ramirez Ortiz wrote:
>> On 09/01/2015 02:27 PM, Dmitriy Cherkasov wrote:
>>> We have seen this before, but so far have not found a way to consistently
>>> reproduce it.
>>>
>>> There are some caveats with the current way context switching is done in Xenomai.
>>>
>>> On armv7, there was a completely separate implementation of xnarch_switch_to(),
>>> as well as separate fpu and TLS switching. At this stage for armv8 however, we
>>> are doing a direct call to the kernel's __switch_to(), which then performs the
>>> relevant register switches.
>>>
>>> Another possibility is that this may be lurking in the mm switch. During calls 
>>> to check_and_switch_context(), the interrupt state may not always
>>> match what is expected according to the comments in the original
>>> version of that function. 
>>>
>>> Overall, this mechanism could definitely benefit from further review.
>>>
>>
>>
>> Thanks Dimitry.
>> I'll also try to look into this for this specific platform (however I am hopping
>> that either Gilles or Philippe can spend some time on this soon)
>>
>> Just for completeness, as things stand this is pretty much that value I am
>> seeing over consecutive runs.
>>
>> RTD|      4.167|      6.099|     11.667|       0|     0|      3.333|     66.667
>> RTD|      4.167|      6.217|     13.333|       0|     0|      3.333|     66.667
>> RTD|      4.167|      6.259|     11.667|       0|     0|      3.333|     66.667
>> RTD|      4.167|      6.226|     12.500|       0|     0|      3.333|     66.667
>> RTD|      4.167|      6.105|     11.667|       0|     0|      3.333|     66.667
>> RTD|      4.167|      6.181|     12.500|       0|     0|      3.333|     66.667
>> RTD|      4.167|      6.232|     12.500|       0|     0|      3.333|     66.667
>> RTD|      4.167|      6.483|     20.000|       0|     0|      3.333|     66.667
>> RTD|      4.167|      6.409|      9.167|       0|     0|      3.333|     66.667
>> RTD|      4.167|      5.707|     12.500|       0|     0|      3.333|     66.667
>> RTD|      4.167|      5.026|      6.667|       0|     0|      3.333|     66.667
>> RTT|  00:15:04  (periodic user-mode task, 1000 us period, priority 99)
>> RTH|----lat min|----lat avg|----lat max|-overrun|---msw|---lat best|--lat worst
>> RTD|      4.167|      5.025|      6.667|       0|     0|      3.333|     66.667
>> Load script terminated, terminating checked scripts
>> ---|-----------|-----------|-----------|--------|------|-------------------------
>> RTS|      3.333|      6.540|     66.667|       0|     0|    00:15:04/00:15:04
>> pipe_in: /tmp/xeno-test-in-9797
>>
> 
> This won't address the current issue, but the notification of the head
> domain on memory aborts needs fixing:
> 
> diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
> index f363368..e59da5e 100644


applied.
yes I still see the random translation faults (3 in total during the 15 minutes
test run)

feel free to post me suggestions of course. I have been chasing this one for
nearly three days now.

Re: [Xenomai] xenomai/ipipe arm64 port

[Gilles Chanteperdrix]

On Tue, Sep 01, 2015 at 03:11:05PM -0400, Jorge Ramirez Ortiz wrote:
> On 09/01/2015 02:27 PM, Dmitriy Cherkasov wrote:
> > We have seen this before, but so far have not found a way to consistently
> > reproduce it.
> > 
> > There are some caveats with the current way context switching is done in Xenomai.
> > 
> > On armv7, there was a completely separate implementation of
> > xnarch_switch_to(), as well as separate fpu and TLS switching.
> > At this stage for armv8 however, we are doing a direct call to
> > the kernel's __switch_to(), which then performs the relevant
> > register switches.

On arm 32 bits, the only difference between xnarch_switch_to and
switch_to is the FPU switch, IIRC, switch_to triggers a notifier
which I believe can not be called from primary domain. Or maybe it
can, but the FPU switch, can not be the same as Xenomai "FPU state
state machine", does not work the same as Linux.

-- 
					    Gilles.
https://click-hack.org