[Xenomai] ARM bus error due to alignment

[Xenomai] ARM bus error due to alignment

[Lennart Sorensen]

We are trying to move from gcc 4.6 to gcc 4.9 (as part of moving from
being Debian wheezy based to jessie based).

This has now brought up an unexpected change in behaviour on arm (powerpc
is fine).  We are getting bus errors with kernel complaints about
unaligned memory accesses.  Now it is certainly true that we are doing
some unaligned memory accesses, but on armv7 that is supposed to be
allowed, as long as the memory is marked as "normal memory".

gcc 4.6 and earlier never generated unaligned accesses as far as I
understand it, while gcc 4.7 and higher do generate unaligned accesses
by default on armv7 according to the gcc documentation.

Is it expected that the memory in xenomai would not be flagged as normal
memory (the other options as far as I understand it are device memory
and strongly ordered memory).

This is with 3.14 kernel and xenomai 2.6.4 with LPAE enabled on a
Cortex-A15 system.

So same code, same xenomai, same kernel with gcc 4.6 does not see this,
while with 4.9 it does.  I tried building the application with
-nno-unaligned-accesses, but I didn't rebuild the libraries that way,
so that might not be enough.  Really I think it should work fine, since I
can't see why the pages would be flagged as something other than normal
memory (unless DMA pages are somehow getting involved, since some of
this is dealing with network packets being received).

Any ideas where to investigate?

-- 
Len Soirensen

Re: [Xenomai] ARM bus error due to alignment

[Gilles Chanteperdrix]

On Wed, Nov 04, 2015 at 08:51:14PM -0500, Lennart Sorensen wrote:
> We are trying to move from gcc 4.6 to gcc 4.9 (as part of moving from
> being Debian wheezy based to jessie based).
> 
> This has now brought up an unexpected change in behaviour on arm (powerpc
> is fine).  We are getting bus errors with kernel complaints about
> unaligned memory accesses.  Now it is certainly true that we are doing
> some unaligned memory accesses, but on armv7 that is supposed to be
> allowed, as long as the memory is marked as "normal memory".
> 
> gcc 4.6 and earlier never generated unaligned accesses as far as I
> understand it, while gcc 4.7 and higher do generate unaligned accesses
> by default on armv7 according to the gcc documentation.
> 
> Is it expected that the memory in xenomai would not be flagged as normal
> memory (the other options as far as I understand it are device memory
> and strongly ordered memory).
> 
> This is with 3.14 kernel and xenomai 2.6.4 with LPAE enabled on a
> Cortex-A15 system.
> 
> So same code, same xenomai, same kernel with gcc 4.6 does not see this,
> while with 4.9 it does.  I tried building the application with
> -nno-unaligned-accesses, but I didn't rebuild the libraries that way,
> so that might not be enough.  Really I think it should work fine, since I
> can't see why the pages would be flagged as something other than normal
> memory (unless DMA pages are somehow getting involved, since some of
> this is dealing with network packets being received).
> 
> Any ideas where to investigate?

You need some piece of code, somewhere, to program the processor to
not get the fault. I think Linux historically set the processor to
generate a fault so as to be able to control the behaviour with what
is explained in Documentation/arm/mem_alignment, but there is a way
to avoid the fault on armv7. Maybe a kernel parameter, maybe you
need a more recent version of u-boot.

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] ARM bus error due to alignment

[Lennart Sorensen]

On Thu, Nov 05, 2015 at 12:01:41PM +0100, Gilles Chanteperdrix wrote:
> On Wed, Nov 04, 2015 at 08:51:14PM -0500, Lennart Sorensen wrote:
> > We are trying to move from gcc 4.6 to gcc 4.9 (as part of moving from
> > being Debian wheezy based to jessie based).
> > 
> > This has now brought up an unexpected change in behaviour on arm (powerpc
> > is fine).  We are getting bus errors with kernel complaints about
> > unaligned memory accesses.  Now it is certainly true that we are doing
> > some unaligned memory accesses, but on armv7 that is supposed to be
> > allowed, as long as the memory is marked as "normal memory".
> > 
> > gcc 4.6 and earlier never generated unaligned accesses as far as I
> > understand it, while gcc 4.7 and higher do generate unaligned accesses
> > by default on armv7 according to the gcc documentation.
> > 
> > Is it expected that the memory in xenomai would not be flagged as normal
> > memory (the other options as far as I understand it are device memory
> > and strongly ordered memory).
> > 
> > This is with 3.14 kernel and xenomai 2.6.4 with LPAE enabled on a
> > Cortex-A15 system.
> > 
> > So same code, same xenomai, same kernel with gcc 4.6 does not see this,
> > while with 4.9 it does.  I tried building the application with
> > -nno-unaligned-accesses, but I didn't rebuild the libraries that way,
> > so that might not be enough.  Really I think it should work fine, since I
> > can't see why the pages would be flagged as something other than normal
> > memory (unless DMA pages are somehow getting involved, since some of
> > this is dealing with network packets being received).
> > 
> > Any ideas where to investigate?
> 
> You need some piece of code, somewhere, to program the processor to
> not get the fault. I think Linux historically set the processor to
> generate a fault so as to be able to control the behaviour with what
> is explained in Documentation/arm/mem_alignment, but there is a way
> to avoid the fault on armv7. Maybe a kernel parameter, maybe you
> need a more recent version of u-boot.

Hmm, I thought I had read that as of 2.6.38 and higher arm systems were
supposed to not generate the fault anymore, but maybe I misunderstood
it and they meant it sets fixup to enabled by default.

I should check the SCTLR.A flag.

-- 
Len Sorensen

Re: [Xenomai] ARM bus error due to alignment

[Lennart Sorensen]

On Thu, Nov 05, 2015 at 09:51:59AM -0500, Lennart Sorensen wrote:
> Hmm, I thought I had read that as of 2.6.38 and higher arm systems were
> supposed to not generate the fault anymore, but maybe I misunderstood
> it and they meant it sets fixup to enabled by default.
> 
> I should check the SCTLR.A flag.

Of course the arm docs say it is off by default at reset, so something
in either u-boot or the kernel seems like it must have to enable it
explicitly.  Well once I figure out how to read it I guess I will know.

Of course if the memory happens to be flagged as something other than
normal memory, then it should also fault.  I wonder if a page used to dma
data to/from a network driver would be flagged as normal memory or not.

-- 
Len Sorensen

Re: [Xenomai] ARM bus error due to alignment

[Lennart Sorensen]

On Thu, Nov 05, 2015 at 09:57:25AM -0500, Lennart Sorensen wrote:
> On Thu, Nov 05, 2015 at 09:51:59AM -0500, Lennart Sorensen wrote:
> > Hmm, I thought I had read that as of 2.6.38 and higher arm systems were
> > supposed to not generate the fault anymore, but maybe I misunderstood
> > it and they meant it sets fixup to enabled by default.
> > 
> > I should check the SCTLR.A flag.
> 
> Of course the arm docs say it is off by default at reset, so something
> in either u-boot or the kernel seems like it must have to enable it
> explicitly.  Well once I figure out how to read it I guess I will know.
> 
> Of course if the memory happens to be flagged as something other than
> normal memory, then it should also fault.  I wonder if a page used to dma
> data to/from a network driver would be flagged as normal memory or not.

Would using this cause such a problem:

/usr/include/xenomai/native/heap.h:#define H_DMA      0x100     /* Use memory suitable for DMA. */

-- 
Len Sorensen

Re: [Xenomai] ARM bus error due to alignment

[Gilles Chanteperdrix]

On Thu, Nov 05, 2015 at 10:02:26AM -0500, Lennart Sorensen wrote:
> On Thu, Nov 05, 2015 at 09:57:25AM -0500, Lennart Sorensen wrote:
> > On Thu, Nov 05, 2015 at 09:51:59AM -0500, Lennart Sorensen wrote:
> > > Hmm, I thought I had read that as of 2.6.38 and higher arm systems were
> > > supposed to not generate the fault anymore, but maybe I misunderstood
> > > it and they meant it sets fixup to enabled by default.
> > > 
> > > I should check the SCTLR.A flag.
> > 
> > Of course the arm docs say it is off by default at reset, so something
> > in either u-boot or the kernel seems like it must have to enable it
> > explicitly.  Well once I figure out how to read it I guess I will know.
> > 
> > Of course if the memory happens to be flagged as something other than
> > normal memory, then it should also fault.  I wonder if a page used to dma
> > data to/from a network driver would be flagged as normal memory or not.
> 
> Would using this cause such a problem:
> 
> /usr/include/xenomai/native/heap.h:#define H_DMA      0x100     /* Use memory suitable for DMA. */

Do you have alignment issues with the same version of Linux with
exactly the same configuration, but without Xenomai?

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] ARM bus error due to alignment

[Lennart Sorensen]

On Thu, Nov 05, 2015 at 04:15:44PM +0100, Gilles Chanteperdrix wrote:
> On Thu, Nov 05, 2015 at 10:02:26AM -0500, Lennart Sorensen wrote:
> > On Thu, Nov 05, 2015 at 09:57:25AM -0500, Lennart Sorensen wrote:
> > > On Thu, Nov 05, 2015 at 09:51:59AM -0500, Lennart Sorensen wrote:
> > > > Hmm, I thought I had read that as of 2.6.38 and higher arm systems were
> > > > supposed to not generate the fault anymore, but maybe I misunderstood
> > > > it and they meant it sets fixup to enabled by default.
> > > > 
> > > > I should check the SCTLR.A flag.
> > > 
> > > Of course the arm docs say it is off by default at reset, so something
> > > in either u-boot or the kernel seems like it must have to enable it
> > > explicitly.  Well once I figure out how to read it I guess I will know.
> > > 
> > > Of course if the memory happens to be flagged as something other than
> > > normal memory, then it should also fault.  I wonder if a page used to dma
> > > data to/from a network driver would be flagged as normal memory or not.
> > 
> > Would using this cause such a problem:
> > 
> > /usr/include/xenomai/native/heap.h:#define H_DMA      0x100     /* Use memory suitable for DMA. */
> 
> Do you have alignment issues with the same version of Linux with
> exactly the same configuration, but without Xenomai?

Given the only application with alignment issues is a xenomai application,
then I don't know.  All the non xenomai code is running happily though
and the kernel never flags any alignment issues there although I don't
remember if the default kernel setting would even report alignment
problems or just silently fix them.

-- 
Len Sorensen

Re: [Xenomai] ARM bus error due to alignment

[Gilles Chanteperdrix]

On Thu, Nov 05, 2015 at 10:24:27AM -0500, Lennart Sorensen wrote:
> On Thu, Nov 05, 2015 at 04:15:44PM +0100, Gilles Chanteperdrix wrote:
> > On Thu, Nov 05, 2015 at 10:02:26AM -0500, Lennart Sorensen wrote:
> > > On Thu, Nov 05, 2015 at 09:57:25AM -0500, Lennart Sorensen wrote:
> > > > On Thu, Nov 05, 2015 at 09:51:59AM -0500, Lennart Sorensen wrote:
> > > > > Hmm, I thought I had read that as of 2.6.38 and higher arm systems were
> > > > > supposed to not generate the fault anymore, but maybe I misunderstood
> > > > > it and they meant it sets fixup to enabled by default.
> > > > > 
> > > > > I should check the SCTLR.A flag.
> > > > 
> > > > Of course the arm docs say it is off by default at reset, so something
> > > > in either u-boot or the kernel seems like it must have to enable it
> > > > explicitly.  Well once I figure out how to read it I guess I will know.
> > > > 
> > > > Of course if the memory happens to be flagged as something other than
> > > > normal memory, then it should also fault.  I wonder if a page used to dma
> > > > data to/from a network driver would be flagged as normal memory or not.
> > > 
> > > Would using this cause such a problem:
> > > 
> > > /usr/include/xenomai/native/heap.h:#define H_DMA      0x100     /* Use memory suitable for DMA. */
> > 
> > Do you have alignment issues with the same version of Linux with
> > exactly the same configuration, but without Xenomai?
> 
> Given the only application with alignment issues is a xenomai application,
> then I don't know.  All the non xenomai code is running happily though
> and the kernel never flags any alignment issues there although I don't
> remember if the default kernel setting would even report alignment
> problems or just silently fix them.

As I said, that depends on the contents of the /proc file described
in Documentation/arm/mem_alignment

If that file is configured to fixup faults in kernel, then every
alignment fault in user-space triggers a trap, the trap handler
simulates the faulting instruction and returns to user-space. If you
do that from a Xenomai task running in primary mode, it should cause
the task to switch to secondary mode, unless there is a bug in the
I-pipe patch regarding that particular issue (like a fault not
transmitted to Xenomai).

Xenomai does not change the type of the RAM memory, the H_DMA flag
only has an effect on heaps created with Xenomai shared user/kernel
heaps interface. Xenomai only use that flag automatically on arm
machines with VIVT cache. armv7 do not have VIVT cache as far as I
know.

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] ARM bus error due to alignment

[Lennart Sorensen]

On Thu, Nov 05, 2015 at 04:55:49PM +0100, Gilles Chanteperdrix wrote:
> As I said, that depends on the contents of the /proc file described
> in Documentation/arm/mem_alignment
> 
> If that file is configured to fixup faults in kernel, then every
> alignment fault in user-space triggers a trap, the trap handler
> simulates the faulting instruction and returns to user-space. If you
> do that from a Xenomai task running in primary mode, it should cause
> the task to switch to secondary mode, unless there is a bug in the
> I-pipe patch regarding that particular issue (like a fault not
> transmitted to Xenomai).

Well I tried changing that setting to 0 (ignore) and that just crashed
the system instead.

> Xenomai does not change the type of the RAM memory, the H_DMA flag
> only has an effect on heaps created with Xenomai shared user/kernel
> heaps interface. Xenomai only use that flag automatically on arm
> machines with VIVT cache. armv7 do not have VIVT cache as far as I
> know.

Hmm, not sure.

It does appear that when LPAE is enabled H_DMA does do something on arm,
but only when LPAE is enabled since that enables ZONE_DMA.

-- 
Len Sorensen

Re: [Xenomai] ARM bus error due to alignment

[Gilles Chanteperdrix]

On Thu, Nov 05, 2015 at 11:19:02AM -0500, Lennart Sorensen wrote:
> On Thu, Nov 05, 2015 at 04:55:49PM +0100, Gilles Chanteperdrix wrote:
> > As I said, that depends on the contents of the /proc file described
> > in Documentation/arm/mem_alignment
> > 
> > If that file is configured to fixup faults in kernel, then every
> > alignment fault in user-space triggers a trap, the trap handler
> > simulates the faulting instruction and returns to user-space. If you
> > do that from a Xenomai task running in primary mode, it should cause
> > the task to switch to secondary mode, unless there is a bug in the
> > I-pipe patch regarding that particular issue (like a fault not
> > transmitted to Xenomai).
> 
> Well I tried changing that setting to 0 (ignore) and that just crashed
> the system instead.
> 
> > Xenomai does not change the type of the RAM memory, the H_DMA flag
> > only has an effect on heaps created with Xenomai shared user/kernel
> > heaps interface. Xenomai only use that flag automatically on arm
> > machines with VIVT cache. armv7 do not have VIVT cache as far as I
> > know.
> 
> Hmm, not sure.

I am sure. Xenomai does not use H_DMA on armv7, so, it is not used
if your code, it is not used at all.

> 
> It does appear that when LPAE is enabled H_DMA does do something on arm,
> but only when LPAE is enabled since that enables ZONE_DMA.

I never said that H_DMA did nothing.

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] ARM bus error due to alignment

[Lennart Sorensen]

On Thu, Nov 05, 2015 at 05:25:14PM +0100, Gilles Chanteperdrix wrote:
> I am sure. Xenomai does not use H_DMA on armv7, so, it is not used
> if your code, it is not used at all.

Well we are using it for one allocation for talking to a network port.

If I remove it, I get a NULL pointer dereference, so it clearly matters
to something in our code.

> > It does appear that when LPAE is enabled H_DMA does do something on arm,
> > but only when LPAE is enabled since that enables ZONE_DMA.
> 
> I never said that H_DMA did nothing.

OK I misunderstood then.

Setting /proc/cpu/alignment to 0 gives the kernel message:
alignment: ignoring faults is unsafe on this CPU.  Defaulting to fixup mode.

Apparently armv6+ with CR_U set is not allowed to turn off fixups.

The fixups don't work for the xenomai code though it would seem.

I get:
Alignment trap: not handling instruction f8d03002 at [<00020be0>]
Unhandled fault: alignment exception (0x221) at 0xb56ba006

Code looks like this:
   20bd8:       6802            ldr     r2, [r0, #0]
   20bda:       680b            ldr     r3, [r1, #0]
   20bdc:       429a            cmp     r2, r3
   20bde:       d107            bne.n   20bf0 <_ZNK11rc_mac_addreqERKS_+0x18>
-> 20be0:       f8d0 3002       ldr.w   r3, [r0, #2]
   20be4:       f8d1 0002       ldr.w   r0, [r1, #2]
   20be8:       1a1b            subs    r3, r3, r0
   20bea:       4258            negs    r0, r3
   20bec:       4158            adcs    r0, r3
   20bee:       4770            bx      lr
   20bf0:       2000            movs    r0, #0
   20bf2:       4770            bx      lr

As far as I understand it, ldr.w should work for unaligned reads as long
as the page if flagged normal.  Still trying to determine what the page
is marked as in this case.

And looking at the code compiled with gcc-4.6 it is obviously different
and would not have alignment issues (but is much less efficient):
   49a46:       b5f0            push    {r4, r5, r6, r7, lr}
   49a48:       7847            ldrb    r7, [r0, #1]
   49a4a:       7803            ldrb    r3, [r0, #0]
   49a4c:       784e            ldrb    r6, [r1, #1]
   49a4e:       ea43 2707       orr.w   r7, r3, r7, lsl #8
   49a52:       780b            ldrb    r3, [r1, #0]
   49a54:       7884            ldrb    r4, [r0, #2]
   49a56:       ea43 2606       orr.w   r6, r3, r6, lsl #8
   49a5a:       788b            ldrb    r3, [r1, #2]
   49a5c:       78c5            ldrb    r5, [r0, #3]
   49a5e:       78ca            ldrb    r2, [r1, #3]
   49a60:       ea47 4704       orr.w   r7, r7, r4, lsl #16
   49a64:       ea46 4603       orr.w   r6, r6, r3, lsl #16
   49a68:       ea47 6705       orr.w   r7, r7, r5, lsl #24
   49a6c:       ea46 6602       orr.w   r6, r6, r2, lsl #24
   49a70:       42b7            cmp     r7, r6
   49a72:       d114            bne.n   49a9e <_ZNK11rc_mac_addreqERKS_+0x58>
   49a74:       ea43 2302       orr.w   r3, r3, r2, lsl #8
   49a78:       790a            ldrb    r2, [r1, #4]
   49a7a:       ea44 2405       orr.w   r4, r4, r5, lsl #8
   49a7e:       7905            ldrb    r5, [r0, #4]
   49a80:       7940            ldrb    r0, [r0, #5]
   49a82:       ea43 4302       orr.w   r3, r3, r2, lsl #16
   49a86:       794a            ldrb    r2, [r1, #5]
   49a88:       ea44 4405       orr.w   r4, r4, r5, lsl #16
   49a8c:       ea44 6000       orr.w   r0, r4, r0, lsl #24
   49a90:       ea43 6302       orr.w   r3, r3, r2, lsl #24
   49a94:       1ac1            subs    r1, r0, r3
   49a96:       4248            negs    r0, r1
   49a98:       eb40 0001       adc.w   r0, r0, r1
   49a9c:       bdf0            pop     {r4, r5, r6, r7, pc}
   49a9e:       2000            movs    r0, #0
   49aa0:       bdf0            pop     {r4, r5, r6, r7, pc}

Stupid alignment problems. :)

I tried booting with 'noalign' kernel argument which appears in the code
to clear the CR_A flag, but that didn't make any difference.

-- 
Len Sorensen

Re: [Xenomai] ARM bus error due to alignment

[Gilles Chanteperdrix]

On Thu, Nov 05, 2015 at 11:57:50AM -0500, Lennart Sorensen wrote:
> On Thu, Nov 05, 2015 at 05:25:14PM +0100, Gilles Chanteperdrix wrote:
> > I am sure. Xenomai does not use H_DMA on armv7, so, it is not used
> > if your code, it is not used at all.
> 
> Well we are using it for one allocation for talking to a network port.
> 
> If I remove it, I get a NULL pointer dereference, so it clearly matters
> to something in our code.
> 
> > > It does appear that when LPAE is enabled H_DMA does do something on arm,
> > > but only when LPAE is enabled since that enables ZONE_DMA.
> > 
> > I never said that H_DMA did nothing.
> 
> OK I misunderstood then.
> 
> Setting /proc/cpu/alignment to 0 gives the kernel message:
> alignment: ignoring faults is unsafe on this CPU.  Defaulting to fixup mode.
> 
> Apparently armv6+ with CR_U set is not allowed to turn off fixups.

If you have an armv7, you want to turn off all armv6 processors in
the kernel configuration so that the kernel does not try to be
compatible with these obsolete processors.

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] ARM bus error due to alignment

[Lennart Sorensen]

On Thu, Nov 05, 2015 at 06:14:19PM +0100, Gilles Chanteperdrix wrote:
> On Thu, Nov 05, 2015 at 11:57:50AM -0500, Lennart Sorensen wrote:
> > On Thu, Nov 05, 2015 at 05:25:14PM +0100, Gilles Chanteperdrix wrote:
> > > I am sure. Xenomai does not use H_DMA on armv7, so, it is not used
> > > if your code, it is not used at all.
> > 
> > Well we are using it for one allocation for talking to a network port.
> > 
> > If I remove it, I get a NULL pointer dereference, so it clearly matters
> > to something in our code.
> > 
> > > > It does appear that when LPAE is enabled H_DMA does do something on arm,
> > > > but only when LPAE is enabled since that enables ZONE_DMA.
> > > 
> > > I never said that H_DMA did nothing.
> > 
> > OK I misunderstood then.
> > 
> > Setting /proc/cpu/alignment to 0 gives the kernel message:
> > alignment: ignoring faults is unsafe on this CPU.  Defaulting to fixup mode.
> > 
> > Apparently armv6+ with CR_U set is not allowed to turn off fixups.
> 
> If you have an armv7, you want to turn off all armv6 processors in
> the kernel configuration so that the kernel does not try to be
> compatible with these obsolete processors.

I thought I had.  Maybe not quite:

# CONFIG_ARCH_MULTI_V6 is not set
CONFIG_ARCH_MULTI_V7=y
CONFIG_ARCH_MULTI_V6_V7=y

I wonder if I can turn off CONFIG_ARCH_MULTI_V6_V7 somehow, not that I
see anything in the kernel code that would change beecause of it.
Of course LPAE being on pretty much also excludes ARMv6 support.

Certainly as far as I can tell, on any armv6+ with CR_U set (which all
armv7 have hardwired), CR_A appears to always be cleared by the kernel, so I don't think it is causing the issue.

I think our problem is that we have allocated some memory that isn't
considered normal and hence isn't allowed unaligned accesses and with
gcc-4.6 that wasn't a problem, but with 4.9 it is (since it now does
generate unaligned accesses by default)

I just have to figure out why there was any reason to allocate 'weird'
memory.  I can't figure out why that would have been neccesary, but I
didn't write that code.  I am just trying to fix it.

-- 
Len Sorensen

Re: [Xenomai] ARM bus error due to alignment

[Lennart Sorensen]

On Thu, Nov 05, 2015 at 12:24:52PM -0500, Lennart Sorensen wrote:
> On Thu, Nov 05, 2015 at 06:14:19PM +0100, Gilles Chanteperdrix wrote:
> > On Thu, Nov 05, 2015 at 11:57:50AM -0500, Lennart Sorensen wrote:
> > > On Thu, Nov 05, 2015 at 05:25:14PM +0100, Gilles Chanteperdrix wrote:
> > > > I am sure. Xenomai does not use H_DMA on armv7, so, it is not used
> > > > if your code, it is not used at all.
> > > 
> > > Well we are using it for one allocation for talking to a network port.
> > > 
> > > If I remove it, I get a NULL pointer dereference, so it clearly matters
> > > to something in our code.
> > > 
> > > > > It does appear that when LPAE is enabled H_DMA does do something on arm,
> > > > > but only when LPAE is enabled since that enables ZONE_DMA.
> > > > 
> > > > I never said that H_DMA did nothing.
> > > 
> > > OK I misunderstood then.
> > > 
> > > Setting /proc/cpu/alignment to 0 gives the kernel message:
> > > alignment: ignoring faults is unsafe on this CPU.  Defaulting to fixup mode.
> > > 
> > > Apparently armv6+ with CR_U set is not allowed to turn off fixups.
> > 
> > If you have an armv7, you want to turn off all armv6 processors in
> > the kernel configuration so that the kernel does not try to be
> > compatible with these obsolete processors.
> 
> I thought I had.  Maybe not quite:
> 
> # CONFIG_ARCH_MULTI_V6 is not set
> CONFIG_ARCH_MULTI_V7=y
> CONFIG_ARCH_MULTI_V6_V7=y
> 
> I wonder if I can turn off CONFIG_ARCH_MULTI_V6_V7 somehow, not that I
> see anything in the kernel code that would change beecause of it.
> Of course LPAE being on pretty much also excludes ARMv6 support.
> 
> Certainly as far as I can tell, on any armv6+ with CR_U set (which all
> armv7 have hardwired), CR_A appears to always be cleared by the kernel, so I don't think it is causing the issue.
> 
> I think our problem is that we have allocated some memory that isn't
> considered normal and hence isn't allowed unaligned accesses and with
> gcc-4.6 that wasn't a problem, but with 4.9 it is (since it now does
> generate unaligned accesses by default)
> 
> I just have to figure out why there was any reason to allocate 'weird'
> memory.  I can't figure out why that would have been neccesary, but I
> didn't write that code.  I am just trying to fix it.

So when I change the one allocation from H_DMA to H_NONCACHED I get:

[   55.340237] rcksapi: DTB node=int_sm_sw, physical slot number=0, virtual IRQ=245
[   55.395419] Unable to handle kernel NULL pointer dereference at virtual address 00000002
[   55.411662] pgd = eded4ec0, hw pgd = eded4ec0
[   55.420397] [00000002] *pgd=adde8003, *pmd=add49003, *pte=00000000
[   55.432824] Internal error: Oops: 207 [#1] SMP ARM
[   55.442431] Modules linked in: ebtable_filter ebtables ip6table_filter ip6_tables 8021q garp stp mrp llc l2tp_ip6 l2tp_ip l2tp_eth l2tp_netlink ip_gre ip_tunnel gre l2tp_core macvlan mpls ti_pru_eth rcksapi_layer2(O) xhci_plat_hcd xhci_hcd iptable_nat nf_conntrack_ipv4
 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack dummy xeno_posix xeno_rtdm xeno_native dwc3 max6369_wdt phy_ti_pipe3 phy_omap_usb2 omap_aes_driver dwc3_omap at24 omap_des lm75 iptable_filter ip_tables x_tables
[   55.527108] CPU: 0 PID: 2944 Comm: StartTask Tainted: G           O 3.14-2-am5726 #1 Debian 3.14.56-0.1RR1
[   55.546483] task: ee9a3200 ti: edece000 task.ti: edece000
[   55.557325] PC is at get_free_range+0x60/0x1a8
[   55.566234] LR is at xnheap_alloc+0x4a4/0x538
[   55.574971] pc : [<c00f2d78>]    lr : [<c00f47ec>]    psr: 200d0093
[   55.574971] sp : edecfdd8  ip : 00000002  fp : 00000002
[   55.598008] r10: ef4a9d00  r9 : 00000ffe  r8 : 00000000
[   55.608485] r7 : 00001000  r6 : 00000002  r5 : f2051e30  r4 : ffff0f00
[   55.621578] r3 : 00001002  r2 : 00000002  r1 : 0004c000  r0 : f2051e30
[   55.634675] Flags: nzCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment user
[   55.649163] Control: 30c5387d  Table: aded4ec0  DAC: fffffffd
[   55.660687] Process StartTask (pid: 2944, stack limit = 0xedece240)
[   55.673258] Stack: (0xedecfdd8 to 0xeded0000)
[   55.681992] fdc0:                                                       f2051e54 00000000
[   55.698402] fde0: ffff0f00 f2051e30 c0793cc0 c073eb1c 00000001 f2051e60 15e6a4da 00000000
[   55.714812] fe00: eeb1e000 00000002 ee259c00 c04ae6fc 00000000 0000119c 0004c000 c0730cd8
[   55.731223] fe20: 15e6a4da 00000000 eefb5b38 eeb14500 c073eb1c c0756160 c0793cc0 c073eb1c
[   55.747633] fe40: 00000003 00000000 f2051e00 0004b900 00000000 bf099660 00000001 eefb9a80
[   55.764043] fe60: c010b03c c0733a80 e5d24aa6 0000000c c0733a80 c0731ecc c073e5d4 c0793cc0
[   55.780453] fe80: ffffffff ffffffff 00208040 f2051e00 00000000 00000000 edecffb0 edece030
[   55.796864] fea0: c0793cc0 c0756160 00000000 bf084d90 edecfeec c04ac3a8 000099ca c007b794
[   55.813274] fec0: 00000000 2e886000 00000135 00000000 ffff0f00 c073eb1c ee9a3200 c08121fc
[   55.829683] fee0: 00000001 00000000 00000000 00000000 ffffffff ffffffff ee259c38 00000083
[   55.846093] ff00: f2051e30 b56e4000 00095000 f0abb000 b56e4000 00000022 00000001 edecffb0
[   55.862504] ff20: 00000218 c0812220 c08121fc f2041a10 edece038 c010b318 c010b268 00001198
[   55.878912] ff40: 00000000 c07bbcc0 c072fb38 c0793cc0 c0731e78 c07bbcc0 c073eb1c c00c857c
[   55.895323] ff60: c073eb1c eefb5b38 0000119c c00c7428 000f0abb e5a56840 c073eb1c c0793cc0
[   55.911733] ff80: c07bbcc0 edecffb0 e5a56840 b6a6ecbc 00000000 000000c0 000f0042 c0020d68
[   55.928144] ffa0: edece000 00000002 b68036d8 c0020cbc 4301022b b68e8250 0004b900 b6a6ec80
[   55.944553] ffc0: b6a6ecbc 00000000 000000c0 000f0042 000000c0 b68036e4 b68037bc b68036d8
[   55.960964] ffe0: b6f8c221 b6a6ec80 0007b98f b6f8c264 000d0030 4301022b 403ac0c4 e34a4d10
[   55.977383] [<c00f2d78>] (get_free_range) from [<eeb1e000>] (0xeeb1e000)
[   55.990829] Code: e1a02003 9a000016 e1a0c002 e0823007 (e69c4009) 
[   56.003053] ---[ end trace 1dcd48b94aa9e4b5 ]---

So clearly that is upsetting xenomai.  Or at least making it look like
it is upsetting xenomai given the xnheap_alloc+0x4a4/0x538.

I wish I could figure out why the backtraces aren't working, since they
are such a nice debug feature to have.

-- 
Len Sorensen

Re: [Xenomai] ARM bus error due to alignment

[Lennart Sorensen]

On Thu, Nov 05, 2015 at 12:52:25PM -0500, Lennart Sorensen wrote:
> On Thu, Nov 05, 2015 at 12:24:52PM -0500, Lennart Sorensen wrote:
> > On Thu, Nov 05, 2015 at 06:14:19PM +0100, Gilles Chanteperdrix wrote:
> > > On Thu, Nov 05, 2015 at 11:57:50AM -0500, Lennart Sorensen wrote:
> > > > On Thu, Nov 05, 2015 at 05:25:14PM +0100, Gilles Chanteperdrix wrote:
> > > > > I am sure. Xenomai does not use H_DMA on armv7, so, it is not used
> > > > > if your code, it is not used at all.
> > > > 
> > > > Well we are using it for one allocation for talking to a network port.
> > > > 
> > > > If I remove it, I get a NULL pointer dereference, so it clearly matters
> > > > to something in our code.
> > > > 
> > > > > > It does appear that when LPAE is enabled H_DMA does do something on arm,
> > > > > > but only when LPAE is enabled since that enables ZONE_DMA.
> > > > > 
> > > > > I never said that H_DMA did nothing.
> > > > 
> > > > OK I misunderstood then.
> > > > 
> > > > Setting /proc/cpu/alignment to 0 gives the kernel message:
> > > > alignment: ignoring faults is unsafe on this CPU.  Defaulting to fixup mode.
> > > > 
> > > > Apparently armv6+ with CR_U set is not allowed to turn off fixups.
> > > 
> > > If you have an armv7, you want to turn off all armv6 processors in
> > > the kernel configuration so that the kernel does not try to be
> > > compatible with these obsolete processors.
> > 
> > I thought I had.  Maybe not quite:
> > 
> > # CONFIG_ARCH_MULTI_V6 is not set
> > CONFIG_ARCH_MULTI_V7=y
> > CONFIG_ARCH_MULTI_V6_V7=y
> > 
> > I wonder if I can turn off CONFIG_ARCH_MULTI_V6_V7 somehow, not that I
> > see anything in the kernel code that would change beecause of it.
> > Of course LPAE being on pretty much also excludes ARMv6 support.
> > 
> > Certainly as far as I can tell, on any armv6+ with CR_U set (which all
> > armv7 have hardwired), CR_A appears to always be cleared by the kernel, so I don't think it is causing the issue.
> > 
> > I think our problem is that we have allocated some memory that isn't
> > considered normal and hence isn't allowed unaligned accesses and with
> > gcc-4.6 that wasn't a problem, but with 4.9 it is (since it now does
> > generate unaligned accesses by default)
> > 
> > I just have to figure out why there was any reason to allocate 'weird'
> > memory.  I can't figure out why that would have been neccesary, but I
> > didn't write that code.  I am just trying to fix it.
> 
> So when I change the one allocation from H_DMA to H_NONCACHED I get:
> 
> [   55.340237] rcksapi: DTB node=int_sm_sw, physical slot number=0, virtual IRQ=245
> [   55.395419] Unable to handle kernel NULL pointer dereference at virtual address 00000002
> [   55.411662] pgd = eded4ec0, hw pgd = eded4ec0
> [   55.420397] [00000002] *pgd=adde8003, *pmd=add49003, *pte=00000000
> [   55.432824] Internal error: Oops: 207 [#1] SMP ARM
> [   55.442431] Modules linked in: ebtable_filter ebtables ip6table_filter ip6_tables 8021q garp stp mrp llc l2tp_ip6 l2tp_ip l2tp_eth l2tp_netlink ip_gre ip_tunnel gre l2tp_core macvlan mpls ti_pru_eth rcksapi_layer2(O) xhci_plat_hcd xhci_hcd iptable_nat nf_conntrack_ipv4
>  nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack dummy xeno_posix xeno_rtdm xeno_native dwc3 max6369_wdt phy_ti_pipe3 phy_omap_usb2 omap_aes_driver dwc3_omap at24 omap_des lm75 iptable_filter ip_tables x_tables
> [   55.527108] CPU: 0 PID: 2944 Comm: StartTask Tainted: G           O 3.14-2-am5726 #1 Debian 3.14.56-0.1RR1
> [   55.546483] task: ee9a3200 ti: edece000 task.ti: edece000
> [   55.557325] PC is at get_free_range+0x60/0x1a8
> [   55.566234] LR is at xnheap_alloc+0x4a4/0x538
> [   55.574971] pc : [<c00f2d78>]    lr : [<c00f47ec>]    psr: 200d0093
> [   55.574971] sp : edecfdd8  ip : 00000002  fp : 00000002
> [   55.598008] r10: ef4a9d00  r9 : 00000ffe  r8 : 00000000
> [   55.608485] r7 : 00001000  r6 : 00000002  r5 : f2051e30  r4 : ffff0f00
> [   55.621578] r3 : 00001002  r2 : 00000002  r1 : 0004c000  r0 : f2051e30
> [   55.634675] Flags: nzCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment user
> [   55.649163] Control: 30c5387d  Table: aded4ec0  DAC: fffffffd
> [   55.660687] Process StartTask (pid: 2944, stack limit = 0xedece240)
> [   55.673258] Stack: (0xedecfdd8 to 0xeded0000)
> [   55.681992] fdc0:                                                       f2051e54 00000000
> [   55.698402] fde0: ffff0f00 f2051e30 c0793cc0 c073eb1c 00000001 f2051e60 15e6a4da 00000000
> [   55.714812] fe00: eeb1e000 00000002 ee259c00 c04ae6fc 00000000 0000119c 0004c000 c0730cd8
> [   55.731223] fe20: 15e6a4da 00000000 eefb5b38 eeb14500 c073eb1c c0756160 c0793cc0 c073eb1c
> [   55.747633] fe40: 00000003 00000000 f2051e00 0004b900 00000000 bf099660 00000001 eefb9a80
> [   55.764043] fe60: c010b03c c0733a80 e5d24aa6 0000000c c0733a80 c0731ecc c073e5d4 c0793cc0
> [   55.780453] fe80: ffffffff ffffffff 00208040 f2051e00 00000000 00000000 edecffb0 edece030
> [   55.796864] fea0: c0793cc0 c0756160 00000000 bf084d90 edecfeec c04ac3a8 000099ca c007b794
> [   55.813274] fec0: 00000000 2e886000 00000135 00000000 ffff0f00 c073eb1c ee9a3200 c08121fc
> [   55.829683] fee0: 00000001 00000000 00000000 00000000 ffffffff ffffffff ee259c38 00000083
> [   55.846093] ff00: f2051e30 b56e4000 00095000 f0abb000 b56e4000 00000022 00000001 edecffb0
> [   55.862504] ff20: 00000218 c0812220 c08121fc f2041a10 edece038 c010b318 c010b268 00001198
> [   55.878912] ff40: 00000000 c07bbcc0 c072fb38 c0793cc0 c0731e78 c07bbcc0 c073eb1c c00c857c
> [   55.895323] ff60: c073eb1c eefb5b38 0000119c c00c7428 000f0abb e5a56840 c073eb1c c0793cc0
> [   55.911733] ff80: c07bbcc0 edecffb0 e5a56840 b6a6ecbc 00000000 000000c0 000f0042 c0020d68
> [   55.928144] ffa0: edece000 00000002 b68036d8 c0020cbc 4301022b b68e8250 0004b900 b6a6ec80
> [   55.944553] ffc0: b6a6ecbc 00000000 000000c0 000f0042 000000c0 b68036e4 b68037bc b68036d8
> [   55.960964] ffe0: b6f8c221 b6a6ec80 0007b98f b6f8c264 000d0030 4301022b 403ac0c4 e34a4d10
> [   55.977383] [<c00f2d78>] (get_free_range) from [<eeb1e000>] (0xeeb1e000)
> [   55.990829] Code: e1a02003 9a000016 e1a0c002 e0823007 (e69c4009) 
> [   56.003053] ---[ end trace 1dcd48b94aa9e4b5 ]---
> 
> So clearly that is upsetting xenomai.  Or at least making it look like
> it is upsetting xenomai given the xnheap_alloc+0x4a4/0x538.
> 
> I wish I could figure out why the backtraces aren't working, since they
> are such a nice debug feature to have.

OK blowing up here:

                        /*
                         * Search for a range of contiguous pages in
                         * the free page list of the current
                         * extent. The range must be 'bsize' long.
                         */
                        do {
                                lastpage = freepage;
---->                           freepage = *((caddr_t *) freepage);
                                freecont += heap->pagesize;
                        }
                        while (freepage == lastpage + heap->pagesize
                               && freecont < bsize);

I wonder if I am getting to the end of the list and trying to derefence
a NULL pointer as a result.  I see nothing in the code to make sure it
doesn't try to do that, although I hope I am just misreading the code.

Maybe the heap allocated to the application is too small.

-- 
Len Sorensen

Re: [Xenomai] ARM bus error due to alignment

[Lennart Sorensen]

On Thu, Nov 05, 2015 at 04:18:58PM -0500, Lennart Sorensen wrote:
> OK blowing up here:
> 
>                         /*
>                          * Search for a range of contiguous pages in
>                          * the free page list of the current
>                          * extent. The range must be 'bsize' long.
>                          */
>                         do {
>                                 lastpage = freepage;
> ---->                           freepage = *((caddr_t *) freepage);
>                                 freecont += heap->pagesize;
>                         }
>                         while (freepage == lastpage + heap->pagesize
>                                && freecont < bsize);
> 
> I wonder if I am getting to the end of the list and trying to derefence
> a NULL pointer as a result.  I see nothing in the code to make sure it
> doesn't try to do that, although I hope I am just misreading the code.

Well it does handle NULL checking at the top with while (freepage !=
NULL), so how is this derefencing doing that.   Hmm.

-- 
Len Sorensen

Re: [Xenomai] ARM bus error due to alignment

[Gilles Chanteperdrix]

On Thu, Nov 05, 2015 at 12:52:25PM -0500, Lennart Sorensen wrote:
> I wish I could figure out why the backtraces aren't working, since they
> are such a nice debug feature to have.

You can try disabling unwinding, this should enable frame pointers
and maybe make the backtraces more reliable.

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] ARM bus error due to alignment

[Lennart Sorensen]

On Thu, Nov 05, 2015 at 10:40:42PM +0100, Gilles Chanteperdrix wrote:
> On Thu, Nov 05, 2015 at 12:52:25PM -0500, Lennart Sorensen wrote:
> > I wish I could figure out why the backtraces aren't working, since they
> > are such a nice debug feature to have.
> 
> You can try disabling unwinding, this should enable frame pointers
> and maybe make the backtraces more reliable.

I can try that.  I always thought unwinding was part of making backtraces
work.  Not sure why I thought so.

-- 
Len Sorensen

Re: [Xenomai] ARM bus error due to alignment

[Lennart Sorensen]

On Thu, Nov 05, 2015 at 05:08:51PM -0500, Lennart Sorensen wrote:
> On Thu, Nov 05, 2015 at 10:40:42PM +0100, Gilles Chanteperdrix wrote:
> > On Thu, Nov 05, 2015 at 12:52:25PM -0500, Lennart Sorensen wrote:
> > > I wish I could figure out why the backtraces aren't working, since they
> > > are such a nice debug feature to have.
> > 
> > You can try disabling unwinding, this should enable frame pointers
> > and maybe make the backtraces more reliable.
> 
> I can try that.  I always thought unwinding was part of making backtraces
> work.  Not sure why I thought so.

So I removed H_DMA from the alloc cal and the alignment problem went away.
Never actually did need the DMA flag in the case anyhow so it shouldn't
have been there in the first place.  Now while poking at it, I did find
that using H_NONCACHED instead causes a NULL pointer crash in the kernel,
which seems like a bug (how can user space making a system call cause
a NULL pointer dereference in the kernel unless there is a bug?).
With 0 for the flags, it just works.

-- 
Len Sorensen

Re: [Xenomai] ARM bus error due to alignment

[Gilles Chanteperdrix]

On Fri, Nov 06, 2015 at 02:25:57PM -0500, Lennart Sorensen wrote:
> On Thu, Nov 05, 2015 at 05:08:51PM -0500, Lennart Sorensen wrote:
> > On Thu, Nov 05, 2015 at 10:40:42PM +0100, Gilles Chanteperdrix wrote:
> > > On Thu, Nov 05, 2015 at 12:52:25PM -0500, Lennart Sorensen wrote:
> > > > I wish I could figure out why the backtraces aren't working, since they
> > > > are such a nice debug feature to have.
> > > 
> > > You can try disabling unwinding, this should enable frame pointers
> > > and maybe make the backtraces more reliable.
> > 
> > I can try that.  I always thought unwinding was part of making backtraces
> > work.  Not sure why I thought so.
> 
> So I removed H_DMA from the alloc cal and the alignment problem went away.
> Never actually did need the DMA flag in the case anyhow so it shouldn't
> have been there in the first place.
>  Now while poking at it, I did find
> that using H_NONCACHED instead causes a NULL pointer crash in the kernel,
> which seems like a bug (how can user space making a system call cause
> a NULL pointer dereference in the kernel unless there is a bug?).
> With 0 for the flags, it just works.


Well, mapping the same piece of RAM with cacheable and non-cacheable
mappings on ARM is "undefined behaviour" territory. If you want to
map some RAM with H_NONCACHED, you should reserve it with something
like the CMA allocator so that the kernel does not map it in any
other way. That is one possible reason.

H_NONCACHED is routinely used on armv4/armv5 (actually, I mixed
H_DMA with H_NONCACHED, Xenomai core never sets H_DMA, it sets
H_NONCACHED for shared user/kernel mappings for armv4/armv5, because
this is required with these architectures). So, it works. However,
that maybe an issue with LPAE, that would be a second possible reason.

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] ARM bus error due to alignment

[Lennart Sorensen]

On Sat, Nov 07, 2015 at 11:13:03PM +0100, Gilles Chanteperdrix wrote:
> Well, mapping the same piece of RAM with cacheable and non-cacheable
> mappings on ARM is "undefined behaviour" territory. If you want to
> map some RAM with H_NONCACHED, you should reserve it with something
> like the CMA allocator so that the kernel does not map it in any
> other way. That is one possible reason.
> 
> H_NONCACHED is routinely used on armv4/armv5 (actually, I mixed
> H_DMA with H_NONCACHED, Xenomai core never sets H_DMA, it sets
> H_NONCACHED for shared user/kernel mappings for armv4/armv5, because
> this is required with these architectures). So, it works. However,
> that maybe an issue with LPAE, that would be a second possible reason.

Sounds plausible.  Well I am not going to worry about it, since after
removing the unneeded flags (all of them), things work great.

-- 
Len Sorensen