Bus Error on MPC8313

Bus Error on MPC8313

[Ron Madrid]

I wrote a custom driver for an MPC8313 based board.  Everything seems to
have been working for a long time, but now one of my IOCTL functions seems
to be having a problem.  I'm getting a "Bus error" reported by my
application and also a kernel "Oops".  Here's what the kernel is reporting. 

Oops: Machine check, sig: 7 [#1]
MPC831x RDB
Modules linked in: simpc
NIP: e10687d8 LR: e106880c CTR: 00000001
REGS: dfbcbdd0 TRAP: 0200   Not tainted  (2.6.27)
MSR: 00041000 <ME>  CR: 28000424  XER: 20000000
TASK = df864400[795] 'sisample.out' THREAD: dfbca000
GPR00: ff0865a9 dfbcbe80 df864400 00000000 bfdd47f8 00000014 dfbcbeb8 00080400
GPR08: bfdd4880 e1070000 00000000 e1180400 88000428 1005ecbc 1fffb000 00000000
GPR16: 1fff06e8 1fff5a30 00000000 100b3e7c 100a6358 100fb364 bf8d95b8 00000000
GPR24: 4800e71c 00000001 bfe51e74 dfbb2380 fffffff7 40047217 dfbb2380 dfbcbea8
NIP [e10687d8] SIMPC_TargetRead+0xc8/0x198 [simpc]
LR [e106880c] SIMPC_TargetRead+0xfc/0x198 [simpc]
Call Trace:
[dfbcbe80] [22000000] 0x22000000 (unreliable)
[dfbcbea0] [e1068368] SIMPC_ioctl+0x104/0x260 [simpc]
[dfbcbed0] [c00857b0] vfs_ioctl+0x94/0x98
[dfbcbee0] [c0085aac] do_vfs_ioctl+0x2f8/0x428
[dfbcbf10] [c0085c1c] sys_ioctl+0x40/0x74
[dfbcbf40] [c0011cec] ret_from_syscall+0x0/0x38
--- Exception: c01 at 0xfc99b10
    LR = 0xfd4d8a8
Instruction dump:
XXXXXXXX XXXXXXXX XXXXXXXX 2b830001 XXXXXXXX XXXXXXXX XXXXXXXX 7d4903a6
XXXXXXXX XXXXXXXX XXXXXXXX 800b0000 XXXXXXXX XXXXXXXX XXXXXXXX 396b0004
---[ end trace faf66608f10b7074 ]---
Bus error

I'm not convinced that this is a problem with the driver as it has been
functioning without this problem for a while on our first two revisions of
hardware.  This problem is happening on new hardware, but everything seems
to work fine on the new hardware for a while (an hour or so) and then this
begins to happen.

Does anyone have any thoughts or ideas?  Let me know if you need a bit
more information.  Thanks in advance for any response.

Ron

Re: Bus Error on MPC8313

[Scott Wood]

Ron Madrid wrote:
> I wrote a custom driver for an MPC8313 based board.  Everything seems to
> have been working for a long time, but now one of my IOCTL functions seems
> to be having a problem.  I'm getting a "Bus error" reported by my
> application and also a kernel "Oops".  Here's what the kernel is reporting. 
> 
> Oops: Machine check, sig: 7 [#1]

Were there any messages before this?  Is there anything interesting in 
the arbiter registers?

Is this custom driver using the localbus, and are there any NAND 
operations going on at the same time?

-Scott

Re: Bus Error on MPC8313

[Ron Madrid]

--- On Fri, 11/13/09, Scott Wood <scottwood@freescale.com> wrote:

> From: Scott Wood <scottwood@freescale.com>
> Subject: Re: Bus Error on MPC8313
> To: "Ron Madrid" <ron_madrid@sbcglobal.net>
> Cc: linuxppc-dev@ozlabs.org
> Date: Friday, November 13, 2009, 11:33 AM
> Ron Madrid wrote:
> > I wrote a custom driver for an MPC8313 based
> board.  Everything seems to
> > have been working for a long time, but now one of my
> IOCTL functions seems
> > to be having a problem.  I'm getting a "Bus
> error" reported by my
> > application and also a kernel "Oops".  Here's
> what the kernel is reporting. 
> > Oops: Machine check, sig: 7 [#1]
> 
> Were there any messages before this?  Is there
> anything interesting in the arbiter registers?

No messages prior, only the normal kernel boot messages.  And when this
does happen in usually causes a lockup, so I haven't been able to check the
status of any arbiter registers.
 
> Is this custom driver using the localbus, and are there any
> NAND operations going on at the same time?

This driver does use the localbus.  It seems to happen (when it does) when
performing an access to our FPGA that is connected via UPM on the localbus. 
I do not know if there are any NAND operations going on at that same time. 
If there are, they are not being directly caused by any of my apps or
driver calls.

Ron

Re: Bus Error on MPC8313

[Scott Wood]

Ron Madrid wrote:
> --- On Fri, 11/13/09, Scott Wood <scottwood@freescale.com> wrote:
> 
>> From: Scott Wood <scottwood@freescale.com>
>> Subject: Re: Bus Error on MPC8313
>> To: "Ron Madrid" <ron_madrid@sbcglobal.net>
>> Cc: linuxppc-dev@ozlabs.org
>> Date: Friday, November 13, 2009, 11:33 AM
>> Ron Madrid wrote:
>>> I wrote a custom driver for an MPC8313 based
>> board.  Everything seems to
>>> have been working for a long time, but now one of my
>> IOCTL functions seems
>>> to be having a problem.  I'm getting a "Bus
>> error" reported by my
>>> application and also a kernel "Oops".  Here's
>> what the kernel is reporting. 
>>> Oops: Machine check, sig: 7 [#1]
>> Were there any messages before this?  Is there
>> anything interesting in the arbiter registers?
> 
> No messages prior, only the normal kernel boot messages. 

Hmm, for some reason e300c3's cputable entry has no machine_check field, 
while all the other e300s point to machine_check_generic.

> And when this
> does happen in usually causes a lockup, so I haven't been able to check the
> status of any arbiter registers.

The system is up long enough to print an oops; you could stick some 
output in the trap handler.

>> Is this custom driver using the localbus, and are there any
>> NAND operations going on at the same time?
> 
> This driver does use the localbus. 

OK, check the localbus error registers as well.

> It seems to happen (when it does) when
> performing an access to our FPGA that is connected via UPM on the localbus. 
> I do not know if there are any NAND operations going on at that same time. 
> If there are, they are not being directly caused by any of my apps or
> driver calls.

Are there any filesystems mounted on NAND?  There are some issues with 
transactions timing out if there's contention with a long-running NAND 
operation.

-Scott

Re: Bus Error on MPC8313

[Ron Madrid]

--- On Fri, 11/13/09, Scott Wood <scottwood@freescale.com> wrote:
> > And when this
> > does happen in usually causes a lockup, so I haven't
> > been able to check the
> > status of any arbiter registers.
> 
> The system is up long enough to print an oops; you could
> stick some output in the trap handler.

I've never done anything like that before so this could
take me a while to figure out and try.
 
> OK, check the localbus error registers as well.

OK

> Are there any filesystems mounted on NAND?  There are
> some issues with transactions timing out if there's
> contention with a long-running NAND operation.

Yes, a JFFS2.  If that is the case, is there any sort of
workaround?

Ron

Re: Bus Error on MPC8313

[Scott Wood]

Ron Madrid wrote:
>> Are there any filesystems mounted on NAND?  There are
>> some issues with transactions timing out if there's
>> contention with a long-running NAND operation.
> 
> Yes, a JFFS2.  If that is the case, is there any sort of
> workaround?

I've just posted some patches to linuxppc-dev and linux-mtd.

However, this can still result in large latencies -- you may want to add 
some software synchronization between NAND and other localbus users.

-Scott