[Xenomai-core] Strange behaviour of RTDM CAN driver

[Xenomai-core] Strange behaviour of RTDM CAN driver

[Sebastian Smolorz]

Hello all,

the following is a really nasty problem I am trying to solve for months now. I 
really hope that someone on the list knows the solution.

As you may remember some months ago I announced a RTDM CAN driver for SJA1000 
based cards (see 
https://mail.gna.org/public/xenomai-core/2005-11/msg00108.html). The driver 
works well with the cards at our institute. Some time ago I was asked to 
extend the driver so that it would work also with a card from another vendor 
(Advantech PCM3680). By now, the driver does its job as expected, with one 
exception: It gets stuck in the interrupt handler. The problem is that the 
interrupt register of the SJA1000 chip does not get cleared after a read 
command like it should. So the driver never knows that there is no interrupt 
anymore and therefore never leaves the interrupt handler.

After I installed a break after 20 loops inside the handler and an additional 
read of the interrupt register in the following syscall routine issued by the 
test program, I noticed that the register is cleared! So I strongly think 
that the reason for clearing this register lies in actions taken after the 
driver left the interrupt handler. But I am not very sure about this. Maybe 
there is another reason I do not see (or in the end the hardware is 
faulty ...).

Did any of you experienced a similar problem in the past? Any hint would be 
appreciated!

Interestingly, the driver does not show the above behaviour with all interrupt 
numbers. E.g. with interrupt number 12 the driver gets no interrupt at all.

Some technical details:
- Linux Kernel 2.6.16
- adeos-ipipe-2.6.16-i386-1.3-01.patch
- Xenomai 2.1.50


Sebastian

Re: [Xenomai-core] Strange behaviour of RTDM CAN driver

[Sebastian Smolorz]

Hello Ulrich!

On Tue, 16 May 2006 u.marx@domain.hid wrote:
> I don't know your hardware, but i think it's a not a hardware problem.
> Your desctription sounds more than a compiler optimization problem.
>
> Ok, this is your READ_REG definition,
>
>  /** Read access to a register of the SJA1000 CAN controller */
> #define READ_REG(base, register) \
>     readb((void *)(base+register))
>
> please try this one:
>
>  /** Read access to a register of the SJA1000 CAN controller */
> #define READ_REG(base, register) \
>     readb((volatile void *)(base+register))

Thank you for your suggestion. In my initial mail I forgot to mention that
the problem of an over-optimising compiler came into my mind, too. I tried
several workarounds to exclude every effect of the compiler (compiling
with -O0 instead of -O2, replacing readb directly with the definition
used in the Linux Kernel sources, and lastly disassembling the driver to
assure myself that there are really memory access commands to the IO
memory). Also, when I compile the driver with the change you suggested,
the kernel module is not different to that compiled without that change.
But I am desperate enough to try this one out. (That means mailing your
suggestion to the person who owns the Advantech CAN card; developing a
driver without the possibility to test it is really difficult.)


Regards,

Sebastian

Re: [Xenomai-core] Strange behaviour of RTDM CAN driver

[Sebastian Smolorz]

> On Tue, 16 May 2006 u.marx@domain.hid wrote:
> > I don't know your hardware, but i think it's a not a hardware problem.
> > Your desctription sounds more than a compiler optimization problem.
> >
> > Ok, this is your READ_REG definition,
> >
> >  /** Read access to a register of the SJA1000 CAN controller */
> > #define READ_REG(base, register) \
> >     readb((void *)(base+register))
> >
> > please try this one:
> >
> >  /** Read access to a register of the SJA1000 CAN controller */
> > #define READ_REG(base, register) \
> >     readb((volatile void *)(base+register))
>
> But I am desperate enough to try this one out. (That means mailing your
> suggestion to the person who owns the Advantech CAN card; developing a
> driver without the possibility to test it is really difficult.)

OK, i got the results. As I expected the patch above does not make any 
difference.


Sebastian

Re: [Xenomai-core] Strange behaviour of RTDM CAN driver

[Philippe Gerum]

Sebastian Smolorz wrote:
> Hello all,
> 
> the following is a really nasty problem I am trying to solve for months now. I 
> really hope that someone on the list knows the solution.
> 
> As you may remember some months ago I announced a RTDM CAN driver for SJA1000 
> based cards (see 
> https://mail.gna.org/public/xenomai-core/2005-11/msg00108.html). The driver 
> works well with the cards at our institute. Some time ago I was asked to 
> extend the driver so that it would work also with a card from another vendor 
> (Advantech PCM3680). By now, the driver does its job as expected, with one 
> exception: It gets stuck in the interrupt handler. The problem is that the 
> interrupt register of the SJA1000 chip does not get cleared after a read 
> command like it should. So the driver never knows that there is no interrupt 
> anymore and therefore never leaves the interrupt handler.
> 
> After I installed a break after 20 loops inside the handler and an additional 
> read of the interrupt register in the following syscall routine issued by the 
> test program, I noticed that the register is cleared! So I strongly think 
> that the reason for clearing this register lies in actions taken after the 
> driver left the interrupt handler. But I am not very sure about this. Maybe 
> there is another reason I do not see (or in the end the hardware is 
> faulty ...).
> 
> Did any of you experienced a similar problem in the past? Any hint would be 
> appreciated!
> 
> Interestingly, the driver does not show the above behaviour with all interrupt 
> numbers. E.g. with interrupt number 12 the driver gets no interrupt at all.
> 
> Some technical details:
> - Linux Kernel 2.6.16
> - adeos-ipipe-2.6.16-i386-1.3-01.patch
> - Xenomai 2.1.50
> 
> 

It would be interesting to know how Adeos is asked to deal with such 
interrupt upon receipt, e.g. does it relay it to Linux? What do the 
following say? And also, which is the number of your interrupt in the 
dumps below?

$ cat /proc/ipipe/Xenomai
$ cat /proc/ipipe/Linux
$ cat /proc/xenomai/irq

-- 

Philippe.

Re: [Xenomai-core] Strange behaviour of RTDM CAN driver

[Sebastian Smolorz]

Hello Philippe,

on Tue, 16 May 2006, Philippe Gerum wrote:
> It would be interesting to know how Adeos is asked to deal with such
> interrupt upon receipt, e.g. does it relay it to Linux?

In the init_module routine the driver calls rtdm_irq_request() with flags
RTDM_IRQTYPE_SHARED | RTDM_IRQTYPE_EDGE.

The interrupt handler returns RTDM_IRQ_NONE if the interrupt register of
the CAN chip reports zero (which is not the case here) or RTDM_IRQ_HANDLED
if it is not zero.


> What do the
> following say? And also, which is the number of your interrupt in the
> dumps below?
>
> $ cat /proc/ipipe/Xenomai
> $ cat /proc/ipipe/Linux
> $ cat /proc/xenomai/irq

I've sent a mail to Roland Riepl who has got that strange piece of CAN
hardware and also suggested him to directly post his answer to this ML.


Sebastian

Re: [Xenomai-core] Strange behaviour of RTDM CAN driver

[Sebastian Smolorz]

Hello Philippe,

I've got the results from Roland now.


> > Interestingly, the driver does not show the above behaviour with all
> > interrupt numbers. E.g. with interrupt number 12 the driver gets no
> > interrupt at all.

Now IRQ 12 works ...

> > Some technical details:
> > - Linux Kernel 2.6.16
> > - adeos-ipipe-2.6.16-i386-1.3-01.patch
> > - Xenomai 2.1.50
>
> It would be interesting to know how Adeos is asked to deal with such
> interrupt upon receipt, e.g. does it relay it to Linux? What do the
> following say? And also, which is the number of your interrupt in the
> dumps below?
>
> $ cat /proc/ipipe/Xenomai
> $ cat /proc/ipipe/Linux
> $ cat /proc/xenomai/irq

First, for IRQ 11:
# cat /proc/ipipe/Xenomai
Invariant head, Id=0x58454e4f
irq0-10: passed
irq11: wired
irq12-215: passed
irq216: wired
irq217-255: passed
irq256-257: passed, virtual
irq258: wired, virtual

# cat /proc/ipipe/Linux
Priority=100, Id=0x00000000
irq0-238: accepted
irq256-257: grabbed, virtual
irq258: passed, virtual

# cat /proc/xenomai/irq
IRQ         CPU0
 11:          10
216:          29
258:          20


And now, the card is assigned to IRQ12:
# cat /proc/ipipe/Xenomai
Invariant head, Id=0x58454e4f
irq0-11: passed
irq12: wired
irq13-215: passed
irq216: wired
irq217-255: passed
irq256-257: passed, virtual
irq258: wired, virtual

# cat /proc/ipipe/Linux
Priority=100, Id=0x00000000
irq0-238: accepted
irq256-257: grabbed, virtual
irq258: passed, virtual

# cat /proc/xenomai/irq
IRQ         CPU0
 12:           8
216:          33
258:          20

It should be noted that only 3 interrupts are registered by my driver.

Another phenomenon reported by Roland: His test program sends three messages 
via rt_dev_send(). After a fresh reboot, when the program sends it directly 
in a row, only the first one makes it. When he adds a printf() statement 
after each rt_dev_send() to get the return value, the program works nicely 
and sends 3 messages. And when he removes the printf() statements afterwards 
without booting freshly, the first behaviour cannot be reproduced.

Strange! I guess this behaviour has to do with the fact that no interrupts 
appear the first time. But why does they appear during the second run? Could 
it be related to the switch between primary/secondary domain and back? And 
has it to do with the problem that the interrupt register of the SJA1000 is 
not cleared?

Maybe someone can bring some light to this dark place ...


Sebastian