[Xenomai-help] rtcan I/O hang bug with /proc/rtcan/rtcan0/registers

[Xenomai-help] rtcan I/O hang bug with /proc/rtcan/rtcan0/registers

[Andrew Tannenbaum]

I'm running

	Linux 2.6.38.8
	Xenomai 2.6.0
	Atom N270 CPU

I'm talking to CAN with a PEAK PCI SJA1000 CAN adapter.

I have a control loop running at 200 Hz, and I use it to drive a motor
and read an encoder using rtcan.  My code is based on the
rtcansend/rtcanrecv examples.

I found a bug where my control loop was hanging, and I've isolated the
problem so that it may be reproduced with Xenomai supplied tools,
without using my own code.

The problem is:

When I have a loop that is sending CAN packets, if I read
/proc/rtcan/rtcan0/registers at the same time, it causes the loop to
hang.  The hang is in rt_dev_sendto.

To reproduce:

Configure the CAN controller

	$ rtcanconfig rtcan0 --baudrate=1000000 start

Send packets, in this case, SYNC messages at 1000 Hz
with tty output 1 Hz.

	$ rtcansend rtcan0 -i 0x80 -p 1000 -l 1000000

It will print CAN SYNC (0x80) messages:

	<0x080> [0]
	<0x080> [0]
	...

In another window, read rtcan0/registers in a loop (10 Hz):

	$ while :; do cat /proc/rtcan/rtcan0/registers; sleep .1; done

After a few seconds or a minute, the rtcansend window will stop printing
SYNC messages, because it is hung.

gdb stack trace from hung rtcansend:

	(gdb) bt
	#0  0xb7862813 in rt_dev_sendmsg (fd=0, msg=0xbfe4d3ec, flags=0) at
core.c:84
	#1  0x08048b9b in rt_dev_sendto () at ../../../include/rtdm/rtdm.h:359
	#2  rt_task () at rtcansend.c:89
	#3  0x080492d3 in main (argc=8, argv=0xbfe4d654) at rtcansend.c:301

This example uses a (default) 1000 Hz rtcansend loop and a 10 Hz cat
from /proc, but in my code I found it with a 200 Hz rtcan I/O loop and a
/usr/bin/watch script that was running at .5 Hz (but it took longer to
reproduce the bug).

I think I see that the rtcan0/registers are read in:

	ksrc/drivers/can/sja1000/rtcan_sja1000_proc.c:rtcan_sja_proc_regs()

but I don't understand the details enough to debug further.

For debugging convenience, I found that if I ^C out of the rtcansend and
try to rerun it, the rtcan0 I/O will still be hung, but if I rerun the
rtcanconfig command above, that will unhang the I/O.  Of course, that's
not a workable fix.

I found this problem by running my CAN I/O app while running a "canstat"
shell script from a "watch" command.  The canstat script looks like this:

	#! /bin/bash

	pcat() {
		for i in $*
		do
			echo "###" $i
			cat $i
			echo
		done
	}
	cd /proc/rtcan
	pcat version devices sockets rtcan0/filters rtcan0/registers

	echo "### rtps"
	rtps

This script cats a bunch of data from /proc/rtcan/ but I believe the
rtcan0/registers data is the only part that causes the rt_dev_sendto to
hang.

-Andy

Re: [Xenomai-help] rtcan I/O hang bug with /proc/rtcan/rtcan0/registers

[Gilles Chanteperdrix]

On 04/24/2012 08:47 PM, Andrew Tannenbaum wrote:
> I'm running
> 
> 	Linux 2.6.38.8
> 	Xenomai 2.6.0
> 	Atom N270 CPU
> 
> I'm talking to CAN with a PEAK PCI SJA1000 CAN adapter.
> 
> I have a control loop running at 200 Hz, and I use it to drive a motor
> and read an encoder using rtcan.  My code is based on the
> rtcansend/rtcanrecv examples.
> 
> I found a bug where my control loop was hanging, and I've isolated the
> problem so that it may be reproduced with Xenomai supplied tools,
> without using my own code.
> 
> The problem is:
> 
> When I have a loop that is sending CAN packets, if I read
> /proc/rtcan/rtcan0/registers at the same time, it causes the loop to
> hang.  The hang is in rt_dev_sendto.
> 
> To reproduce:
> 
> Configure the CAN controller
> 
> 	$ rtcanconfig rtcan0 --baudrate=1000000 start
> 
> Send packets, in this case, SYNC messages at 1000 Hz
> with tty output 1 Hz.
> 
> 	$ rtcansend rtcan0 -i 0x80 -p 1000 -l 1000000
> 
> It will print CAN SYNC (0x80) messages:
> 
> 	<0x080> [0]
> 	<0x080> [0]
> 	...
> 
> In another window, read rtcan0/registers in a loop (10 Hz):
> 
> 	$ while :; do cat /proc/rtcan/rtcan0/registers; sleep .1; done

Hi Andrew,

I am probably not the one which will answer your question, because I
know nothing about CAN hardware. What I know however is that some
hardware registers, when read, have side effects. For instance reading
an interrupt status may deassert the hardware interrupt.

The code for /proc/rtcan/rtcan0/registers is compiled if
CONFIG_XENO_DRIVERS_CAN_DEBUG is #defined.

This would seem to indicate that perhaps some registers have side
effects, and so this /proc file more a debug tool than something which
should be used on a regular basis.

Regards.

-- 
                                                                Gilles.

Re: [Xenomai-help] rtcan I/O hang bug with /proc/rtcan/rtcan0/registers

[Andrew Tannenbaum]

On 04/24/2012 02:57 PM, Gilles Chanteperdrix wrote:
> On 04/24/2012 08:47 PM, Andrew Tannenbaum wrote:
>> I'm running
>>
>> 	Linux 2.6.38.8
>> 	Xenomai 2.6.0
>> 	Atom N270 CPU
>>
>> I'm talking to CAN with a PEAK PCI SJA1000 CAN adapter.
>>
>> I have a control loop running at 200 Hz, and I use it to drive a motor
>> and read an encoder using rtcan.  My code is based on the
>> rtcansend/rtcanrecv examples.
>>
>> I found a bug where my control loop was hanging, and I've isolated the
>> problem so that it may be reproduced with Xenomai supplied tools,
>> without using my own code.
>>
>> The problem is:
>>
>> When I have a loop that is sending CAN packets, if I read
>> /proc/rtcan/rtcan0/registers at the same time, it causes the loop to
>> hang.  The hang is in rt_dev_sendto.
>>
>> To reproduce:
>>
>> Configure the CAN controller
>>
>> 	$ rtcanconfig rtcan0 --baudrate=1000000 start
>>
>> Send packets, in this case, SYNC messages at 1000 Hz
>> with tty output 1 Hz.
>>
>> 	$ rtcansend rtcan0 -i 0x80 -p 1000 -l 1000000
>>
>> It will print CAN SYNC (0x80) messages:
>>
>> 	<0x080> [0]
>> 	<0x080> [0]
>> 	...
>>
>> In another window, read rtcan0/registers in a loop (10 Hz):
>>
>> 	$ while :; do cat /proc/rtcan/rtcan0/registers; sleep .1; done
> 
> Hi Andrew,
> 
> I am probably not the one which will answer your question, because I
> know nothing about CAN hardware. What I know however is that some
> hardware registers, when read, have side effects. For instance reading
> an interrupt status may deassert the hardware interrupt.
> 
> The code for /proc/rtcan/rtcan0/registers is compiled if
> CONFIG_XENO_DRIVERS_CAN_DEBUG is #defined.
> 
> This would seem to indicate that perhaps some registers have side
> effects, and so this /proc file more a debug tool than something which
> should be used on a regular basis.
> 
> Regards.
> 

Thanks, Gilles, I see.

It seems that the registers are the only member of /proc/rtcan that is
#ifdef CONFIG_XENO_DRIVERS_CAN_DEBUG so it looks like the rest should be
safe.

Re: [Xenomai-help] rtcan I/O hang bug with /proc/rtcan/rtcan0/registers

[Wolfgang Grandegger]

On 04/24/2012 08:57 PM, Gilles Chanteperdrix wrote:
> On 04/24/2012 08:47 PM, Andrew Tannenbaum wrote:
>> I'm running
>>
>> 	Linux 2.6.38.8
>> 	Xenomai 2.6.0
>> 	Atom N270 CPU
>>
>> I'm talking to CAN with a PEAK PCI SJA1000 CAN adapter.
>>
>> I have a control loop running at 200 Hz, and I use it to drive a motor
>> and read an encoder using rtcan.  My code is based on the
>> rtcansend/rtcanrecv examples.
>>
>> I found a bug where my control loop was hanging, and I've isolated the
>> problem so that it may be reproduced with Xenomai supplied tools,
>> without using my own code.
>>
>> The problem is:
>>
>> When I have a loop that is sending CAN packets, if I read
>> /proc/rtcan/rtcan0/registers at the same time, it causes the loop to
>> hang.  The hang is in rt_dev_sendto.
>>
>> To reproduce:
>>
>> Configure the CAN controller
>>
>> 	$ rtcanconfig rtcan0 --baudrate=1000000 start
>>
>> Send packets, in this case, SYNC messages at 1000 Hz
>> with tty output 1 Hz.
>>
>> 	$ rtcansend rtcan0 -i 0x80 -p 1000 -l 1000000
>>
>> It will print CAN SYNC (0x80) messages:
>>
>> 	<0x080> [0]
>> 	<0x080> [0]
>> 	...
>>
>> In another window, read rtcan0/registers in a loop (10 Hz):
>>
>> 	$ while :; do cat /proc/rtcan/rtcan0/registers; sleep .1; done
> 
> Hi Andrew,
> 
> I am probably not the one which will answer your question, because I
> know nothing about CAN hardware. What I know however is that some
> hardware registers, when read, have side effects. For instance reading
> an interrupt status may deassert the hardware interrupt.

Yes, reading the registers while running is not a good idea. The
"register" proc file is for debugging and should not be use under normal
operation.

Wolfgang.