[Xenomai-help] rtcan bufferoverflow but no evidence

[Xenomai-help] rtcan bufferoverflow but no evidence

[Roland Tollenaar]

Hi,

I seem to be getting major buffer overflow. Yet when I check the 
messages with rtcanrecv with relative time stamp I get confirmation of 
the fact that there are only two messages on the bus every ms and the 
EASILY fit withing the 1ms task. I have tried to mask all errors but see 
nothing. Why would i get about one billion messagebuffer overflow errors 
to syslog? How can I make these mysterious messages visible?

Or is there a setting that i may have wrong such that the rtcan rev 
function does not empty the ring buffer?

I hope there is something obvious I am doing wrong because this rtcan is 
really giving me a headache.

Kind regards,

Roland

Re: [Xenomai-help] rtcan bufferoverflow but no evidence

[Wolfgang Grandegger]

Roland Tollenaar wrote:
> Hi,
> 
> I seem to be getting major buffer overflow. Yet when I check the 
> messages with rtcanrecv with relative time stamp I get confirmation of 
> the fact that there are only two messages on the bus every ms and the 
> EASILY fit withing the 1ms task. I have tried to mask all errors but see 
> nothing. Why would i get about one billion messagebuffer overflow errors 
> to syslog? How can I make these mysterious messages visible?
> 
> Or is there a setting that i may have wrong such that the rtcan rev 
> function does not empty the ring buffer?
> 
> I hope there is something obvious I am doing wrong because this rtcan is 
> really giving me a headache.

When you have bound a RTCAN socket, it will receive messages and put 
them into the RX socket queue. If no task reads them via recv* function 
calls, the buffer will overflow sooner than later. Do you use a bound 
socket just for sending messages? Then you could avoid buffer overflows 
by defining an empty receive filter list as show here
http://www.rts.uni-hannover.de/xenomai/lxr/source/src/utils/can/rtcansend.c#251.
Or use rt_dev_sendto() without binding the socket.

You can (or even should) suppress the overflow messages by disabling the 
kernel option CONFIG_XENO_DRIVERS_CAN_DEBUG.

Wolfgang.

Re: [Xenomai-help] rtcan bufferoverflow but no evidence

[Wolfgang Grandegger]

Roland Tollenaar wrote:
> Hi Wolfgang,
> 
>>> See below. This does not look good. Does it mean that I am only 
>>> always reading out the last socket and the rest are having their 
>>> buffers pumped full? Should I be unbinding somehow?
>>
>> Don't know, check your code ;-).
> Thanks that is exactly what I did because I thought I was unbinding. 
> rt_dev_close () should do that for me correct? Its not. At least not 
> reliably. :(
> 
>>> 778518     1
>>
>> There is a default filter defined for all sockets. If you just want to 
>> receive error frames, there is no need to bind the socket, IIRC.
>>
> I am lost here. I don;t want to receive ANY error frames. My sensor is 
> working, I am getting the readings in rapicly and all these error frames 
> which I cannot make visible anyhow are just causing grievance. I'd feel 
> a lot happier if I knew where the rogue can messages were coming from 
> that are overflowing the buffer. Mind you, if the Buf overflow count in 
> /proc/rtcan/sockets is zero while the application is running (the latest 
> socket will always have that) then syslog still gets messages saying 
> buffer overflow. I'm doing something wrong and I don;t just want to turn 
> off the messaging. If I can help it Id like to know what is wrong. Is 
> there any manner in which I can make visible ANY message error or not 
> that appears on the bus?
> 
> Is there a possibility that there is some loopback route which gets 
> messages intot he buffer which are NOT on the physical bus?

Yes, of course. You have enabled the TX loopback feature via kernel option:

config XENO_DRIVERS_CAN_TX_LOOPBACK
         depends on XENO_DRIVERS_CAN
         bool "Enable TX loopback to local sockets"
         default n
         help

         This options adds support for TX loopback to local sockets. 
Normally,
         messages sent to the CAN bus are not visible to sockets 
listening to
         the same local device. When this option is enabled, TX messages are
         looped back locally when the transmit has been done by default. 
This
         behaviour can be deactivated or reactivated with "setsockopt". 
Enable
         this option, if you want to have a "net-alike" behaviour.


 From your output:

root@domain.hid# cat /proc/rtcan/sockets
fd Name___________ Filter ErrMask RX_Timeout_ns TX_Timeout_ns RX_BufFull 
TX_Lo
  7 rtcan2               1 0x00008      infinite      infinite 0     1

TX_Lo tell you that TX messages are loopback to local sockets.

Wolfgang.

Re: [Xenomai-help] rtcan bufferoverflow but no evidence

[Wolfgang Grandegger]

Roland Tollenaar wrote:
> Hi Wolfgang,
> 
>>> See below. This does not look good. Does it mean that I am only 
>>> always reading out the last socket and the rest are having their 
>>> buffers pumped full? Should I be unbinding somehow?
>>
>> Don't know, check your code ;-).
> Thanks that is exactly what I did because I thought I was unbinding. 
> rt_dev_close () should do that for me correct? Its not. At least not 
> reliably. :(

That's wired. You mean you called rt_dev_close() and the socket still 
does show up in /proc/rtcan_sockets? Is it possible that there is 
another task still using that socket? Could you show the code fragment 
doing that.

Wolfgang.

Re: [Xenomai-help] rtcan bufferoverflow but no evidence

[Roland Tollenaar]

Hi,
>> Is there a possibility that there is some loopback route which gets 
>> messages intot he buffer which are NOT on the physical bus?
> 
> Yes, of course. You have enabled the TX loopback feature via kernel option:
> 
> config XENO_DRIVERS_CAN_TX_LOOPBACK
>         depends on XENO_DRIVERS_CAN
>         bool "Enable TX loopback to local sockets"
>         default n
>         help
> 
>         This options adds support for TX loopback to local sockets. 
> Normally,
>         messages sent to the CAN bus are not visible to sockets 
> listening to
>         the same local device. When this option is enabled, TX messages are
>         looped back locally when the transmit has been done by default. 
> This
>         behaviour can be deactivated or reactivated with "setsockopt". 
I'll have a look. But I understnad you correctly that even if I have 
activated the loopback behaviour in the kernel I can deactivate at 
runtime with setsockopt?

I still don;t see why this would generate overflows in the case when i 
only have one socket active? For the case that there are still lingering 
sockets which have not been closed it is obvious.


> TX_Lo tell you that TX messages are loopback to local sockets.
I'll turn this off in the morning if i can figure out how to with 
setsockopt.

Thanks a-plenty!

Roland


> 
> Wolfgang.
> 
> 

Re: [Xenomai-help] rtcan bufferoverflow but no evidence

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1000 bytes --]

Wolfgang Grandegger wrote:
> Roland Tollenaar wrote:
>> Hi Wolfgang,
>>
>>>> See below. This does not look good. Does it mean that I am only 
>>>> always reading out the last socket and the rest are having their 
>>>> buffers pumped full? Should I be unbinding somehow?
>>> Don't know, check your code ;-).
>> Thanks that is exactly what I did because I thought I was unbinding. 
>> rt_dev_close () should do that for me correct? Its not. At least not 
>> reliably. :(
> 
> That's wired. You mean you called rt_dev_close() and the socket still 
> does show up in /proc/rtcan_sockets? Is it possible that there is 
> another task still using that socket? Could you show the code fragment 
> doing that.

Recall that up to Xenomai 2.4 RTDM device/socket closing may fail (with
-EAGAIN) when some other thread is currently using it and doesn't
release it immediately when being "asked". That means for now you should
loop over [rt_dev_]close as long as EAGAIN is returned.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-help] rtcan bufferoverflow but no evidence

[Roland Tollenaar]

Hi,

> Recall that up to Xenomai 2.4 RTDM device/socket closing may fail (with
> -EAGAIN) when some other thread is currently using it and doesn't
> release it immediately when being "asked". That means for now you should
> loop over [rt_dev_]close as long as EAGAIN is returned.

Excellent thanks. I did not know this. Is xenomai 2.4 stable yet? Would 
you advise it as the version of choice at the moment?


Roland


> 
> Jan
> 

Re: [Xenomai-help] rtcan bufferoverflow but no evidence

[Wolfgang Grandegger]

Jan Kiszka wrote:
> Wolfgang Grandegger wrote:
>> Roland Tollenaar wrote:
>>> Hi Wolfgang,
>>>
>>>>> See below. This does not look good. Does it mean that I am only 
>>>>> always reading out the last socket and the rest are having their 
>>>>> buffers pumped full? Should I be unbinding somehow?
>>>> Don't know, check your code ;-).
>>> Thanks that is exactly what I did because I thought I was unbinding. 
>>> rt_dev_close () should do that for me correct? Its not. At least not 
>>> reliably. :(
>> That's wired. You mean you called rt_dev_close() and the socket still 
>> does show up in /proc/rtcan_sockets? Is it possible that there is 
>> another task still using that socket? Could you show the code fragment 
>> doing that.
> 
> Recall that up to Xenomai 2.4 RTDM device/socket closing may fail (with
> -EAGAIN) when some other thread is currently using it and doesn't
> release it immediately when being "asked". That means for now you should
> loop over [rt_dev_]close as long as EAGAIN is returned.

Roland, could you please check the return code of [rt_dev_]close. 
Nevertheless, I'm still puzzled why the socket shows up in 
/proc/rtcan/sockets because rtcan_raw_close() is called before returning 
-EAGAIN and the error mask shown there is weired as well.
Roland, do you perform the open and close in a serialized way (same 
task) and in nrt context?

Wolfgang.

Re: [Xenomai-help] rtcan bufferoverflow but no evidence

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 830 bytes --]

Roland Tollenaar wrote:
> Hi,
> 
>> Recall that up to Xenomai 2.4 RTDM device/socket closing may fail (with
>> -EAGAIN) when some other thread is currently using it and doesn't
>> release it immediately when being "asked". That means for now you should
>> loop over [rt_dev_]close as long as EAGAIN is returned.
> 
> Excellent thanks. I did not know this. Is xenomai 2.4 stable yet? Would
> you advise it as the version of choice at the moment?

2.4 is in -rc cycle, means there can still be bugs (ie. more bugs than
in stable ;) ). If you have more than just a few weeks to go until your
system role-out, having a look and giving it a try is not wrong. We'd
love to read your reports! ;) And you can quickly switch back again.

However, solely for the sake of this feature, I wouldn't throw 2.3.x away.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-help] rtcan bufferoverflow but no evidence

[Roland Tollenaar]

Hi,

>> Recall that up to Xenomai 2.4 RTDM device/socket closing may fail (with
>> -EAGAIN) when some other thread is currently using it and doesn't
>> release it immediately when being "asked". That means for now you should
>> loop over [rt_dev_]close as long as EAGAIN is returned.
> 
> Roland, could you please check the return code of [rt_dev_]close. 
> Nevertheless, I'm still puzzled why the socket shows up in 
> /proc/rtcan/sockets because rtcan_raw_close() is called before returning 
I have been battling with this all morning. I close with

do{
	ret=rt_dev_close(can_fd);
	//printf("%d",ret)
while(ret!=-9)

but this only works (without fail) if I uncomment the printf line. I 
have now tried to put in a sleep(1) command instead but due to the crash 
caused  by commenting out hte crash (does not come out of loop) I am now 
trying to save my filesystem. :( :(.





> -EAGAIN and the error mask shown there is weired as well.
> Roland, do you perform the open and close in a serialized way (same 
> task) and in nrt context?
I call the rt_dev_close in non-real time code if that is what you are 
asking. Same with the init AFAIK. I will post the code if I can retrieve it.

Roland



> 
> Wolfgang.
> 
> 
> 

Re: [Xenomai-help] rtcan bufferoverflow but no evidence

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1242 bytes --]

Roland Tollenaar wrote:
> Hi,
> 
>>> Recall that up to Xenomai 2.4 RTDM device/socket closing may fail (with
>>> -EAGAIN) when some other thread is currently using it and doesn't
>>> release it immediately when being "asked". That means for now you should
>>> loop over [rt_dev_]close as long as EAGAIN is returned.
>>
>> Roland, could you please check the return code of [rt_dev_]close.
>> Nevertheless, I'm still puzzled why the socket shows up in
>> /proc/rtcan/sockets because rtcan_raw_close() is called before returning 
> I have been battling with this all morning. I close with
> 
> do{
>     ret=rt_dev_close(can_fd);
>     //printf("%d",ret)
> while(ret!=-9)
> 
> but this only works (without fail) if I uncomment the printf line. I
> have now tried to put in a sleep(1) command instead but due to the crash
> caused  by commenting out hte crash (does not come out of loop) I am now
> trying to save my filesystem. :( :(.
> 

That really sounds like this issue, just applied on CAN:

http://sourceforge.net/mailarchive/forum.php?thread_name=9203.194.114.62.34.1186585293.squirrel%40webmail.krokoworld.de&forum_name=rtnet-users

If yes, you would have found a second reason to look at 2.4... ;)

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-help] rtcan bufferoverflow but no evidence

[Roland Tollenaar]

Hi,

> Roland, could you please check the return code of [rt_dev_]close. 
> Nevertheless, I'm still puzzled why the socket shows up in 
> /proc/rtcan/sockets because rtcan_raw_close() is called before returning 
> -EAGAIN and the error mask shown there is weired as well.
> Roland, do you perform the open and close in a serialized way (same 
> task) and in nrt context?

I have now got the close loop with the printf in it. You are probably 
right that I was closing in the rt context because syslog shows an error 
from rtdm stating that one cannot close from rea time. I understand from 
a thread elsewhere that printf brings the program to secondary mode 
hence helps with the closing?

Its working now infallibly, no more open sockets remain so I can work 
like this. Also applied the suggestion to turn off the loopback with 
setsockopt and that functions well too (no more buffer overflows)

I only still have the problem with EML which does not seem happy in 
combination with rtcan. Even if I increase the rtskbf_cache_size value 
(now on 2048). It still occurs incidentally and even after the program 
has been running for a while. Anyhow that seems to have nothing to do 
with this topic....drifting a bit.


Thanks.

Roland


> 
> Wolfgang.
> 
> 
>