[Xenomai-help] catching CAN errors

[Xenomai-help] catching CAN errors

[roland Tollenaar]

Hi,

What I cannot find an example of is how to filter for more than one error?

ret = rt_dev_setsockopt(can_fd, SOL_CAN_RAW, CAN_RAW_ERR_FILTER,
&err_mask, sizeof(err_mask));

will pick out the messages that correspond to err_mask. At the moment
I seem to be picking up CAN_ERR_BUSERROR but not for example
CAN_ERR_CRTL (mind you I presume the latter should have been
CAN_ERR_CTRL which is a bit more intuitive). If I want to pick up both
can I do this


ret = rt_dev_setsockopt(can_fd, SOL_CAN_RAW, CAN_RAW_ERR_FILTER,
&err_mask1, sizeof(err_mask1));

ret = rt_dev_setsockopt(can_fd, SOL_CAN_RAW, CAN_RAW_ERR_FILTER,
&err_mask2, sizeof(err_mask2));

to set up to get errors of two types or can I only pick up one at a time.?

This is what my CanID and the data sequence looks like:

!0x00000088!<0x088> [8] 00 00 80 19 00 00 00 00

0x00000080 is CAN_ERR_BUSERROR  and
0x00000008 is protocol violation CAN_ERR_PROT.

So I seem to have those two errors. Correct?

-What is a protocol error?

The values in data 2 and 3 seem to say

data 2:  80 Error occurred on transmission
data 3 19 Ack Slot?

The first one I can understand I have nothing connected to the bus,
the second one
What does the second one mean?


Regards,

Roland.

Re: [Xenomai-help] catching CAN errors

[Sebastian Smolorz]

roland Tollenaar wrote:
> Hi,
>
> What I cannot find an example of is how to filter for more than one error?
>
> ret = rt_dev_setsockopt(can_fd, SOL_CAN_RAW, CAN_RAW_ERR_FILTER,
> &err_mask, sizeof(err_mask));
>
> will pick out the messages that correspond to err_mask. At the moment
> I seem to be picking up CAN_ERR_BUSERROR but not for example
> CAN_ERR_CRTL (mind you I presume the latter should have been
> CAN_ERR_CTRL which is a bit more intuitive). If I want to pick up both
> can I do this
>
>
> ret = rt_dev_setsockopt(can_fd, SOL_CAN_RAW, CAN_RAW_ERR_FILTER,
> &err_mask1, sizeof(err_mask1));
>
> ret = rt_dev_setsockopt(can_fd, SOL_CAN_RAW, CAN_RAW_ERR_FILTER,
> &err_mask2, sizeof(err_mask2));
>
> to set up to get errors of two types or can I only pick up one at a time.?

The error mask is called a *mask* because you can mask those errors you are 
interested in. Of course you can set the error *bits* in one mask. Just OR 
them.


>
> This is what my CanID and the data sequence looks like:
>
> !0x00000088!<0x088> [8] 00 00 80 19 00 00 00 00
>
> 0x00000080 is CAN_ERR_BUSERROR  and
> 0x00000008 is protocol violation CAN_ERR_PROT.
>
> So I seem to have those two errors. Correct?
>
> -What is a protocol error?

Either consult the CAN specification or look at the protocol error types in 
the RT-Socket-CAN documentation [1] which give you a rough overview what a 
protocol error is. In short: It is a violation against the defined CAN 
protocol.

>
> The values in data 2 and 3 seem to say
>
> data 2:  80 Error occurred on transmission
> data 3 19 Ack Slot?
>
> The first one I can understand I have nothing connected to the bus,
> the second one
> What does the second one mean?

It means that no one has ack'ed your transmission. This makes sense since you 
didn't connect anything.


[1] 
http://www.xenomai.org/documentation/trunk/html/api/group__rtcan.html#g6cf7ee3a3d83315ed2589bb3bdb816b3

--
Sebastian

Re: [Xenomai-help] catching CAN errors

[Roland Tollenaar]

Hi ,

thanks for this. Is it correct that the errors come in on the 
messagebuffer where normal can frames also get read from? If so I 
presume this is normal CAN behavior but it does cause the message-buffer 
to overflow which in turn causes system messages to be generated which 
in turn messes up the RT behaviour of the entire application. If I have 
problems on the CAN bus I do not want the RT behavior of other parts of 
the application to be affected. So how can i prevent the error messages 
causing the message-buffer to overflow?

P.S. this question is related to another thread where the current 
standing is that I would have to switch off whatever is writing the 
system messages, i.e. syslogd?

If I can prevent the CAN errors coming in on the message buffer or 
rather maybe reduce their intensity (once is really enough I don;t need 
to be notified 100 billion times) that would be an elegant manner too.

Thanks for your time.

Regards,

Roland

Sebastian Smolorz wrote:
> roland Tollenaar wrote:
>> Hi,
>>
>> What I cannot find an example of is how to filter for more than one error?
>>
>> ret = rt_dev_setsockopt(can_fd, SOL_CAN_RAW, CAN_RAW_ERR_FILTER,
>> &err_mask, sizeof(err_mask));
>>
>> will pick out the messages that correspond to err_mask. At the moment
>> I seem to be picking up CAN_ERR_BUSERROR but not for example
>> CAN_ERR_CRTL (mind you I presume the latter should have been
>> CAN_ERR_CTRL which is a bit more intuitive). If I want to pick up both
>> can I do this
>>
>>
>> ret = rt_dev_setsockopt(can_fd, SOL_CAN_RAW, CAN_RAW_ERR_FILTER,
>> &err_mask1, sizeof(err_mask1));
>>
>> ret = rt_dev_setsockopt(can_fd, SOL_CAN_RAW, CAN_RAW_ERR_FILTER,
>> &err_mask2, sizeof(err_mask2));
>>
>> to set up to get errors of two types or can I only pick up one at a time.?
> 
> The error mask is called a *mask* because you can mask those errors you are 
> interested in. Of course you can set the error *bits* in one mask. Just OR 
> them.
> 
> 
>> This is what my CanID and the data sequence looks like:
>>
>> !0x00000088!<0x088> [8] 00 00 80 19 00 00 00 00
>>
>> 0x00000080 is CAN_ERR_BUSERROR  and
>> 0x00000008 is protocol violation CAN_ERR_PROT.
>>
>> So I seem to have those two errors. Correct?
>>
>> -What is a protocol error?
> 
> Either consult the CAN specification or look at the protocol error types in 
> the RT-Socket-CAN documentation [1] which give you a rough overview what a 
> protocol error is. In short: It is a violation against the defined CAN 
> protocol.
> 
>> The values in data 2 and 3 seem to say
>>
>> data 2:  80 Error occurred on transmission
>> data 3 19 Ack Slot?
>>
>> The first one I can understand I have nothing connected to the bus,
>> the second one
>> What does the second one mean?
> 
> It means that no one has ack'ed your transmission. This makes sense since you 
> didn't connect anything.
> 
> 
> [1] 
> http://www.xenomai.org/documentation/trunk/html/api/group__rtcan.html#g6cf7ee3a3d83315ed2589bb3bdb816b3
> 
> --
> Sebastian
> 

Re: [Xenomai-help] catching CAN errors

[Sebastian Smolorz]

Roland Tollenaar wrote:
> Hi ,
>
> thanks for this. Is it correct that the errors come in on the
> messagebuffer where normal can frames also get read from?

Yes, this is the internal socket ring buffer. But the error messages are 
generated inside the driver on-the-fly when errors are signaled by the CAN 
controller. They are pseudo-messages not really coming in from the bus.

> If so I 
> presume this is normal CAN behavior

The delivery of the error messages is a behaviour of the RT-Socket-CAN stack.

> but it does cause the message-buffer 
> to overflow which in turn causes system messages to be generated which
> in turn messes up the RT behaviour of the entire application. If I have
> problems on the CAN bus I do not want the RT behavior of other parts of
> the application to be affected. So how can i prevent the error messages
> causing the message-buffer to overflow?

If you can do without the reception of error messages simply use an empty 
error mask. No error message will then appear in the socket buffer.

>
> P.S. this question is related to another thread where the current
> standing is that I would have to switch off whatever is writing the
> system messages, i.e. syslogd?

The system messages you refer to are generated by the CAN stack. They will be 
generated also without syslogd because they use rtdm_printk() in kernel 
space.

>
> If I can prevent the CAN errors coming in on the message buffer or
> rather maybe reduce their intensity (once is really enough I don;t need
> to be notified 100 billion times) that would be an elegant manner too.

The behaviour of RT-Socket-CAN is: generate one error frame for errors that 
appear simultaneously. Consecutive errors therefore lead to the generation of 
several error frames. In your case the transmission to a disconnected bus 
leads to errors as long as you don't connect it. That's it. The hardware is 
busy with trying to send the message *you* want it to send. So it does and so 
it reports an error on every attempt to a disconnected bus.

--
Sebastian

Re: [Xenomai-help] catching CAN errors

[Sebastian Smolorz]

Roland Tollenaar wrote:

> > The system messages you refer to are generated by the CAN stack. They
> > will be generated also without syslogd because they use rtdm_printk() in
> > kernel space.
>
> Here I have a problem. Why then is the business of syslogd related to
> the overflowing of the buffer.

CAN socket buffer overflow --> kernel prints warnings --> syslogd logs these 
warnings

This is not exclusively RT-Socket-CAN related. Normally, syslogd logs all 
kernel messages.

> Its a fact that if my buffer over flows 
> (even if it overflows with normal messages being sent in from another
> node) that syslogd becomes very busy (when viewed in top) and the
> hardness of the rt app starts faltering.
>
> Btw I am not explicitly aware of ANY CAN error degrading the RT
> behaviour only of the relation between buffer overflow and reduction in
> RT performance.

As I stated in another mail you can silence the overflow message by 
configuring it out. But ask yourself if this is really what you want. Losing 
messages due to an overflowing buffer is nothing that should appear regularly 
but rather as exception. The kernel message is meant to inform you of this 
exceptional situation. If you lose messages under RT conditions you very 
likely have more severe problems than a kernel printing out too much 
information.

> Forget the errors for a moment because I have the same problem with too
> many messages coming in on a perfectly healthy bus from a perfectly
> healthy node.

Right. In fact, for the RAW protocol layer this is the very same situation as 
with the error frames. In my opinion, a clean design approach is not to 
silence the warnings because the discarding of CAN frames takes place 
regardless of this. I think you have to options:
1. If messages are lost sporadically try to enlarge the socker buffer size.
2. If messages are lost permanently your real-time task is overloaded. Try to 
lower the baud rate, set an appropriate filter or, more likely, rethink your 
whole system design. If CAN frames are piling up in the socket buffer and 
your application isn't quick enough to read them out although it does nothing 
else than receiving them, then there is something going wrong.

--
Sebastian

Re: [Xenomai-help] catching CAN errors

[Roland Tollenaar]

Hi,

>> the overflowing of the buffer.
> 
> CAN socket buffer overflow --> kernel prints warnings --> syslogd logs these 
> warnings
thanks this helps.

> As I stated in another mail you can silence the overflow message by 
> configuring it out. But ask yourself if this is really what you want. Losing 
> messages due to an overflowing buffer is nothing that should appear regularly 
> but rather as exception. The kernel message is meant to inform you of this 
> exceptional situation. If you lose messages under RT conditions you very 
> likely have more severe problems than a kernel printing out too much 
> information.
True. Very true. Well lets pursue this line of thought then. My node is 
set to sending 1 message every 1ms if the position changes. I am reading 
out the message buffer and it seems to work. But in theory I should only 
be findin 1 or zero messages in the buffer every time I read. Why don't 
I? Is the messagebuffer perhaps not cleaned out when I do a read? I only 
get an overflow when the sensor moves and emits readings. the readings 
are fine and I don;t have any masks set for errors anyway so I assume 
that my reads are not cleaning out the entire buffer. I have pasted in 
the code in which I do the reading. Am I doing something wrong?



	while(ret>=0){
		ret = rt_dev_recv (can_fd, (void *)&Rxframe, 			  			 
sizeof(can_frame_t), MSG_DONTWAIT);
		if (ret < 0) {
			switch (ret) {
			case -ETIMEDOUT:
				printf("rt_dev_recv: timed out");
				break;
			case -EBADF:
				printf("rt_dev_recv: aborted because
                                             socket was closed");
				break;
			}
		}

	}//while message buffer not empty
	//when exiting this loop the message buffer will be empty??????

I have cleared out some of the code in the loop for selecting the 
message I want and so on but it will in essence not influence the 
beahvior above.

> 
> Right. In fact, for the RAW protocol layer this is the very same situation as 
> with the error frames. In my opinion, a clean design approach is not to 
> silence the warnings because the discarding of CAN frames takes place 
> regardless of this. I think you have to options:
> 1. If messages are lost sporadically try to enlarge the socker buffer size.
Have considered this but it will slow down the reading routine (while 
loop in above snippet runs longer)

> 2. If messages are lost permanently your real-time task is overloaded. Try to 
> lower the baud rate, set an appropriate filter or, more likely, rethink your 
> whole system design. 
The design should be right. At the moment I am only looking for one 
message and only one message seems to be filling the message buffer. 
Filtering that will give me no reading.

If CAN frames are piling up in the socket buffer and
> your application isn't quick enough to read them out although it does nothing 
> else than receiving them, then there is something going wrong.
Probably. Hope you see something wrong in the reading loop.

According to the documentation a normal recv will remove the message 
from the buffer unless PEAK_something flag is used. So I assume that I 
am clearing out the buffer in the above code.

Regard,s

Roland

> 
> --
> Sebastian
> 

Re: [Xenomai-help] catching CAN errors

[Sebastian Smolorz]

Roland Tollenaar wrote:
> True. Very true. Well lets pursue this line of thought then. My node is
> set to sending 1 message every 1ms if the position changes. I am reading
> out the message buffer and it seems to work. But in theory I should only
> be findin 1 or zero messages in the buffer every time I read. Why don't
> I?

How many messages are you receiving? What are their contents compared to the 
messages sent? Are there also overflows if you use rtcanrecv shipped with 
Xenomai instead of your app?

> Is the messagebuffer perhaps not cleaned out when I do a read?

The message buffer is cleared unless you use MSG_PEEK.

> I only  
> get an overflow when the sensor moves and emits readings.

Hm, maybe you have another socket bound which you don't read from?

> the readings 
> are fine and I don;t have any masks set for errors anyway so I assume
> that my reads are not cleaning out the entire buffer. I have pasted in
> the code in which I do the reading. Am I doing something wrong?
>
>
>
> 	while(ret>=0){
> 		ret = rt_dev_recv (can_fd, (void *)&Rxframe,
> sizeof(can_frame_t), MSG_DONTWAIT);
> 		if (ret < 0) {
> 			switch (ret) {
> 			case -ETIMEDOUT:
> 				printf("rt_dev_recv: timed out");
> 				break;
> 			case -EBADF:
> 				printf("rt_dev_recv: aborted because
>                                              socket was closed");
> 				break;
> 			}
> 		}
>
> 	}//while message buffer not empty
> 	//when exiting this loop the message buffer will be empty??????

Are you sure this loop gets executed more than a few times? The flag 
MSG_DONTWAIT indicates that rt_dev_recv shall return with -EAGAIN if the 
socket buffer is empty. If this is the case the while-loop is left. Further 
CAN messages would not be read. Maybe this is the reason for the overflows 
which appear *after* you read a message for the last time.

--
Sebastian

Re: [Xenomai-help] catching CAN errors

[Sebastian Smolorz]

Roland Tollenaar wrote:
> Hi,
>
> Sebastian Smolorz wrote:
> > How many messages are you receiving?
>
> How do I count them?

Can you make your sending node send only, let's say, 10 messages and no more? 
Then print out the number of received messages by your application and 
compare.


> > messages sent? Are there also overflows if you use rtcanrecv shipped with
> > Xenomai instead of your app?
>
> Still overflows. I know because I have had it receiving while my
> application was running and when I first picked up the RT degradation I
> turned it off just to see if rtcanrecv was not causing the problem.

Eh ... and what is the situation if you run rtcanrecv without your app?

> >> Is the messagebuffer perhaps not cleaned out when I do a read?
> >
> > The message buffer is cleared unless you use MSG_PEEK.
>
> Ok so we agree that my code snippet as posed should clear out the
> buffer. That doubt is eliminated then.

Yes.

>
> >> I only
> >> get an overflow when the sensor moves and emits readings.
> >
> > Hm, maybe you have another socket bound which you don't read from?
>
> No. I am 100% certain that I only call rt_dev_bind (if it is called
> that) once for my entire application in a kind of init routine. Ah sorry
> I see what you are saying. Well I would have to have another bind to
> rtcan2 as well I presume. And as I say, I only call bind once. But let
> me understand this properly, does that mean that every socket created
> has its own buffer ring?

You got it. And for this reason there can be more than one buffer that 
overflows.

>
> >> the readings
> >> are fine and I don;t have any masks set for errors anyway so I assume
> >> that my reads are not cleaning out the entire buffer. I have pasted in
> >> the code in which I do the reading. Am I doing something wrong?
> >>
> >>
> >>
> >> 	while(ret>=0){
> >> 		ret = rt_dev_recv (can_fd, (void *)&Rxframe,
> >> sizeof(can_frame_t), MSG_DONTWAIT);
> >> 		if (ret < 0) {
> >> 			switch (ret) {
> >> 			case -ETIMEDOUT:
> >> 				printf("rt_dev_recv: timed out");
> >> 				break;
> >> 			case -EBADF:
> >> 				printf("rt_dev_recv: aborted because
> >>                                              socket was closed");
> >> 				break;
> >> 			}
> >> 		}
> >>
> >> 	}//while message buffer not empty
> >> 	//when exiting this loop the message buffer will be empty??????
> >
> > Are you sure this loop gets executed more than a few times? The flag
> > MSG_DONTWAIT indicates that rt_dev_recv shall return with -EAGAIN if the
> > socket buffer is empty. If this is the case the while-loop is left.
> > Further CAN messages would not be read. Maybe this is the reason for the
> > overflows which appear *after* you read a message for the last time.
>
> Sorry Sebastian, I lose you here. I know that -EAGAIN is returned if hte
> socket buffer is empty. But that is the point. I read all messages in
> the buffer until ret=-EAGAIN and that clears out the message buffer.
> Indeed after I have read for the last time my application waits 1ms (or
> what is left of it) then only reads again.

The code above does not state this. I assume this only is the inner snippet 
and the outer loop is missing?

> If my node sends so many 
> messages that I get an overflow in the mean time I will get a syslog
> telling me the buffer is overflowing. This is what I assumed was
> happening. This is what SEEMS to be happening. The thing is that the
> node is supposed to only send one message every ms itself. I will have
> to validate that somehow,

I have another suspicion. Maybe the sending node does not get an acknowledge? 
You don't run the CAN controller in listen only mode, do you? And what about 
termination of your CAN bus?

-- 
Sebastian

Re: [Xenomai-help] catching CAN errors

[roland Tollenaar]

Hi,

in the mean time I have started the other thread. I have also still
refering to this one, validated the number of messages coming from the
sensor. I would say that 1 message per ms is about correct. 11,000 in
about 11 seconds. With me clearing out the buffer every ms it is as
yet unclear where the rt breaking is taking place.

i have killed syslogd and that improves matters slightly. however i
can still see (via ksysguard) that some system process still starts
hammering away and doing things if I turn the encoder for a while.

And the message buffer overflow warnings are back. Even at 500 KBs.
That was before I killed syslogd of course.

Tomorrow i will try making the buffer larger, then I will switch off
that interrupt switch whatever it was called.

Do I only need to recompile the peak_can_dongle module or replace the
entire kernel after changing the buffer size and the interrupt
disabling??

Regards,

Roland

On 3/5/07, Sebastian Smolorz <ssm@domain.hid> wrote:
> Roland Tollenaar wrote:
> > Hi,
> >
> > Sebastian Smolorz wrote:
> > > How many messages are you receiving?
> >
> > How do I count them?
>
> Can you make your sending node send only, let's say, 10 messages and no more?
> Then print out the number of received messages by your application and
> compare.
>
>
> > > messages sent? Are there also overflows if you use rtcanrecv shipped with
> > > Xenomai instead of your app?
> >
> > Still overflows. I know because I have had it receiving while my
> > application was running and when I first picked up the RT degradation I
> > turned it off just to see if rtcanrecv was not causing the problem.
>
> Eh ... and what is the situation if you run rtcanrecv without your app?
>
> > >> Is the messagebuffer perhaps not cleaned out when I do a read?
> > >
> > > The message buffer is cleared unless you use MSG_PEEK.
> >
> > Ok so we agree that my code snippet as posed should clear out the
> > buffer. That doubt is eliminated then.
>
> Yes.
>
> >
> > >> I only
> > >> get an overflow when the sensor moves and emits readings.
> > >
> > > Hm, maybe you have another socket bound which you don't read from?
> >
> > No. I am 100% certain that I only call rt_dev_bind (if it is called
> > that) once for my entire application in a kind of init routine. Ah sorry
> > I see what you are saying. Well I would have to have another bind to
> > rtcan2 as well I presume. And as I say, I only call bind once. But let
> > me understand this properly, does that mean that every socket created
> > has its own buffer ring?
>
> You got it. And for this reason there can be more than one buffer that
> overflows.
>
> >
> > >> the readings
> > >> are fine and I don;t have any masks set for errors anyway so I assume
> > >> that my reads are not cleaning out the entire buffer. I have pasted in
> > >> the code in which I do the reading. Am I doing something wrong?
> > >>
> > >>
> > >>
> > >>    while(ret>=0){
> > >>            ret = rt_dev_recv (can_fd, (void *)&Rxframe,
> > >> sizeof(can_frame_t), MSG_DONTWAIT);
> > >>            if (ret < 0) {
> > >>                    switch (ret) {
> > >>                    case -ETIMEDOUT:
> > >>                            printf("rt_dev_recv: timed out");
> > >>                            break;
> > >>                    case -EBADF:
> > >>                            printf("rt_dev_recv: aborted because
> > >>                                              socket was closed");
> > >>                            break;
> > >>                    }
> > >>            }
> > >>
> > >>    }//while message buffer not empty
> > >>    //when exiting this loop the message buffer will be empty??????
> > >
> > > Are you sure this loop gets executed more than a few times? The flag
> > > MSG_DONTWAIT indicates that rt_dev_recv shall return with -EAGAIN if the
> > > socket buffer is empty. If this is the case the while-loop is left.
> > > Further CAN messages would not be read. Maybe this is the reason for the
> > > overflows which appear *after* you read a message for the last time.
> >
> > Sorry Sebastian, I lose you here. I know that -EAGAIN is returned if hte
> > socket buffer is empty. But that is the point. I read all messages in
> > the buffer until ret=-EAGAIN and that clears out the message buffer.
> > Indeed after I have read for the last time my application waits 1ms (or
> > what is left of it) then only reads again.
>
> The code above does not state this. I assume this only is the inner snippet
> and the outer loop is missing?
>
> > If my node sends so many
> > messages that I get an overflow in the mean time I will get a syslog
> > telling me the buffer is overflowing. This is what I assumed was
> > happening. This is what SEEMS to be happening. The thing is that the
> > node is supposed to only send one message every ms itself. I will have
> > to validate that somehow,
>
> I have another suspicion. Maybe the sending node does not get an acknowledge?
> You don't run the CAN controller in listen only mode, do you? And what about
> termination of your CAN bus?
>
> --
> Sebastian
>