[Xenomai-help] syslogd breaking rt behaviour of xenomai?

[Xenomai-help] syslogd breaking rt behaviour of xenomai?

[roland Tollenaar]

Hi,

I am using rtcan and so far its working. I have a device on a 1MB/s
CAN line and its working well.

However my 1ms and 2ms periodic tasks start to have badly oscillating
period-times as soon as my device puts so many messages in the message
buffer that it starts overflowing.

At least that is my analysis of the situation. Because as soon as the
bufferoverflows  syslogd starts writing system error messages telling
(if dmesg is called) that the buffer is overflowing and that a message
is discarded.

Now I can make the buffer larger but that will take longer to read out
and I don;t mind losing the messages. So how do i tell syslogd that it
is acceptable that the buffer is overflowing to make it less busy?

How could I make syslogd more quiet in general. It seems that it can
be a major real-time destroyer?

Hope someone can help.

Kind regards,

Roland Tollenaar.

Re: [Xenomai-help] syslogd breaking rt behaviour of xenomai?

[Eric Noulard]

2007/3/4, roland Tollenaar <rolandtollenaar@domain.hid>:
>
> Now I can make the buffer larger but that will take longer to read out
> and I don;t mind losing the messages. So how do i tell syslogd that it
> is acceptable that the buffer is overflowing to make it less busy?
>
> How could I make syslogd more quiet in general. It seems that it can
> be a major real-time destroyer?
>
> Hope someone can help.

Which process is calling syslog ? Your RT task? Another process/daemon
running on your system?

Be it with Xenomai or not, using syslog system call
in realtime application is a bad idea, moreover standard
syslogd daemon has really bad performance regarding
high volume of message.

If it is your RT task which calls syslog you  should
find a way to send message in another way.
May be adding a message queue and a non-RT task,
the RT task sends messages through message queue
and the non-RT task finally call syslog.

Your RT task may be prepared to receive
-ENOMEM if the queue if full.

You may want to try alternatives syslog daemon
implementation like syslog-ng (http://www.balabit.com/products/syslog_ng/)
which exhibits far better performance than "standard" syslogd.


-- 
Erk

Re: [Xenomai-help] syslogd breaking rt behaviour of xenomai?

[roland Tollenaar]

Hi

> Which process is calling syslog ? Your RT task? Another process/daemon
> running on your system?

I don't  know. Before this started happening I was never aware of
syslogd. I certainly never called it myself explicitly. Apparently it
is called by something monitoring the CAN message buffer? Don't know.

>
> Be it with Xenomai or not, using syslog system call
> in realtime application is a bad idea,

evidently.

>moreover standard
> syslogd daemon has really bad performance regarding
> high volume of message.
Ok.

>
> If it is your RT task which calls syslog you  should
> find a way to send message in another way.

Its not me. Or not that I know of.

> May be adding a message queue and a non-RT task,
> the RT task sends messages through message queue
> and the non-RT task finally call syslog.

Sounds like a good idea. But first I must find out what is calling
syslogd in the first place and stop that happening.

>
> Your RT task may be prepared to receive
> -ENOMEM if the queue if full.
Ah, this sounds like a lead. How would I confirm this and then how do
I change that setting?


>
> You may want to try alternatives syslog daemon
> implementation like syslog-ng (http://www.balabit.com/products/syslog_ng/)
> which exhibits far better performance than "standard" syslogd.

I'd much rather just switch it off entirely when I am running my
rt-task. If I have a problem and need to debug with it I can always
switch on. During debugging I am not so worried about the hardness of
the RT behaviour.

Anyone know how to switch off syslogd or block all calls to it?

Thanks.

Regards,

Roland.


>
>
> --
> Erk
>

Re: [Xenomai-help] syslogd breaking rt behaviour of xenomai?

[Sebastian Smolorz]

roland Tollenaar wrote:
> Hi
>
> > Which process is calling syslog ? Your RT task? Another process/daemon
> > running on your system?
>
> I don't  know. Before this started happening I was never aware of
> syslogd. I certainly never called it myself explicitly. Apparently it
> is called by something monitoring the CAN message buffer? Don't know.

Which messages are you referring to? Could you post them? Maybe you are 
referring to the Kernel log messages printed by the RT-CAN-Stack. Then there 
is no syslogd involved at all.

--
Sebastian

Re: [Xenomai-help] syslogd breaking rt behaviour of xenomai?

[Eric Noulard]

2007/3/4, roland Tollenaar <rolandtollenaar@domain.hid>:
> > May be adding a message queue and a non-RT task,
> > the RT task sends messages through message queue
> > and the non-RT task finally call syslog.
>
> Sounds like a good idea. But first I must find out what is calling
> syslogd in the first place and stop that happening.

You are right.

> > Your RT task may be prepared to receive
> > -ENOMEM if the queue if full.
> Ah, this sounds like a lead. How would I confirm this and then how do
> I change that setting?

I think I did not tell it right.
IF YOU set up a message queue between YOUR RT task and
YOUR non-RT task that call syslog then the RT task should
be prepared to full message queue.

I'm NOT telling that full message WOULD generate syslog call
I DON'T know.

> > You may want to try alternatives syslog daemon
> > implementation like syslog-ng (http://www.balabit.com/products/syslog_ng/)
> > which exhibits far better performance than "standard" syslogd.
>
> I'd much rather just switch it off entirely when I am running my
> rt-task. If I have a problem and need to debug with it I can always
> switch on.

For this then simply try

/etc/init.d/syslogd stop

> During debugging I am not so worried about the hardness of
> the RT behaviour.
>
> Anyone know how to switch off syslogd or block all calls to it?

May be you can post the log excerpt you have in order
fo us to know where they come from?

log should show if they are coming from kernel or user process.



-- 
Erk

Re: [Xenomai-help] syslogd breaking rt behaviour of xenomai?

[Roland Tollenaar]

Hi
  > You may want to try alternatives syslog daemon
>> > implementation like syslog-ng 
>> (http://www.balabit.com/products/syslog_ng/)
>> > which exhibits far better performance than "standard" syslogd.
>>
>> I'd much rather just switch it off entirely when I am running my
>> rt-task. If I have a problem and need to debug with it I can always
>> switch on.
> 
> For this then simply try
> 
> /etc/init.d/syslogd stop

I'll try this later to see whether it improves matters.

> 
>> During debugging I am not so worried about the hardness of
>> the RT behaviour.
>>
>> Anyone know how to switch off syslogd or block all calls to it?
> 
> May be you can post the log excerpt you have in order
> fo us to know where they come from?
> 
> log should show if they are coming from kernel or user process.

Off the top of my head (am not at the linux box right now)

xeno-(something): buffer overflow bla bla (fd=[number]), message discarded


The [numer has been anything between 0 and 5. I presume the useful part 
is the xeno-(something) where the something will tell us what is making 
the calls?

I will not be able to run the system today so I post this memory version 
so long. :)

Regards,

Roland.

> 
> 
>