RE: Help on tuning the Linux kernel for soft real-time requiremen ts

RE: Help on tuning the Linux kernel for soft real-time requiremen ts

[Zajac Adam-AAZ004]

Eugene,

It sounds like you're doing exactly what we've been trying to achieve.
We're trying to meet soft real-time deadlines that are in order of seconds
(2sec).
The only differences between our systems are:
- kernel 2.4.20
- HZ = 100
- Compact Flash

We tried to do a similar thing to the scenario you described where we were
running all other threads with SCHED_OTHER, and the thread with real-time
priority with RoundRobin scheduler policy. We keep experiencing the greatest
slips for such file system intensive operations like gzip and tar.
Also, for an experiment, we tried to turn disk caching off, which helped
meeting the deadlines a little but slowed down all file transfers
unacceptably.
To eliminate the question on the compact flash performance we tried using
NFS. Using NFS  yielded similar results (missed deadlines) to performing the
transfers with the cflash card.

Did you have to set anything special at your kernel config file (.config)?
Did you need to apply low-latency patch separately or the MV kernel you have
came pre-patched?

Thanks,
Adam Zajac

Re: Help on tuning the Linux kernel for soft real-time requiremen ts

[Eugene Surovegin]

On Mon, Oct 18, 2004 at 05:44:52PM -0500, Zajac Adam-AAZ004 wrote:
> It sounds like you're doing exactly what we've been trying to achieve.
> We're trying to meet soft real-time deadlines that are in order of seconds
> (2sec).

Hmm, 2s is a lot. You shouldn't have any problems with getting 2 
second accuracy.

> The only differences between our systems are:
> - kernel 2.4.20
> - HZ = 100
> - Compact Flash

And O(1) scheduler + SCHED_FIFO instead of round-robin (I don't like 
this scheduling policy :), SCHED_FIFO is true priority based 
scheduling.

BTW, we have systems with CF card, writing to it doesn't seem to 
affect real-time threads either.

> We tried to do a similar thing to the scenario you described where we were
> running all other threads with SCHED_OTHER, and the thread with real-time
> priority with RoundRobin scheduler policy. We keep experiencing the greatest
> slips for such file system intensive operations like gzip and tar.
> Also, for an experiment, we tried to turn disk caching off, which helped
> meeting the deadlines a little but slowed down all file transfers
> unacceptably.
> To eliminate the question on the compact flash performance we tried using
> NFS. Using NFS  yielded similar results (missed deadlines) to performing the
> transfers with the cflash card.
> 
> Did you have to set anything special at your kernel config file (.config)?

Nothing special I can think of.

> Did you need to apply low-latency patch separately or the MV kernel you have
> came pre-patched?

I applied _all_ patches myself, although I used MV kernel as a 
reference in some cases, e.g. where rml PPC preempt patches were 
buggy.

--
Eugene

RE: Help on tuning the Linux kernel for soft real-time requiremen ts

[Zajac Adam-AAZ004]

Gerald, Eugene,

Thanks a bunch for your inputs.
I've done a lot of different experiments that included using different
storage media and the problem has been narrowed down to a serial driver
contention.

>I would be suspicious that one of your drivers is blocking interrupts
>for long durations, your mass storage driver in particular, although
>serial consoles are also known for rude behavior as well.
>
Gerald, you may be right about this. Although I still don't know which of
the threads in our system cause the serial resource contention, I
temporarily alleviated the problem with limiting the messages that are being
printed out to the screen.

BTW, are you guys aware of any tool that could help me debug the threads
that would concurrently try to access shared resources?
Do you think LTT would be helpful for this particular problem (serial driver
contention that leads to missing soft real-time deadlines)?

Adam

Re: Help on tuning the Linux kernel for soft real-time requiremen ts

[Wolfgang Denk]

In message <EFB813091B18D511BD3600508B644F820A618202@tx14exm06.ftw.mot.com> you wrote:
> 
> Gerald, you may be right about this. Although I still don't know which of
> the threads in our system cause the serial resource contention, I

Check the console driver first.

> BTW, are you guys aware of any tool that could help me debug the threads
> that would concurrently try to access shared resources?
> Do you think LTT would be helpful for this particular problem (serial driver

LTT is certaily useful; but be aware that it may generate a  of  data
in  short  time,  which then needs a lot of work to really understand
what's going on ;-)

> contention that leads to missing soft real-time deadlines)?

I'm not exaclty sure I really understand what  you  are  asking  for.
Isn't  sometimes  missing  the  deadlines what the "soft" is standing
for? ;-)

Best regards,

Wolfgang Denk

-- 
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
There's another way to survive.  Mutual trust -- and help.
	-- Kirk, "Day of the Dove", stardate unknown

RE: Help on tuning the Linux kernel for soft real-time requiremen ts

[Zajac Adam-AAZ004]

Wolfgang,

>I'm not exaclty sure I really understand what  you  are  asking  for.
>Isn't  sometimes  missing  the  deadlines what the "soft" is standing
>for? ;-)

I'm certainly not the one who would attempt to define the distinction
between hard and soft real-time requirements (beyond the obvious notion of
an absolute deterministic and an average response time). Just used that term
casually as we wanted to meet application deadlines that were in order of
seconds with some small room for a response variation (+/- milliseconds).
What I referred to as "missed deadlines" were slips in order of +/- 20
seconds. This was way out of line.

Do you know if sysstat RPM is available for PPC?
It'd be nice to have tools like sar and iostat for debugging my problem.

Regards,
Adam

Re: Help on tuning the Linux kernel for soft real-time requiremen ts

[Wolfgang Denk]

Dear Adam,

in message <EFB813091B18D511BD3600508B644F820A618204@tx14exm06.ftw.mot.com> you wrote:
> 
> What I referred to as "missed deadlines" were slips in order of +/- 20
> seconds. This was way out of line.

20 seconds? 

> Do you know if sysstat RPM is available for PPC?
> It'd be nice to have tools like sar and iostat for debugging my problem.

It is included with YDL (for YDL 3.0 it was on CD # 3 in file
YellowDog/RPMS/sysstat-4.0.5-3.ppc.rpm )

Best regards,

Wolfgang Denk

-- 
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
"Who is the oldest inhabitant of this village?"
"We haven't got one; we had one, but he died three weeks ago."