How to talk to DAQ Hardware?

How to talk to DAQ Hardware?

[Calin Culianu]

Hi,


I'm coming from the RTAI and RTLinux world and I just learned about the great
strides you guys have made in hard RT.  Anyway, I was wondering -- currently
how would one go about talking to DAQ hardware using the RT_PREEMPT patch(es)?
Will this "just work" with COMEDI?  What happens when a userspace hard rt app
calls into a kernel device driver in the simple case of an inb() or outb()
instruction?

I am hoping I can use this patch to do some basic input/output from a data
acquisition card that is driver by COMEDI -- using just synchronous acquisition
and output (basically writing to registers on the card, etc.. no IRQ's).

Will this work?

Thanks..

PS: I also posted on the comedi mailing list but wanted to ask you guys too.

Thanks again,

-Calin



       
____________________________________________________________________________________
Yahoo! oneSearch: Finally, mobile search 
that gives answers, not web links. 
http://mobile.yahoo.com/mobileweb/onesearch?refer=1ONXIC

Re: How to talk to DAQ Hardware?

[Daniel Walker]

On Sat, 2007-08-25 at 08:45 -0700, Calin Culianu wrote:
> Hi,
> 
> 
> I'm coming from the RTAI and RTLinux world and I just learned about the great
> strides you guys have made in hard RT.  Anyway, I was wondering -- currently
> how would one go about talking to DAQ hardware using the RT_PREEMPT patch(es)?
> Will this "just work" with COMEDI?  What happens when a userspace hard rt app
> calls into a kernel device driver in the simple case of an inb() or outb()
> instruction?

PREEMPT_RT doesn't change the way that users call into drivers .. The
users perspective of things is just same as regular linux .. 

Daniel

Re: How to talk to DAQ Hardware?

[Calin Culianu]

--- Daniel Walker <dwalker@mvista.com> wrote:

> On Sat, 2007-08-25 at 08:45 -0700, Calin Culianu wrote:
> > Hi,
> > 
> > 
> > I'm coming from the RTAI and RTLinux world and I just learned about the
> great
> > strides you guys have made in hard RT.  Anyway, I was wondering --
> currently
> > how would one go about talking to DAQ hardware using the RT_PREEMPT
> patch(es)?
> > Will this "just work" with COMEDI?  What happens when a userspace hard rt
> app
> > calls into a kernel device driver in the simple case of an inb() or outb()
> > instruction?
> 
> PREEMPT_RT doesn't change the way that users call into drivers .. The
> users perspective of things is just same as regular linux .. 
> 
> Daniel
> 
> 

Cool!  That's what I figured but I wanted to double-check.

Now, there is no way to raise the priority of particular driver's ISRs is
there?  So that a specific ISR can preempt anything including a hard realtime
process?

I know in the shared IRQ situation it would introduce lots of jitter but having
really high priority interrupts would be nice.  Is that possible?

Anyway thanks for answering my question -- this basically means I can throw
away rtlinux and use the latest 2.6 kernels with the RT_PREEMPT patches!  :)

-Calin





       
____________________________________________________________________________________
Building a website is a piece of cake. Yahoo! Small Business gives you all the tools to get online.
http://smallbusiness.yahoo.com/webhosting 

Re: How to talk to DAQ Hardware?

[Daniel Walker]

On Sat, 2007-08-25 at 12:49 -0700, Calin Culianu wrote:

> Cool!  That's what I figured but I wanted to double-check.
> 
> Now, there is no way to raise the priority of particular driver's ISRs is
> there?  So that a specific ISR can preempt anything including a hard realtime
> process?

ISR's are kernel threads, and you can change the priority of them .. If
you look 'ps' or 'top' you'll notice some processes that looks like,

m0nster-mash ~ # ps alx | grep IRQ
5     0   119     2 -51   -      0     0 irqd   S<   ?          0:00 [IRQ-20]
5     0   898     2 -51   -      0     0 irqd   S<   ?          0:00 [IRQ-8]
5     0   930     2 -51   -      0     0 irqd   S<   ?          0:00 [IRQ-14]
5     0   948     2 -51   -      0     0 irqd   S<   ?          0:00 [IRQ-18]
5     0   957     2 -51   -      0     0 irqd   S<   ?          0:00 [IRQ-16]
5     0   964     2 -51   -      0     0 irqd   S<   ?          0:00 [IRQ-19]
5     0   974     2 -51   -      0     0 irqd   S<   ?          0:00 [IRQ-12]
5     0   975     2 -51   -      0     0 irqd   S<   ?          0:00 [IRQ-1]
5     0  1002     2 -51   -      0     0 irqd   S<   ?          0:00 [IRQ-4]
m0nster-mash ~ #

So you change the priority just like with any process.. Using chrt from
sched utils ..

> I know in the shared IRQ situation it would introduce lots of jitter but having
> really high priority interrupts would be nice.  Is that possible?

yeah, if they're shared then both handlers will get the same priority .

> Anyway thanks for answering my question -- this basically means I can throw
> away rtlinux and use the latest 2.6 kernels with the RT_PREEMPT patches!  :)

Yeah, hopefully it works out for you..

Daniel

Re: How to talk to DAQ Hardware?

[Calin Culianu]

--- Daniel Walker <dwalker@mvista.com> wrote:

> > process?
> 
> ISR's are kernel threads, and you can change the priority of them .. If
> you look 'ps' or 'top' you'll notice some processes that looks like,

Wow, thanks for the thorough answer.  So basically interrupts come in from
hardware, are dispatched by waking up a thread.. and then the thread decides to
call a device driver's ISR or not?  So if another lower priority interupt calls
in, the thread is woken up but processing resumes again in the
previously-running higher priority thread?  I guess I should just read the
sources..

Anyway thanks for anwering my questions.  Now I must rave:

I seriously think you guys are AMAZING -- or at least your work on this is.  I
am so happy to be rid of RTLinux.  I liked RTAI, but it was really horribly
unstable and the API was all over the place.  So I was forced to use RTLinux.

RTLinux was not free, unless you used an ancient version that only really ran
on 2.4 kernels.

This RT patch is amazing.  I really want to say.. KUDOS.  You guys are my
heroes!  I have been pulling my hair out for 6 years now with sub-standard open
RTOS's like RTAI and RTLinux and having this done "the right way" by you guys
feels awesome..  I know RTAI had userspace realtime for some time but its
administration and configuration was messy and buggy and RTLinux's userspace
realtime was not open.  In the end it always felt like a hack somehow.. And if
you called into libraries or kernel facilities that weren't hard-rt.. BOOM,
your app exploded!  So kludgy..

Thanks again!!

-Calin



      ____________________________________________________________________________________
Luggage? GPS? Comic books? 
Check out fitting gifts for grads at Yahoo! Search
http://search.yahoo.com/search?fr=oni_on_mail&p=graduation+gifts&cs=bz

Re: How to talk to DAQ Hardware?

[Gregory Haskins]

On Sat, 2007-08-25 at 19:08 -0700, Calin Culianu wrote:
> --- Daniel Walker <dwalker@mvista.com> wrote:
> 
> > > process?
> > 
> > ISR's are kernel threads, and you can change the priority of them .. If
> > you look 'ps' or 'top' you'll notice some processes that looks like,
> 
> Wow, thanks for the thorough answer.  So basically interrupts come in from
> hardware, are dispatched by waking up a thread.. and then the thread decides to
> call a device driver's ISR or not?

Close.  There is one thread per IRQ, so each interrupt vector wakes a
specific thread which executes the registered handler unconditionally
once scheduled to run.  By default, all IRQ threads run at priority
RT50, but you can tweak this via tools like chrt like Daniel mentioned.


>   So if another lower priority interupt calls
> in, the thread is woken up but processing resumes again in the
> previously-running higher priority thread?  I guess I should just read the
> sources..

That's exactly right, but note that sometimes the opposite is not true
if you are not careful.  The interrupt controller itself often supports
its own concept of priority, and disparity between the hardware/software
priority can cause inadvertent masking.  In this scenario, the hardware
fails to dispatch what it perceives as the lower-priority vector until
the perceived higher one completes its EOI.  What it perceives as higher
and lower may differ from your RT priorities.

If you find yourself configuring your vectors to a RT priority
relationship that differs from their hardware relationship, you have a
few choices.  

1) Change the hardware relationship (if you are on a platform that
allows programmable priority in the interrupt-controller: e.g. some
powerpcs).

OR

2) Make sure the vectors in question are using fasteoi which (IIUC)
effectively moves the priority back into the control of the scheduler by
EOI'ing the vector immediately.

HTH

Regards,
-Greg

Re: How to talk to DAQ Hardware?

[Sven-Thorsten Dietrich]

On Sat, 2007-08-25 at 12:49 -0700, Calin Culianu wrote: 
> --- Daniel Walker <dwalker@mvista.com> wrote:
> 
> > On Sat, 2007-08-25 at 08:45 -0700, Calin Culianu wrote:
> > > Hi,
> > > 
> > > 
> > > I'm coming from the RTAI and RTLinux world and I just learned about the
> > great
> > > strides you guys have made in hard RT.  Anyway, I was wondering --
> > currently
> > > how would one go about talking to DAQ hardware using the RT_PREEMPT
> > patch(es)?
> > > Will this "just work" with COMEDI?  What happens when a userspace hard rt
> > app
> > > calls into a kernel device driver in the simple case of an inb() or outb()
> > > instruction?
> > 
> > PREEMPT_RT doesn't change the way that users call into drivers .. The
> > users perspective of things is just same as regular linux .. 
> > 
> > Daniel
> > 
> > 
> 
> Cool!  That's what I figured but I wanted to double-check.
> 
> Now, there is no way to raise the priority of particular driver's ISRs is
> there?  So that a specific ISR can preempt anything including a hard realtime
> process?
> 

there is chrt

(I had started to compose this earlier, so I defer to Greg H's comments
on the low level stuff.) - just a few add-ons below:

> I know in the shared IRQ situation it would introduce lots of jitter but having
> really high priority interrupts would be nice.  Is that possible?
> 

I looked at that problem with some folks before. 

The current RT IRQ model masks the IRQ until the thread corresponding to
the top-level handler (HARDIRQ) processes it.
For shared IRQs, that means there can be quite some jitter when one
thread is handling a shared IRQ on a certain priority level, and it gets
preempted by another thread at a higher prio - all before the shared IRQ
you are interested in runs.

Its possible to configure an IRQ handler to run in IRQ space, so it
vectores and runs in the "classical" IRQ paradigm. In a shared IRQ
model, however, where you have a shared IRQ in thread-space, and another
IRQ in IRQ context, things get messy...

Currently, internal priorities on shared irqs are not supported. 
To make this work, not only the shared-IRQ code must be modified to add
prio awareness, but also in many cases, the driver model falls short.

We were looking at a shared device similar to your DAQ's requirements,
but - the BIOS was stuffing a USB device in there (on the same SH_IRQ),
and on different revisions, there were other subsystems, ACPI, IIRC,
amongst others.

The workaround required adding a layer to the USB/ACPI layer, that would
allow you to clear the IRQ assert on the respective chip-set, so that it
would be possible to receive, and process a concurrently asserted high
prio interrupt on the same IRQ line (the DAQ), without constantly
re-entering the USB/ACPI vector while the IRQ was being asserted...

(because just masking the IRQ would block the DAQ from being processed
until some arbitrary IRQ handler runs - e.g. USB)

Essentially, the workaround required a 3-tiered driver model, where tier
1 ack-s the IRQ and triages the chipset to "shut up". 
Tier 2 handles the classic IRQ top-level, either in thread context, or
in IRQ context 
- and tier 3 is the "bottom half".

This doesn't translate well to a general IRQ model for all devices,
isn't supported in the general Linux driver model at this time, and
creates a lot of complexity to debug.

But it can be done, and if you can avoid the shared IRQ stuff, and can
get a dedicated IRQ in IRQ-space for your DAQ, you will see performance
similar to any sub-kernel. 

Happy hacking.


Sven

> Anyway thanks for answering my question -- this basically means I can
> throw
> away rtlinux and use the latest 2.6 kernels with the RT_PREEMPT
> patches!  :)
> 

Enjoy.

Sven


-- 
Sven-Thorsten Dietrich
+1 415 694-2930
( www.novell.com/linux )

Re: How to talk to DAQ Hardware?

[Guennadi Liakhovetski]

On Sat, 25 Aug 2007, Calin Culianu wrote:

> Now, there is no way to raise the priority of particular driver's ISRs is
> there?  So that a specific ISR can preempt anything including a hard realtime
> process?

It's the second time you say "hard" real-time, so, just wanted to warn - 
it is not hard. It is still only soft real time, but a pretty good one, I 
think.

Thanks
Guennadi
---------------------------------
Guennadi Liakhovetski, Ph.D.
DSA Daten- und Systemtechnik GmbH
Pascalstr. 28
D-52076 Aachen
Germany

Re: How to talk to DAQ Hardware?

[Carsten Emde]

Guennadi Liakhovetski wrote:
> On Sat, 25 Aug 2007, Calin Culianu wrote:
>> Now, there is no way to raise the priority of particular driver's ISRs is
>> there?  So that a specific ISR can preempt anything including a hard realtime
>> process?
> It's the second time you say "hard" real-time, so, just wanted to warn - 
> it is not hard. It is still only soft real time, but a pretty good one, I 
> think.
I don't know what exactly "soft real-time" is, and I don't think there 
is an exact definition of it. The term "hard real-time", however, is 
well defined. It is the same as "real-time". It means that a system 
never ever exceeds a previously defined maximum latency. If, for 
example, you define 100 microseconds as such, you run your system over a 
long time period (several days), and you observe several billions of 
single external events that trigger a user space process at real-time 
priority, then there must be no single event that fails to do so within 
100 microseconds. We did this constantly and successfully with an -rt 
kernel. So why are we not allowed to call this "hard real-time"?

If you are aware of a situation where this is not the case, then this 
may be a specific problem related to a specific situation. Please help 
us to improve the -rt patches further and submit a bug report. I am 
confident that it will be fixed as soon as possible.

	--cbe

Re: How to talk to DAQ Hardware?

[Sven-Thorsten Dietrich]

On Mon, 2007-08-27 at 09:56 +0200, Guennadi Liakhovetski wrote:
> On Sat, 25 Aug 2007, Calin Culianu wrote:
> 
> > Now, there is no way to raise the priority of particular driver's ISRs is
> > there?  So that a specific ISR can preempt anything including a hard realtime
> > process?
> 
> It's the second time you say "hard" real-time, so, just wanted to warn - 
> it is not hard. It is still only soft real time, but a pretty good one, I 
> think.


Hard / Soft is a property of the application's deadline requirements,
not a property of the operating system.

The consequences of missing a deadline determine determine whether an RT
>application< is hard or soft RT. 

Therefore, it is inappropriate to qualify an operating system as hard or
soft RT without deadline context.

Sven

Re: How to talk to DAQ Hardware?

[Guennadi Liakhovetski]

On Mon, 27 Aug 2007, Sven-Thorsten Dietrich wrote:

> Therefore, it is inappropriate to qualify an operating system as hard or
> soft RT without deadline context.

It is. An OS is not hard real-time (capable) if it cannot guarantee you 
with 100% certainty to meet ANY hard dead-line.

There is a philosophical side of this question, whether _any_ OS can 
guaratee that, but we don't want to discuss it here.

Thanks
Guennadi
---------------------------------
Guennadi Liakhovetski, Ph.D.
DSA Daten- und Systemtechnik GmbH
Pascalstr. 28
D-52076 Aachen
Germany

Re: How to talk to DAQ Hardware?

[Guennadi Liakhovetski]

On Mon, 27 Aug 2007, Carsten Emde wrote:

> I don't know what exactly "soft real-time" is, and I don't think there is an
> exact definition of it. The term "hard real-time", however, is well defined.
> It is the same as "real-time". It means that a system never ever exceeds a
> previously defined maximum latency.

As long as your non-realtime tasks are allowed to take locks, that also 
prevent real-time tasks from preempting them, and as long as you haven't 
proved that all those locks in all your non-realtime sources can never 
under any circumstances last longer than your pre-defined limit, it is not 
a hard-realtime OS.

This doesn't mean I don't like rt-opreempt:-) One just has to understand 
clearly its features and limitations.

Thanks
Guennadi
---------------------------------
Guennadi Liakhovetski, Ph.D.
DSA Daten- und Systemtechnik GmbH
Pascalstr. 28
D-52076 Aachen
Germany

Re: How to talk to DAQ Hardware?

[Bernhard Kuhn]

Carsten Emde wrote:
> So why are we not allowed to call this "hard real-time"?

:-)

I guess the disconnect is that even an RTOS can't guarantee
timing when the user didn't analysed his application software
with due diligence before going into production.

For example, someone can very well do something like
"for(;;);" at fifo priority 99, locking out all other threads
completely. The RTOS might think this is what the user likes
to do - means: an RTOS doesn't prevent a user to do silly things!

In this respect, it is even more dangerous if i.e. multimedia
application make use of real time priorties, because it can
easily lookup the whole system whereas "nice" priorities only
reduce system responsivness in case something went wrong.

Even if the developer carefully reviews what the application
is doing, then there is still the risk that he has overlooked
something. This is the reason why RTOSes like OSEK have a
"timing protection" that checks if a certain thread doesn't
exceed a given amount of allowed time per system cycle.
Not truly the best solution, either, but at least a means
of detecting that something went wrong ...

Concerning locking: the amount of time a low priority
thread holds a lock and in turn prevents a high priority
thread from acquiring it is a pure application problem
(means, something the developer has to take care about).
However, -rt implements the priority inheritance protocol
to soften that issue.

BTW.: Avoiding locks is a good real time programming
practice, *g* (However this is not always achivable).

regards

Bernhard

Re: How to talk to DAQ Hardware?

[Guennadi Liakhovetski]

On Mon, 27 Aug 2007, Sven-Thorsten Dietrich wrote:

> On Mon, 2007-08-27 at 12:02 +0200, Guennadi Liakhovetski wrote:
> > 
> > As long as your non-realtime tasks are allowed to take locks, that also 
>                                ^^^^^
> > prevent real-time tasks from preempting them, and as long as you haven't 
> > proved that all those locks in all your non-realtime sources can never 
> > under any circumstances last longer than your pre-defined limit, it is not 
> > a hard-realtime OS.
> 
> So you're talking about a property of the application !!!

Wrong emphasis. Here's what is meant:

> > As long as your non-realtime tasks are allowed to take locks, that also 
                                       ^^^^^^^^^^^
> > prevent real-time tasks from preempting them, and as long as you haven't 

Thanks
Guennadi
---------------------------------
Guennadi Liakhovetski, Ph.D.
DSA Daten- und Systemtechnik GmbH
Pascalstr. 28
D-52076 Aachen
Germany

Re: How to talk to DAQ Hardware?

[Robert Schwebel]

On Mon, Aug 27, 2007 at 09:56:15AM +0200, Guennadi Liakhovetski wrote:
> It's the second time you say "hard" real-time, so, just wanted to warn - 
> it is not hard. It is still only soft real time, but a pretty good one, I 
> think.

It is no more or less hard than other realtime operating systems with
the same set of features.

Calin, did you notice our OSADL fieldbus framework activities? It is in
an early stage but looks very promising for industrial I/O (not only
fieldbus, but also local type).

Robert
-- 
Pengutronix - Linux Solutions for Science and Industry
Entwicklungszentrum Nord     http://www.pengutronix.de

Re: How to talk to DAQ Hardware?

[Robert Schwebel]

On Mon, Aug 27, 2007 at 12:02:24PM +0200, Guennadi Liakhovetski wrote:
> As long as your non-realtime tasks are allowed to take locks, that also
> prevent real-time tasks from preempting them, and as long as you haven't
> proved that all those locks in all your non-realtime sources can never
> under any circumstances last longer than your pre-defined limit, it is not
> a hard-realtime OS.

Yes, but that's only true for locks shared between rt and non-rt
threads. You surely have to design an interface between high and low
priority parts of an RT system following certain rules, but that's the
same with all hard realtime systems.

Robert
-- 
Pengutronix - Linux Solutions for Science and Industry
Entwicklungszentrum Nord     http://www.pengutronix.de