rt file i/o

rt file i/o

[Tim Blechmann]

[-- Attachment #1: Type: text/plain, Size: 609 bytes --]

hi all,

having read the howto build an rt-application, i am trying to think of a
way to handle disk i/o in my application.

the howto suggest to avoid file i/o in non-rt helper threads, but to
move them to another process. while the howto suggests using network
socket, i'm curious, if other ipc primitives like shared memory would
give better throughput? or would i experience any issues, like is it
possible to lock shared memory to physical ram?

thanks, tim

-- 
tim@klingt.org
http://tim.klingt.org

Art is either a complaint or do something else
  John Cage quoting Jasper Johns



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: rt file i/o

[Mark Knecht]

On Wed, Nov 4, 2009 at 7:25 AM, Tim Blechmann <tim@klingt.org> wrote:
> hi all,
>
> having read the howto build an rt-application, i am trying to think of a
> way to handle disk i/o in my application.
>
> the howto suggest to avoid file i/o in non-rt helper threads, but to
> move them to another process. while the howto suggests using network
> socket, i'm curious, if other ipc primitives like shared memory would
> give better throughput? or would i experience any issues, like is it
> possible to lock shared memory to physical ram?
>
> thanks, tim
>
> --
> tim@klingt.org
> http://tim.klingt.org
>
> Art is either a complaint or do something else
>  John Cage quoting Jasper Johns
>

Tim,
   I'm not a developer and cannot really address your specific
question but for my use of the rt-kernel I put important rt-audio
files on a 1394 drive and give the 1394 driver higher priorities using
the IRQ scheduling tools. I don't have any trouble running 2 or 3 1394
drives recording and playing back 48 channels in Ardour.

   While I agree you should do everything the right way technically in
the code maybe part of your solution is outside of the app you are
writing and in the use of these support tools?

Take care,
Mark
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rt file i/o

[Tim Blechmann]

[-- Attachment #1: Type: text/plain, Size: 1048 bytes --]

>    I'm not a developer and cannot really address your specific
> question but for my use of the rt-kernel I put important rt-audio
> files on a 1394 drive and give the 1394 driver higher priorities using
> the IRQ scheduling tools. I don't have any trouble running 2 or 3 1394
> drives recording and playing back 48 channels in Ardour.
> 
>    While I agree you should do everything the right way technically in
> the code maybe part of your solution is outside of the app you are
> writing and in the use of these support tools?

well, if i understand the rt howto correctly, _no_ disc access is
allowed, neither from rt nor from non-rt threads, since it may produce
page faults, which introduce latencies ...
according to this definition, ardour is not an rt-safe application ...
as for rt-safety, i prefer analyzing the latency hotspots instead of
relying on test results

tim

-- 
tim@klingt.org
http://tim.klingt.org

Avoid the world, it's just a lot of dust and drag and means nothing in
the end.
  Jack Kerouac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: rt file i/o

[Nicholas Mc Guire]

On Wed, 04 Nov 2009, Tim Blechmann wrote:

> >    I'm not a developer and cannot really address your specific
> > question but for my use of the rt-kernel I put important rt-audio
> > files on a 1394 drive and give the 1394 driver higher priorities using
> > the IRQ scheduling tools. I don't have any trouble running 2 or 3 1394
> > drives recording and playing back 48 channels in Ardour.
> > 
> >    While I agree you should do everything the right way technically in
> > the code maybe part of your solution is outside of the app you are
> > writing and in the use of these support tools?
> 
> well, if i understand the rt howto correctly, _no_ disc access is
> allowed, neither from rt nor from non-rt threads, since it may produce
> page faults, which introduce latencies ...

I would be supprised if the rt howto states that page-faults in non-rt
threads is a critical problem - that would not significantly impact RT
performance - atelast not the worst case - it will (as every other system
load) impact the average case. so having a non-rt thread reading disk-files
to a buffer and a rt-thread processing this buffer should be perfectly fine.

hofrat

Re: rt file i/o

[Tim Blechmann]

[-- Attachment #1: Type: text/plain, Size: 1042 bytes --]

>> well, if i understand the rt howto correctly, _no_ disc access is
>> allowed, neither from rt nor from non-rt threads, since it may produce
>> page faults, which introduce latencies ...
> 
> I would be supprised if the rt howto states that page-faults in non-rt
> threads is a critical problem - that would not significantly impact RT
> performance - atelast not the worst case - it will (as every other system
> load) impact the average case. so having a non-rt thread reading disk-files
> to a buffer and a rt-thread processing this buffer should be perfectly fine.

that is what i thought for years ... but according to [1] a page fault
in an rt process freezes the entire process with both rt and non-rt
threads until the page fault is handled ...

tim

[1]
http://rt.wiki.kernel.org/index.php/HOWTO:_Build_an_RT-application#Latencies_caused_by_Page-faults

-- 
tim@klingt.org
http://tim.klingt.org

Your mind will answer most questions if you learn to relax and wait
for the answer.
  William S. Burroughs


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: rt file i/o

[Nicholas Mc Guire]

On Wed, 04 Nov 2009, Tim Blechmann wrote:

> >> well, if i understand the rt howto correctly, _no_ disc access is
> >> allowed, neither from rt nor from non-rt threads, since it may produce
> >> page faults, which introduce latencies ...
> > 
> > I would be supprised if the rt howto states that page-faults in non-rt
> > threads is a critical problem - that would not significantly impact RT
> > performance - atelast not the worst case - it will (as every other system
> > load) impact the average case. so having a non-rt thread reading disk-files
> > to a buffer and a rt-thread processing this buffer should be perfectly fine.
> 
> that is what i thought for years ... but according to [1] a page fault
> in an rt process freezes the entire process with both rt and non-rt
> threads until the page fault is handled ...
>
I proved to my self only a few days ago that I know nothing about RT_Preempt
so at the risk that this is completly wrong - for a simple test I added a
reader/writer thread in cyclictest.

with SCHED_OTHER or priority below 50 with SCHED_FIFO the impact is as expected
with priority above 50 (left all the irq-threads at there usual default) I see 
no impact at all in the worst case (average - as can be expected goes up a bit
compared to an idle system).

here are the results of a short run - the effect pops up imediately for SCHED_OTHER or -p 48, the -p 99 run is about 15 minutes.

/* SCHED_OTHER */
rtl26:~/rt/rt-tests# ./cyclictest -n
0.71 0.27 0.16 1/128 6907

T: 0 ( 6906) P: 0 I:1000 C:   9383 Min:      2 Act:   52 Avg:  315 Max:   73704

/* priority below any I/O activity in the kernel */
rtl26:~/rt/rt-tests# ./cyclictest -n -p 48
0.44 0.25 0.16 1/128 6911

T: 0 ( 6910) P:48 I:1000 C:   8610 Min:      1 Act:    2 Avg:  322 Max:   73607

/* priority above any I/O activity in the kernel */

rtl26:~/rt/rt-tests.mod# ./cyclictest -n -p 99
1.60 1.59 1.23 1/128 7314

T: 0 ( 7079) P:99 I:1000 C:1067096 Min:      1 Act:    2 Avg:    2 Max:      31


The thread added simply opens a large file (linux-2.6.31.tar.bz2, 62MB), reading 1k at a time, and writing it to a file in /tmp in an while(1) loop - the thread runs with default attr thus PTHREAD_SCOPE_SYSTEM as SCHED_OTHER thread. so maybe someone with more insight could comment on this "good" result.

HW: UP AMD Sempron 1.6GHz, NN IDE disk, 256MB RAM
SW: Debian 5.0.3 (Default Desktop) kernel 2.6.31.4-rt14 

thx!
hofrat

Re: rt file i/o

[Clark Williams]

[-- Attachment #1: Type: text/plain, Size: 1217 bytes --]

On Wed, 04 Nov 2009 18:32:41 +0100
Tim Blechmann <tim@klingt.org> wrote:

> >> well, if i understand the rt howto correctly, _no_ disc access is
> >> allowed, neither from rt nor from non-rt threads, since it may produce
> >> page faults, which introduce latencies ...
> > 
> > I would be supprised if the rt howto states that page-faults in non-rt
> > threads is a critical problem - that would not significantly impact RT
> > performance - atelast not the worst case - it will (as every other system
> > load) impact the average case. so having a non-rt thread reading disk-files
> > to a buffer and a rt-thread processing this buffer should be perfectly fine.
> 
> that is what i thought for years ... but according to [1] a page fault
> in an rt process freezes the entire process with both rt and non-rt
> threads until the page fault is handled ...
> 

No, we need to get that fixed in the wiki. It may have been the case
that the entire process was frozen while a fault was being handled, but
I don't believe that is the case now. If you're in a thread on core0
and a sibling thread on core1 faults, you shouldn't be impacted unless
you're trying to reference the same memory. 

Clark

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: rt file i/o

[Shane M Smith]

Clark Williams <williams@redhat.com> wrote on 11/04/2009 03:44:13 PM:
> No, we need to get that fixed in the wiki. It may have been the case
> that the entire process was frozen while a fault was being handled, but
> I don't believe that is the case now. If you're in a thread on core0
> and a sibling thread on core1 faults, you shouldn't be impacted unless
> you're trying to reference the same memory. 
> 
> Clark

I had seen this myself in the wiki and have been telling many people about 
it.  I finally tested this a couple of months ago and found out it was not 
true.  I tried having a thread malloc and touch memory, write to the disk, 
and read from the disk.  No matter what I did, I couldn't affect a second 
thread.

I have read briefly on some of the differences with the current and 
previous user model.  I don't know all the facts, but I believe that it 
used to be a user library.  At that time, I'm sure that a page fault would 
effect every thread.

Shane

Re: rt file i/o

[Tim Blechmann]

[-- Attachment #1: Type: text/plain, Size: 1625 bytes --]

On 11/04/2009 11:44 PM, Clark Williams wrote:
> On Wed, 04 Nov 2009 18:32:41 +0100
> Tim Blechmann <tim@klingt.org> wrote:
> 
>>>> well, if i understand the rt howto correctly, _no_ disc access is
>>>> allowed, neither from rt nor from non-rt threads, since it may produce
>>>> page faults, which introduce latencies ...
>>>
>>> I would be supprised if the rt howto states that page-faults in non-rt
>>> threads is a critical problem - that would not significantly impact RT
>>> performance - atelast not the worst case - it will (as every other system
>>> load) impact the average case. so having a non-rt thread reading disk-files
>>> to a buffer and a rt-thread processing this buffer should be perfectly fine.
>>
>> that is what i thought for years ... but according to [1] a page fault
>> in an rt process freezes the entire process with both rt and non-rt
>> threads until the page fault is handled ...
>>
> 
> No, we need to get that fixed in the wiki. It may have been the case
> that the entire process was frozen while a fault was being handled, but
> I don't believe that is the case now. If you're in a thread on core0
> and a sibling thread on core1 faults, you shouldn't be impacted unless
> you're trying to reference the same memory. 

well, then my application seems to be real-time safe ... it would be
nice if the wiki would reflect the current state of the rt-preempt
kernel, though ...

thnx, tim

-- 
tim@klingt.org
http://tim.klingt.org

All we composers really have to work with is time and sound - and
sometimes I'm not even sure about sound.
  Morton Feldman


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: rt file i/o

[Remy Bohmer]

Hi,

>>> that is what i thought for years ... but according to [1] a page fault
>>> in an rt process freezes the entire process with both rt and non-rt
>>> threads until the page fault is handled ...
>>
>> No, we need to get that fixed in the wiki. It may have been the case
>> that the entire process was frozen while a fault was being handled, but
>> I don't believe that is the case now. If you're in a thread on core0
>> and a sibling thread on core1 faults, you shouldn't be impacted unless
>> you're trying to reference the same memory.
>
> well, then my application seems to be real-time safe ... it would be
> nice if the wiki would reflect the current state of the rt-preempt
> kernel, though ...

Well, I wrote this part of the wiki quite some time ago, and yes it
might be outdated on some points.
I will look into it, and update it in a couple of days.

Kind regards,

Remy

Re: rt file i/o

[Sven-Thorsten Dietrich]

On 11/04/2009 09:14 AM, Nicholas Mc Guire wrote:
> On Wed, 04 Nov 2009, Tim Blechmann wrote:
>
>    
>>>     I'm not a developer and cannot really address your specific
>>> question but for my use of the rt-kernel I put important rt-audio
>>> files on a 1394 drive and give the 1394 driver higher priorities using
>>> the IRQ scheduling tools. I don't have any trouble running 2 or 3 1394
>>> drives recording and playing back 48 channels in Ardour.
>>>
>>>     While I agree you should do everything the right way technically in
>>> the code maybe part of your solution is outside of the app you are
>>> writing and in the use of these support tools?
>>>        
>> well, if i understand the rt howto correctly, _no_ disc access is
>> allowed, neither from rt nor from non-rt threads, since it may produce
>> page faults, which introduce latencies ...
>>      
> I would be supprised if the rt howto states that page-faults in non-rt
> threads is a critical problem - that would not significantly impact RT
> performance - atelast not the worst case - it will (as every other system
> load) impact the average case. so having a non-rt thread reading disk-files
> to a buffer and a rt-thread processing this buffer should be perfectly fine.
>
>    

Ack. I agree with Herr Hofrat, and would only add, that it is helpful to 
pre-allocate and mlock() memory you use to pipe data from RT tasks to 
the I/O subsystem.

It is possible to trigger delays in a priority-agnostic manner when a 
large number of tasks end up all hammering heavily on malloc().

Sven

> hofrat
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>    

Re: rt file i/o

[Mark Knecht]

On Wed, Nov 4, 2009 at 7:52 AM, Tim Blechmann <tim@klingt.org> wrote:
>>    I'm not a developer and cannot really address your specific
>> question but for my use of the rt-kernel I put important rt-audio
>> files on a 1394 drive and give the 1394 driver higher priorities using
>> the IRQ scheduling tools. I don't have any trouble running 2 or 3 1394
>> drives recording and playing back 48 channels in Ardour.
>>
>>    While I agree you should do everything the right way technically in
>> the code maybe part of your solution is outside of the app you are
>> writing and in the use of these support tools?
>
> well, if i understand the rt howto correctly, _no_ disc access is
> allowed, neither from rt nor from non-rt threads, since it may produce
> page faults, which introduce latencies ...
> according to this definition, ardour is not an rt-safe application ...
> as for rt-safety, i prefer analyzing the latency hotspots instead of
> relying on test results
>
> tim
>
> --
> tim@klingt.org
> http://tim.klingt.org
>
> Avoid the world, it's just a lot of dust and drag and means nothing in
> the end.
>  Jack Kerouac

In the limit you may be right. Possibly unbounded disk IO would never
be real-time safe. However in practice I see xruns when I use normal
priorities and none when I give my 1394 driver higher priorities.

Logically it doesn't make sense to me that *no* disk access is
allowed. I think the rt-threads just do their work and the drives
catch up when they can. Apps like Ardour are reading and writing the
hard drive all the time. Lots of files are open and streaming, in my
case maybe 40 files running for hours.

I think that disk access in an rt-thread make no sense at it cannot
response real-time, but non-rt disk activity just gets interrupted
like any other non-rt process, right?

Good luck with what you are up to.

Cheers,
Mark
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rt file i/o

[Nicholas Mc Guire]

On Wed, 04 Nov 2009, Tim Blechmann wrote:

> hi all,
> 
> having read the howto build an rt-application, i am trying to think of a
> way to handle disk i/o in my application.
> 
> the howto suggest to avoid file i/o in non-rt helper threads, but to
> move them to another process. while the howto suggests using network
> socket, i'm curious, if other ipc primitives like shared memory would
> give better throughput? or would i experience any issues, like is it
> possible to lock shared memory to physical ram?
>
this is probably a bit off topic - but both RTLinux/GPL (RTLFS) and L4 have 
implemented RT IDE drivers - limited to dedicated discs - the performance
is really bad (aprox. 800k/s) - so determinism on a device that really 
lives from cache/pre-fetch and large-block access is very expensive.
What such a driver can give you at best is a guaranteed bandwidth, and
a worst case latency that is not really too atractive (tens of milliseconds)

So probably the best option is to have a user-space/non-RT driver access the
disk and pre-fetch/cache a "large" chunk of what you want to read/write to
de-couple RT from non-RT - this has been the model used i.e. for RAT (Real-time 
Audio Tools see proceedings of the 6th Real Time Linux Workshop - Singapur)
quite successfully - as long as the I/O rate your RT-app expects is 
sufficiently much smaller than the I/O capability of the device driver AND you
ensure the bandwidth (i.e. by raising the respective irq-thread priority, and
restricting other I/O to that controler/disk) this is actually a quite 
deterministic approach provided your buffering is not too small.

Generally you will not be able to analyze paths of complex hardware/software
like disk-subsystems in a GPOS to the point were you can give absolute 
guarantees - what you can achieve though is statistic guarantees, and in
combination with monitoring of the cached data you can (atleast for 
some access patters) give guarantees about the behavior of the system.

hofrat

RAT http://www.osadl.org/Papers.rtlws-2004-papers.0.html#PAPER_2590
RTLFS http://www.osadl.org/Papers.rtlws-2004-papers.0.html#PAPER_1889