Is it possible to implement a real-time API to read the files on the disk?

Is it possible to implement a real-time API to read the files on the disk?

[孙世龙 sunshilong]

Is it possible to implement a real-time API to read the files on the disk?

As far as I know, there is no such an interface now, but is it
possible to achieve this goal?

I would be grateful to have some help with this question.

Best Regards
Sunshilong

Re: Is it possible to implement a real-time API to read the files on the disk?

[Julien Blanc]

Le lundi 04 janvier 2021 à 19:58 +0800, 孙世龙 sunshilong via Xenomai a
écrit :
> Is it possible to implement a real-time API to read the files on the disk?
> 
> As far as I know, there is no such an interface now, but is it
> possible to achieve this goal?

I doubt it could be. It depends on what you mean by "real-time", but
disks (hard-disks and ssds as well) typically have non-deterministic
response times, which breaks any real-time promise.

There may be a way to define an api using only asynchronous operations.
I don't know enough of the low-level stuff involved in sata to have a
strong opinion on this. It's possible with some flash devices and
microcontrollers.

The other way is: do your i/o in a linux thread, communicating
asynchronously with your rt threads.

Regards,

Julien

Re: Is it possible to implement a real-time API to read the files on the disk?

[Fino Meng]

On Mon, Jan 04, 2021 at 07:58:53PM +0800, 孙世龙 sunshilong via Xenomai wrote:
> Is it possible to implement a real-time API to read the files on the disk?
> 
> As far as I know, there is no such an interface now, but is it
> possible to achieve this goal?
> 
> I would be grateful to have some help with this question.
> 
> Best Regards
> Sunshilong
>

my first thinking, disks are heavily shared resources in a general OS, if
realtime API occupy it for a long time, non-realtime thread all have to
wait.

read a file is too much slow compare a realtime thread's latency/jitter
scope (< 20us)

depends on your use case, maybe possible to add a raw disk, only used by
the time critical code?

BR/fino 

Re: Is it possible to implement a real-time API to read the files on the disk?

[孙世龙 sunshilong]

>depends on your use case, maybe possible to add a raw disk, only used by
>the time critical code?
It seems a good solution.
It's possible to add a raw disk.
One more question, does it could guarantee the read operation achieve a
real-time performance by this method(i.e occupying another disk)?
Is there still some problem that I should be aware of?

On Wed, Jan 6, 2021 at 8:05 AM Fino Meng <fino.meng@linux.intel.com> wrote:
>
> On Mon, Jan 04, 2021 at 07:58:53PM +0800, 孙世龙 sunshilong via Xenomai wrote:
> > Is it possible to implement a real-time API to read the files on the disk?
> >
> > As far as I know, there is no such an interface now, but is it
> > possible to achieve this goal?
> >
> > I would be grateful to have some help with this question.
> >
> > Best Regards
> > Sunshilong
> >
>
> my first thinking, disks are heavily shared resources in a general OS, if
> realtime API occupy it for a long time, non-realtime thread all have to
> wait.
>
> read a file is too much slow compare a realtime thread's latency/jitter
> scope (< 20us)
>
> depends on your use case, maybe possible to add a raw disk, only used by
> the time critical code?
>
> BR/fino

Re: Is it possible to implement a real-time API to read the files on the disk?

[Fino Meng]

On Thu, Jan 07, 2021 at 10:03:06AM +0800, 孙世龙 sunshilong wrote:
> >depends on your use case, maybe possible to add a raw disk, only used by
> >the time critical code?
> It seems a good solution.
> It's possible to add a raw disk.
> One more question, does it could guarantee the read operation achieve a
> real-time performance by this method(i.e occupying another disk)?
> Is there still some problem that I should be aware of?
> 

raw disk is not shared, then should not mount it.
the drivers of file system and harddisk belongs to non-realtime vanilla
Linux. Choose a fastest disk (PCIE NVMe disk?), then need to implement a
RTDM driver for it, that's quite a lot of work,
those raised in my mind by far,

BR/fino

> On Wed, Jan 6, 2021 at 8:05 AM Fino Meng <fino.meng@linux.intel.com> wrote:
> >
> > On Mon, Jan 04, 2021 at 07:58:53PM +0800, 孙世龙 sunshilong via Xenomai wrote:
> > > Is it possible to implement a real-time API to read the files on the disk?
> > >
> > > As far as I know, there is no such an interface now, but is it
> > > possible to achieve this goal?
> > >
> > > I would be grateful to have some help with this question.
> > >
> > > Best Regards
> > > Sunshilong
> > >
> >
> > my first thinking, disks are heavily shared resources in a general OS, if
> > realtime API occupy it for a long time, non-realtime thread all have to
> > wait.
> >
> > read a file is too much slow compare a realtime thread's latency/jitter
> > scope (< 20us)
> >
> > depends on your use case, maybe possible to add a raw disk, only used by
> > the time critical code?
> >
> > BR/fino

Re: Is it possible to implement a real-time API to read the files on the disk?

[Wolfgang Denk]

Dear 孙世龙 sunshilong,

In message <CAAvDm6bxvLaeRYvTpQM9CgewnK5qRrpV8ExQSuJ72d3axfcYRA@mail.gmail.com> you wrote:
> >depends on your use case, maybe possible to add a raw disk, only used by
> >the time critical code?
> It seems a good solution.
> It's possible to add a raw disk.

This will not help you at all.  The key problem here is that common
storage devices (conventional rotating hard disk drives, SSD,
SDCards, eMMC, USB sticks, CF Cards, ....) are all not real-time
capable itself.

They all contain controllers to manage the storage device, and
strict real-time behaviour has never been a design criteria for the
firmware running on these controllers.

Rotating harddisks can do all kind of maintainance tasks, including
retries when they detect positioning and/or I/O errors,
re-allocation of defective sectores, recalibration, etc. Not to
mention that - depending on your configuration - they may even spin
down to save power and will need many seconds to spin up when waking
up.

All other storage devices are flash based, so the controller will
have to take care of wear levelling, trimming, garbage collection
etc.

Try to get any guaranteed data for response times from any of the
vendors of your prefereed sotage data (i. e. ask: what is the
_guaranteed_ maximum time between sending a read request for a
specific block of data and receiving this data on the interface?)
I would not expect any precise technical answers.


You can guarantee _average_ data for I/O to storage devices, like a
guaranteed sustained throughput (*), but you will not be able to
guarantee any hard real-time behaviour (like latencies or such).

(*) And this guarantee will hold only as long as you do not
encounter any errors on the storage device, which can _always_
happen, so your real-time processes should better not depend on it
if they control any critical equipment.


Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
As long as we're going to reinvent the wheel again, we might as  well
try making it round this time.                        - Mike Dennison

Re: Is it possible to implement a real-time API to read the files on the disk?

[孙世龙 sunshilong]

I see. Thank you for your detailed explanation.

My understanding of this question is at a different level with your
generous help.

>You can guarantee _average_ data for I/O to storage devices, like a
>guaranteed sustained throughput (*), but you will not be able to
>guarantee any hard real-time behaviour (like latencies or such).
It seems that real-time OS could not guarantee a hard real-time
behaviour for reading files.

Two more questions arise:
1. A lot of real-time applications based on Xenomai have to meet this
 problem (i.e reading files), how do you solve it?
Reading files by non-rt thread and sending the data(or the pointer to
the data) to the rt thread by XDDP?

2.Can I draw the conclusion that there is no real-time OS that could
guarantee a hard real-time behaviour for reading?

Best regards.
Sunshilong

On Thu, Jan 7, 2021 at 7:14 PM Wolfgang Denk <wd@denx.de> wrote:
>
> Dear 孙世龙 sunshilong,
>
> In message <CAAvDm6bxvLaeRYvTpQM9CgewnK5qRrpV8ExQSuJ72d3axfcYRA@mail.gmail.com> you wrote:
> > >depends on your use case, maybe possible to add a raw disk, only used by
> > >the time critical code?
> > It seems a good solution.
> > It's possible to add a raw disk.
>
> This will not help you at all.  The key problem here is that common
> storage devices (conventional rotating hard disk drives, SSD,
> SDCards, eMMC, USB sticks, CF Cards, ....) are all not real-time
> capable itself.
>
> They all contain controllers to manage the storage device, and
> strict real-time behaviour has never been a design criteria for the
> firmware running on these controllers.
>
> Rotating harddisks can do all kind of maintainance tasks, including
> retries when they detect positioning and/or I/O errors,
> re-allocation of defective sectores, recalibration, etc. Not to
> mention that - depending on your configuration - they may even spin
> down to save power and will need many seconds to spin up when waking
> up.
>
> All other storage devices are flash based, so the controller will
> have to take care of wear levelling, trimming, garbage collection
> etc.
>
> Try to get any guaranteed data for response times from any of the
> vendors of your prefereed sotage data (i. e. ask: what is the
> _guaranteed_ maximum time between sending a read request for a
> specific block of data and receiving this data on the interface?)
> I would not expect any precise technical answers.
>
>
> You can guarantee _average_ data for I/O to storage devices, like a
> guaranteed sustained throughput (*), but you will not be able to
> guarantee any hard real-time behaviour (like latencies or such).
>
> (*) And this guarantee will hold only as long as you do not
> encounter any errors on the storage device, which can _always_
> happen, so your real-time processes should better not depend on it
> if they control any critical equipment.
>
>
> Best regards,
>
> Wolfgang Denk
>
> --
> DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
> Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
> As long as we're going to reinvent the wheel again, we might as  well
> try making it round this time.                        - Mike Dennison

Re: Is it possible to implement a real-time API to read the files on the disk?

[Wolfgang Denk]

Dear 孙世龙 sunshilong,

In message <CAAvDm6ZsQwrFoLfjnespyx65cmzWgYXGk=dyUE_UD5-dAUKaNQ@mail.gmail.com> you wrote:
>
> >You can guarantee _average_ data for I/O to storage devices, like a
> >guaranteed sustained throughput (*), but you will not be able to
> >guarantee any hard real-time behaviour (like latencies or such).
> It seems that real-time OS could not guarantee a hard real-time
> behaviour for reading files.
>
> Two more questions arise:
> 1. A lot of real-time applications based on Xenomai have to meet this
>  problem (i.e reading files), how do you solve it?
> Reading files by non-rt thread and sending the data(or the pointer to
> the data) to the rt thread by XDDP?

Yes, use another task (process or thread).

> 2.Can I draw the conclusion that there is no real-time OS that could
> guarantee a hard real-time behaviour for reading?

Thisis correct, assuming "reading" means reading from conventional
rotating hard disks, SSD drives or any other kind of NAND flash
based storage devices (USB sticks, eMMC, SDCard, CompactFlash
etc.).  None of these devices guarantee any RT capabilites (*).

(*) As far as I can tell; there may be special models with
alternative firmware which is tailored for RT responses -
technically that would be possible, but I have never seen any such
thing, and a quick search on your favorite search engine comes up
empty.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
It may be bad manners to talk with your mouth full, but it isn't  too
good either if you speak when your head is empty.