Real-time problem due to IO congestion.

Real-time problem due to IO congestion.

[Takashi Ikebe]

Hello,
I have been encountering big real-time problem due to IO congestion, and
I want some advices.

-The problem description-
There are 2 type processes in test environment.
1. The real-time needed process (run on with high static priority)
    The process wake up every 10ms, and wake up, write some log (the
test case is current CPU clock via tsc) to the file.

2. The process which make IO load
    The process have large memory size, and kill the process with dumping.
    The process's memory area exceeds 70% of whole physical
RAM.(Actually 1.5GB memory area while whole RAM is 2GB)

Whenever during dumping, the real-time needed process sometimes stop for
long time during write system call. (sometimes exceeds 1000ms)
I tested every IO scheduler but the same problem occurs.
I also seek this problem into the code, and find that the stops are
mainly occurring on blk_congestion_wait/get_request/get_request_wait
functions located on drivers/block/ll_rw_blk.c.

-My assumption-
The design of IO(read/write) queue and queuing is not well match to
real-time needed processes.
If there are many IO requests by low priority processes already, then
the IO request by high priority process should wait until queue goes
clean, and this cause some kind of priority inversions.

-My suggestion-
Add the new IO scheduler or change current IO scheduler to reflect
process's priority on queuing.

I don't know my assumption and suggestion are correct and you like,
would you give me some advices?

Thank you.
-- 
Takashi Ikebe

Re: Real-time problem due to IO congestion.

[Jens Axboe]

On Fri, Jun 10 2005, Takashi Ikebe wrote:
> Hello,
> I have been encountering big real-time problem due to IO congestion, and
> I want some advices.
> 
> -The problem description-
> There are 2 type processes in test environment.
> 1. The real-time needed process (run on with high static priority)
>     The process wake up every 10ms, and wake up, write some log (the
> test case is current CPU clock via tsc) to the file.
> 
> 2. The process which make IO load
>     The process have large memory size, and kill the process with dumping.
>     The process's memory area exceeds 70% of whole physical
> RAM.(Actually 1.5GB memory area while whole RAM is 2GB)
> 
> Whenever during dumping, the real-time needed process sometimes stop for
> long time during write system call. (sometimes exceeds 1000ms)
> I tested every IO scheduler but the same problem occurs.
> I also seek this problem into the code, and find that the stops are
> mainly occurring on blk_congestion_wait/get_request/get_request_wait
> functions located on drivers/block/ll_rw_blk.c.
> 
> -My assumption-
> The design of IO(read/write) queue and queuing is not well match to
> real-time needed processes.
> If there are many IO requests by low priority processes already, then
> the IO request by high priority process should wait until queue goes
> clean, and this cause some kind of priority inversions.
> 
> -My suggestion-
> Add the new IO scheduler or change current IO scheduler to reflect
> process's priority on queuing.
> 
> I don't know my assumption and suggestion are correct and you like,
> would you give me some advices?

This basically needs io priorities to work, so that request allocation
is prioritized as well. I didn't actually add request allocation groups
in the cfq-ts posted with priority support, however I have some patches
from years ago that did so. I'll see if I can find the time to brush
those off.

-- 
Jens Axboe

Re: Real-time problem due to IO congestion.

[Andrew Morton]

Takashi Ikebe <ikebe.takashi@lab.ntt.co.jp> wrote:
>
> There are 2 type processes in test environment.
>  1. The real-time needed process (run on with high static priority)
>      The process wake up every 10ms, and wake up, write some log (the
>  test case is current CPU clock via tsc) to the file.
> 
>  2. The process which make IO load
>      The process have large memory size, and kill the process with dumping.
>      The process's memory area exceeds 70% of whole physical
>  RAM.(Actually 1.5GB memory area while whole RAM is 2GB)
> 
>  Whenever during dumping, the real-time needed process sometimes stop for
>  long time during write system call. (sometimes exceeds 1000ms)

The writeback code does attempt to give some preference to realtime tasks
(in get_dirty_limits()), but it can only work up to a point.

Frankly, your application is poorly designed.  If you want sub-10ms
responsiveness you shouldn't be doing disk I/O.  The realtime task should
hand the data off to a non-realtime task for writeout, with suitable
amounts of buffering in between.

Re: Real-time problem due to IO congestion.

[Takashi Ikebe]

I see.
The program which I tested is just sample, and I wanted to know the 
phenomena is spec or bug.
I also understand that this problem is spec, and need to apply some 
buffering to such applications.

Thank you.

Andrew Morton wrote:
> Takashi Ikebe <ikebe.takashi@lab.ntt.co.jp> wrote:
> 
>>There are 2 type processes in test environment.
>> 1. The real-time needed process (run on with high static priority)
>>     The process wake up every 10ms, and wake up, write some log (the
>> test case is current CPU clock via tsc) to the file.
>>
>> 2. The process which make IO load
>>     The process have large memory size, and kill the process with dumping.
>>     The process's memory area exceeds 70% of whole physical
>> RAM.(Actually 1.5GB memory area while whole RAM is 2GB)
>>
>> Whenever during dumping, the real-time needed process sometimes stop for
>> long time during write system call. (sometimes exceeds 1000ms)
> 
> 
> The writeback code does attempt to give some preference to realtime tasks
> (in get_dirty_limits()), but it can only work up to a point.
> 
> Frankly, your application is poorly designed.  If you want sub-10ms
> responsiveness you shouldn't be doing disk I/O.  The realtime task should
> hand the data off to a non-realtime task for writeout, with suitable
> amounts of buffering in between.

-- 
Takashi Ikebe

Re: Real-time problem due to IO congestion.

[Jens Axboe]

(please don't top post)

On Fri, Jun 10 2005, Takashi Ikebe wrote:
> 
> I see.
> The program which I tested is just sample, and I wanted to know the 
> phenomena is spec or bug.
> I also understand that this problem is spec, and need to apply some 
> buffering to such applications.

Additionally, following up on Andrew, even with prioritized request
allocations you could get equally long stuck if you just had lots of
high prio allocaters queueing io. So rethinking the setup is definitely
good advice.

-- 
Jens Axboe

Re: Real-time problem due to IO congestion.

[Takashi Ikebe]

Jens Axboe wrote:
> On Fri, Jun 10 2005, Takashi Ikebe wrote:
> 
> This basically needs io priorities to work, so that request allocation
> is prioritized as well. I didn't actually add request allocation groups
> in the cfq-ts posted with priority support, however I have some patches
> from years ago that did so. I'll see if I can find the time to brush
> those off.
> 

As you and andrew said, basically application based approach seems 
reasonable,
but I'm so interesting your patch, if you have time, please show me :-)

Thank you.

-- 
Takashi Ikebe

Re: Real-time problem due to IO congestion.

[Lee Revell]

On Fri, 2005-06-10 at 16:49 +0900, Takashi Ikebe wrote:
> Jens Axboe wrote:
> > On Fri, Jun 10 2005, Takashi Ikebe wrote:
> > 
> > This basically needs io priorities to work, so that request allocation
> > is prioritized as well. I didn't actually add request allocation groups
> > in the cfq-ts posted with priority support, however I have some patches
> > from years ago that did so. I'll see if I can find the time to brush
> > those off.
> > 
> 
> As you and andrew said, basically application based approach seems 
> reasonable,
> but I'm so interesting your patch, if you have time, please show me :-)

Take a look at the lock-free ringbuffers in JACK.  No point reinventing
the wheel...

I think it's a bit strange that the disk IO issue comes up so often in
these RT discussions, it's something of a red herring, because there's
rarely an RT constraint in getting the data to disk; the RT constraint
is in getting the data from the device to memory.

Lee

Re: Real-time problem due to IO congestion.

[Helge Hafting]

On Fri, Jun 10, 2005 at 04:13:09PM +0900, Takashi Ikebe wrote:
> 
> I see.
> The program which I tested is just sample, and I wanted to know the 
> phenomena is spec or bug.
> I also understand that this problem is spec, and need to apply some 
> buffering to such applications.
> 
There is an alternative.  Get a separate disk just for the RT-job.
You can then run your very heavy IO on other disks, the RT disk won't
be delayed by that.

Helge Hafting

Re: Real-time problem due to IO congestion.

[Takashi Ikebe]

Thank you for your kind advices!

Our system has characteristics such as embedded system, and, 
unfortunately, such systems have the restriction that available disk is 
only 1(raid 1 mirrored one).
But the system has large memory (maximum 16GB), and SMP. (such as blade 
type system)

I think this problem can also be solved by modifying page cache 
transactions(assume there is huge page cache!), and this may guarantee 
somewhat real-time write even on single normal disk.
but this may be only the answer in case of huge page-cache, and may be 
not for user desktops.

what I'm considering is that the this approach is acceptable by kernel 
community or not?? (as you mentioned, kernel modify is one approach, and 
app level buffering is another approach).
If I or somebody made some patch, is there any possibility to be 
accepted? or this type of problem should not go into kernel?


Helge Hafting wrote:
> On Fri, Jun 10, 2005 at 04:13:09PM +0900, Takashi Ikebe wrote:
> 
>>I see.
>>The program which I tested is just sample, and I wanted to know the 
>>phenomena is spec or bug.
>>I also understand that this problem is spec, and need to apply some 
>>buffering to such applications.
>>
> 
> There is an alternative.  Get a separate disk just for the RT-job.
> You can then run your very heavy IO on other disks, the RT disk won't
> be delayed by that.
> 
> Helge Hafting
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/


-- 
Takashi Ikebe
NTT Network Service Systems Laboratories
9-11, Midori-Cho 3-Chome Musashino-Shi,
Tokyo 180-8585 Japan
Tel : +81 422 59 4246, Fax : +81 422 60 4012
e-mail : ikebe.takashi@lab.ntt.co.jp