Re: Process Scheduling Issue using sg/libata

[James Chapman <jchapman@katalix.com>]

Mark Lord wrote:
> Fajun Chen wrote:
>>
>> As a matter of fact, I'm using /dev/sg*.  Due to the size of my test
>> application, I have not be able to compress it into a small and
>> publishable form. However, this issue can be easily reproduced on my
>> ARM XScale target using sg3_util code as follows:
>> 1. Run printtime.c attached,  which prints message to console in a loop.
>> 2. Run sgm_dd (part of sg3_util package, source code attached) on the
>> same system as follows:
>>> sgm_dd if=/dev/sg0 of=/dev/null count=10M bpt=1
>> The print task can be delayed for as many as 25 seconds. Surprisingly,
>> I can't reproduce the problem in an i386 test system with a more
>> powerful processor.
>>
>> Some clarification to MAP_ANONYMOUS option in mmap(). After fixing a
>> bug and more testing, this option seems to make no difference to cpu
>> load. Sorry about previous report. Back to the drawing board now :-)
> ..
> 
> Okay, I don't see anything unusual here.  The code is on a slow CPU,
> and is triggering 10MBytes of PIO over a (probably) slow bus to an ATA
> device.
> 
> This *will* tie up the CPU at 100% for the duration of the I/O,
> because the I/O happens in interrupt handlers, which are outside
> of the realm of the CPU scheduler.
> 
> This is a known shortcoming of Linux for real-time uses.
> 
> When the I/O uses DMA transfers, it *may* still have a similar effect,
> depending upon the caching in the ATA device, and on how the DMA shares
> the memory bus with the CPU.
> 
> Again, no surprise here.
> 
> One way to deal with it in an embedded device, is to force the
> application that's generating the I/O to self-throttle.
> Or modify the device driver to self-throttle.

Does disk access have to be so interrupt driven? Could disk interrupt
handling be done in a softirq/kthread like the networking guys deal with
network device interrupts? This would prevent the system from
live-locking when it is being bombarded with disk IO events. It doesn't
seem right that the disk IO subsystem can cause interrupt live-lock on
relatively slow CPUs...

> You may want to find an embedded Linux consultant to help out
> with this situation if it's beyond your expertise.

Check out the rtlinux patch, which pushes all interrupt handling out to
per-cpu kernel threads (irqd). The kernel scheduler then regains control
of what runs when.

Another option is to change your ATA driver to do interrupt processing
at task level using a workqueue or similar.

-- 
James Chapman
Katalix Systems Ltd
http://www.katalix.com
Catalysts for your Embedded Linux software development