Re: [rfc] direct IO submission and completion scalability issues

["Siddha, Suresh B" <suresh.b.siddha@intel.com>]

On Wed, Aug 01, 2007 at 02:41:18AM +0200, Nick Piggin wrote:
> On Tue, Jul 31, 2007 at 10:14:03AM -0700, Suresh B wrote:
> > Yes, softirq context is one way. But just didn't want to penalize the running
> > task by taking away some of its cpu time. With CFS micro accounting, perhaps
> > we can track irq, softirq time and avoid penalizing the running task's cpu
> > time.
> 
> But you "penalize" the running task in the completion handler as well
> anyway.

Yes.

Ingo, in general with CFS micro accounting, we should be able to avoid
penalizing the running task by tracking irq/softirq time. Isn't it?

> Doing this with a SCHED_FIFO task is sort of like doing interrupt
> threading which AFAIK has not been accepted (yet).

I am not recommending SCHED_FIFO. I will take a look at softirq
infrastructure for this.

> > This workload is using direct IO and there is no batching at the block layer
> > for direct IO. IO is submitted to the HW as it arrives.
> 
> So you aren't putting concurrent requests into the queue? Sounds like
> userspace should be improved.

Nick remember that there are hundreds of disks in this setup and at
an instance, there will be max 1 or 2 requests per disk.

> > It is applicable for both direct IO and buffered IO. But the implementations
> > will differ. For example in buffered IO, we can setup in such a way that the
> > block plug timeout function runs on the IO completion cpu.
> 
> It would be nice to be doing that anyway. But unplug via request submission
> rather than timeout is fairly common in buffered loads too.

Ok. Currently the patch handles both direct and buffered IO. While making
improvements to this patch I will make sure that both the paths take
advantage of this.

thanks,
suresh