Re: elevator priorities vs. full request queues

[Jens Axboe <axboe@suse.de>]

On Tue, Jun 22 2004, Werner Almesberger wrote:
> Another issue regarding priorities: let's suppose we have "high"
> and "low" priority requests. Those priorities are indicated by
> some means, and do not depend on CPU scheduler priorities. Let's
> say that the elevator always handles high priority requests
> first, unless barriers dictate it to do otherwise.
> 
> So far so good. Next, let's suppose that the request queue is
> filling up. Now all the processes doing IO will compete for
> request slots, and the outcome of this competition is
> determined by the CPU scheduler - which is blissfully unaware
> of those request priorities.
> 
> Has anybody thought of a solution for this problem ?

Well yes, different parties have worked on io priorities in the past.
Request allocation is a important part of that.

> I can think of a few possible approaches, but they're all
> relatively intrusive or plain ugly. A few of them:
> 
>  1) temporarily raise the CPU priority of processes waiting
>     for request queue slots
>  2) implement some priority scheme in ll_rw_blk.c
>  3) give elv_may_queue more information about what the request
>     will look like, so that it can decide to accept "high"
>     priority requests even if the queue is officially full
>  4) allow elv_add_request to refuse requests, e.g. to keep
>     some reserve for "high" priority requests
> 
> 1) and 2) would have the advantage that they also work if
> requests are delayed due to a lack of memory. However, in this
> case, all bets are off anyway, so I wouldn't worry too much
> about it.
> 
> 3) and 4) have the nice property of keeping all the priority
> logic entirely in the elevator. For 4), one would also have to
> force an interface change (besides giving elv_add_request a
> return value), to make sure there are no enqueuers left in the
> kernel or elsewhere which don't know that elv_add_request may
> now fail.

#4 doesn't work for various reasons, the obvious being that the request
is already allocated at this point. Or request could be coming from
outside get_request().

CFQ with priorities used a combination of 2+3. It seems you haven't
looked any of this up, maybe that would be a good starting point. You
can do this internally in your io scheduler in several ways. CFQ used
per-process (-group) priorities, so it would get the priority from
there. A bio has room for priority info, so you could also pass it in
from submit_bio() -> __make_request() -> get_request() -> may_queue().

> Opinions ?

#1 can complement a solution of 2+3, it's not a solution in itself imo.
You absolutely need some way of reserving requests for some processes,
or denying requests for other and priorities alone does not accomplish
that.

#4 is a definite no-go. Maybe you meant to specify a different function,
I don't think your suggestion makes any sense.

-- 
Jens Axboe