Re: [patch 2.5] ips queue depths

Patrick Mansfield
2002-10-16T19:04:42Z

On Tue, Oct 15, 2002 at 10:32:31PM -0400, Doug Ledford wrote:
> On Tue, Oct 15, 2002 at 04:30:57PM -0700, Patrick Mansfield wrote:
>
> > If the adapter is setting queue depth based upon what
> > the adapter knows, that is completely wrong
>
> I strongly disagree here. Only the adapter driver knows if the card
> itself has a hard limit of X commands at a time (so that deeper queues are
> wasted). Only the adapter driver knows if the particular interconnect has
> any inherent queue limitations or speed limitations. There are a thousand
> things only the adapter driver knows that should be factored into a sane
> queue depth.

OK, the adapter does not get it completely wrong, but it does not
know about special scsi device limitations, block layer limits, usage
patterns, or total number of scsi devices on the system.

Your changes (set high, adjust lower as we hit queue fulls) should
work fine in most cases, and is much better than the previous state.

Some example cases where we might want to lower queue depth:

System with small amounts of memory compared to the number of devices.

With many disks on a system, some with a very light load, it could be
give the lightly loaded disks a lower queue depth so they use less
memory, or so they can do less IO.

In a 2 node cluster with shared devices, the queue depth could be set to
half of some hard limit on each node of the cluster, and avoid hitting
any hard queue fulls.

(It would really nice if we could modify the number of struct request's
allocated for little used or unused devices, or on character devices like
tape that don't even use the requests. Current block code allocates 2*128
of these on systems with lots of memory, this could save way more space
than lowering the queue depth.)

> > It would nice to have more device model/kernfs device attributes.
> >
> > With your new queueing code, if we expose and allow setting new_queue_depth,
> > user code can easily modify the queue depth.
>
> No. For the reasons that the driver knows what it's capable of, there
> really is *very* little tunability here. But, should you really need it,
> the drivers should provide it (both mine and Justin's aic7xxx drivers do
> so run time via module options or boot command options). Make the default
> a good common default, let users override if necessary. But I wouldn't
> make it a fiddle knob that people can tweak without thinking. Regardless
> of where you make it adjustable though, it has to be passed through the
> low level drivers so that they can adjust their internal data structs as
> needed (some won't have to do anything, some will have to do allocations
> or frees on every adjustment, so you *have* to pass all adjustments
> through them).

I was suggesting a common interface via a Scsi_Device device attribute
so the default depth can be modified as needed, rather than a fixed
boot or module load option that is fixed (once the driver is loaded)
and might be a different option for every adapter driver.

It's too bad we can't modify new_queue_depth and have all the layers
(well mid and lower) adjust accordingly.

-- Patrick Mansfield