scsi_track_queue_full function - static values ?

scsi_track_queue_full function - static values ?

[Frederic TEMPORELLI]

Hi,


I was looking at the scsi_track_queue_full (driver/scsi/scsi.c) function.

Can someone tell me how have been defined all the static values in this function ?

- we may have (max) 16 (>>4) jiffies between calls (else there's no need to call 
this function...),

- queue_full_depth_count should be > 10 (else queue depth still not changed),

- if lun queue depth < 8, lun queue depth is set with cmd_per_lun
(what's happen if cmd_per_lun > 8 ???)


May someone add some #define for these values ?
Is it a way to use 'auto-adapted' values ?

Best regards.

-- 
Frederic TEMPORELLI

Re: scsi_track_queue_full function - static values ?

[Christoph Hellwig]

On Tue, Mar 07, 2006 at 02:32:44PM +0100, Frederic TEMPORELLI wrote:
> I was looking at the scsi_track_queue_full (driver/scsi/scsi.c) function.
> 
> Can someone tell me how have been defined all the static values in this 
> function ?
> 
> - we may have (max) 16 (>>4) jiffies between calls (else there's no need to 
> call this function...),
> 
> - queue_full_depth_count should be > 10 (else queue depth still not 
> changed),
> 
> - if lun queue depth < 8, lun queue depth is set with cmd_per_lun
> (what's happen if cmd_per_lun > 8 ???)
> 
> 
> May someone add some #define for these values ?
> Is it a way to use 'auto-adapted' values ?

I think Doug Ledford wrote that code, I've added him to the cc list
as he's probably the best one to answer your question.

While we're at it, it would be nice if more drivers used this functionality..

Re: scsi_track_queue_full function - static values ?

[Doug Ledford]

On Fri, Mar 10, 2006 at 04:40:27PM +0000, Christoph Hellwig wrote:
> On Tue, Mar 07, 2006 at 02:32:44PM +0100, Frederic TEMPORELLI wrote:
> > I was looking at the scsi_track_queue_full (driver/scsi/scsi.c) function.
> > 
> > Can someone tell me how have been defined all the static values in this 
> > function ?

Painful experience is how they were defined.  That said, I'll explain said
experience.

> > - we may have (max) 16 (>>4) jiffies between calls (else there's no need to 
> > call this function...),

QUEUE_FULLs happen in bunches.  When you have 10 commands waiting to go to a
drive, and you fill its queue, depending on the driver you will either block
the remaining 9 commands or all 10 commands will end up getting sent back to
back and all 10 will QUEUE_FULL out.  You want these mass QUEUE_FULL events
to be treated as a single QUEUE_FULL for the purpose of tracking the
device's queue depth.  In addition, you want to know the depth the device
was at, not how many commands the mid layer has created.  Only the driver
can now that since different drivers queue things differently internally,
there may be commands that are paused and not yet sent to the device but are
present on the card, etc.  Only the driver can know how many commands are
*truly* outstanding, and even then it can only really know when it has
confirmed that all currently outstanding commands besides the one it is
currently processing have been accepted by the device and not returned with
QUEUE_FULL as well.

> > - queue_full_depth_count should be > 10 (else queue depth still not 
> > changed),

There are three distinct scenarios resulting in QUEUE_FULL issues:

1)  A fixed command depth on a device.  This is the same each and every
time.

2) A variable command depth on a device (Quantum Atlas II/III drives with
write behind caching are really bad here).

3) Multi-initiator mixed with both of the above where the depth that we see
may not be the depth the device sees due to other SCSI hosts also sending
commands.

In order to avoid artificially throttling drives for momentary issues versus
fixed issues, track the queue depth count of the last queue full and if it
is the same repeatedly, then assume it's a fixed depth.  With the Quantum
drives previously mentioned, they have a fixed depth of 64, but will reduce
that as needed when too many write commands have been cached.  The heuristic
in that code will take a while (usually within a few minutes of starting
heavy load) to get the 64 hard limit on those drives, but it eventually
succeeds.

> > - if lun queue depth < 8, lun queue depth is set with cmd_per_lun
> > (what's happen if cmd_per_lun > 8 ???)

cmd_per_lun is (was?) defined as the driver's allowable queue depth on
untagged devices.  Since all untagged devices can never have more than 1
command outstanding at a time, any driver that sets cmd_per_lun > 1 must, by
definition, be able to do it's own internal queueing and respect the limit
of 1 command at a time on untagged devices.  In addition, we are clearing
the tagged operation bit for the device when we set it to cmd_per_lun.  This
is based on more experience, specifically that I have, in all my testing of
some really *crappy* scsi drives, never found a single drive that both A)
supported tagged queueing and B) had a hard limit of less than 8 (although a
few models, Quantum Fireballs in particular, did have a limit of 8, even
that was a rarity and most drives were either 32 or 64 or higher).  So, if
we ever get a drive that tells us a limit of less than 8 repeatedly, we
either have a bogus firmware that's horked, or we have a heavily
multi-initiator environment with starvation issues.  So, be on the safe side
and go untagged in case it's the firmware problem.

> > 
> > May someone add some #define for these values ?
> > Is it a way to use 'auto-adapted' values ?
> 
> I think Doug Ledford wrote that code, I've added him to the cc list
> as he's probably the best one to answer your question.
> 
> While we're at it, it would be nice if more drivers used this functionality..

Using it well requires a little care.  Due to the jitter problem you get
when you have a queue full barrage, the driver should really only call this
once it has a final count for the real depth, not on each QUEUE_FULL.  If
the driver doesn't want to do that, then the other option would be to modify
this routine so that at the beginning it does something like this:

	/*
	 * Catch repeated QUEUE_FULLs in a short period of time, but
	 * if depth is 1 less than previous depth, assume we are
	 * trickling in all the QUEUE_FULLs from a single batch and
	 * we need the lowest number, so let it fall through.
	 */
        if ((jiffies >> 4) == sdev->last_queue_full_time &&
	    (sdev->last_queue_full_depth - 1) != depth)
		return 0;

But doing this *greatly* increases the complexity of tracking the final
queue full depth as now you need a current queue full depth and a last final
queue full depth so you can compare where the trickling stops to where it
stopped last time in order to see if you have a repeat of the same depth.


-- 
  Doug Ledford <dledford@redhat.com>     919-754-3700 x44233
         Red Hat, Inc. 
         1801 Varsity Dr.
         Raleigh, NC 27606
  

Re: scsi_track_queue_full function - static values ?

[Arjan van de Ven]

> QUEUE_FULLs happen in bunches.  When you have 10 commands waiting to go to a
> drive, and you fill its queue, depending on the driver you will either block
> the remaining 9 commands or all 10 commands will end up getting sent back to
> back and all 10 will QUEUE_FULL out.  You want these mass QUEUE_FULL events
> to be treated as a single QUEUE_FULL for the purpose of tracking the
> device's queue depth.  In addition, you want to know the depth the device
> was at, not how many commands the mid layer has created.  Only the driver
> can now that since different drivers queue things differently internally,

drivers no longer queue thankfully ;)

> there may be commands that are paused and not yet sent to the device but are
> present on the card, etc. 

that'll be hard to find out anyway


> command outstanding at a time, any driver that sets cmd_per_lun > 1 must, by
> definition, be able to do it's own internal queueing and respect the limit
> of 1 command at a time on untagged devices. 


Hmmmmmmmm are you sure?

Re: scsi_track_queue_full function - static values ?

[Doug Ledford]

On Fri, Mar 10, 2006 at 07:39:50PM +0100, Arjan van de Ven wrote:
> 
> > QUEUE_FULLs happen in bunches.  When you have 10 commands waiting to go to a
> > drive, and you fill its queue, depending on the driver you will either block
> > the remaining 9 commands or all 10 commands will end up getting sent back to
> > back and all 10 will QUEUE_FULL out.  You want these mass QUEUE_FULL events
> > to be treated as a single QUEUE_FULL for the purpose of tracking the
> > device's queue depth.  In addition, you want to know the depth the device
> > was at, not how many commands the mid layer has created.  Only the driver
> > can now that since different drivers queue things differently internally,
> 
> drivers no longer queue thankfully ;)

Drivers always queue and always will ever since the age of the 1542 ISA
cards.  There isn't a SCSI host in use today that doesn't have some sort of
smarts or scripts or other things that mean that a command on the card !=
command on the drive.  Move things up to the mid layer that make sense there
and where you can do them well there, but don't move them there for the sake
of making driver writing a lowest common demoninator profession, especially
when doing so cripples a driver's ability to do the "Right Thing".  Counting
the actual queue depth is a driver specific thing and it will be until the
entire current crop of hardware drastically changes.

> > there may be commands that are paused and not yet sent to the device but are
> > present on the card, etc. 
> 
> that'll be hard to find out anyway

Not for the card driver.  Not to speak for the aic7xxx, but at least in the
aic7xxx_old driver, on QUEUE_FULL, my driver did the following things:

1) looking for any commands on the device queue (this is where commands from
aic7xxx_queue go to first, it's in the driver itself, not on the card) and
move them to the waiting queue (another driver internal queue that was
monitored and when it wasn't empty, a backup timer was set to make sure it
didn't stall permanently).

2) pause sequencer on card, look for the current position of the sequencer
on the QIN_FIFO.  If it isn't at the head, then search from current position
to head looking for commands for the problem device, if found, move those to
the head of the waiting queue, update head pointer of queue.

3) check current command in sequencer to make sure it isn't a command for
target device that hasn't been sent yet and may be in selection, if so, kill
it and set sequencer to restart on unpause

4) check waiting list on card to see if any commands are waiting for
selection (only happens on first command send, never for reselection as
that's target driven and it's the disconnected list instead), if they match
target, kill them as well.  If no command left on list or in process, make
sure selection is killed on the bus.

5) check QOUT_FIFO for commands that have already completed but not yet
serviced by an interrupt that belong to the device

After clearing all that stuff out, then and only then do I have an accurate
count of what depth the device was at when it sent a QUEUE_FULL.

For every single card with a firmware or scripts this is going to be
different depending on how that firmware handles things.  So, I say again,
drivers *ALWAYS* queue, whether you choose to acknowledge it or not is your
business.  Handling queue fulls properly will, therefore, *never* be
something that can be genericized fully.  My code was an attempt to
genericize just that part which is common across all devices, the
host/target interaction.  The host specific stuff was all left up to the
driver.

> 
> > command outstanding at a time, any driver that sets cmd_per_lun > 1 must, by
> > definition, be able to do it's own internal queueing and respect the limit
> > of 1 command at a time on untagged devices. 
> 
> 
> Hmmmmmmmm are you sure?

Try sending more than 1 untagged command to a device at a time.  Instant
error.  So, of course if cmd_per_lun is > 1 and the device is untagged it
must be queued somewhere.  Where may be a question now, but it used to be
either in the driver or the hardware.  In my case, it was the driver and
that was specifically so that on command completion of an untagged command I
would have the next command ready to go and could stick it in the cards
QIN_FIFO during the interrupt handler, thereby improving things like round
trip time on tape drives so as to not interrupt streaming.  If it's at the
mid layer level now, then you've taken away a driver's ability to do what I
just mentioned.


-- 
  Doug Ledford <dledford@redhat.com>     919-754-3700 x44233
         Red Hat, Inc. 
         1801 Varsity Dr.
         Raleigh, NC 27606
  

Re: scsi_track_queue_full function - static values ?

[Arjan van de Ven]

On Fri, 2006-03-10 at 14:16 -0500, Doug Ledford wrote:
> > 
> > > command outstanding at a time, any driver that sets cmd_per_lun > 1 must, by
> > > definition, be able to do it's own internal queueing and respect the limit
> > > of 1 command at a time on untagged devices. 
> > 
> > 
> > Hmmmmmmmm are you sure?
> 
> Try sending more than 1 untagged command to a device at a time.  Instant
> error.  So, of course if cmd_per_lun is > 1 and the device is untagged it
> must be queued somewhere. 

sure

>  Where may be a question now, but it used to be
> either in the driver or the hardware.  In my case, it was the driver and
> that was specifically so that on command completion of an untagged command I
> would have the next command ready to go and could stick it in the cards
> QIN_FIFO during the interrupt handler, thereby improving things like round
> trip time on tape drives so as to not interrupt streaming.  If it's at the
> mid layer level now, then you've taken away a driver's ability to do what I
> just mentioned.

drivers nowadays do NOT have queues (or if they do, they're left over
and should be fixed). Of course a hardware based queue doesn't count in
this sense...

Re: scsi_track_queue_full function - static values ?

[Hannes Reinecke]

Doug Ledford wrote:
[ .. ]
> Not for the card driver.  Not to speak for the aic7xxx, but at least in the
> aic7xxx_old driver, on QUEUE_FULL, my driver did the following things:
> 
[ Lots of instructive text deleted ]
> 
> After clearing all that stuff out, then and only then do I have an accurate
> count of what depth the device was at when it sent a QUEUE_FULL.
> 
Thanks Doug, that was exactly that bit of information I was looking for.
Currently this mechanism doesn't work anymore with aic79xx and now I
have to figure out what I've broken during upgrade. Your description
will come in handy here.

Thanks a lot :-)

Cheers,

Hannes
-- 
Dr. Hannes Reinecke			hare@suse.de
SuSE Linux Products GmbH		S390 & zSeries
Maxfeldstraße 5				+49 911 74053 688
90409 Nürnberg				http://www.suse.de

-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html