Re: per device queues for cciss 2.6.0

[Jeff Garzik <jgarzik@pobox.com>]

Miller, Mike (OS Dev) wrote:
> Yes, the controller has a single command buffer. It can hold 1024 outstanding commands.

Ok, great.  Well then the carmel.c I sent you should be a good model -- 
carmel.c has per-device queues, and there are no starvation issues.  The 
code is contained within only a few LOC, in carm_push_q(), carm_pop_q(), 
and carm_round_robin().

As an aside, you should probably make the call to cciss_round_robin() 
conditional on the hardware's command buffer being at least 1/2 empty. 
(or pick whatever low water mark you like)

The command buffer size, 1024, is quite nice.  Given the same model as 
carmel.c, I predict that blk_{start,stop}_queue will be called quite 
infrequently -- that translates to _high_ performance on the cciss hardware.

Note the blk_{start,stop}_queue() were only recently fixed (grab latest 
2.6.4-rc), so that may have introduced noise into whatever testing and 
design you've done.

Now, per-queue locking, rather than per-HBA locking, definitely 
introduces some additional complexity.  I've got a good idea how to do 
that, which involves the each queue's request function kicking a common 
tasklet that queues commands to hardware.  But there's a lot of deadlock 
potentional if it's not done right, since you still need a common lock 
for the HBA when submitting and completing hardware commands.  So I 
would be interested to see some evidence of actual SMP contention on the 
per-HBA lock...

Regards,

	Jeff