Re: Actually using the sg table/chain code

[James Bottomley <James.Bottomley@HansenPartnership.com>]

On Tue, 2008-01-15 at 14:52 -0500, Jeff Garzik wrote:
> James Bottomley wrote:
> > 	if (BLK_DEF_MAX_SECTORS > max_sectors)
> > 		q->max_hw_sectors = q->max_sectors = max_sectors;
> >  	else {
> > 		q->max_sectors = BLK_DEF_MAX_SECTORS;
> > 		q->max_hw_sectors = max_sectors;
> > 	}
> > }
> > 
> > So it imposes a maximum possible setting of BLK_DEF_MAX_SECTORS which is
> > defined in blkdev.h to .... 1024, thus also forcing the queue down to
> > 128 scatterlist entries.
> > 
> > Once I raised this limit as well, I was able to transfer over 128
> > scatterlist elements during benchmark test runs of normal I/O (actually
> > kernel compiles seem best, they hit 608 scatterlist entries).
> > 
> > So my question, is there any reason not to raise this limit to something
> > large (like 65536) or even eliminate it altogether?
> 
> ISTR a thread long ago, perhaps including Andrea A (as well as Jens), 
> where 1024 sectors was arrived upon as a reasonable balance between 
> tying up gobs of VM memory on a single command (multiplied, then, across 
> N commands), and getting decent per-command throughput.
> 
> Jens probably recalls better than I...  but I'm pretty sure that the 
> 1024 limit played into "being nice with the VM" somehow.

There's certainly the writeout deadlock avoidance issue.  1024 sectors
is 128 scatterlist entries (at minimum physical merging), is currently
the maximum SCSI scatterlist from the mempools, so would always have a
logically provable forward progress clear path.  Once we go over this,
we start to get into corner cases where we have to hope that having deep
enough mempools makes the issue magically disappear.

The question is, how should we police all of this?  Should be block
layer really be blindly enforcing the 1024 sector limit ... particularly
now the default allocator won't begin chaining until we get 2048 sectors
or higher?

James