Re: bigalloc and max file size

[Ted Ts'o <tytso@mit.edu>]

On Mon, Oct 31, 2011 at 06:27:25PM +0800, Tao Ma wrote:
> In the new bigalloc case if chunk size=64k, and with the linux-3.0
> source, every file will be allocated a chunk, but they aren't contiguous
> if we only write the 1st 4k bytes. In this case, writeback and the block
> layer below can't merge all the requests sent by ext4. And in our test
> case, the total io will be around 20000. While with the cluster size, we
> have to zero the whole cluster. From the upper point of view. we have to
> write more bytes. But from the block layer, the write is contiguous and
> it can merge them to be a big one. In our test, it will only do around
> 2000 ios. So it helps the test case.

This is test case then where there are lot of sub-64k files, and so
the system administrator would be ill-advised to use a 64k bigalloc
cluster size in the first place.  So don't really consider that a
strong argument; in fact, if the block device is a SSD or a
thin-provisioned device with an allocation size smaller than the
cluster size, the behaviour you describe would in fact be detrimental,
not a benefit.

In the case of a hard drive where seeks are expensive relative to
small writes, this is something which we could do (zero out the whole
cluster) with the current bigalloc file system format.  I could
imagine trying to turn this on automatically with a hueristic, but
since we can't know the underlying allocation size of a
thin-provisioned block device, that would be tricky at best...

Regards,

		       	       	    	     - Ted