[LSF/MM TOPIC] Sparseness in storage

[Douglas Gilbert <dgilbert@interlog.com>]

There are a lot of zeros out there. Efficient use of sparseness
involves techniques to detect large quantities of zeros in
advance rather than just reading them all. And on the write side
there are standard techniques to append zeros to a file without
actually writing them.

Seems a damn shame to read a terabyte of zeros and then write them
to another device or file. Carrying the idea further: if we know
random data has no meaning *** and we are asked to copy it,
why not "write" zeros to the output file?

Over the last few years various commands have been added to the
SCSI and ATA command sets to better handle sparseness (and
trim/unmap/write_same can be viewed in this light). File systems
are improving their sparseness handling as well, with Linux
playing "catch up" to NTFS in this regard (e.g. the new
FALLOC_FL_PUNCH_HOLE flag in fallocate() ).

So I am proposing a discussion of the:
   - existing SCSI commands to support sparseness
   - existing ATA commands to support sparseness
   - suggestions for more sparseness support to be
     added to the SCSI and ATA command sets
   - user space tools that support sparseness
   - file system support for sparseness

Perhaps the latter point should involve the file system track as
well.

Doug Gilbert
20100202


*** For example: after ATA CRYPTO SCRAMBLE EXT command (which
     is one of the "sanitize device" commands and is fast) the
     data read will be random and meaningless. If the disk does
     "read zero after trim" why not follow the scramble with a
     trim/unmap of the whole disk?