SEEK_HOLE and SEEK_DATA

SEEK_HOLE and SEEK_DATA

[linux]

Solaris 10 has added a moderately useful new feature...  lseek now
supports whence = 3 (SEEK_DATA) and 4 (SEEK_HOLE).  What these do is
advance the file pointer to the start of the next run of the appropriate
kind past the given (absolute) offset.

This is, of course, to make backing up and copying sparse files more
efficient.

I'm still figuring out the fine details of semantics.  EOF is considered
the start of a hole.  If the seek position is past EOF, they return ENXIO.

I'm still trying to figure out if they search > the given offset or >=.
Reading the code, it actually looks like lseek(fd, 13, SEEK_DATA) will
return 0 on a non-sparse file, because they round down to blocks and
then search by blocks.

Not that this affects the usual case where you start at offset 0 and
alternate SEEK_DATA/SEEK_HOLE to find ranges to copy.


I was just wondering if it's an extension worth adopting.

Re: SEEK_HOLE and SEEK_DATA

[Chris Wedgwood]

On Thu, Feb 02, 2006 at 04:03:49AM -0500, linux@horizon.com wrote:

> Solaris 10 has added a moderately useful new feature...  lseek now
> supports whence = 3 (SEEK_DATA) and 4 (SEEK_HOLE).  What these do is
> advance the file pointer to the start of the next run of the
> appropriate kind past the given (absolute) offset.

Some filesystems in linux (XFS for one, probably others?) already have
a mechanism (albeit more complex the way it's exposed) to find holes &
data.

Re: SEEK_HOLE and SEEK_DATA

[Szakacsits Szabolcs]

On Fri, 3 Feb 2006, Chris Wedgwood wrote:
> On Thu, Feb 02, 2006 at 04:03:49AM -0500, linux@horizon.com wrote:
> 
> > Solaris 10 has added a moderately useful new feature...  lseek now
> > supports whence = 3 (SEEK_DATA) and 4 (SEEK_HOLE).  What these do is
> > advance the file pointer to the start of the next run of the
> > appropriate kind past the given (absolute) offset.
> 
> Some filesystems in linux (XFS for one, probably others?) already have
> a mechanism (albeit more complex the way it's exposed) to find holes &
> data.

NTFS, though not the Linux driver. Both XFS_IOC_GETBMAPX 
and FSCTL_QUERY_ALLOCATED_RANGE are much more efficient 
than SEEK_HOLE/SEEK_DATA (and let's forget about FIBMAP).

	Szaka

Re: SEEK_HOLE and SEEK_DATA

[Chris Wedgwood]

On Fri, Feb 03, 2006 at 08:23:51PM +0200, Szakacsits Szabolcs wrote:

> NTFS, though not the Linux driver. Both XFS_IOC_GETBMAPX and
> FSCTL_QUERY_ALLOCATED_RANGE are much more efficient than
> SEEK_HOLE/SEEK_DATA (and let's forget about FIBMAP).

If there was something consistent across all filesystems albeit it
different to SEEK_HOLE & SEEK_DATA those could be implemented in
glibc.

Given we've gotten along fine without them for some time I don't see
that a little bit of thought and taking some time to chew over this
will do any harm.

Re: SEEK_HOLE and SEEK_DATA

[Bryan Henderson]

The popular XDSM interface (commonly incorrectly called DMAPI) provides 
dm_probe_hole() for this (in fact, XFS on Linux implements XDSM).

But dm_probe_hole() is considerably harder to use (as a programmer) than 
SEEK_HOLE/SEEK_DATA.  The worst thing about services like dm_probe_hole is 
the fact that you get back variable size data; SEEK_HOLE/SEEK_DATA doesn't 
have that problem.  It isn't obvious to me that a series of seek/tell are 
noticeably slower when copying a file than a single system call that 
returns a list.

I usually prefer a special version of read() that shows you holes 
explicitly to either of the other approaches, but SEEK_HOLE/SEEK_DATA has 
the definite advantage of being a smaller perturbance of the API and 
easier to understand.

For those not familiar with XDSM and DMAPI, they're intended to facilitate 
file migration and backup and date back to 1993.  DMAPI (Data Management 
API) is the general term for an API that meets certain requirements of 
data management, while XDSM is an X/Open standard that was the first and 
essentially the only DMAPI.  Lots of commerical grade filesystem drivers 
implement at least an approximation of XDSM.

--
Bryan Henderson                     IBM Almaden Research Center
San Jose CA                         Filesystems

Re: SEEK_HOLE and SEEK_DATA

[Szakacsits Szabolcs]

On Fri, 3 Feb 2006, Bryan Henderson wrote:

> It isn't obvious to me that a series of seek/tell are noticeably slower 
> when copying a file than a single system call that returns a list.

It depends on how fast one is able to copy.

A special but not so unusual case of this if one doesn't want to copy at 
all and just interested in querying this information for analyses, future 
decisions, etc. Then the numbers would be something like this for e.g. a 
200 GB sparse file:

  JFS  + FIBMAP      	   137.80 sec  97% CPU
  XFS  + FIBMAP            117.81 sec  97% CPU
  ext3 + FIBMAP             88.40 sec  96% CPU
  XFS  + XFS_IOC_GETBMAPX    0.05 sec  95% CPU

SEEK_HOLE/SEEK_DATA is somewhere between XFS_IOC_GETBMAPX and FIBMAP, in 
the worst case half-way.

> I usually prefer a special version of read() that shows you holes 
> explicitly to either of the other approaches, but SEEK_HOLE/SEEK_DATA has 
> the definite advantage of being a smaller perturbance of the API and 
> easier to understand.

Absolutely. For the price being less useful and efficient.
  
	Szaka

Re: SEEK_HOLE and SEEK_DATA

[Chris Wedgwood]

On Fri, Feb 03, 2006 at 02:00:31PM -0800, Bryan Henderson wrote:

> But dm_probe_hole() is considerably harder to use (as a programmer)
> than SEEK_HOLE/SEEK_DATA.

It's also not in mainline nor likely to ever get merged.