[RFC] FIEMAP on ext3

[RFC] FIEMAP on ext3

[Josef Bacik]

Hello,

Obviously ext3 doesn't have extents, however it would still be usefull to use it
on sparse files in order to find data/holes within a file.  What I'm thinking is
treat contigous allocated blocks as an extent, and contiguous non-allocated
blocks as extents.  So for example, with Eric's fiemap test program, on a normal
non-sparse file you would only see one large extent for the entire file.  In the
case of a file that has data, hole and then data you would see 3 extents, two
for the two spans of data and one for the span of holes.
FIEMAP_FLAG_NUM_EXTENTS would in that case return 3.  Does this sound like an
acceptable thing to do?  If not, are there any other suggestions on how to do
fiemap in ext3?  Thanks much,

Josef

Re: [RFC] FIEMAP on ext3

[Eric Sandeen]

Josef Bacik wrote:
> Hello,
> 
> Obviously ext3 doesn't have extents, however it would still be usefull to use it
> on sparse files in order to find data/holes within a file.  What I'm thinking is
> treat contigous allocated blocks as an extent, and contiguous non-allocated
> blocks as extents.  So for example, with Eric's fiemap test program, on a normal
> non-sparse file you would only see one large extent for the entire file.  In the
> case of a file that has data, hole and then data you would see 3 extents, two
> for the two spans of data and one for the span of holes.
> FIEMAP_FLAG_NUM_EXTENTS would in that case return 3.  Does this sound like an
> acceptable thing to do?  If not, are there any other suggestions on how to do
> fiemap in ext3?  Thanks much,

Yes, regardless of whether the fs itself manages blocks as "extents" you
would just return contiguous ranges of blocks, allocated or unallocated,
in the structures...

FWIW I think currently the definition says FLAG_NUM_EXTENTS only returns
data extents not holes, so in your case above it'd be "2" (this bugs me
a little)  :)

The current FIEMAP definition has enough other bells and whistles
(flags), ext3 won't support most of them but that's ok; you won't have
to mark blocks delalloc or unwritten or offline or ....

I was playing with using getblock to do this a bit more efficiently; I
owe Josef my hacky patch for that, which I was playing with on ext4... I
did run into the situation where at least today, getblock will return
you a nice length of blocks for contiguous data, but for holes IIRC I
had to go one block at a time (basically falling back to FIBMAP-like
operation)

-Eric

Re: [RFC] FIEMAP on ext3

[Andreas Dilger]

On Apr 16, 2008  14:29 -0500, Eric Sandeen wrote:
> Josef Bacik wrote:
> > Obviously ext3 doesn't have extents, however it would still be usefull to use it
> > on sparse files in order to find data/holes within a file.  What I'm thinking is
> > treat contigous allocated blocks as an extent, and contiguous non-allocated
> > blocks as extents.  So for example, with Eric's fiemap test program, on a normal
> > non-sparse file you would only see one large extent for the entire file.  In the
> > case of a file that has data, hole and then data you would see 3 extents, two
> > for the two spans of data and one for the span of holes.
> > FIEMAP_FLAG_NUM_EXTENTS would in that case return 3.  Does this sound like an
> > acceptable thing to do?  If not, are there any other suggestions on how to do
> > fiemap in ext3?  Thanks much,

Well, this is already somewhat possible with "filefrag" though if you
want an API to do FIEMAP from applications (easier to program) I'd suggest
just having the fallback in glibc doing a loop of bmap ioctls.

Note that the updated filefrag tool we have for FIEMAP also allows a much
more useful output even for block-mapped files with the "-e" option.

> Yes, regardless of whether the fs itself manages blocks as "extents" you
> would just return contiguous ranges of blocks, allocated or unallocated,
> in the structures...
> 
> FWIW I think currently the definition says FLAG_NUM_EXTENTS only returns
> data extents not holes, so in your case above it'd be "2" (this bugs me
> a little)  :)

Well, if you really think that is important, feel free to complain now
rather than later.  We haven't landed the patch into Lustre yet, and I
agree it is inconsistent that we return 3 extents when we do a full scan
but 2 when we do NUM_EXTENTS...

> The current FIEMAP definition has enough other bells and whistles
> (flags), ext3 won't support most of them but that's ok; you won't have
> to mark blocks delalloc or unwritten or offline or ....

Right, for simple filesystem most of the flags can be ignored.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.