How does ext2 implement sparse files?

How does ext2 implement sparse files?

[Lars Noschinski]

Hello!

For an university project, we had to write a toy filesystem (ext2-like),
for which I would like to implement sparse file support. For this, I
digged through the ext2 source code; but I could not find the point,
where ext2 detects holes.

As far as I can see from fs/buffer.c, an hole is a buffer_head which is
not mapped, but uptodate. But I cannot find a relevant source line,
where ext2 makes usage of this information.

Any hints would be greatly appreciated,
     Lars.

[Please CC me on answers, I'm not subscribed]

Re: How does ext2 implement sparse files?

[Lennart Sorensen]

On Thu, Jan 31, 2008 at 04:28:23PM +0100, Lars Noschinski wrote:
> For an university project, we had to write a toy filesystem (ext2-like),
> for which I would like to implement sparse file support. For this, I
> digged through the ext2 source code; but I could not find the point,
> where ext2 detects holes.
> 
> As far as I can see from fs/buffer.c, an hole is a buffer_head which is
> not mapped, but uptodate. But I cannot find a relevant source line,
> where ext2 makes usage of this information.
> 
> Any hints would be greatly appreciated,

While I don't actually know, I always thought it was up to the
application to create files with holes by seeking over the holes before
writing data.  That seems like the simplest way to me.

--
Len Sorensen

Re: How does ext2 implement sparse files?

[Chris Snook]

Lars Noschinski wrote:
> 
> Hello!
> 
> For an university project, we had to write a toy filesystem (ext2-like),
> for which I would like to implement sparse file support. For this, I
> digged through the ext2 source code; but I could not find the point,
> where ext2 detects holes.
> 
> As far as I can see from fs/buffer.c, an hole is a buffer_head which is
> not mapped, but uptodate. But I cannot find a relevant source line,
> where ext2 makes usage of this information.

In ext2 (and most other block filesystems) all files are sparse files. 
If you write to an address in the file for which no block is allocated, 
the filesystem allocates a block and writes the contents to disk, 
regardless of whether that block is at the end of the file (the usual 
case of lengthening a non-sparse file), in the middle of the file 
(filling in holes in a sparse file), or past the the end of the file 
(making a file sparse).

	-- Chris

Re: How does ext2 implement sparse files?

[Andi Kleen]

Lars Noschinski <lklml@ml.noschinski.de> writes:

> For an university project, we had to write a toy filesystem (ext2-like),
> for which I would like to implement sparse file support. For this, I
> digged through the ext2 source code; but I could not find the point,
> where ext2 detects holes.
>
> As far as I can see from fs/buffer.c, an hole is a buffer_head which is
> not mapped, but uptodate. But I cannot find a relevant source line,
> where ext2 makes usage of this information.

It does not explicitely detect holes; holey data is just never written
so no space for it is allocated.

-Andi

Re: How does ext2 implement sparse files?

[Shuduo Sang]

On Feb 1, 2008 2:14 AM, Andi Kleen <andi@firstfloor.org> wrote:
> Lars Noschinski <lklml@ml.noschinski.de> writes:
>
> > For an university project, we had to write a toy filesystem (ext2-like),
> > for which I would like to implement sparse file support. For this, I
> > digged through the ext2 source code; but I could not find the point,
> > where ext2 detects holes.
> >
> > As far as I can see from fs/buffer.c, an hole is a buffer_head which is
> > not mapped, but uptodate. But I cannot find a relevant source line,
> > where ext2 makes usage of this information.
>
> It does not explicitely detect holes; holey data is just never written
> so no space for it is allocated.
>

does anybody know how to make a hole in a large file which already has
real content from user space application?
In my project I need this function to delete a piece of content from
an exist large effectively.
thanks.

> -Andi
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: How does ext2 implement sparse files?

[Eric Dumazet]

Shuduo Sang a écrit :
> On Feb 1, 2008 2:14 AM, Andi Kleen <andi@firstfloor.org> wrote:
>   
>> Lars Noschinski <lklml@ml.noschinski.de> writes:
>>
>>     
>>> For an university project, we had to write a toy filesystem (ext2-like),
>>> for which I would like to implement sparse file support. For this, I
>>> digged through the ext2 source code; but I could not find the point,
>>> where ext2 detects holes.
>>>
>>> As far as I can see from fs/buffer.c, an hole is a buffer_head which is
>>> not mapped, but uptodate. But I cannot find a relevant source line,
>>> where ext2 makes usage of this information.
>>>       
>> It does not explicitely detect holes; holey data is just never written
>> so no space for it is allocated.
>>
>>     
>
> does anybody know how to make a hole in a large file which already has
> real content from user space application?
> In my project I need this function to delete a piece of content from
> an exist large effectively.
> thanks.
>
>   
Some OSes use fcntl() F_FREESP/F_FREESP64 to be able to free allocated 
space in files (ie make holes if supported by underlying fs)

AFAIK, linux can generically do this only at the end of a file 
(ftruncate()) , not at random points.

XFS has special support for FREESP (it comes from IRIX), implemented as 
an ioctl()

Check for XFS_IOC_FREESP and XFS_IOC_UNRESVSP in fs/xfs/xfs_fs.h

Re: How does ext2 implement sparse files?

[Lars Noschinski]

[Sorry for using an invalid mail address in my previous post]

* Andi Kleen <andi@firstfloor.org> [08-02-01 10:28]:
>Lars Noschinski <lklml@ml.noschinski.de> writes:
>
>> For an university project, we had to write a toy filesystem (ext2-like),
>> for which I would like to implement sparse file support. For this, I
>> digged through the ext2 source code; but I could not find the point,
>> where ext2 detects holes.
>>
>> As far as I can see from fs/buffer.c, an hole is a buffer_head which is
>> not mapped, but uptodate. But I cannot find a relevant source line,
>> where ext2 makes usage of this information.
>
>It does not explicitely detect holes; holey data is just never written
>so no space for it is allocated.

Thanks for the answers. No that I know that, it is kind of obvious. I
was irritated yesterday by a bug a bug in our implementation (number of
used blocks is reported as filesize/blocksize); so I was expecting the
VFS to issue write requests for empty pages (which would be sillly, I
know).

Thanks,
     Lars.