Feature request: e2fsck -z

Feature request: e2fsck -z

[H. Peter Anvin]

Hi all,

This is something I've wanted to see for a very long time, and it
finally occurred to me that perhaps I should say something about it!

It would be a very nice thing to have a flag to e2fsck, presumably -z,
to zero out any unused data blocks, inodes and so on.  The goal is to
minimize the amount of space required after compressing a virtual disk
image or similar, and to make sure any non-data isn't lying around.

	-hpa

Re: Feature request: e2fsck -z

[Ric Wheeler]

On 08/09/2011 06:52 PM, H. Peter Anvin wrote:
> Hi all,
>
> This is something I've wanted to see for a very long time, and it
> finally occurred to me that perhaps I should say something about it!
>
> It would be a very nice thing to have a flag to e2fsck, presumably -z,
> to zero out any unused data blocks, inodes and so on.  The goal is to
> minimize the amount of space required after compressing a virtual disk
> image or similar, and to make sure any non-data isn't lying around.
>
> 	-hpa
>

Do you need it to be in the fsck tool?

If you have a sparsely allocated block map under your file system, doing a zero 
of all blocks could add hours for a big, slow S-ATA drives (2-3 hours for a 1TB 
drive).

An alternative for SSD's and devices that do TRIM/UNMAP would be to use one of 
the batched discard tools (that would make discarded data read back as zeroed).

Thanks!

Ric

Re: Feature request: e2fsck -z

[Andreas Dilger]

On 2011-08-10, at 12:34 AM, Ric Wheeler wrote:
> On 08/09/2011 06:52 PM, H. Peter Anvin wrote:
>> Hi all,
>> 
>> This is something I've wanted to see for a very long time, and it
>> finally occurred to me that perhaps I should say something about it!
>> 
>> It would be a very nice thing to have a flag to e2fsck, presumably -z,
>> to zero out any unused data blocks, inodes and so on.  The goal is to
>> minimize the amount of space required after compressing a virtual disk
>> image or similar, and to make sure any non-data isn't lying around.
> 
> Do you need it to be in the fsck tool?
> 
> If you have a sparsely allocated block map under your file system, doing a zero of all blocks could add hours for a big, slow S-ATA drives (2-3 hours for a 1TB drive).

I think Ted has a tool that does this already.  It should be relatively simple oo do, like "dd if=/dev/zero of=/mountpoint/temp_zero_file && rm /mountpoint/temp_zero_file.

> An alternative for SSD's and devices that do TRIM/UNMAP would be to use one of the batched discard tools (that would make discarded data read back as zeroed).

In fact, I thought Lukas has already made a tool for sending BLKDISCARD for all unused parts of the filesystem?

Cheers, Andreas

Re: Feature request: e2fsck -z

[Ric Wheeler]

On 08/10/2011 09:16 AM, Andreas Dilger wrote:
> On 2011-08-10, at 12:34 AM, Ric Wheeler wrote:
>> On 08/09/2011 06:52 PM, H. Peter Anvin wrote:
>>> Hi all,
>>>
>>> This is something I've wanted to see for a very long time, and it
>>> finally occurred to me that perhaps I should say something about it!
>>>
>>> It would be a very nice thing to have a flag to e2fsck, presumably -z,
>>> to zero out any unused data blocks, inodes and so on.  The goal is to
>>> minimize the amount of space required after compressing a virtual disk
>>> image or similar, and to make sure any non-data isn't lying around.
>> Do you need it to be in the fsck tool?
>>
>> If you have a sparsely allocated block map under your file system, doing a zero of all blocks could add hours for a big, slow S-ATA drives (2-3 hours for a 1TB drive).
> I think Ted has a tool that does this already.  It should be relatively simple oo do, like "dd if=/dev/zero of=/mountpoint/temp_zero_file&&  rm /mountpoint/temp_zero_file.

This will work but will be a potential multi-hour long process (and cause out of 
space errors at some point for other applications for a very brief window :))

>
>> An alternative for SSD's and devices that do TRIM/UNMAP would be to use one of the batched discard tools (that would make discarded data read back as zeroed).
> In fact, I thought Lukas has already made a tool for sending BLKDISCARD for all unused parts of the filesystem?
>
> Cheers, Andreas

Right, I think he has. That tool (not part of fsck) would do the job much 
quicker for enabled devices,

Ric

Re: Feature request: e2fsck -z

[Lukas Czerner]

On Wed, 10 Aug 2011, Ric Wheeler wrote:

> On 08/10/2011 09:16 AM, Andreas Dilger wrote:
> > On 2011-08-10, at 12:34 AM, Ric Wheeler wrote:
> > > On 08/09/2011 06:52 PM, H. Peter Anvin wrote:
> > > > Hi all,
> > > > 
> > > > This is something I've wanted to see for a very long time, and it
> > > > finally occurred to me that perhaps I should say something about it!
> > > > 
> > > > It would be a very nice thing to have a flag to e2fsck, presumably -z,
> > > > to zero out any unused data blocks, inodes and so on.  The goal is to
> > > > minimize the amount of space required after compressing a virtual disk
> > > > image or similar, and to make sure any non-data isn't lying around.
> > > Do you need it to be in the fsck tool?
> > > 
> > > If you have a sparsely allocated block map under your file system, doing a
> > > zero of all blocks could add hours for a big, slow S-ATA drives (2-3 hours
> > > for a 1TB drive).
> > I think Ted has a tool that does this already.  It should be relatively
> > simple oo do, like "dd if=/dev/zero of=/mountpoint/temp_zero_file&&  rm
> > /mountpoint/temp_zero_file.
> 
> This will work but will be a potential multi-hour long process (and cause out
> of space errors at some point for other applications for a very brief window
> :))
> 
> > 
> > > An alternative for SSD's and devices that do TRIM/UNMAP would be to use
> > > one of the batched discard tools (that would make discarded data read back
> > > as zeroed).
> > In fact, I thought Lukas has already made a tool for sending BLKDISCARD for
> > all unused parts of the filesystem?
> > 
> > Cheers, Andreas
> 
> Right, I think he has. That tool (not part of fsck) would do the job much
> quicker for enabled devices,

You're right. It is a part of e2fsck and you can enable it with -E
discard.

Thanks!
-Lukas

Re: Feature request: e2fsck -z

[Florian Weimer]

* Lukas Czerner:

> You're right. It is a part of e2fsck and you can enable it with -E
> discard.

If the block device does not support TRIM, does this result in zeros
being written instead?

-- 
Florian Weimer                <fweimer@bfk.de>
BFK edv-consulting GmbH       http://www.bfk.de/
Kriegsstraße 100              tel: +49-721-96201-1
D-76133 Karlsruhe             fax: +49-721-96201-99
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Feature request: e2fsck -z

[Lukas Czerner]

On Wed, 10 Aug 2011, Florian Weimer wrote:

> * Lukas Czerner:
> 
> > You're right. It is a part of e2fsck and you can enable it with -E
> > discard.
> 
> If the block device does not support TRIM, does this result in zeros
> being written instead?
> 

Of course not, discard works only on devices which does support it. If
the device does not support discard it will to nothing.

Thanks!
-Lukas

Re: Feature request: e2fsck -z

[Florian Weimer]

* Lukas Czerner:

> On Wed, 10 Aug 2011, Florian Weimer wrote:
>
>> * Lukas Czerner:
>> 
>> > You're right. It is a part of e2fsck and you can enable it with -E
>> > discard.
>> 
>> If the block device does not support TRIM, does this result in zeros
>> being written instead?

> Of course not, discard works only on devices which does support it. If
> the device does not support discard it will to nothing.

Oh, then it doesn't address hpa's use case (improving compression).

-- 
Florian Weimer                <fweimer@bfk.de>
BFK edv-consulting GmbH       http://www.bfk.de/
Kriegsstraße 100              tel: +49-721-96201-1
D-76133 Karlsruhe             fax: +49-721-96201-99
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Feature request: e2fsck -z

[Ric Wheeler]

On 08/10/2011 12:04 PM, Florian Weimer wrote:
> * Lukas Czerner:
>
>> On Wed, 10 Aug 2011, Florian Weimer wrote:
>>
>>> * Lukas Czerner:
>>>
>>>> You're right. It is a part of e2fsck and you can enable it with -E
>>>> discard.
>>> If the block device does not support TRIM, does this result in zeros
>>> being written instead?
>> Of course not, discard works only on devices which does support it. If
>> the device does not support discard it will to nothing.
> Oh, then it doesn't address hpa's use case (improving compression).
>

I think that there are multiple ways to help compression.

One would be to extract only the used data blocks (like file system specific 
dump utilities can do), the other would be to zero with the discard or by 
writing zeros to the unused blocks.

Ric

Re: Feature request: e2fsck -z

[Lukas Czerner]

On Wed, 10 Aug 2011, Florian Weimer wrote:

> * Lukas Czerner:
> 
> > On Wed, 10 Aug 2011, Florian Weimer wrote:
> >
> >> * Lukas Czerner:
> >> 
> >> > You're right. It is a part of e2fsck and you can enable it with -E
> >> > discard.
> >> 
> >> If the block device does not support TRIM, does this result in zeros
> >> being written instead?
> 
> > Of course not, discard works only on devices which does support it. If
> > the device does not support discard it will to nothing.
> 
> Oh, then it doesn't address hpa's use case (improving compression).
> 

It does if he will use SSD's with "discard_zeroes_data" capability. But
my original answer was to simply confirm that e2fsck has that feature.

Thanks!
-Lukas

Re: Feature request: e2fsck -z

[Ted Ts'o]

On Wed, Aug 10, 2011 at 07:34:46AM +0100, Ric Wheeler wrote:
> 
> Do you need it to be in the fsck tool?

As others have noted, (a) zerofree does this already, and (b) there's
also -E discard.

My take on it though is that it's a reasonable request.  It's much
like there is "sort -u" even though you could do this via "sort |
uniq".  Part of the Unix philosophy is to use tools that are
composable, yes --- but optimizing for common cases is also a good
thing, and with the advent of virtualization being more and more
popular, zeroing free blocks for virtualization images is good and
useful thing.

This is also why I agitated for adding support so that e2fsprogs tools
could operate directly on qemu-img files, and not just have support
which is hacked into e2image.  Yes, you can always take an qemu-img
file, decompress and expand it into a raw file, run debugfs or e2fsck
on it, and then convert it back to a qemu-img file --- but if lots of
people are doing that, it does make sense to optimize for the most
common use cases.

						- Ted

Re: Feature request: e2fsck -z

[Lukas Czerner]

On Wed, 10 Aug 2011, Ted Ts'o wrote:

> On Wed, Aug 10, 2011 at 07:34:46AM +0100, Ric Wheeler wrote:
> > 
> > Do you need it to be in the fsck tool?
> 
> As others have noted, (a) zerofree does this already, and (b) there's
> also -E discard.
> 
> My take on it though is that it's a reasonable request.  It's much
> like there is "sort -u" even though you could do this via "sort |
> uniq".  Part of the Unix philosophy is to use tools that are
> composable, yes --- but optimizing for common cases is also a good
> thing, and with the advent of virtualization being more and more
> popular, zeroing free blocks for virtualization images is good and
> useful thing.

And since we already have punch hole support in ext4 I am going to use
that to "discard" - "punch out" unused blocks from the image. It is
probably not exactly what you are talking about, but it can be useful.

> 
> This is also why I agitated for adding support so that e2fsprogs tools
> could operate directly on qemu-img files, and not just have support
> which is hacked into e2image.  Yes, you can always take an qemu-img
> file, decompress and expand it into a raw file, run debugfs or e2fsck
> on it, and then convert it back to a qemu-img file --- but if lots of
> people are doing that, it does make sense to optimize for the most
> common use cases.

I am in favor if this, however I do not want to do it. I seems to me
that it is just rewriting the support that qemu already has. However if
qemu guys (or anyone) would create an library which we can use from
both projects, that would be certainly very useful.

Thanks!
-Lukas

> 
> 						- Ted
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 

Re: Feature request: e2fsck -z

[Ron Yorston]

"H. Peter Anvin" wrote:

>It would be a very nice thing to have a flag to e2fsck, presumably -z,
>to zero out any unused data blocks, inodes and so on.  The goal is to
>minimize the amount of space required after compressing a virtual disk
>image or similar, and to make sure any non-data isn't lying around.

I wrote a separate utility, zerofree[1], that zeroes out free blocks.

Ron
--
[1] http://intgat.tigress.co.uk/rmy/uml/