Zero-clearing all zero-clearable bytes.

Zero-clearing all zero-clearable bytes.

[Tetsuo Handa]

Hello.

I often create a Live CD.
When creating a Live CD, loopback mounted ext3 image files are used.
These image files are then compressed so that the ISO image will fit within
a CD-R's capacity (i.e. 700MB).

Compressing whole image files includes compressing deleted/unused bytes within
a block. This means that non-zero bytes in deleted/unused blocks affect
compression ratio.

I'm using the below program to improve compression ratio.

----------
#define _FILE_OFFSET_BITS 64
#define _LARGEFILE_SOURCE
#define _LARGEFILE64_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

int main(int argc, char *argv[]) {
    int fd;
    static char buffer[4096];
    memset(buffer, 0, sizeof(buffer));
    snprintf(buffer, sizeof(buffer) - 1, "%s/XXXXXX", argc > 1 ? argv[1] : "");
    if ((fd = mkstemp(buffer)) != EOF) {
        unlink(buffer);
        memset(buffer, 255, sizeof(buffer));
        while (write(fd, buffer, sizeof(buffer)) > 0);
        sync();
        close(fd);
    }
    return 0;
}
----------

But this program cannot clear some bytes within a block for file data (e.g.
offset from 1 to 511 of a data block used by a file with only 1 byte data)
and blocks for directory entries.

Any suggestions?

Regards.

Re: Zero-clearing all zero-clearable bytes.

[Andreas Dilger]

On Nov 22, 2008  12:36 +0900, Tetsuo Handa wrote:
> Compressing whole image files includes compressing deleted/unused bytes within
> a block. This means that non-zero bytes in deleted/unused blocks affect
> compression ratio.
> 
>     static char buffer[4096];
>     memset(buffer, 0, sizeof(buffer));
>     snprintf(buffer, sizeof(buffer) - 1, "%s/XXXXXX", argc > 1 ? argv[1] : "");
>     if ((fd = mkstemp(buffer)) != EOF) {
>         unlink(buffer);
>         memset(buffer, 255, sizeof(buffer));
>         while (write(fd, buffer, sizeof(buffer)) > 0);

Why would you fill the buffer with 0xff instead of 0?
In fact no such program is needed, just "dd if=/dev/zero of=/{fs}/tmp"
and then delete the file.


Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

Re: Zero-clearing all zero-clearable bytes.

[Tetsuo Handa]

Hello.

Andreas Dilger wrote:
> Why would you fill the buffer with 0xff instead of 0?
> In fact no such program is needed, just "dd if=/dev/zero of=/{fs}/tmp"
> and then delete the file.
> 
To avoid that the /{fs}/tmp is created as a sparse file.

I want to clear not only not-in-use blocks but also in-use blocks,
while keeping files in /{fs}/tmp intact.

Regards.

Re: Zero-clearing all zero-clearable bytes.

[Phillip Lougher]

Tetsuo Handa wrote:
> Hello.
> 
> Andreas Dilger wrote:
>> Why would you fill the buffer with 0xff instead of 0?
>> In fact no such program is needed, just "dd if=/dev/zero of=/{fs}/tmp"
>> and then delete the file.
>>
> To avoid that the /{fs}/tmp is created as a sparse file.
> 

Most filesystems will not create a sparse file if zero-byte filled 
blocks are written.  To create a sparse file you normally have to seek 
beyond the file end and then write blocks, leaving a hole in-between the 
positions.

The information from stat can tell you if a file has been stored 
sparsely, because the blocks used count will be less than the file size 
suggests.

For example:

$ dd if=/dev/zero of=small-file bs=512 count=16
16+0 records in
16+0 records out
8192 bytes (8.2 kB) copied, 0.000187744 seconds, 43.6 MB/s
phillip@dylan:/tmp$ stat small-file
   File: `small-file'
   Size: 8192            Blocks: 16         IO Block: 4096   regular file
Device: 801h/2049d      Inode: 847831      Links: 1

The zero filled file uses 16 blocks (16 * 512 bytes = 8K), and so we 
know it isn't sparsely stored.

You can get dd to seek a number of blocks into the output file before 
writing, this will create a sparse file...

$ dd if=/dev/zero of=small-file bs=512 count=16 seek=16
16+0 records in
16+0 records out
8192 bytes (8.2 kB) copied, 0.000212609 seconds, 38.5 MB/s
phillip@dylan:/tmp$ stat small-file
   File: `small-file'
   Size: 16384           Blocks: 16         IO Block: 4096   regular file
Device: 801h/2049d      Inode: 847831      Links: 1

A 16K file is only using 16 blocks, and so 8K is stored sparsely.

Phillip

Re: Zero-clearing all zero-clearable bytes.

[Phillip Lougher]

Tetsuo Handa wrote:
> Hello.
> 
> I often create a Live CD.
> When creating a Live CD, loopback mounted ext3 image files are used.
> These image files are then compressed so that the ISO image will fit within
> a CD-R's capacity (i.e. 700MB).
> 

You don't mention what you're using to compress the ext3 image file.  If 
you're using Squashfs then it's much better to zero-fill the blocks 
rather than use 255, because Squashfs detects zero-filled blocks and 
stores them sparsely.  This not only gets slightly better compression 
(than a compressed zero-filled block), but reading is also slightly faster.

Phillip

Re: Zero-clearing all zero-clearable bytes.

[Tetsuo Handa]

Hello.

Phillip Lougher wrote:
> Most filesystems will not create a sparse file if zero-byte filled 
> blocks are written.  To create a sparse file you normally have to seek 
> beyond the file end and then write blocks, leaving a hole in-between the 
> positions.
> 
> The information from stat can tell you if a file has been stored 
> sparsely, because the blocks used count will be less than the file size 
> suggests.
>
Then, I can use "dd if=/dev/zero" without worrying sparse files.

Phillip Lougher wrote:
> You don't mention what you're using to compress the ext3 image file.  If 
> you're using Squashfs then it's much better to zero-fill the blocks 
> rather than use 255, because Squashfs detects zero-filled blocks and 
> stores them sparsely.  This not only gets slightly better compression 
> (than a compressed zero-filled block), but reading is also slightly faster.
> 
Yes, I use squashfs.



What I wanted to do is "Zero-clearing *all zero-clearable* bytes".
Suppose blocks for inode are 8192 bytes and only 128 bytes are in use,
I think there are 8064 bytes I can zero-fill.
Suppose blocks for file data are 8192 bytes and a file's size is 1 byte,
I think there are 8191 bytes I can zero-fill.
Such bytes may have non-zero values because directory entries can be
unlink()ed and file data can be truncate()d.
I wished there is a utility to zero-fill such bytes.


Thanks a lot.

Re: Zero-clearing all zero-clearable bytes.

[Henrique de Moraes Holschuh]

On Sun, 23 Nov 2008, Tetsuo Handa wrote:
> What I wanted to do is "Zero-clearing *all zero-clearable* bytes".

Other than acessing the fs directly with something fs-specific that knows
how to do it, the following trick comes to mind:

1. compress all files to a tar.bz2.
2. remove all files
3. zero fs using dd to a file (will zero all blocks except for the ones used
by the tar.bz2
4. unpack tar.bz2
5. remove tar.bz2
6. redo the dd trick. This will now zero all blocks that were in use by
the tar.bz2.

Of course, this only works if the (kernel, glibc, tar) are not writing
random junk to the unused parts of a fs block.

The cost is from 1 to 3 rm'ed inodes left behind.  If you use two
filesystems (i.e. tar to outside the filesystem), you avoid that
possibility.

> I wished there is a utility to zero-fill such bytes.

So do I.  And a IOCTL/syscall/whatever that we could use to sanitize (i.e.
fill with an user-supplied byte) half-used blocks, so that we could have it
on the most used filesystems (ext2, ext3, xfs, reiser...), and that we could
implement scrub-erasing of unused filesystem areas properly.

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Re: Zero-clearing all zero-clearable bytes.

[Martin Steigerwald]

[-- Attachment #1: Type: text/plain, Size: 3500 bytes --]


Hi!

Am Sonntag 23 November 2008 schrieben Sie:
> On Sun, 23 Nov 2008, Tetsuo Handa wrote:
> > What I wanted to do is "Zero-clearing *all zero-clearable* bytes".
> Other than acessing the fs directly with something fs-specific that
> knows how to do it, the following trick comes to mind:
>
> 1. compress all files to a tar.bz2.
> 2. remove all files
> 3. zero fs using dd to a file (will zero all blocks except for the ones
> used by the tar.bz2
> 4. unpack tar.bz2
> 5. remove tar.bz2
> 6. redo the dd trick. This will now zero all blocks that were in use by
> the tar.bz2.
>
> Of course, this only works if the (kernel, glibc, tar) are not writing
> random junk to the unused parts of a fs block.
>
> The cost is from 1 to 3 rm'ed inodes left behind.  If you use two
> filesystems (i.e. tar to outside the filesystem), you avoid that
> possibility.
>
> > I wished there is a utility to zero-fill such bytes.
>
> So do I.  And a IOCTL/syscall/whatever that we could use to sanitize
> (i.e. fill with an user-supplied byte) half-used blocks, so that we
> could have it on the most used filesystems (ext2, ext3, xfs,
> reiser...), and that we could implement scrub-erasing of unused
> filesystem areas properly.

There are at least 2 tools - I think there was even another one. I did not 
test them and do not know whether they work on singular inode basis.

Debian Package zerofree:

Description: zero free blocks from ext2/3 file-systems
 Zerofree finds the unallocated, non-zeroed blocks in an ext2 or ext3
 file-system and fills them with zeroes. This is useful if the device
 on which this file-system resides is a disk image. In this case,
 depending on the type of disk image, a secondary utility may be able
 to reduce the size of the disk image after zerofree has been
 run. Zerofree requires the file-system to be unmounted or mounted
 read-only.
 .
 The usual way to achieve the same result (zeroing the unallocated
 blocks) is to run "dd" do create a file full of zeroes that takes up
 the entire free space on the drive, and then delete this file. This
 has many disadvantages, which zerofree alleviates:
  * it is slow;
  * it makes the disk image (temporarily) grow to its maximal extent;
  * it (temporarily) uses all free space on the disk, so other
    concurrent write actions may fail.
 .
 Zerofree has been written to be run from GNU/Linux systems installed
 as guest OSes inside a virtual machine. If this is not your case, you
 almost certainly don't need this package.



Debian Package wipe2fs:

Package: wipe2fs
Priority: extra
Section: admin
Installed-Size: 112
Maintainer: Martin A. Godisch <godisch@debian.org>
Architecture: i386
Version: 0.2.1-1
Depends: e2fslibs, libc6 (>= 2.7-1), libcomerr2 (>= 1.33-3)
Filename: pool/main/w/wipe2fs/wipe2fs_0.2.1-1_i386.deb
Size: 14150
MD5sum: da0c0d4319fda08a1f5fc7a92f5b2535
SHA1: 2f7d55fbfda34b328b67f17e5e2e26d8f853bdac
SHA256: 569bc452331b1067f49ba5aaf36dd7bb357036307b79156c4d7c5beaa2a17545
Description: Securely wipe unused space in ext2/3 filesystems
 This users-pace program locates unused space in ext2/3 filesystems
 and overwrites the space with zero. wipe2fs also reaches the slack
 space at the end of files, it does not require kernel filesystem
 support.
Homepage: http://web.cecs.pdx.edu/~cklin/wipe2fs/


Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Zero-clearing all zero-clearable bytes.

[Tetsuo Handa]

Hello.

Martin Steigerwald wrote:
> There are at least 2 tools - I think there was even another one. I did not 
> test them and do not know whether they work on singular inode basis.

> Debian Package zerofree:
> Debian Package wipe2fs:
Great!

They must be what I wished.

Thank you.