Re: created ext4 disk image differs depending on the underlying filesystem

[Johannes Schauer Marin Rodrigues <josch@mister-muffin.de>]

[-- Attachment #1: Type: text/plain, Size: 2795 bytes --]

Hi Ted,

thank you very much for your (as usual) very detailed and comprehensive reply!
:D

Quoting Theodore Ts'o (2024-05-05 02:10:20)
> If your goal is to create completely reproducible image files, one question
> is whether keeping the checksums identical is enough, or do you care about
> whether the underlying file is being more efficiently stored by using sparse
> files or extents marked unitialized?
> 
> Depending on how much you care about reproducibility versus file
> storage efficiency, I could imagine adding some kind of option which
> disables the zeroout function, and forces e2fsprogs to always write
> zeros, even if that increases the write wearout rate of the underlying
> flash file system, and increasing the size of the image file.  Or I
> could imageine some kind of extended option which hacks mke2fs to zero out
> the lifetime writes counter.;

the good news is, that the fix in my situation is very simple: create the
filesystem on a tmpfs first and then copy it into 9p fs afterwards. Tada, the
created images will be reproducible. I think there are multiple ways forward
with which I'd be happy with:

 1. leave everything as it is. It's just one more copy operation on my end.  I
    can just document that if your underlying file system is stupid, you might
    not get the same identical image as somebody with a more intelligent
    filesystem does.

 2. allow resetting fs->super->s_kbytes_written to zero. This patch worked for
    me:

--- a/lib/ext2fs/closefs.c
+++ b/lib/ext2fs/closefs.c
@@ -504,6 +504,7 @@ errcode_t ext2fs_close2(ext2_filsys fs, int flags)
                                (fs->blocksize / 1024);
                if ((fs->flags & EXT2_FLAG_DIRTY) == 0)
                        fs->flags |= EXT2_FLAG_SUPER_ONLY | EXT2_FLAG_DIRTY;
+               fs->super->s_kbytes_written = 0;
        }
        if (fs->flags & EXT2_FLAG_DIRTY) {
                retval = ext2fs_flush2(fs, flags);


    If my goal is to create disk images, one could argue that what the end user
    is interested in, is the filesystem writes that *they* performed and that
    the disk image they receive should therefor have the counter start at zero.

 3. Somehow do magic with the zeroout function. If anybody has too much
    free-time... ;)

As an end-user I am very interested in keeping the functionality of mke2fs
which keeps track of which parts are actually sparse and which ones are not.
This functionality can be used with tools like "bmaptool" (a more clever dd) to
only copy those parts of the image to the flash drive which are actually
supposed to contain data.

Would you be happy about a patch for (2.)? If yes, I can send something over
once I find some time. :)

Thanks!

cheers, josch

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEElFhU6KL81LF4wVq58sulx4+9g+EFAmY/A2oACgkQ8sulx4+9
g+GY4RAAu5QOgV8+BVPzH966NS9RpJu7TR/rtj9XjnNJF1yU+YlMY7TU9zD40Gq4
B8aKsg1o4mEj2pOSfIPEwL4CkCZ/AfOWQAt2pxfAtWoh1iOjeQdaz6ZBVjkt6/2O
HMznP6w1eCrcIjr8AanAUAT7KxZWxxYki25LCcsiehzXvY59Yu6vUdundbrOo6Eh
AahB0NXIYC52ZcwTKrltPSt7JRt5jBE3qgkRpenzln0o6JKU9UBOUqTgqW9ENsL2
d5RL1YUV/UrkTpyPKKMUmj86oHixmlNm0qzvMXtgnAIhXEmsflytT9c+kMb2JXYN
WbbwdI+w1YOXDhwPc53ZDsiNGfDSrHJeAmnuhHu3mm28PtRZirJYexYvZuzf6dMu
v+c/01EwAqSuzrW4bGi2CkGKHiWjmXqeO+f9/hVOCA8+MVzonAgBU5woROTwtQEx
h21rLYtbLdfy5bf+YUo22WZ1Fq6+JI0LpeU8qXb0cBB0Sox4x/kEDfCV4CR3RdlL
Ey4Z4n689h3612wBMXLzKQ4lwJMVOjGt+yzTTfmFTeASSBmtBpRqGfydwinvbopQ
pfUc6rdaYLPimS5kVK44fe40eSZQBkZQL2YflRBsom8H1jDAQEff4XKQ68EmmQ+4
bAa/c7DjdjpSwgjOxf86f5eu9mj7a2cZKXsvu3d6An3y1STaQNI=
=JLUI
-----END PGP SIGNATURE-----