e2fsck and backup superblocks

e2fsck and backup superblocks

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I have noticed that e2fsck always complains about block group descriptor checksums being invalid when run on a backup superblock, even when run again immediately after being told to fix them.  Is there some odd intentional behavior here I'm not aware of?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJPPxQ5AAoJEJrBOlT6nu75dgUH/19Sv2UASFGCLTfsvJcBBFjC
z6w0LgI/GB37mlan7g06mT3UYSEN5LlD7VYAqWUEOgchu3Y+t735w1MRZy7WmVrc
9PopsJVTUZnc8vCcGAoh/T4OcJkf5zU7EAoL+BQUInTOkermLL/Bp9yGVIVsDXN1
gXnf6BdozDkm4LRCIbN1RStg6IqUClIxbvLwItDYhgdwmMFIVsd0dUPTPhb8s8c3
tMQclqLTx+9lYJXC7ZBKVC5BZhaMmuT4ITrKd66h2goRY4wEWz/v1iwhuLyNfgUo
enXjG65PrxiiqirGiLPS5UtEvoqMQ8PsyaSILnYqpgJKd9AdP9NQjvjnG6PxA/Y=
=TfhG
-----END PGP SIGNATURE-----

Re: e2fsck and backup superblocks

[Eric Sandeen]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2/17/12 7:00 PM, Phillip Susi wrote:
> I have noticed that e2fsck always complains about block group descriptor checksums being invalid when run on a backup superblock, even when run again immediately after being told to fix them.  Is there some odd intentional behavior here I'm not aware of?

Check the list for:

"[PATCH] libext2fs: quiet spurious group checksum errors"

- -Eric
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJPPxV/AAoJECCuFpLhPd7gul4P/3OOyVnLqAapkkApwJf1cRq0
SDNH/1d8cbldwLMizgAuE3omoJRbzgpy6jghOtq573e1IOIskbQFDxG8HBSg6GGn
E1Yw8UcaCqKERRiemTESmC6Idtj6EAusZJWeTlC9XRx0Cbx8bGJenlU6eR/vpxH0
AvOPmgJhNMcRyMdsHLsvXXZNER5j9St1AEEVa2Wzpo2PGyJjVP8jDnJiuhvz2bK3
nv5u28WSuF5y8zzTs0j6DT40XQlpT/OVNflqUGelhxKHBB+sT2c0VvaVhVy63hw/
yuaIMmnwggJSs6p//9iOtKJhTSmR6qSGuuT65ajMgixy66efRLI0C0e5Y/U4stS4
cpT8m8+FGILhIe+Q+sJ2nOxC1JUvZW3AQvi0/gLZDCpzBid9RrmigfU/FTxs4MQB
CD9QDkmhz7A7rUvRulC4NSSXcgErMtwC5zAXG9MKiFAtvMBqYXWhWoKfxslaZXOU
5sOIvoB+TOwlWL2StQgwnJjJMCZ9/Eju2VhgEFbZxEgpInaFpk/FUm2k4hzQpHY9
M1ItllWFrR/XMXxozmN1/KmynOLdBEf4rW/BouC+ZCGZLeUOg8M+obJFNICFQEXq
VLSvUfIfsLaLb2ApBUyq75/mvIQIo5mCjRYqi1Ev7Rx8XrFWHfNR62Q/VwAVU6X+
vqDKt1dPO06sBPpzOdiP
=jdg1
-----END PGP SIGNATURE-----

Re: e2fsck and backup superblocks

[Ted Ts'o]

On Fri, Feb 17, 2012 at 07:05:36PM -0800, Eric Sandeen wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 2/17/12 7:00 PM, Phillip Susi wrote:
> > I have noticed that e2fsck always complains about block group descriptor checksums being invalid when run on a backup superblock, even when run again immediately after being told to fix them.  Is there some odd intentional behavior here I'm not aware of?
> 
> Check the list for:
> 
> "[PATCH] libext2fs: quiet spurious group checksum errors"

Which is in e2fsprogs 1.42.1....

       	       	  	       	      	  - Ted

Re: e2fsck and backup superblocks

[Eric Sandeen]

On Feb 18, 2012, at 4:31 PM, "Ted Ts'o" <tytso@mit.edu> wrote:

> On Fri, Feb 17, 2012 at 07:05:36PM -0800, Eric Sandeen wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>> 
>> On 2/17/12 7:00 PM, Phillip Susi wrote:
>>> I have noticed that e2fsck always complains about block group descriptor checksums being invalid when run on a backup superblock, even when run again immediately after being told to fix them.  Is there some odd intentional behavior here I'm not aware of?
>> 
>> Check the list for:
>> 
>> "[PATCH] libext2fs: quiet spurious group checksum errors"
> 
> Which is in e2fsprogs 1.42.1....
> 
Was that the release announcement?  :)

- Eric

>                                                  - Ted
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

E2fsprogs 1.42.1 release

[Ted Ts'o]

On Sun, Feb 19, 2012 at 10:11:12PM -0500, Eric Sandeen wrote:
> > Which is in e2fsprogs 1.42.1....
> > 
> Was that the release announcement?  :)

I knew I forgot something.  :-)

Over the weekend I released e2fsprogs 1.42.1.  It can be found in the
usual places: on sourceforge, a signed git tag, and on
ftp.kernel.org:/pub/linux/kernel/people/tytso/e2fsprogs/v1.42.1.

							- Ted

E2fsprogs 1.42.1 (February 17, 2012)
===================================

The mke2fs and e2fsck now use significantly less memory when creating
or checking very large file systems.  This was enabled by adding
extent-based bitmaps which are stored using a red-block tree, since
block and inode allocations tend to be contiguous.

The command mke2fs -S is used as a last ditch recovery command to
write new superblock and block group descriptors, but _not_ to destroy
the inode table in hopes of recovering from a badly corrupted file
system.  So if the uninit_bg feature is enabled, mke2fs -S will now
set the unused inodes count field to zero.  Otherwise, e2fsck -fy
after using mke2fs -S would leave the file system completely empty.

Since mke2fs recognizes mke3fs in argv[0] to mean "mkfs.ext3", also
honor "mke4fs" to work the same as "mke2fs.ext4", since RHEL5 has
installed an mke2fs binary using that name.

The usage and help messages for the -G, -t and -T options in mke2fs
have been fixed.

If e2fsck needs to use the backup group descriptors, the
ext2fs_open2() function clears the UNINIT bits to ensure all of the
inodes in the file systems get scanned.  However, the code which reset
the UNINIT flags did not also recalculate the checksum, which produced
many spurious (and scary) e2fsck messages.  This has been fixed by
resetting cheksums when the UNINIT bits are cleared.

Relax a check in e2fsck which required that the block bitmap to be
initialized when the inode bitmap is in use.  This will allow us to
eventually eliminate code from the kernel which forcibly initialized
the block bitmap when the inode bitmap is first used, requiring an
extra journal credit and disk write.  (Addresses Google Bug: #5944440)
    
Make sure rdebugfs (which may be installed setuid or setgid disk) does
not honor environment variables if euid != uid or egid != gid.

Debugfs's ncheck command has been optimized and now is much more
robust with faced with corrupted file systems.  The ncheck command
also now has a -c option which will verify the file type information
in the directory entry to see if matches the inode's mode bits.  This
is extremely useful when trying to use debugsfs to determine which
parts of the file system metadata can be trusted.

E2image will try to use ftruncate64() to set the i_size for raw
images, instead of writing a single null byte.  This avoid allocating
an extra block to the raw image, for those file systems and/or
operating systems that support this.  (Linux does.)  In addition, fix
a logic bug that caused the file to not be properly extended if the
size of the last hole was exactly an multiple of a megabyte.

Fixed a bug in resize2fs where for 1k and 2k file systems, where
s_first_data_block is non-zero, this wasn't taken into account when
calculate the minimum file system size for use with the -M option.

Fixed the badblocks program to honor the -s flag when in read-only -t
mode.  (Addresses Debian Bug #646629)

Update Czech, Dutch, French, Polish, and Sweedish translation from the
Translation Project.

Fixed various Debian Packaging issues so that dpkg-buildflags is used
if present, which allows e2fsprogs to be built with security hardening
flags.  (Addresses Debian Bugs: #654457)

Programmer's Notes
------------------

Fix a bug in ext2fs_clear_generic_bmap() when used for 32-bit bitmaps.
This was only an issue for programs compiled against e2fsprogs 1.41
that manipulate bitmaps directly.  (Addresses Sourceforge Bugs:
#3451486)

The libext2fs library now uses sysconf() to fetch the page size, instead
of the deprecated getpagesize().

The ext2fs_get_pathname() function will return a partial path if an a
directory in the path is not a directory, displaying it as an inode
number in angle brackets instead of giving up and displaying an error.
This is much more helpful when a user is trying to debug a corrupted
file system.

Codepoints for the RO_COMPAT_REPLICA feature has been reserved.

Added a new library function, ext2fs_file_get_inode_num(), for use by
fuse2fs.

Fixed a bug in ext2fs_file_set_size2() so that when it is truncating a
file, it actually works.

The block iterator now properly honors the BLOCK_ABORT flag for
extent-based flags.  Previously, it didn't, which generally made code
be less efficient, but it could cause bugs in ext2fs_link(), for
example, by causing it to insert multiple directory entries.

Fixed an (harmless other than causing a compiler warning) use of an
uninitialized variable in e2fsck's MMP code.

Re: E2fsprogs 1.42.1 release

[Eric Sandeen]

On 2/19/12 9:19 PM, Ted Ts'o wrote:
> On Sun, Feb 19, 2012 at 10:11:12PM -0500, Eric Sandeen wrote:
>>> Which is in e2fsprogs 1.42.1....
>>>
>> Was that the release announcement?  :)
> 
> I knew I forgot something.  :-)
> 
> Over the weekend I released e2fsprogs 1.42.1.  It can be found in the
> usual places: on sourceforge, a signed git tag, and on
> ftp.kernel.org:/pub/linux/kernel/people/tytso/e2fsprogs/v1.42.1.

The tarball is a little messed up:

[esandeen@neon e2fsprogs]$ tar xvzf e2fsprogs-1.42.1.tar.gz
e2fsprogs-libs-1.42.1/
e2fsprogs-libs-1.42.1/.gitignore
e2fsprogs-libs-1.42.1/.release-checklist
e2fsprogs-libs-1.42.1/lib/

why does it unpack into e2fsprogs-libs-1.42.1?
                                  ^^^^

I could work around that but it's unexpected.

-Eric

> 							- Ted
> 
> E2fsprogs 1.42.1 (February 17, 2012)
> ===================================
> 
> The mke2fs and e2fsck now use significantly less memory when creating
> or checking very large file systems.  This was enabled by adding
> extent-based bitmaps which are stored using a red-block tree, since
> block and inode allocations tend to be contiguous.
> 
> The command mke2fs -S is used as a last ditch recovery command to
> write new superblock and block group descriptors, but _not_ to destroy
> the inode table in hopes of recovering from a badly corrupted file
> system.  So if the uninit_bg feature is enabled, mke2fs -S will now
> set the unused inodes count field to zero.  Otherwise, e2fsck -fy
> after using mke2fs -S would leave the file system completely empty.
> 
> Since mke2fs recognizes mke3fs in argv[0] to mean "mkfs.ext3", also
> honor "mke4fs" to work the same as "mke2fs.ext4", since RHEL5 has
> installed an mke2fs binary using that name.
> 
> The usage and help messages for the -G, -t and -T options in mke2fs
> have been fixed.
> 
> If e2fsck needs to use the backup group descriptors, the
> ext2fs_open2() function clears the UNINIT bits to ensure all of the
> inodes in the file systems get scanned.  However, the code which reset
> the UNINIT flags did not also recalculate the checksum, which produced
> many spurious (and scary) e2fsck messages.  This has been fixed by
> resetting cheksums when the UNINIT bits are cleared.
> 
> Relax a check in e2fsck which required that the block bitmap to be
> initialized when the inode bitmap is in use.  This will allow us to
> eventually eliminate code from the kernel which forcibly initialized
> the block bitmap when the inode bitmap is first used, requiring an
> extra journal credit and disk write.  (Addresses Google Bug: #5944440)
>     
> Make sure rdebugfs (which may be installed setuid or setgid disk) does
> not honor environment variables if euid != uid or egid != gid.
> 
> Debugfs's ncheck command has been optimized and now is much more
> robust with faced with corrupted file systems.  The ncheck command
> also now has a -c option which will verify the file type information
> in the directory entry to see if matches the inode's mode bits.  This
> is extremely useful when trying to use debugsfs to determine which
> parts of the file system metadata can be trusted.
> 
> E2image will try to use ftruncate64() to set the i_size for raw
> images, instead of writing a single null byte.  This avoid allocating
> an extra block to the raw image, for those file systems and/or
> operating systems that support this.  (Linux does.)  In addition, fix
> a logic bug that caused the file to not be properly extended if the
> size of the last hole was exactly an multiple of a megabyte.
> 
> Fixed a bug in resize2fs where for 1k and 2k file systems, where
> s_first_data_block is non-zero, this wasn't taken into account when
> calculate the minimum file system size for use with the -M option.
> 
> Fixed the badblocks program to honor the -s flag when in read-only -t
> mode.  (Addresses Debian Bug #646629)
> 
> Update Czech, Dutch, French, Polish, and Sweedish translation from the
> Translation Project.
> 
> Fixed various Debian Packaging issues so that dpkg-buildflags is used
> if present, which allows e2fsprogs to be built with security hardening
> flags.  (Addresses Debian Bugs: #654457)
> 
> Programmer's Notes
> ------------------
> 
> Fix a bug in ext2fs_clear_generic_bmap() when used for 32-bit bitmaps.
> This was only an issue for programs compiled against e2fsprogs 1.41
> that manipulate bitmaps directly.  (Addresses Sourceforge Bugs:
> #3451486)
> 
> The libext2fs library now uses sysconf() to fetch the page size, instead
> of the deprecated getpagesize().
> 
> The ext2fs_get_pathname() function will return a partial path if an a
> directory in the path is not a directory, displaying it as an inode
> number in angle brackets instead of giving up and displaying an error.
> This is much more helpful when a user is trying to debug a corrupted
> file system.
> 
> Codepoints for the RO_COMPAT_REPLICA feature has been reserved.
> 
> Added a new library function, ext2fs_file_get_inode_num(), for use by
> fuse2fs.
> 
> Fixed a bug in ext2fs_file_set_size2() so that when it is truncating a
> file, it actually works.
> 
> The block iterator now properly honors the BLOCK_ABORT flag for
> extent-based flags.  Previously, it didn't, which generally made code
> be less efficient, but it could cause bugs in ext2fs_link(), for
> example, by causing it to insert multiple directory entries.
> 
> Fixed an (harmless other than causing a compiler warning) use of an
> uninitialized variable in e2fsck's MMP code.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: E2fsprogs 1.42.1 release

[Eric Sandeen]

On 2/20/12 12:59 PM, Eric Sandeen wrote:
> On 2/19/12 9:19 PM, Ted Ts'o wrote:
>> On Sun, Feb 19, 2012 at 10:11:12PM -0500, Eric Sandeen wrote:
>>>> Which is in e2fsprogs 1.42.1....
>>>>
>>> Was that the release announcement?  :)
>>
>> I knew I forgot something.  :-)
>>
>> Over the weekend I released e2fsprogs 1.42.1.  It can be found in the
>> usual places: on sourceforge, a signed git tag, and on
>> ftp.kernel.org:/pub/linux/kernel/people/tytso/e2fsprogs/v1.42.1.
> 
> The tarball is a little messed up:
> 
> [esandeen@neon e2fsprogs]$ tar xvzf e2fsprogs-1.42.1.tar.gz
> e2fsprogs-libs-1.42.1/
> e2fsprogs-libs-1.42.1/.gitignore
> e2fsprogs-libs-1.42.1/.release-checklist
> e2fsprogs-libs-1.42.1/lib/
> 
> why does it unpack into e2fsprogs-libs-1.42.1?
>                                   ^^^^
> 
> I could work around that but it's unexpected.

Oh actually I can't work around it, it really is just the libs.

-Eric

Re: E2fsprogs 1.42.1 release

[Ted Ts'o]

On Mon, Feb 20, 2012 at 01:01:52PM -0600, Eric Sandeen wrote:
> > The tarball is a little messed up:
> > 
> > [esandeen@neon e2fsprogs]$ tar xvzf e2fsprogs-1.42.1.tar.gz
> > e2fsprogs-libs-1.42.1/
> > e2fsprogs-libs-1.42.1/.gitignore
> > e2fsprogs-libs-1.42.1/.release-checklist
> > e2fsprogs-libs-1.42.1/lib/
> > 
> > why does it unpack into e2fsprogs-libs-1.42.1?
> >                                   ^^^^
> > 
> > I could work around that but it's unexpected.
> 
> Oh actually I can't work around it, it really is just the libs.

Whoops, kup screwup.  I uploaded e2fsprogs-1.42.tar.gz and
e2fsprogs-libs-1.42.tar.gz.... into the same destination file.  I just
fixed it on master.kernel.org; it should propagate over in 15 minutes
or so.

Or you can grab the files from sourceforge.  (Note that the
sourceforge tarballs are actually signed by me.  In contrast the
kernel.org tarballs are signed by the ftp.kernel.org machine key, and
the signature covers the uncompressed tar file.  That way they can use
the same signature file for the tar.gz and tar.bz2 tarball.)

    	 	   	    	       	   	   - Ted