Filesystem corrupted by offline resize

Filesystem corrupted by offline resize

[Chris Naunton]

I think I'm the victim of a resize2fs bug that was addressed in
1.42.7: http://e2fsprogs.sourceforge.net/e2fsprogs-release.html#1.42.7

"Fix resize2fs so that it can handle off-line resizes of file systems
with the flex_bg feature but without a resize_inode (or if we run out
of reserved gdt blocks). This also fixes a problem where if the user
creates a filesystem with a restricted number of reserved gdt blocks,
an off-line resize which grows the file system could potentially
result in file system corruption."

My senario:

* Running Ubuntu 12.10 with the latest x64 e2fsprogs package, which is 1.47.5

* I have a 3.6T ext4 filesystem that had been created with the "resize" option:

  mkfs -t ext4 -T ext4 -E stripe-width=64,resize=20T -i 1048576 -L
data -m 0 /dev/raid5/data

  (This was three years ago. I think at the time I assumed if I didn't
specify this option my ability to resize would be limited.)

* Added an extra disk to the mdadm RAID5 array and extended the LVM
logical volume.

* Decided to perform resize offline as I assumed it would be the safer option.

* Offline fsck and resize completed successfully. No errors reported.
Didn't immediately perform a fsck after resize, just rebooted.

* Upon boot, filesytem errors detected and manual fsck is required. It
reports what looks like massive and serious corruption.


I've placed dumps of relevant output here as it's quite a few
megabytes, even compressed:

http://members.optusnet.com.au/~naunivans/ext4/


So far I've left the corrupted filesystem untouched in the hope
something can be done to repair it or recover data. I'm worried that
not many (any?) usable files will be left after a "fsck -y".

Is it likely that the bug was the cause of the corruption and if so,
what is the likely extent of data loss?

Thanks very much.
Chris Naunton

Re: Filesystem corrupted by offline resize

[Theodore Ts'o]

On Thu, Feb 21, 2013 at 04:40:42PM +1100, Chris Naunton wrote:
> I think I'm the victim of a resize2fs bug that was addressed in
> 1.42.7: http://e2fsprogs.sourceforge.net/e2fsprogs-release.html#1.42.7
> 
> My senario:
> 
> * Running Ubuntu 12.10 with the latest x64 e2fsprogs package, which is 1.47.5
> 
> * I have a 3.6T ext4 filesystem that had been created with the "resize" option:
> 
>   mkfs -t ext4 -T ext4 -E stripe-width=64,resize=20T -i 1048576 -L
> data -m 0 /dev/raid5/data
> 
>   (This was three years ago. I think at the time I assumed if I didn't
> specify this option my ability to resize would be limited.)
>.....

Yup, I think you're correct.  You hit the bug which we only discovered
and fixed in time for the 1.42.7 release.  ;-(

Unfortunately, the corruption that happened included wiping out parts
of the inode table.  This means the data loss is going to be
non-trivial, and possibly extremely bad.   Sorry.  :-(

I don't know how big your RAID array is, and how recent was your last
backups, but basically, if you can afford to make a disk image copy of
the file system, I would recommend doing that first.  Then trying
running fsck -y on the copy, and see if that allows you to recover all
of the data that you really need.  If it doesn't, it may be possible
to get more data back by doing either (a) using tools that search for
critical keywords (i.e., if there is an extremely critical file, say
like a Ph.D. thesis for which ten years of work was un backed up), or
(b) by using tools that search for file signatures in the data blocks.

The tool I recommend for the latter is called PhotoRec:

    http://www.cgsecurity.org/wiki/PhotoRec

It was originally designed to extract digital photos from damaged SD
cards, but it now understands hundreds of file formats, including
OpenOffice/LibreOffice, zip files, MS Office files, etc.:

    http://www.cgsecurity.org/wiki/File_Formats_Recovered_By_PhotoRec

I'm sorry to have to tell you this, and I'm really sorry you ran
across this bug before we had a chance to get it fixed and pushed out
to the distributions.

Regards,

					- Ted

Re: Filesystem corrupted by offline resize

[Chris Naunton]

Thanks so much for the detailed reply, Ted. And no need to apologise -
no lost Ph.D. thesis or anything! I have a backup of the most
important data. But there was over 3T of data and my backup capacity
is only 500G, so there's still a lot that will be lost.

I'll give photorec a try. It's very lucky I have the backup - most of
the critical data is encrypted with eCryptfs, so photorec wouldn't
have been able to help there.

Really appreciate all the work you've put into Linux and the
filesystem over the course of your career. I've been using and
administering Linux systems for 13 yrs and this is the first time I've
experienced major ext2/3/4 corruption that hasn't been the fault of
hardware, so I'm still a happy customer, so to speak.

Chris

Re: Filesystem corrupted by offline resize

[Spider]

Hey chris, 

I've a quiete similar problem and i tried to user Photorec which retrieved
thousands of .ecryptfs files. Could you solve your problem ? uncrypt those
files or other ? 

Thank you.