Run mkfs.ext4 on an already existing ext4 filesystem. Seeking professional help or programming hints for the recovery.

Run mkfs.ext4 on an already existing ext4 filesystem. Seeking professional help or programming hints for the recovery.

[Andrea Lo Pumo]

Sorry for writing on this development mailing list, we are seeking an
expert in ext4, we are also willing to offer a professional contract
to solve this issue.

On /dev/sda1(*) there was an ext4 file system with a lot of large
files. Now, by mistake, mkfs.ext4 has been run on /dev/sda1. The
result is that now /dev/sda1 is "empty": mounting it shows no files.

Note *: actually, it is /dev/mapper/secure_storage, an encrypted
volume opened with LUKS: cryptsetup luksOpen /dev/sda1 secure_storage

Note 2: Ubuntu 18.04

This is my current understanding:

- Originally, /dev/sda1 was created with "mkfs.ext4 /dev/sda1"
- Now, the second "mkfs.ext4 /dev/sda1" has overwritten the
superblock, and all backups of the superblock (because it has created
the backups of the NEW superblock at the exact same locations of the
previous superblock backups).
- The inode map has been overwritten too.
- However, the data is still there in the disk, and also the related
inode structures. (Just the inode map is missing right?). So, if one
is able to locate these inode structures, the relative files could be
recovered. We know the name of important directories and files to be
recovered. Could this help?

We are willing to discuss a professional contract tailored to solve
this issue. So if you have the right knowledge, or you know someone,
please let us know.

I could also invest some programming efforts to solve this issue, by
hacking some available tools, if this could help and is not too
complex. In this regard, I have this question: given that I know the
name of some important directories and files to be recovered,
theoretically I could write a program that "greps" the name of the
file in /dev/sda1 and, around that point, I should locate the inode
structure, and with the inode recover the whole file? Any hint toward
this direction? I don't have experience with ext programming, but I am
willing to hack.

Final question: are there tools to handle this situation? testdisk and
ext4magic do not seem to give good results. Photorec is useless to
recover large .tar.gz and .ogg files, and more importantly the name of
the file, which we also need.

Thank you.

Best regards.

Re: Run mkfs.ext4 on an already existing ext4 filesystem. Seeking professional help or programming hints for the recovery.

[Reindl Harald]

Am 11.04.19 um 11:37 schrieb Andrea Lo Pumo:
> Sorry for writing on this development mailing list, we are seeking an
> expert in ext4, we are also willing to offer a professional contract
> to solve this issue.
> 
> On /dev/sda1(*) there was an ext4 file system with a lot of large
> files. Now, by mistake, mkfs.ext4 has been run on /dev/sda1. The
> result is that now /dev/sda1 is "empty": mounting it shows no files.
> 
> Note *: actually, it is /dev/mapper/secure_storage, an encrypted
> volume opened with LUKS: cryptsetup luksOpen /dev/sda1 secure_storage
> 
> This is my current understanding:
> 
> - Originally, /dev/sda1 was created with "mkfs.ext4 /dev/sda1"
> - Now, the second "mkfs.ext4 /dev/sda1" has overwritten the
> superblock, and all backups of the superblock (because it has created
> the backups of the NEW superblock at the exact same locations of the
> previous superblock backups)

no

you said you had a LUKS volume on /dev/sda1 which means by directly
create a fs on /dev/sda1 you have overwritten parts of the encryption
layer *and* at the same time the filesystem *on top* of it

in other words: you are done, can stop seek solutions and start restore
your backups

Re: Run mkfs.ext4 on an already existing ext4 filesystem. Seeking professional help or programming hints for the recovery.

[Andrea Lo Pumo]

Il giorno gio 11 apr 2019 alle ore 11:42 Reindl Harald
<h.reindl@thelounge.net> ha scritto:
>
> Am 11.04.19 um 11:37 schrieb Andrea Lo Pumo:
> > On /dev/sda1(*) there was an ext4 file system with a lot of large
> > files. Now, by mistake, mkfs.ext4 has been run on /dev/sda1. The
> > result is that now /dev/sda1 is "empty": mounting it shows no files.
> >
> > Note *: actually, it is /dev/mapper/secure_storage, an encrypted
> > volume opened with LUKS: cryptsetup luksOpen /dev/sda1 secure_storage
>
> no
>
> you said you had a LUKS volume on /dev/sda1 which means by directly
> create a fs on /dev/sda1 you have overwritten parts of the encryption
> layer *and* at the same time the filesystem *on top* of it

Sorry, I explained it badly. The commands were:

  cryptsetup luksOpen /dev/sda1 secure_storage
  mkfs.ext4 /dev/mapper/secure_storage

Re: Run mkfs.ext4 on an already existing ext4 filesystem. Seeking professional help or programming hints for the recovery.

[Andrea Lo Pumo]

To clarify, the commands run on the machine were:

   cryptsetup luksOpen /dev/sda1 secure_storage
   mkfs.ext4 /dev/mapper/secure_storage  <--- this messed everything

Through the emails I used "/dev/sda1" to make the story simpler, as I
believe that /dev/mapper/secure_storage is the same as a normal
partition.

Thank you again.

Il giorno gio 11 apr 2019 alle ore 11:45 Andrea Lo Pumo
<alopumo@movia.biz> ha scritto:
>
> Il giorno gio 11 apr 2019 alle ore 11:42 Reindl Harald
> <h.reindl@thelounge.net> ha scritto:
> >
> > Am 11.04.19 um 11:37 schrieb Andrea Lo Pumo:
> > > On /dev/sda1(*) there was an ext4 file system with a lot of large
> > > files. Now, by mistake, mkfs.ext4 has been run on /dev/sda1. The
> > > result is that now /dev/sda1 is "empty": mounting it shows no files.
> > >
> > > Note *: actually, it is /dev/mapper/secure_storage, an encrypted
> > > volume opened with LUKS: cryptsetup luksOpen /dev/sda1 secure_storage
> >
> > no
> >
> > you said you had a LUKS volume on /dev/sda1 which means by directly
> > create a fs on /dev/sda1 you have overwritten parts of the encryption
> > layer *and* at the same time the filesystem *on top* of it
>
> Sorry, I explained it badly. The commands were:
>
>   cryptsetup luksOpen /dev/sda1 secure_storage
>   mkfs.ext4 /dev/mapper/secure_storage

Re: Run mkfs.ext4 on an already existing ext4 filesystem. Seeking professional help or programming hints for the recovery.

[Theodore Ts'o]

On Thu, Apr 11, 2019 at 11:37:55AM +0200, Andrea Lo Pumo wrote:
> - The inode map has been overwritten too.
> - However, the data is still there in the disk, and also the related
> inode structures. (Just the inode map is missing right?). So, if one
> is able to locate these inode structures, the relative files could be
> recovered. We know the name of important directories and files to be
> recovered. Could this help?

Unfortunately, what gets overwritten is the inode table, which
contains the inode structures.  So all of the information which says,
"logical block N of inode M is located on physical block P" is gone.

So your only hope is going to be to use a program which looks at
individual data blocks, and assumes that (for the most part) files
tend to be allocated contiguously on the storage device.  Fortunately,
such a tool has already been written, and it is an open source tool
called PhotoRec[1].  I see however, you've already tried PhotoRec.

> I could also invest some programming efforts to solve this issue, by
> hacking some available tools, if this could help and is not too
> complex. In this regard, I have this question: given that I know the
> name of some important directories and files to be recovered,
> theoretically I could write a program that "greps" the name of the
> file in /dev/sda1 and, around that point, I should locate the inode
> structure, and with the inode recover the whole file? Any hint toward
> this direction? I don't have experience with ext programming, but I am
> willing to hack.

Yeah, unfortunately, no.  You'll be able to find the directory data
block, sure.  And that will contain the inode number.  But mke2fs
overwrites the entire inode table, so there's nothing that you can
find.

> Final question: are there tools to handle this situation? testdisk and
> ext4magic do not seem to give good results. Photorec is useless to
> recover large .tar.gz and .ogg files, and more importantly the name of
> the file, which we also need.

You've listed the primary tools that are available already.  It is
possible to configure mke2fs to create an undo file, and then when
something screws up they can use the e2undo file to unwind the
modifications made by mke2fs (or e2fsck, if the undo file generation
is enabled by e2fsck).

This feature is not enabled by default, mainly because (a) it slows
down the mke2fs and e2fsck operations, and that tends to make system
administrators cranky, and (b) you have to put the e2undo file
somewhere, and you need to have some kind of scheme to delete old
e2undo files.  So there is a lot of distribution integration changes
that has never been done.

Telling you this now isn't particularly helpful, since it's basically
suggesting that you close the barn door after the horse has escaped.
However, along with other changes you might want to make to your
procedures (such as doing regular backups) to avoid future mistakes of
this ilk, it might be something to consider.

Good luck, and sorry there's not much else help we can offer you,

					- Ted