how to recover filesystem after clobbering array?

how to recover filesystem after clobbering array?

[Vasco Névoa]

Hello people.

I've messed up good, and now I need you nice folks to help me recover  
500GB of irreplaceable home video, the full 7 years of my family  
trove. :(

I mistakenly used mdadm to *create* an array instead of *starting* the  
array (big Duh!). Now the array has no partition table. I shudder to  
think I may have clobbered not only the array but also the file  
system. I hardly slept last night. Acceptance is a process. :/

I've learned my lesson there, no more fiddling with "--assume-clean"  
(which was a stupid idea in the first place), but the issue remains:  
how do I get to the file system that I know is still there? The array  
is up but obviously it was never mounted.

Can I just recreate the partition table and it works?.... or do I have  
to use some complex form of forensics to recover the data?

Thank you very much,
Vasco.

Re: how to recover filesystem after clobbering array?

[Vasco Névoa]

Oh, the type of array is Raid 1.

Citando Vasco Névoa <vasco.nevoa@sapo.pt>:

>
> Hello people.
>
> I've messed up good, and now I need you nice folks to help me  
> recover 500GB of irreplaceable home video, the full 7 years of my  
> family trove. :(
>
> I mistakenly used mdadm to *create* an array instead of *starting*  
> the array (big Duh!). Now the array has no partition table. I  
> shudder to think I may have clobbered not only the array but also  
> the file system. I hardly slept last night. Acceptance is a process.  
> :/
>
> I've learned my lesson there, no more fiddling with "--assume-clean"  
> (which was a stupid idea in the first place), but the issue remains:  
> how do I get to the file system that I know is still there? The  
> array is up but obviously it was never mounted.
>
> Can I just recreate the partition table and it works?.... or do I  
> have to use some complex form of forensics to recover the data?
>
> Thank you very much,
> Vasco.
>
>
>


--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: how to recover filesystem after clobbering array?

[David Brown]

On 19/07/2011 11:43, Vasco Névoa wrote:
>
> Hello people.
>
> I've messed up good, and now I need you nice folks to help me recover
> 500GB of irreplaceable home video, the full 7 years of my family trove. :(
>
> I mistakenly used mdadm to *create* an array instead of *starting* the
> array (big Duh!). Now the array has no partition table. I shudder to
> think I may have clobbered not only the array but also the file system.
> I hardly slept last night. Acceptance is a process. :/
>
> I've learned my lesson there, no more fiddling with "--assume-clean"
> (which was a stupid idea in the first place), but the issue remains: how
> do I get to the file system that I know is still there? The array is up
> but obviously it was never mounted.
>
> Can I just recreate the partition table and it works?.... or do I have
> to use some complex form of forensics to recover the data?
>
> Thank you very much,
> Vasco.
>

The first thing you should do is write out "RAID is not a backup 
solution" 100 times!

Then write down all the information you know - exactly how was the array 
built up, how were the disks partitioned, what filesystems, etc.  And 
what was the exact command you used to mess up the array, and how far 
did you let it go with resynchronisation, etc.  Make good notes here 
before you forget.

The next step is to get a couple spare disks that are bigger than your 
original disks (get 2 TB disks - they cost almost the same as anything 
smaller).  Make a direct copy of the entire original disks to a file on 
the new disks with something like:

	dd if=/dev/sda of=/mnt/backup/disk1.img bs=1M

Once you have got image files for each of your disks, make copies of 
these image files to another spare disk.  Keep careful notes of exactly 
what you have done here, and which files are which.  And put your 
original disks, carefully labelled, on a shelf somewhere.

Now you are in a position to attempt data recovery on your copied files. 
  If you do something wrong, you can simply re-copy the image files and 
try again.  You still have absolutely no guarantees that you'll get 
anything back - but at least you can be sure you are not going to make 
anything worse.

If you have started re-syncing the two disks as a RAID1 pair, there is a 
good chance that one of the disks contains the original data and 
filesystem, except where it was overwritten by the new metadata for the 
newly created array.

Then you sit back and hope that someone on this list can give you ideas 
about getting the data back from the image file(s).


--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: how to recover filesystem after clobbering array?

[Tyler J. Wagner]

On 2011-07-19 11:17, David Brown wrote:
> Once you have got image files for each of your disks, make copies of
> these image files to another spare disk. Keep careful notes of exactly
> what you have done here, and which files are which. And put your
> original disks, carefully labelled, on a shelf somewhere.
>
> Now you are in a position to attempt data recovery on your copied
> files. If you do something wrong, you can simply re-copy the image
> files and try again. You still have absolutely no guarantees that
> you'll get anything back - but at least you can be sure you are not
> going to make anything worse.

Follow David's advice.

What was the filesystem on the array?

Now, use testdisk, photorec, and foremost to seek through the raw images
and extract files. The good news is, most video formats are detectable
by these tools. All can be installed with your package manager.

Regards,
Tyler

-- 
"Offending fundamentalists isn't my goal – but if it is an inevitable
side-effect of defending human rights, so be it."
   -- Johann Hari

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: how to recover filesystem after clobbering array?

[Vasco Névoa]

Thank you very much, Tyler and David.
Good advice.

The array is level 1, built upon primary partitions of the devices.
The file system is EXT4.
The mdadm command that I stupidly used to create instead of starting  
the array included "--assume-clean" but no "--build".
I checked via /proc/mdadm that the array came up without syncing  
anything (or at least it was ultra-fast, less than 5 seconds).
So I firmly believe the data is all there, on both disks.
I just need to rebuild the metadata to point to the data again somehow.
Right?...

Citando "Tyler J. Wagner" <tyler@tolaris.com>:

> On 2011-07-19 11:17, David Brown wrote:
>> Once you have got image files for each of your disks, make copies of
>> these image files to another spare disk. Keep careful notes of exactly
>> what you have done here, and which files are which. And put your
>> original disks, carefully labelled, on a shelf somewhere.
>>
>> Now you are in a position to attempt data recovery on your copied
>> files. If you do something wrong, you can simply re-copy the image
>> files and try again. You still have absolutely no guarantees that
>> you'll get anything back - but at least you can be sure you are not
>> going to make anything worse.
>
> Follow David's advice.
>
> What was the filesystem on the array?
>
> Now, use testdisk, photorec, and foremost to seek through the raw images
> and extract files. The good news is, most video formats are detectable
> by these tools. All can be installed with your package manager.
>
> Regards,
> Tyler
>
> --
> "Offending fundamentalists isn't my goal - but if it is an inevitable
> side-effect of defending human rights, so be it."
>    -- Johann Hari
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: how to recover filesystem after clobbering array?

[Tyler J. Wagner]

On 2011-07-19 12:01, Vasco Névoa wrote:
> So I firmly believe the data is all there, on both disks.
> I just need to rebuild the metadata to point to the data again somehow.
> Right?...

Yes. Or no. Do you want to risk losing your data? Clone one of the disks
now, then experiment with trying to re-assemble the array.

Regards,
Tyler

-- 
"It is an interesting and demonstrable fact, that all children are atheists
and were religion not inculcated into their minds, they would remain so."
   -- Ernestine Rose

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: how to recover filesystem after clobbering array?

[Vasco Névoa]

I've successfully recovered my array data. :)
All it took was, as I expected, to rebuild the right pointers. The data 
was always untouched.
Here it goes, for the sake of completeness.

I initially used "mdadm --create --assume-clean ..." on a level1 array 
of 2 disks that came from another machine. I didn't know any other way 
of starting an array in this situation. While this is an acceptable 
practice in some cases, it is better to use "--assemble" and pass the 
necessary info (like uuid).

Unfortunately I apparently lost all the data because the newly created 
metadata superblock was the default 0.9 version, and the original array 
metadata version was 1.2, and this resulted in a missing partition table 
once the array was run.

Then I retrieved the mdadm.conf file from the original machine, and 
there I found the correct metadata version, uuid, array name, array 
device name. I also double-checked the volume partition and filesystem 
type from that machine's /etc/fstab.

So, I zeroed-out the metadata superblocks with "mdadm --zero-superblock 
...", and then proceeded to restore the original metadata superblock 
with "mdadm --create --assume-clean --metadata=1.2 --uuid=... 
--name=...  --level=1 --raid-devices=2 /dev/sde1 missing", and it worked 
just fine. The array was started and I could see the original data 
partition with fdisk and actually mount it. After this successful test, 
I just added the other disk to the array.

The one information I expected from this list was: "No problem, your 
data is still there. If you can recreate the superblock with the same 
metadata version as it used to be, everything reverts to normal." 
Unfortunately I had no such support.

Cheers,
Vasco.

On 19-07-2011 12:01, Vasco Névoa wrote:
> Thank you very much, Tyler and David.
> Good advice.
>
> The array is level 1, built upon primary partitions of the devices.
> The file system is EXT4.
> The mdadm command that I stupidly used to create instead of starting 
> the array included "--assume-clean" but no "--build".
> I checked via /proc/mdadm that the array came up without syncing 
> anything (or at least it was ultra-fast, less than 5 seconds).
> So I firmly believe the data is all there, on both disks.
> I just need to rebuild the metadata to point to the data again somehow.
> Right?...
>
> Citando "Tyler J. Wagner" <tyler@tolaris.com>:
>
>> On 2011-07-19 11:17, David Brown wrote:
>>> Once you have got image files for each of your disks, make copies of
>>> these image files to another spare disk. Keep careful notes of exactly
>>> what you have done here, and which files are which. And put your
>>> original disks, carefully labelled, on a shelf somewhere.
>>>
>>> Now you are in a position to attempt data recovery on your copied
>>> files. If you do something wrong, you can simply re-copy the image
>>> files and try again. You still have absolutely no guarantees that
>>> you'll get anything back - but at least you can be sure you are not
>>> going to make anything worse.
>>
>> Follow David's advice.
>>
>> What was the filesystem on the array?
>>
>> Now, use testdisk, photorec, and foremost to seek through the raw images
>> and extract files. The good news is, most video formats are detectable
>> by these tools. All can be installed with your package manager.
>>
>> Regards,
>> Tyler
>>
>> -- 
>> "Offending fundamentalists isn't my goal - but if it is an inevitable
>> side-effect of defending human rights, so be it."
>>    -- Johann Hari
>>
>> -- 
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
>

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: how to recover filesystem after clobbering array?

[Mikael Abrahamsson]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 792 bytes --]

On Sat, 23 Jul 2011, Vasco Névoa wrote:

> I initially used "mdadm --create --assume-clean ..." on a level1 array 
> of 2 disks that came from another machine. I didn't know any other way 
> of starting an array in this situation. While this is an acceptable 
> practice in some cases, it is better to use "--assemble" and pass the 
> necessary info (like uuid).

In case someone finds this from the archives:

mdadm --create --assume-clean is a LAST RESORT. It's to be used when 
everything else fails. I've seen so many start doing this way too early in 
the process and who then subsequently lost their data because of 
superblock version and mdadm changes over time (especially when they 
forgot --assume-clean and got the layout wrong).

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: how to recover filesystem after clobbering array?

[CoolCold]

Just to make data restoring a bit safer - one can use device mappers's
snapshot feature as it can snapshot _any_ block device, you can create
snapshots of say /dev/sda1 & /dev/sdb1 , then do your raid experiments
on that snapshotted devices.

More info http://www.mjmwired.net/kernel/Documentation/device-mapper/snapshot.txt
https://www.redhat.com/archives/dm-devel/2004-July/msg00071.html

2011/7/23 Mikael Abrahamsson <swmike@swm.pp.se>:
> On Sat, 23 Jul 2011, Vasco Névoa wrote:
>
>> I initially used "mdadm --create --assume-clean ..." on a level1 array of
>> 2 disks that came from another machine. I didn't know any other way of
>> starting an array in this situation. While this is an acceptable practice in
>> some cases, it is better to use "--assemble" and pass the necessary info
>> (like uuid).
>
> In case someone finds this from the archives:
>
> mdadm --create --assume-clean is a LAST RESORT. It's to be used when
> everything else fails. I've seen so many start doing this way too early in
> the process and who then subsequently lost their data because of superblock
> version and mdadm changes over time (especially when they forgot
> --assume-clean and got the layout wrong).
>
> --
> Mikael Abrahamsson    email: swmike@swm.pp.se



-- 
Best regards,
[COOLCOLD-RIPN]
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html