Erased start of voulme...

Erased start of voulme...

[Louis Erickson]

I don't know how it happened, but the start of the /var partition of one 
of the machines I run has gotten itself zeroed out.  Only the first little 
bit - 4k or so - is nulls, then a large amount of very plausible data 
appears.

I can't get reiserfsck to even think this is a reiserfs volume - the 
program doesn't think this is a reiserfs volume any more.

Is there any chance of getting the data back off of this disk?  Can 
reiserfsck be told to look harder and recover more deeply or something?  
Or is this a lesson in why backups are important?

Any suggestions you can give are welcome.

I'm currently running dd from the mangled partition to a file to be able 
to try some things without losing my only copy of the data.  If there's 
anything special I have to do to make that work, please let me know.

Thank you!

-- 
Louis Erickson - lerickson@rdwarf.net - http://www.rdwarf.com/~wwonko/

Oh Dad!  We're ALL Devo!

Re: Erased start of voulme...

[Sander]

Louis Erickson wrote (ao):
> I don't know how it happened, but the start of the /var partition of
> one of the machines I run has gotten itself zeroed out. Only the first
> little bit - 4k or so - is nulls, then a large amount of very
> plausible data appears.
> 
> I can't get reiserfsck to even think this is a reiserfs volume - the
> program doesn't think this is a reiserfs volume any more.

What arguments did you use? What version of reiserfsck? And what
filesystem?

> Is there any chance of getting the data back off of this disk? Can
> reiserfsck be told to look harder and recover more deeply or
> something? Or is this a lesson in why backups are important?

_And_ it is a lesson why backups are important :-)

> I'm currently running dd from the mangled partition to a file to be
> able to try some things without losing my only copy of the data. If
> there's anything special I have to do to make that work, please let me
> know.

Good start.

-- 
Humilis IT Services and Solutions
http://www.humilis.net

Re: Erased start of voulme...

[Alex Zarochentsev]

Hello,

On Thu, Jan 13, 2005 at 05:03:30PM -0800, Louis Erickson wrote:
> 
> I don't know how it happened, but the start of the /var partition of one 
> of the machines I run has gotten itself zeroed out.  Only the first little 
> bit - 4k or so - is nulls, then a large amount of very plausible data 
> appears.
> 
> I can't get reiserfsck to even think this is a reiserfs volume - the 
> program doesn't think this is a reiserfs volume any more.

> 
> Is there any chance of getting the data back off of this disk?  Can 
> reiserfsck be told to look harder and recover more deeply or something?  
> Or is this a lesson in why backups are important?
> 
> Any suggestions you can give are welcome.

reiserfsck --rebuild-sb can re-create the super block.

> I'm currently running dd from the mangled partition to a file to be able 
> to try some things without losing my only copy of the data.  If there's 
> anything special I have to do to make that work, please let me know.
> 
> Thank you!
> 
> -- 
> Louis Erickson - lerickson@rdwarf.net - http://www.rdwarf.com/~wwonko/
> 
> Oh Dad!  We're ALL Devo!

-- 
Alex.

Re: Erased start of voulme...

[E.Gryaznova]

Alex Zarochentsev wrote:

>Hello,
>
>On Thu, Jan 13, 2005 at 05:03:30PM -0800, Louis Erickson wrote:
>  
>
>>I don't know how it happened, but the start of the /var partition of one 
>>of the machines I run has gotten itself zeroed out.  Only the first little 
>>bit - 4k or so - is nulls, then a large amount of very plausible data 
>>appears.
>>
>>I can't get reiserfsck to even think this is a reiserfs volume - the 
>>program doesn't think this is a reiserfs volume any more.
>>    
>>
>
>  
>
>>Is there any chance of getting the data back off of this disk?  Can 
>>reiserfsck be told to look harder and recover more deeply or something?  
>>Or is this a lesson in why backups are important?
>>
>>Any suggestions you can give are welcome.
>>    
>>
>
>reiserfsck --rebuild-sb can re-create the super block.
>  
>
I would prefer to do the following :
dd if=/dev/problem_partition of=/dev/sparedevice
sparedevice can be usual disk partition
then
reiserfsck --rebuild-sb /dev/sparedevice
reiserfsck --check /dev/sparedevice

Thanks,
Lena.

>  
>
>>I'm currently running dd from the mangled partition to a file to be able 
>>to try some things without losing my only copy of the data.  If there's 
>>anything special I have to do to make that work, please let me know.
>>
>>Thank you!
>>
>>-- 
>>Louis Erickson - lerickson@rdwarf.net - http://www.rdwarf.com/~wwonko/
>>
>>Oh Dad!  We're ALL Devo!
>>    
>>
>
>  
>

Re: Erased start of voulme...

[Louis Erickson]

Wow, thanks for the fast responses, everyone.  I'll try and consolidate 
them all in to one reply, in the order I got them here.

Silly me forgot to start ssh on the afflicted box, so I can't get to it 
until I get home.  I do have another near-identical machine that I've 
checked versions and such on, and can use to build tools if need be.

On Fri, 14 Jan 2005, Sander <sander@humilis.net> wrote:

> What arguments did you use?

reiserfsck --check /dev/md2

> What version of reiserfsck?

How do I find that out?  It isn't in the help or visible as an option.  
If I ask mkfs.reiserfs it tells me it's 3.x.1b (2002).  I suspect it's 
quite old.

Should I build new tools (on another similar machine) and use those 
instead?

> And what filesystem?

It's /dev/md2, mounting on /var.  Linux is unhappy without it, although it 
does come up to single user mode.  Much of the important data is on /home, 
and I can therefore get off if I have to completely rebuild, but there's a 
couple of things (/var/mail, for instance) that would be good to rescue.  
I'd like to try and get a gander at /var/log too, to see if I can suss out 
what happened.

All of this is running on software raid, but the volumes are all up and 
good, and the raid component isn't complaining at all.

Or is that not the question you're asking?

> _And_ it is a lesson why backups are important :-)

Yes it is.  It is past time to fix this astonishing lack in our 
infrastructure.

On Fri, 14 Jan 2005, Alex Zarochentsev <zam@namesys.com> wrote:

> reiserfsck --rebuild-sb can re-create the super block.

I saw that, but it looked dangerous, and I didn't want to try it until my 
copy had finished.

On Fri, 14 Jan 2005, E.Gryaznova <grev@namesys.com> wrote:

> I would prefer to do the following :
> dd if=/dev/problem_partition of=/dev/sparedevice
> sparedevice can be usual disk partition

I don't have a spare partition large enough in this system.  I have done:

dd if=/dev/problem_partition of=/home/scratch/rescue.dat

I'll then use the loopback file system to create a device entry for that 
file, and work on that.  It was a suggestion made earlier on this very 
mailing list, and it sounded like a wise one to me.  If the loopback 
doesn't work, I'll fiddle around with hardware to make a spare partition 
large enough available.

> then
> reiserfsck --rebuild-sb /dev/sparedevice
> reiserfsck --check /dev/sparedevice

The help screen for --rebuild-sb says that a --rebuild-sb will require a 
--rebuild-tree afterwards.  Should I try that, or the --check?  Or will 
--check do the --rebuild-sb as needed?

Also, there's a -S/--scan-whole-partition option... should I use that to 
make sure no files are missed, or would it best not to add that?

Again, thank you all for your quick replies.  It's been very helpful, and 
very positive to hear there are things I can at least try, rather than 
that I am simply hosed.

-- 
Louis Erickson - lerickson@rdwarf.net - http://www.rdwarf.com/~wwonko/

"The human brain is like an enormous fish -- it is flat and slimy and
has gills through which it can see."
		-- Monty Python

Re: Erased start of voulme...

[E.Gryaznova]

Louis Erickson wrote:

>Wow, thanks for the fast responses, everyone.  I'll try and consolidate 
>them all in to one reply, in the order I got them here.
>
>Silly me forgot to start ssh on the afflicted box, so I can't get to it 
>until I get home.  I do have another near-identical machine that I've 
>checked versions and such on, and can use to build tools if need be.
>
>On Fri, 14 Jan 2005, Sander <sander@humilis.net> wrote:
>
>  
>
>>What arguments did you use?
>>    
>>
>
>reiserfsck --check /dev/md2
>
>  
>
>>What version of reiserfsck?
>>    
>>
>
>How do I find that out?  It isn't in the help or visible as an option.  
>If I ask mkfs.reiserfs it tells me it's 3.x.1b (2002).  I suspect it's 
>quite old.
>  
>
please try the latest reiserfsprogs-3.6.19.tar.gz from 
ftp://namesys.com/pub/reiserfsprogs

>Should I build new tools (on another similar machine) and use those 
>instead?
>  
>
>  
>
>>And what filesystem?
>>    
>>
>
>It's /dev/md2, mounting on /var.  Linux is unhappy without it, although it 
>does come up to single user mode.  Much of the important data is on /home, 
>and I can therefore get off if I have to completely rebuild, but there's a 
>couple of things (/var/mail, for instance) that would be good to rescue.  
>I'd like to try and get a gander at /var/log too, to see if I can suss out 
>what happened.
>
>All of this is running on software raid, but the volumes are all up and 
>good, and the raid component isn't complaining at all.
>
>Or is that not the question you're asking?
>
>  
>
>>_And_ it is a lesson why backups are important :-)
>>    
>>
>
>Yes it is.  It is past time to fix this astonishing lack in our 
>infrastructure.
>
>On Fri, 14 Jan 2005, Alex Zarochentsev <zam@namesys.com> wrote:
>
>  
>
>>reiserfsck --rebuild-sb can re-create the super block.
>>    
>>
>
>I saw that, but it looked dangerous, and I didn't want to try it until my 
>copy had finished.
>
>On Fri, 14 Jan 2005, E.Gryaznova <grev@namesys.com> wrote:
>
>  
>
>>I would prefer to do the following :
>>dd if=/dev/problem_partition of=/dev/sparedevice
>>sparedevice can be usual disk partition
>>    
>>
>
>I don't have a spare partition large enough in this system.  I have done:
>
>dd if=/dev/problem_partition of=/home/scratch/rescue.dat
>
>I'll then use the loopback file system to create a device entry for that 
>file, and work on that.  It was a suggestion made earlier on this very 
>mailing list, and it sounded like a wise one to me.  If the loopback 
>doesn't work, I'll fiddle around with hardware to make a spare partition 
>large enough available.
>
>  
>
>>then
>>reiserfsck --rebuild-sb /dev/sparedevice
>>reiserfsck --check /dev/sparedevice
>>    
>>
>
>The help screen for --rebuild-sb says that a --rebuild-sb will require a 
>--rebuild-tree afterwards.  Should I try that, or the --check? 
>
new reiserfsck (3.6.19)  asks to run  --check after --rebuild-sb

> Or will 
>--check do the --rebuild-sb as needed?
>
>Also, there's a -S/--scan-whole-partition option... should I use that to 
>make sure no files are missed, or would it best not to add that?
>  
>
if you have a copy of corrupted fs -- you may try both ways (w/ and w/o 
-S) and see what way is more successful

Good luck!
Lena

>Again, thank you all for your quick replies.  It's been very helpful, and 
>very positive to hear there are things I can at least try, rather than 
>that I am simply hosed.
>
>  
>

Re: Erased start of voulme...

[Sander]

Louis Erickson wrote (ao):
> On Fri, 14 Jan 2005, Sander <sander@humilis.net> wrote:
> > What arguments did you use?
> 
> reiserfsck --check /dev/md2

Good. Then the options Lena suggested will most likely make your data
appear.

> > What version of reiserfsck?
> 
> How do I find that out? It isn't in the help or visible as an option.
> If I ask mkfs.reiserfs it tells me it's 3.x.1b (2002). I suspect it's
> quite old.

Normally with -V, but 3.x.1b is the version and it indeed is very old.
Newest is 3.6.19.

Is this because you use an old distribution, and do you have such an old
kernel too? (sorry if I missed the kernel/distribution).

> Should I build new tools (on another similar machine) and use those
> instead?

Yes. Reiserfsck has seen a _lot_ improvement since then.

> > And what filesystem?
> 
> It's /dev/md2, mounting on /var.

...

> All of this is running on software raid, but the volumes are all up
> and good, and the raid component isn't complaining at all.
> 
> Or is that not the question you're asking?

Well, I meant Reiser3 of Reiser4, but it is Reiser3 most likely.


> On Fri, 14 Jan 2005, Alex Zarochentsev <zam@namesys.com> wrote:
> > reiserfsck --rebuild-sb can re-create the super block.
> 
> I saw that, but it looked dangerous, and I didn't want to try it until
> my copy had finished.

Of course :-)  Perform al your reiserfsck commands on the copy, not on
the original. If somehow the copy action failed you still have the
original, while if you perform all your actions on the original and have
a broken copy, you are lost.

> On Fri, 14 Jan 2005, E.Gryaznova <grev@namesys.com> wrote:
> > I would prefer to do the following :
> > dd if=/dev/problem_partition of=/dev/sparedevice
> > sparedevice can be usual disk partition
> 
> I don't have a spare partition large enough in this system. I have
> done:
> 
> dd if=/dev/problem_partition of=/home/scratch/rescue.dat
> 
> I'll then use the loopback file system to create a device entry for
> that file, and work on that. It was a suggestion made earlier on this
> very mailing list, and it sounded like a wise one to me. If the
> loopback doesn't work, I'll fiddle around with hardware to make a
> spare partition large enough available.

It should be oke.

The rest is answered by Lena :-)


-- 
Humilis IT Services and Solutions
http://www.humilis.net