[linux-lvm] Desater Recovery <urgent>

[linux-lvm] Desater Recovery <urgent>

[Ralf Zerres]

[-- Attachment #1.1: Type: text/plain, Size: 5853 bytes --]

hello everybody,

on a production system we do have a serious data destruction problem. 
Any help leading us to restore the filesystem will be honored. We can dicuss financial details later.
First I'd like to describe the situation giving you a picture wether it is possible to restore the data.

Please answer as soon as possible to give me a hint wether I should wait for a detailed analysis,
if it make sense to dig deeper or if it is more likely that people have to reinsert the data from printed 
information.

0. PROBLEM:
=========
Destructed Filesystem after resizing LVG running a Postgres Database. 

###
# There is no actual backup, which is the mad point!!!! 
### 

24 days of DATA updates are lost right now!

I. Environment
===========

System:    Dual Prozessor PIII (2 CPU's 1Mhz)
OS:    Linux 2.4.9 

Harddisk:    SCSI-3 (as Raid-5)
Controller:    GDT 
LVM version:     1.0.1-rc2(30/08/2001)

Kernel sees one harddrive (/dev/scsi/host2/bus0/target0)

Root is on standard ext3 fs (partition 5)

LVM has a Physical Volumegroup (KI) on partition 7
The system runs a PostgreSQL Database (version 7.1.3). The Data-Structures live on a Logical-Volume-Group
which was mounted as /var/lib/postgres as a ext3 filesystem.
Everything was working until 4 hours ago, before LVG was resized.

II. Changes

The LVG /dev/ki/postgres was reduced from 8 GB to 800MB. Before we did that a df has shown a used size
of 366 MB data.
Postgres was shut down. No other process was writing to /dev/ki/postgres.

     # e2fsadm --size - 4,5G 

Worked out, but remounting has shown incorrect structure.

A new LVG was created 

    # lvcreate    --size 2,5G --name opt /dev/ki/opt

Here the dump-information from the actual Volume-Group after resizing was successful.

----------------------------------------------schnipp / schnapp ------------------------------------------------


#vgdisplay -v /dev/ki
--- Volume group ---
VG Name               ki
VG Access             read/write
VG Status             available/resizable
VG #                  0
MAX LV                256
Cur LV                7
Open LV               6
MAX LV Size           255.99 GB
Max PV                256
Cur PV                1
Act PV                1
VG Size               48.77 GB
PE Size               4 MB
Total PE              12484
Alloc PE / Size       11968 / 46.75 GB
Free  PE / Size       516 / 2.02 GB
VG UUID               P6LodN-bQUg-u5WF-03Cu-A22a-UUhE-NsKhvZ

--- Logical volume ---
LV Name                /dev/ki/export
VG Name                ki
LV Write Access        read/write
LV Status              available
LV #                   1
# open                 1
LV Size                6 GB
Current LE             1536
Allocated LE           1536
Allocation             next free
Read ahead sectors     120
Block device           58:0

--- Logical volume ---
LV Name                /dev/ki/var
VG Name                ki
LV Write Access        read/write
LV Status              available
LV #                   2
# open                 1
LV Size                1 GB
Current LE             256
Allocated LE           256
Allocation             next free
Read ahead sectors     120
Block device           58:1

--- Logical volume ---
LV Name                /dev/ki/tmp
VG Name                ki
LV Write Access        read/write
LV Status              available
LV #                   3
# open                 1
LV Size                1 GB
Current LE             256
Allocated LE           256
Allocation             next free
Read ahead sectors     120
Block device           58:2

--- Logical volume ---
LV Name                /dev/ki/home
VG Name                ki
LV Write Access        read/write
LV Status              available
LV #                   4
# open                 1
LV Size                30 GB
Current LE             7680
Allocated LE           7680
Allocation             next free
Read ahead sectors     120
Block device           58:3

--- Logical volume ---
LV Name                /dev/ki/postgres
VG Name                ki
LV Write Access        read/write
LV Status              available
LV #                   5
# open                 0
LV Size                3.78 GB
Current LE             968
Allocated LE           968
Allocation             next free
Read ahead sectors     120
Block device           58:4

--- Logical volume ---
LV Name                /dev/ki/www
VG Name                ki
LV Write Access        read/write
LV Status              available
LV #                   6
# open                 1
LV Size                2.44 GB
Current LE             625
Allocated LE           625
Allocation             next free
Read ahead sectors     120
Block device           58:5

--- Logical volume ---
LV Name                /dev/ki/opt
VG Name                ki
LV Write Access        read/write
LV Status              available
LV #                   7
# open                 1
LV Size                2.53 GB
Current LE             647
Allocated LE           647
Allocation             next free
Read ahead sectors     120
Block device           58:6


--- Physical volumes ---
PV Name (#)           /dev/scsi/host2/bus0/target0/lun0/part7 (1)
PV Status             available / allocatable
Total PE / Free PE    12484 / 516

Volume-Group: KI

----------------------------------------------schnipp / schnapp ------------------------------------------------


The sysadmin was running a vgscan an lvscan. Afterwards he runs 

    # e2fsck -y -f /dev/ki/postgres

now the mount on /var/lib/postgres was successful, but all date were gone!
I found a long list of dir-entries in the lost+found subdir. No file-entries.

What can we do?

Ralf


[-- Attachment #1.2: Type: text/html, Size: 16963 bytes --]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Ralf, Zerres.vcf --]
[-- Type: text/x-vcard; name="Ralf, Zerres.vcf", Size: 324 bytes --]

BEGIN:VCARD
VERSION:2.1
N:Ralf;Zerres
FN:Ralf, Zerres
ORG:Networkx GmbH
TEL;WORK;VOICE:+49 221 937725 - 0
TEL;WORK;FAX:+49 221 937725 - 18
ADR;WORK:;;Marktstr. 8;Köln;;50968
LABEL;WORK;ENCODING=QUOTED-PRINTABLE:Marktstr. 8=0D=0AK=F6ln 50968
EMAIL;PREF;INTERNET:rzerres@networkx.de
REV:20020422T151525Z
END:VCARD

Re: [linux-lvm] Desater Recovery <urgent>

[Joe Thornber]

On Mon, Apr 22, 2002 at 05:15:25PM +0200, Ralf Zerres wrote:
> 
> 0. PROBLEM:
> =========
> Destructed Filesystem after resizing LVG running a Postgres Database. 

Did you resize the filesystem before shrinking the LV ?

- Joe

Re: [linux-lvm] Desater Recovery <urgent>

[Ralf Zerres]

> > Destructed Filesystem after resizing LVG running a Postgres Database. 
> 
> Did you resize the filesystem before shrinking the LV ?
> 
no, i guess not.
I was using e2fsadm which is serializing the steps. As far as I know ...

Re: [linux-lvm] Desater Recovery <urgent>

[Joe Thornber]

On Mon, Apr 22, 2002 at 07:14:05PM +0200, Ralf Zerres wrote:
> 
> > > Destructed Filesystem after resizing LVG running a Postgres Database. 
> > 
> > Did you resize the filesystem before shrinking the LV ?
> > 
> no, i guess not.
> I was using e2fsadm which is serializing the steps. As far as I know ...

Yes e2fsadm should have resized the filesystem itself, so that's not
the problem.

- Joe

Re: [linux-lvm] Desater Recovery <urgent>

[Andreas Dilger]

On Apr 22, 2002  17:15 +0200, Ralf Zerres wrote:
> on a production system we do have a serious data destruction problem. 
> Any help leading us to restore the filesystem will be honored. We can dicuss
> financial details later. First I'd like to describe the situation giving you
> a picture wether it is possible to restore the data. 
> Please answer as soon as possible to give me a hint wether I should wait for
> a detailed analysis, if it make sense to dig deeper or if it is more likely
> that people have to reinsert the data from printed information.
> 
> 0. PROBLEM:
> =========
> Destructed Filesystem after resizing LVG running a Postgres Database. 
> 
> ###
> # There is no actual backup, which is the mad point!!!! 
> ### 

Yes, very mad indeed.  I don't know why people persist in not having
backups, especially for production systems...

> I. Environment
> ===========
> 
> System:    Dual Prozessor PIII (2 CPU's 1Mhz)
> OS:    Linux 2.4.9 
> 
> Harddisk:    SCSI-3 (as Raid-5)
> Controller:    GDT 
> LVM version:     1.0.1-rc2(30/08/2001)
> 
> Kernel sees one harddrive (/dev/scsi/host2/bus0/target0)
> 
> Root is on standard ext3 fs (partition 5)
> 
> LVM has a Physical Volumegroup (KI) on partition 7
> The system runs a PostgreSQL Database (version 7.1.3). The Data-Structures
> live on a Logical-Volume-Group which was mounted as /var/lib/postgres as
> a ext3 filesystem. Everything was working until 4 hours ago, before LVG
> was resized.

> II. Changes
> 
> The LVG /dev/ki/postgres was reduced from 8 GB to 800MB. Before we did that
> a df has shown a used size of 366 MB data.
> Postgres was shut down. No other process was writing to /dev/ki/postgres.
> 
>      # e2fsadm --size - 4,5G 
> 
> Worked out, but remounting has shown incorrect structure.

I have never seen it used with "- 4,5G".  I don't know what the
program would do in this case.

> A new LVG was created 
> 
>     # lvcreate    --size 2,5G --name opt /dev/ki/opt

This was created after the other one was shrunk?  Did you create a
filesystem on it?

> The sysadmin was running a vgscan an lvscan. Afterwards he runs 
> 
>     # e2fsck -y -f /dev/ki/postgres
> 
> now the mount on /var/lib/postgres was successful, but all date were gone!
> I found a long list of dir-entries in the lost+found subdir. No file-entries.
> 
> What can we do?

Do you have any output from e2fsck?  It is likely that by running e2fsck
you have removed any chance of further recovery.  If the filesystem was
not resized properly before the LV was shrunk, and then you ran e2fsck
on it, it will have corrupted much of the filesystem data structures
because it was unable to access the larger part of the filesystem.

What is the structure of the filesystem that was in ki/postgres?  Did it
only have a few large DB files, or many smaller files?  What I would
suggest is to look into the directories under lost+found and see if your
database files are located therein.  It may be that the data in them
is corrupt, or that you are missing a lot of files.

If you did not create any filesystems on ki/opt, then it might be
possible to recover some of the data from the rest of the disk.  That
would be quite hard to do, however.

Cheers, Andreas
--
Andreas Dilger
http://www-mddsp.enel.ucalgary.ca/People/adilger/
http://sourceforge.net/projects/ext2resize/

Re: [linux-lvm] Desater Recovery <urgent>

[Andreas Dilger]

On Apr 22, 2002  18:22 +0100, Joe Thornber wrote:
> On Mon, Apr 22, 2002 at 07:14:05PM +0200, Ralf Zerres wrote:
> > no, i guess not.
> > I was using e2fsadm which is serializing the steps. As far as I know ...
> 
> Yes e2fsadm should have resized the filesystem itself, so that's not
> the problem.

Unless e2fsadm doesn't work properly with "- 4,5G"...

Out of curiosity, which resizer did e2fsadm use?  resize2fs or
ext2resize?  By default it would have picked resize2fs if it is
installed.

I don't suppose you recall (or better, have saved) any of the output
from e2fsadm, resize2fs, e2fsck, anything?

Cheers, Andreas
--
Andreas Dilger
http://www-mddsp.enel.ucalgary.ca/People/adilger/
http://sourceforge.net/projects/ext2resize/

Re: [linux-lvm] Desater Recovery <urgent>

[Bas]

> On Apr 22, 2002  18:22 +0100, Joe Thornber wrote:
> > On Mon, Apr 22, 2002 at 07:14:05PM +0200, Ralf Zerres wrote:
> > > no, i guess not.
> > > I was using e2fsadm which is serializing the steps. As far as I know
...
> >
> > Yes e2fsadm should have resized the filesystem itself, so that's not
> > the problem.
>
> Unless e2fsadm doesn't work properly with "- 4,5G"...
>
> Out of curiosity, which resizer did e2fsadm use?  resize2fs or
> ext2resize?  By default it would have picked resize2fs if it is
> installed.
>
> I don't suppose you recall (or better, have saved) any of the output
> >from e2fsadm, resize2fs, e2fsck, anything?
>
> Cheers, Andreas
> --
> Andreas Dilger

Is it possible to extend the lv to the original size and run fsck on it ?
Make a tarball of the available data in advance !

Good luck,
Bas.

Re: [linux-lvm] Desater Recovery <urgent>

[Heinz J . Mauelshagen]

All,

I supported Ralf on the phone and we analyzed, that the ext2 filesystem was
already *very* messy before e2fsadm could be run.


Brief history was:

- fs run as ext3 (heavily) used for weeks

- try to shrink unmounted fs with e2fsadm (using resize2fs) made e2fsadm
  end early because manual fsck was required; this was the point in time
  where a readonly mount and a backup was needed (but didn't happen) in
  order to save as much data as possible

- manual "fsck -fy" was run exposing piles of messages :-( which
  convinces me, that the fs data was messy already afterwards

- e2fsadm was run "successfully" ending up with smaller fs and larger
  LV size which is most likely caused by a resize2fs exit code != 0.


As already mentioned in other mails of this thread: no backups :-(

"Remind people of taking backup, remind people of taking backup, remind..."

Regards,
Heinz    -- The LVM Guy --



On Tue, Apr 23, 2002 at 08:47:40AM +0200, Bas wrote:
> > On Apr 22, 2002  18:22 +0100, Joe Thornber wrote:
> > > On Mon, Apr 22, 2002 at 07:14:05PM +0200, Ralf Zerres wrote:
> > > > no, i guess not.
> > > > I was using e2fsadm which is serializing the steps. As far as I know
> ...
> > >
> > > Yes e2fsadm should have resized the filesystem itself, so that's not
> > > the problem.
> >
> > Unless e2fsadm doesn't work properly with "- 4,5G"...
> >
> > Out of curiosity, which resizer did e2fsadm use?  resize2fs or
> > ext2resize?  By default it would have picked resize2fs if it is
> > installed.
> >
> > I don't suppose you recall (or better, have saved) any of the output
> > >from e2fsadm, resize2fs, e2fsck, anything?
> >
> > Cheers, Andreas
> > --
> > Andreas Dilger
> 
> Is it possible to extend the lv to the original size and run fsck on it ?
> Make a tarball of the available data in advance !
> 
> Good luck,
> Bas.
> 
> 
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html

*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Sistina Software Inc.
Senior Consultant/Developer                       Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
Mauelshagen@Sistina.com                           +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-