[linux-lvm] URgent - please Help - our main system is corrupt, no one can wor k!

[linux-lvm] URgent - please Help - our main system is corrupt, no one can wor k!

[Gesine Schäfer-Reimers]

Dear Sirs,

this is the second mail. I tried it this noon but got no answer so far, but
our whole 
production is down and I really have to fix it.

We use SUSE 7.3 with LVM. I have got /var and /tmp and /opt  an /home and
many 
Partitions without a filesystem for our database on a logical volume group
named quant.

The system was up and running. I added (with YAST2) one more logical volume
to the 
volume group, applied an restarted. The restart ist "strange", since only
parts of the 
systems come up.

For instance the /var directory claims to be mounted but is empty and cannot
be 
mounted, for they are not mounted !! In /procs/ not filesystem is shown as
mountet.
The / filesystem, which is out of the VM is in a read only state, some files
are there 
others are gone.
When I do vgdisplay the VM is shown as available, when I do vgdisplay -D it
is shown 
as unavailable.

I cannot afford any tests, since I do not have a backup of the database data
from today.

It looks for me as if the addition of the new logical volume wrote some
information in a 
place where it does not belong. Can I recover the old state without loosing
information?

With best regards
Gesine Schäfer-Reimers
quant GmbH
++49/40/254078-70

Re: [linux-lvm] URgent - please Help - our main system is corrupt, no one can wor k!

[Goetz Bock]

On Mon, Jun 10 '02 at 15:16, Gesine Sch�fer-Reimers wrote:
> this is the second mail. I tried it this noon but got no answer so far, but
> our whole production is down and I really have to fix it.
did not reache the list (AFAICT)
 
> The system was up and running. I added (with YAST2) one more logical
> volume to the volume group, applied an restarted. The restart ist
> "strange", since only parts of the systems come up.
Most likely it come up in a "something is broken, bplease try to fix it
manualy" state

> For instance the /var directory claims to be mounted but is empty and
> cannot be mounted, for they are not mounted !! In /procs/ not
The mount command takes it's information from /etc/mtap and not from
/proc, so most likely the console message was something like "unable to
run fsck on /dev/bla" please enter root password and try to fix it,
system will reboot after you left the shell.

> [ ... ]
> It looks for me as if the addition of the new logical volume wrote
> some information in a place where it does not belong. Can I recover
> the old state without loosing information?
possible there are backups in /etc/lvmtab.d, but maybe not.

It would realy help, what the system told you when it booted.

A "simple" fix might be to reboot and pray :-/, but this might fu**up
everything.

Unfortunately I've not enough informations to help you further :-(, and
I don't know how stable yast2 is so I can only guess what you might be
able to do to fix anything, but my guesses might be way off.

Cu,
    Goetz.
--
Goetz Bock                                             bock@blacknet.de
IT Consulting                                          Munich   Germany

Re: [linux-lvm] URgent - please Help - our main system is corrupt, no one can wor k!

[Benjamin Scott]

On Mon, 10 Jun 2002, at 3:16pm, Gesine Schäfer-Reimers wrote:
> this is the second mail. I tried it this noon but got no answer so far,
> but our whole production is down and I really have to fix it.
>
> We use SUSE 7.3 with LVM...

  Call SuSE and pay for technical support.  Or call Sistina and pay for
technical support.  Or find a local consultant who knows their stuff, and
pay them for help.  You get the idea.  You need an expert, in a hurry, and
you are going to have to pay for that.

  If you have a mission critical operation that is non-functional, you need
more intense support than a mailing list staffed with volunteers can give
you.  More pointedly, the symptoms you describe indicate a fairly low-level
and general system failure, which will require direct, one-on-one
trouble-shooting -- not something you can get here.

  I do not speak for SuSE, Sistina, the LVM maintainers, my company, or
anyone else but myself.  However, as a professional system administrator, I
know when to pay for help, and that is where you are right now.

-- 
Ben Scott <bscott@ntisys.com>
| The opinions expressed in this message are those of the author and do not |
| necessarily represent the views or policy of any other person, entity or  |
| organization.  All information is provided without warranty of any kind.  |

Re: [linux-lvm] URgent - please Help - our main system is corrupt, no one can wor k!

[Tim]

Ayup.  $250/hour for Sistina support is jack squat compared to the cost
of a real outage.  I've never had an expense check cut faster than for
the services of one Heinz@Sistina :-).  You get what you pay for!

Quoth Benjamin Scott:
> 
>   Call SuSE and pay for technical support.  Or call Sistina and pay for
> technical support.  Or find a local consultant who knows their stuff, and
> pay them for help.  You get the idea.  You need an expert, in a hurry, and
> you are going to have to pay for that.

-- 
    I've finally found the perfect girl,
    I couldn't ask for more;
    She's deaf and dumb and oversexed,
    and owns a liquor store.

Re: [linux-lvm] URgent - please Help - our main system is corrupt, no one can work!

[Bas]

Hi,

You probably already tried this, but just to make sure.
# vgscan
# vgchange -a y

Try to mount the filesystems.

Good luck,
Bas.

----- Original Message -----
From: "Gesine Sch�fer-Reimers" <G.SchaeferReimers@quant-forum.de>
To: <linux-lvm@sistina.com>
Sent: Monday, June 10, 2002 3:16 PM
Subject: [linux-lvm] URgent - please Help - our main system is corrupt, no
one can work!


Dear Sirs,

this is the second mail. I tried it this noon but got no answer so far, but
our whole
production is down and I really have to fix it.

We use SUSE 7.3 with LVM. I have got /var and /tmp and /opt  an /home and
many
Partitions without a filesystem for our database on a logical volume group
named quant.

The system was up and running. I added (with YAST2) one more logical volume
to the
volume group, applied an restarted. The restart ist "strange", since only
parts of the
systems come up.

For instance the /var directory claims to be mounted but is empty and cannot
be
mounted, for they are not mounted !! In /procs/ not filesystem is shown as
mountet.
The / filesystem, which is out of the VM is in a read only state, some files
are there
others are gone.
When I do vgdisplay the VM is shown as available, when I do vgdisplay -D it
is shown
as unavailable.

I cannot afford any tests, since I do not have a backup of the database data
>from today.

It looks for me as if the addition of the new logical volume wrote some
information in a
place where it does not belong. Can I recover the old state without loosing
information?

With best regards
Gesine Sch�fer-Reimers
quant GmbH
++49/40/254078-70

_______________________________________________
linux-lvm mailing list
linux-lvm@sistina.com
http://lists.sistina.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html

Re: [linux-lvm] URgent - please Help - our main system is corrupt, no one can wor k!

[Andreas Dilger]

On Jun 10, 2002  09:50 -0400, Benjamin Scott wrote:
> On Mon, 10 Jun 2002, at 3:16pm, Gesine Sch?fer-Reimers wrote:
> > this is the second mail. I tried it this noon but got no answer so far,
> > but our whole production is down and I really have to fix it.
> >
> > We use SUSE 7.3 with LVM...
> 
>   Call SuSE and pay for technical support.  Or call Sistina and pay for
> technical support.  Or find a local consultant who knows their stuff, and
> pay them for help.  You get the idea.  You need an expert, in a hurry, and
> you are going to have to pay for that.
> 
>   If you have a mission critical operation that is non-functional, you need
> more intense support than a mailing list staffed with volunteers can give
> you.  More pointedly, the symptoms you describe indicate a fairly low-level
> and general system failure, which will require direct, one-on-one
> trouble-shooting -- not something you can get here.
> 
>   I do not speak for SuSE, Sistina, the LVM maintainers, my company, or
> anyone else but myself.  However, as a professional system administrator, I
> know when to pay for help, and that is where you are right now.

And it sounds like they don't have a backup of their "production"
system.  Sounds like some sysadmin will be out of a job soon.

Cheers, Andreas
--
Andreas Dilger
http://www-mddsp.enel.ucalgary.ca/People/adilger/
http://sourceforge.net/projects/ext2resize/

Re: [linux-lvm] URgent - please Help - our main system is corrupt, no one can wor k!

[Goetz Bock]

On Mon, Jun 10 '02 at 10:00, Andreas Dilger wrote:
> And it sounds like they don't have a backup of their "production"
> system.  Sounds like some sysadmin will be out of a job soon.
They just mailed that they don't have a backup of today, so hopefully
they have backup's of yesterday ...

    Goetz.