[linux-lvm] F7 will not boot after running backup w/snapshot

[linux-lvm] F7 will not boot after running backup w/snapshot

[Gerry Reno]

I ran a backup today with a snapshot.  Sometime after the backup 
completed but before I could unmount and lvremove the snapshot the 
system became hung.  No choice but to cold boot.  When the system came 
up it showed a kernel panic on the screen.  Looking back up the screen I 
could see that it could not find a device with UUID blahblah for 
VolGroup00 (which is where the / drive is located) and then VolGroup00 
not found.  So ok, no problem, I just figure that the VolGroup00 is 
inconsistent and I can deal with this in rescue mode.  Wrong.  When I 
boot the F7 rescue from the dvd it says "You don't have any Linux 
partitions..."  What!?  I go into the shell but there is nothing under 
/mnt/sysimage.  So how do I recover from this?  How could a snapshot 
cause all this?

Gerry

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Gerry Reno]

In rescue, I tried running:
===============================
 lvm vgchange -P -ay
  Partial mode. Incomplete volume groups will be activated read-only.
  Couldn't find device with uuid 'blahblah'
  Couldn't find device with uuid 'blahblah'
  Couldn't find device with uuid 'blahblah'
  Couldn't find device with uuid 'blahblah'
  Device /dev/ioerror not found.
  device-mapper reload ioctl failed: invalid argument
  device-mapper reload ioctl failed: invalid argument
  Couldn't find device with uuid 'blahblah'
  Couldn't find device with uuid 'blahblah'
2 logical volume(s) in volume group "VolGroup00" now active.

lvm vgdisplay -v VolGroup00
Segmentation fault
===============================

In fact every 'lvm' command you try to run after that now gives a 
Segmentation fault.

What I want to do is to try:
lvm vgreduce VolGroup00 --removemissing

which I think might make the volume consistent again but I cannot even 
run the command without segfault.
Any ideas on how to get rid of the segfaulting?

Gerry

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Gerry Reno]

So I got the vgreduce to work:

reboot into rescue and issue it as the very first command:
lvm vgreduce VolGroup00 --removemissing
lvm vgdisplay -v VolGroup00
<no error this time>
but I do see that my LogVol00 is missing (root disk) (this is what got 
corrupted)
lvm lvchange -ay /dev/VolGroup00/LogVol01
<no errors, good>
vgdisplay now shows LogVol01 as available, yea!
so now what about the missing LogVol00?
what I want to try is replacing it at exactly the same extents with a 
new LogVol00
lvm lvcreate --extents 284 --name=/dev/VolGroup00/LogVol00 VolGroup00
<it succeeds>
vgdisplay now shows both log vols with exactly the same size as 
originally before the problem.
vgchange -ay
<succeeds and activates VG and both LV)
mount -t ext3 /dev/mapper/VolGroup00-LogVol00  /mnt/sysimage
<fails: invalid argument>
so now I think that something needs to be done with uuid but I'm not 
familiar with how to do this.
even if I have no access to data on the logvol, if I can mount it I can 
restore it.
Anyone know how to make the new LogVol00 mount?

Gerry

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Gerry Reno]

I ran e2fsck against the new logvol and it found the filesystem and the 
journal but
the filesystem was corrupted with hundreds of errors.  At this point I 
just reformatted
the device and now I'm loading the backup onto it.

How do I handle the LVM metadata?  The restored metadata will not be 
correct.  Or does
LVM have a way to figure this out?


Gerry

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Larry Dickson]

[-- Attachment #1: Type: text/plain, Size: 1156 bytes --]

Gerry, for the benefit of the rest of us who are depending on snapshot -

Was there much writing and/or reading going on between the time your backup
was made, and the time of the hang?

I know that snapshot works by putting an extra load on every write to the
origin volume. Is there any reason, known to knowledgeable LVM designers on
this list, why this should lead to "hundreds of errors" when a system hang
takes place? I assume stuff got caught in a half-committed state, but
_hundreds_... ???

Thanks,

Larry

On 4/30/08, Gerry Reno <greno@verizon.net> wrote:
>
> I ran e2fsck against the new logvol and it found the filesystem and the
> journal but
> the filesystem was corrupted with hundreds of errors.  At this point I
> just reformatted
> the device and now I'm loading the backup onto it.
>
> How do I handle the LVM metadata?  The restored metadata will not be
> correct.  Or does
> LVM have a way to figure this out?
>
>
> Gerry
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>

[-- Attachment #2: Type: text/html, Size: 2008 bytes --]

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Gerry Reno]

Larry,
  There was only maybe ten minutes between the time the backup finished 
and the hang.  The system was very quiet at the time.  The backups run 
overnight when there is almost no system activity.  I had only recently 
begun using snapshots and I was surprised to see a machine get into this 
state while running a backup w/snapshot.  I use a 256M ramdisk as the 
snapshot vol.  Prior to introducing snapshots this machine had run 
without any problems during backups.  I could not tell what exactly 
caused the hang.  There was no indication that the snapshot had run out 
of space and I doubt that it did.  The issue for me is that no matter 
what happened to cause a hang that LVM should never get into a state 
where it cannot start the system.  The backup had completed backing up 
the root filesystem (which was last) when the hang occurred.  What that 
did was to leave the volume that contained the root filesystem in an 
inconsistent state at reboot since the snapshot logvol was not removed.  
LVM should have recognized that the inconsistency was due to a snapshot 
logvol and corrected the problem at boot and brought up the VolGroup 
sans snapshot device since it's worthless at that point anyway.  My 
concern is that there is a bug somewhere in the snapshot mechanism.  
I've seen other people report what they said were corruption problems 
resulting from snapshots.

Regards,
Gerry


Larry Dickson wrote:
> Gerry, for the benefit of the rest of us who are depending on snapshot -
>  
> Was there much writing and/or reading going on between the time your 
> backup was made, and the time of the hang?
>  
> I know that snapshot works by putting an extra load on every write to 
> the origin volume. Is there any reason, known to knowledgeable LVM 
> designers on this list, why this should lead to "hundreds of errors" 
> when a system hang takes place? I assume stuff got caught in a 
> half-committed state, but _hundreds_... ???
>  
> Thanks,
>  
> Larry
>  

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Gerry Reno]

Just to finish documenting the recovery from the corruption:

I was able to fully restore the root logvol and then I rebooted the 
system.  The system came up fine without having to make any special 
adjustments to LVM metadata.  It figured it out.  The only thing I 
noticed was a blizzard of SELinux avc denials.  So I ran a relabel and 
that seemed to cure those.  I did a 'cat /etc/lvm/backup/VolGroup00' and 
took a look at what was in there.  It was in the state that included the 
snapshot0 and my snapshot logvol.  I then did a vgdisplay -v and LVM 
rewrote the meta backup and when I checked it again the snapshot0 and my 
snapshot were gone.  So the system appears to be recovered and working 
again.


Gerry

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Charles Marcus]

On 4/30/2008 1:23 PM, Gerry Reno wrote:
> My concern is that there is a bug somewhere in the snapshot
> mechanism. I've seen other people report what they said were
> corruption problems resulting from snapshots.

This is quite scary to someone like me who is new to LVM and using 
snapshots for backups (whether automated or not)...

I'd really like to hear from some of the devs on this, and what they 
think may have caused it...

Incidentally... this is the first time I've heard of someone using a 
Ramdisk as the snapshot volume... any chance that could have been 
involved somehow?

-- 

Best regards,

Charles

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Gerry Reno]

Charles,
  Here's a snapshot ramdisk reference:
    
http://linuxsoftware.co.nz/blog/2008/03/11/lvm-snapshot-with-no-free-diskspace
   
  The ramdisk has been working without problem.  I tested it quite a bit 
before putting it into production.  But I don't know LVM internals so I 
cannot say whether it could have had anything to do with the hang that 
occurred.

Regards,
Gerry


Charles Marcus wrote:
> On 4/30/2008 1:23 PM, Gerry Reno wrote:
>> My concern is that there is a bug somewhere in the snapshot
>> mechanism. I've seen other people report what they said were
>> corruption problems resulting from snapshots.
>
> This is quite scary to someone like me who is new to LVM and using 
> snapshots for backups (whether automated or not)...
>
> I'd really like to hear from some of the devs on this, and what they 
> think may have caused it...
>
> Incidentally... this is the first time I've heard of someone using a 
> Ramdisk as the snapshot volume... any chance that could have been 
> involved somehow?
>

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Milan Broz]

Gerry Reno wrote:
> Charles,
>   Here's a snapshot ramdisk reference:
>     
> http://linuxsoftware.co.nz/blog/2008/03/11/lvm-snapshot-with-no-free-diskspace
>    
>   The ramdisk has been working without problem.  I tested it quite a bit 
> before putting it into production.  But I don't know LVM internals so I 
> cannot say whether it could have had anything to do with the hang that 
> occurred.

Please, do *not* use this configuration, it is very dangerous!

Note that you extend Volume Group on you physical disk with another PV
in volatile memory.

Now imagine that your system crash (oh, wait, it happened already!)
After reboot you have no ramdisk, and the Volume Group is incomplete
(because you didn't removed PV on ramdisk).

If there is only snapshot, you can recover it, if some operation placed
here part of another volume, you will lose data.

If there is root volume, system will not boot. Period.

Milan
--
mbroz@redhat.com

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Gerry Reno]

Milan Broz wrote:
> ... After reboot you have no ramdisk, and the Volume Group is incomplete
> (because you didn't removed PV on ramdisk).
>   
And this is no different than if snapshot was on any other device such 
as esata-hdd, usb-hdd or usb stick.  LVM should handle this.  If the 
snapshot device goes away, then just vgreduce it on the reboot.
Trying to retrofit snapshot into existing systems is far easier if you 
can use ramdisk, esata-hdd, usb-hdd, usb-stick because most systems have 
allocated all space and rather than struggling through trying to compact 
and reduce VG, LV, PV, partition, filesystem to gain space it is much 
easier to use other devices.  And again this is something that LVM 
should be able to handle.
> If there is only snapshot, you can recover it, if some operation placed
> here part of another volume, you will lose data.
>   
part of another volume placed where?  on the snapshot?  I don't 
understand this part.


Regards,
Gerry

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[malahal]

Gerry Reno [greno@verizon.net] wrote:
> Milan Broz wrote:
>> ... After reboot you have no ramdisk, and the Volume Group is incomplete
>> (because you didn't removed PV on ramdisk).
>>   
> And this is no different than if snapshot was on any other device such as 
> esata-hdd, usb-hdd or usb stick.  LVM should handle this.  If the snapshot 
> device goes away, then just vgreduce it on the reboot.
> Trying to retrofit snapshot into existing systems is far easier if you can 
> use ramdisk, esata-hdd, usb-hdd, usb-stick because most systems have 
> allocated all space and rather than struggling through trying to compact 
> and reduce VG, LV, PV, partition, filesystem to gain space it is much 
> easier to use other devices.  And again this is something that LVM should 
> be able to handle.
>> If there is only snapshot, you can recover it, if some operation placed
>> here part of another volume, you will lose data.
>>   
> part of another volume placed where?  on the snapshot?  I don't understand 
> this part.

LVM is not going to distinguish between your /dev/ram1 and other PV's.
So, if you happen to do any 'lvextend/pvmove/lvcreate/anyother'
operations, LVM may place such data on your ram disk. The only operation
that you do after placing /dev/ram1 under LVM is snapshot creation, I
don't see big problem though. You should be very careful to delete the
ram disk PV before you do any operation that involves allocating space.

--Malahal.

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Mikulas Patocka]

On Thu, 1 May 2008, malahal@us.ibm.com wrote:

> Gerry Reno [greno@verizon.net] wrote:
>> Milan Broz wrote:
>>> ... After reboot you have no ramdisk, and the Volume Group is incomplete
>>> (because you didn't removed PV on ramdisk).
>>>
>> And this is no different than if snapshot was on any other device such as
>> esata-hdd, usb-hdd or usb stick.  LVM should handle this.  If the snapshot
>> device goes away, then just vgreduce it on the reboot.

If you mount the origin device with missing snapshot, you destroy the 
snapshot (even if you don't touch it). The snapshot can no longer be 
repaired.

So it is safer to not activate device in this case then destroy data.

Imagine, for example, you have origin and snapshot, you reconfigure disks 
in some weird way that the snapshot disk is inaccessible, you boot, and 
the system automatically starts without the snapshot. And you lose any 
data that you stored on that snapshot.

>> Trying to retrofit snapshot into existing systems is far easier if you can
>> use ramdisk, esata-hdd, usb-hdd, usb-stick because most systems have
>> allocated all space and rather than struggling through trying to compact
>> and reduce VG, LV, PV, partition, filesystem to gain space it is much
>> easier to use other devices.  And again this is something that LVM should
>> be able to handle.

You can with dmsetup (but it has deadlocks). Maybe someone could write 
non-deadlocky snapshot-managing tool that wouldn't depend on lvm vgs, pvs 
and lvs.

Mikulas

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Gerry Reno]

Mikulas Patocka wrote:
> ... If you mount the origin device with missing snapshot, you destroy 
> the snapshot (even if you don't touch it). The snapshot can no longer 
> be repaired.
>
> So it is safer to not activate device in this case then destroy data.
Why? What value is the old snapshot at this point? You just had a system 
reboot in the middle of a snapshotted backup so all you need to do is 
get the system up, redo another snapshot and retake your backup. I'm not 
interested in the old snapshot.
>
> Imagine, for example, you have origin and snapshot, you reconfigure 
> disks in some weird way that the snapshot disk is inaccessible, you 
> boot, and the system automatically starts without the snapshot. And 
> you lose any data that you stored on that snapshot.
What is on the old snapshot at this point is probably indeterminate anyway.
> You can with dmsetup (but it has deadlocks). Maybe someone could write 
> non-deadlocky snapshot-managing tool that wouldn't depend on lvm vgs, 
> pvs and lvs.
>
> Mikulas
Have not used dmsetup. If it has deadlocks, I don't think I want to use it.


Gerry

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Charles Marcus]

On 5/1/2008 2:15 PM, Gerry Reno wrote:
> Why? What value is the old snapshot at this point?

? It holds all of the changes to your original (snapshotted) volume, so 
I imagine it would be pretty important to most people?

I'm not really understanding all the inner-workings of LVM, but now I 
can see why using a ram disk for an LVM snapshot might not be a very 
good idea for this one reason - it won't survive a reboot...

> You just had a system reboot in the middle of a snapshotted backup so
> all you need to do is get the system up, redo another snapshot and
> retake your backup. I'm not interested in the old snapshot.

I would be... but thats just me...

-- 

Best regards,

Charles

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Gerry Reno]

Charles Marcus wrote:
> On 5/1/2008 2:15 PM, Gerry Reno wrote:
>> Why? What value is the old snapshot at this point?
>
> ? It holds 
(a copy of)
> all of the changes to your original (snapshotted) volume, so I imagine 
> it would be pretty important to most people?
>
> I'm not really understanding all the inner-workings of LVM, but now I 
> can see why using a ram disk for an LVM snapshot might not be a very 
> good idea for this one reason - it won't survive a reboot...
>
>> You just had a system reboot in the middle of a snapshotted backup so
>> all you need to do is get the system up, redo another snapshot and
>> retake your backup. I'm not interested in the old snapshot.
>
> I would be... but thats just me...
It's like yesterday's newspaper.  Why do you need to keep it?


Gerry

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Charles Marcus]

On 5/1/2008 3:37 PM, Gerry Reno wrote:
>>> You just had a system reboot in the middle of a snapshotted backup so
>>> all you need to do is get the system up, redo another snapshot and
>>> retake your backup. I'm not interested in the old snapshot.

>> I would be... but thats just me...

> It's like yesterday's newspaper.  Why do you need to keep it?

A wild guess... because the changes haven't been synced back with the 
original yet (which happens when the snapshot is released)?

-- 

Best regards,

Charles

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Gerry Reno]

Charles Marcus wrote:
> On 5/1/2008 3:37 PM, Gerry Reno wrote:
>>>> You just had a system reboot in the middle of a snapshotted backup so
>>>> all you need to do is get the system up, redo another snapshot and
>>>> retake your backup. I'm not interested in the old snapshot.
>
>>> I would be... but thats just me...
>
>> It's like yesterday's newspaper.  Why do you need to keep it?
>
> A wild guess... because the changes haven't been synced back with the 
> original yet (which happens when the snapshot is released)?
My understanding is that the snapshot is a copy-on-write (COW) to the 
original volume.  If that is not the case, then I guess things must have 
changed and I have some old information.

Gerry

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Charles Marcus]

>>> It's like yesterday's newspaper.  Why do you need to keep it?

>> A wild guess... because the changes haven't been synced back with the 
>> original yet (which happens when the snapshot is released)?

> My understanding is that the snapshot is a copy-on-write (COW) to the 
> original volume.  If that is not the case, then I guess things must have 
> changed and I have some old information.

You could be right... like I said, I'm not totally understanding how it 
all works yet...

It would be nice if someone who knows could clear this up...

-- 

Best regards,

Charles

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Gerry Reno]

At least in the case where the snapshot is read-only (LVM1 default, LVM2 
by config?), if the snapshot is lost, invalid, not removed from VG prior 
to reboot, when LVM comes back up it should see this and immediately 
know that it can just vgreduce VG --removemissing for the old snapshot.  
In the case of rw (no LVM1, LVM2 default), it should be a user choice 
and LVM should prompt the user at boot as to whether to remove this old 
snapshot so it can attempt to activate the VG.  Unless the user knows 
that there were non-backup related lvm mods written during the snapshot 
(eg: pvmove) then the user will just answer yes and the system should 
boot.  This is how LVM should operate in this scenario.

Gerry

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Alasdair G Kergon]

On Thu, May 01, 2008 at 04:21:00PM -0400, Gerry Reno wrote:
> At least in the case where the snapshot is read-only (LVM1 default, LVM2 
> by config?), if the snapshot is lost, invalid, not removed from VG prior 
> to reboot, when LVM comes back up it should see this and immediately 
> know that it can just vgreduce VG --removemissing for the old snapshot.  
> In the case of rw (no LVM1, LVM2 default), it should be a user choice 
> and LVM should prompt the user at boot as to whether to remove this old 
> snapshot so it can attempt to activate the VG.  Unless the user knows 
> that there were non-backup related lvm mods written during the snapshot 
> (eg: pvmove) then the user will just answer yes and the system should 
> boot.  This is how LVM should operate in this scenario.
 
If you want your system to do that, update your initrd/initscripts
accordingly to run the appropriate lvm2 commands to do that!

Alasdair
-- 
agk@redhat.com

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Gerry Reno]

Alasdair G Kergon wrote:
> On Thu, May 01, 2008 at 04:21:00PM -0400, Gerry Reno wrote:
>   
>> At least in the case where the snapshot is read-only (LVM1 default, LVM2 
>> by config?), if the snapshot is lost, invalid, not removed from VG prior 
>> to reboot, when LVM comes back up it should see this and immediately 
>> know that it can just vgreduce VG --removemissing for the old snapshot.  
>> In the case of rw (no LVM1, LVM2 default), it should be a user choice 
>> and LVM should prompt the user at boot as to whether to remove this old 
>> snapshot so it can attempt to activate the VG.  Unless the user knows 
>> that there were non-backup related lvm mods written during the snapshot 
>> (eg: pvmove) then the user will just answer yes and the system should 
>> boot.  This is how LVM should operate in this scenario.
>>     
>  
> If you want your system to do that, update your initrd/initscripts
> accordingly to run the appropriate lvm2 commands to do that!
>
> Alasdair
>   

I can certainly do that. But I think this applies in the general case 
and should be included as standard behavior in LVM. This is a much more 
robust means of dealing with this scenario that having LVM just refuse 
to mount the volume when the only issue is a bad snapshot.


Gerry

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Charles Marcus]

On 5/1/2008 4:30 PM, Gerry Reno wrote:
>>> At least in the case where the snapshot is read-only (LVM1 default, 
>>> LVM2 by config?), if the snapshot is lost, invalid, not removed from 
>>> VG prior to reboot, when LVM comes back up it should see this and 
>>> immediately know that it can just vgreduce VG --removemissing for the 
>>> old snapshot.  In the case of rw (no LVM1, LVM2 default), it should 
>>> be a user choice and LVM should prompt the user at boot as to whether 
>>> to remove this old snapshot so it can attempt to activate the VG.  
>>> Unless the user knows that there were non-backup related lvm mods 
>>> written during the snapshot (eg: pvmove) then the user will just 
>>> answer yes and the system should boot.  This is how LVM should 
>>> operate in this scenario.

>> If you want your system to do that, update your initrd/initscripts
>> accordingly to run the appropriate lvm2 commands to do that!

> I can certainly do that. But I think this applies in the general case 
> and should be included as standard behavior in LVM. This is a much more 
> robust means of dealing with this scenario that having LVM just refuse 
> to mount the volume when the only issue is a bad snapshot.

Question...

In Gerry's scenario here, if the snapshot volume had NOT been on a ram 
disk, would he have had the problem he had or not?

-- 

Best regards,

Charles

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Gerry Reno]

Charles Marcus wrote:
> On 5/1/2008 4:30 PM, Gerry Reno wrote:
>>>> At least in the case where the snapshot is read-only (LVM1 default, 
>>>> LVM2 by config?), if the snapshot is lost, invalid, not removed 
>>>> from VG prior to reboot, when LVM comes back up it should see this 
>>>> and immediately know that it can just vgreduce VG --removemissing 
>>>> for the old snapshot.  In the case of rw (no LVM1, LVM2 default), 
>>>> it should be a user choice and LVM should prompt the user at boot 
>>>> as to whether to remove this old snapshot so it can attempt to 
>>>> activate the VG.  Unless the user knows that there were non-backup 
>>>> related lvm mods written during the snapshot (eg: pvmove) then the 
>>>> user will just answer yes and the system should boot.  This is how 
>>>> LVM should operate in this scenario.
>
>>> If you want your system to do that, update your initrd/initscripts
>>> accordingly to run the appropriate lvm2 commands to do that!
>
>> I can certainly do that. But I think this applies in the general case 
>> and should be included as standard behavior in LVM. This is a much 
>> more robust means of dealing with this scenario that having LVM just 
>> refuse to mount the volume when the only issue is a bad snapshot.
>
> Question...
>
> In Gerry's scenario here, if the snapshot volume had NOT been on a ram 
> disk, would he have had the problem he had or not?
>
Charles,
  Without knowing the exact cause of the hang there is no way to know if 
the ramdisk was at issue.  I doubt it myself, I ran a whole series of 
backups last night using the ramdisk snapshot and everything went fine.

Gerry

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Charles Marcus]

On 5/1/2008 4:36 PM, Gerry Reno wrote:
>  Without knowing the exact cause of the hang there is no way to know if 
> the ramdisk was at issue.  I doubt it myself, I ran a whole series of 
> backups last night using the ramdisk snapshot and everything went fine.

Actually, I'm not talking about the hang as much as I am the problem you 
had recovering from it...

My questions is, had the snapshot volume been on a regular hard drive, 
would you have had the same problems recovering from the situation.

-- 

Best regards,

Charles

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Gerry Reno]

Charles Marcus wrote:
> On 5/1/2008 4:36 PM, Gerry Reno wrote:
>>  Without knowing the exact cause of the hang there is no way to know 
>> if the ramdisk was at issue.  I doubt it myself, I ran a whole series 
>> of backups last night using the ramdisk snapshot and everything went 
>> fine.
>
> Actually, I'm not talking about the hang as much as I am the problem 
> you had recovering from it...
>
> My questions is, had the snapshot volume been on a regular hard drive, 
> would you have had the same problems recovering from the situation.
>
I don't think anyone could possibly say. 

There was corruption involved and that could have occurred for lots of 
reasons.  My thought is that even if the ramdisk were not present on the 
next boot there should be no reason for that to cause corruption.  There 
were no lvm mods made to the rw snapshot which means in essence it was 
read-only.  The VG did not mount and nothing wrote to any filesystem on 
the VG which means the corruption is probably totally unrelated or is a 
bug.  LVM should be robust enough to fully recover from this without any 
corruption.  It just should have discarded the old snapshot.  The 
original volume should not be affected.  But as I said, I suspect there 
may be a bug.  Probably some scenario is not being handled in the code.


Gerry

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Alasdair G Kergon]

On Thu, May 01, 2008 at 04:57:23PM -0400, Gerry Reno wrote:
> LVM should be robust enough to fully recover from this without any 
> corruption.  It just should have discarded the old snapshot.  The 
> original volume should not be affected.  But as I said, I suspect there 
> may be a bug.  Probably some scenario is not being handled in the code.
 
Once again, lvm2 provides the distribution or sysadmin with the facilities to
handle these scenarios.  It's up to your initrd and/or initscripts whether or
not and how to use those facilities.

Alasdair
-- 
agk@redhat.com

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Gerry Reno]

Alasdair G Kergon wrote:
> It's up to your initrd and/or initscripts whether or
> not and how to use those facilities.
>
>   
Alasdair,
  Is it possible to define an init script that would run early enough 
(before LVM tries to activate the volumes) so that I could vgreduce the 
old snapshot?
  Or does this require a new initrd?

Gerry

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Gerry Reno]

Gerry Reno wrote:
> Alasdair G Kergon wrote:
>> It's up to your initrd and/or initscripts whether or
>> not and how to use those facilities.
>>
>>   
> Alasdair,
>  Is it possible to define an init script that would run early enough 
> (before LVM tries to activate the volumes) so that I could vgreduce 
> the old snapshot?
>  Or does this require a new initrd?
>
I started looking at this and then found that Fedora appears to have the 
device mapper and snapshot support as modules:

/lib/modules/2.6.20-1.2952.fc6xen/kernel/drivers/md/dm-snapshot.ko
/lib/modules/2.6.20-1.2952.fc6xen/kernel/drivers/md/dm-mod.ko

So what is going to happen if you have a snapshot of root partition 
active and then a reboot happens (sound familiar?).  The kernel on 
restart can no longer read device mapper or snapshot module or root 
partition and it will panic and who knows what else will happen.  
Corruption maybe???

Gerry

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Stuart D. Gathman]

On Thu, 1 May 2008, Gerry Reno wrote:

> I started looking at this and then found that Fedora appears to have the 
> device mapper and snapshot support as modules:
> 
> /lib/modules/2.6.20-1.2952.fc6xen/kernel/drivers/md/dm-snapshot.ko
> /lib/modules/2.6.20-1.2952.fc6xen/kernel/drivers/md/dm-mod.ko
> 
> So what is going to happen if you have a snapshot of root partition 
> active and then a reboot happens (sound familiar?).  The kernel on 
> restart can no longer read device mapper or snapshot module or root 
> partition and it will panic and who knows what else will happen.  
> Corruption maybe???

The modules are copied to initrd by mkinitrd, and loaded from there.  They
are treated the same as device drivers for the disk containing
the root filesystem.   Mkinitrd can also copy md drivers to initrd
so that root can be on a software raid partition.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Stuart D. Gathman]

ZZZZOn Thu, 1 May 2008, Charles Marcus wrote:

> In Gerry's scenario here, if the snapshot volume had NOT been on a ram 
> disk, would he have had the problem he had or not?

Here is the snip from Fedora/RHEL for activating the VG containing
the root filesystem (note: kernel modules and lvm command are contained in
initrd filesystem, and do not depend on /lib/modules/...):

...
echo "Loading dm-mod.ko module"
insmod /lib/dm-mod.ko
echo "Loading dm-mirror.ko module"
insmod /lib/dm-mirror.ko
echo "Loading dm-zero.ko module"
insmod /lib/dm-zero.ko
echo "Loading dm-snapshot.ko module"
insmod /lib/dm-snapshot.ko
echo Making device-mapper control node
mkdmnod
mkblkdevs
echo Scanning logical volumes
lvm vgscan --ignorelockingfailure
echo Activating logical volumes
lvm vgchange -ay --ignorelockingfailure  rootvg
...

This will fail if any of the physical volumes are missing.  More logic
in initrd is required to boot with missing PVs.

So yes, a ram disk is guaranteed to be missing on reboot, and thus to 
fail in the RH/Fedora distros.  There is no problem taking snapshots
of the root fs otherwise.  If there is a robust way to reduce 
missing PVs automagically in the above script from initrd, Fedora could
use the contribution.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Gerry Reno]

[-- Attachment #1: Type: text/plain, Size: 2154 bytes --]

Stuart D. Gathman wrote:
> ZZZZOn Thu, 1 May 2008, Charles Marcus wrote:
>
>   
>> In Gerry's scenario here, if the snapshot volume had NOT been on a ram 
>> disk, would he have had the problem he had or not?
>>     
>
> Here is the snip from Fedora/RHEL for activating the VG containing
> the root filesystem (note: kernel modules and lvm command are contained in
> initrd filesystem, and do not depend on /lib/modules/...):
>
> ...
> echo "Loading dm-mod.ko module"
> insmod /lib/dm-mod.ko
> echo "Loading dm-mirror.ko module"
> insmod /lib/dm-mirror.ko
> echo "Loading dm-zero.ko module"
> insmod /lib/dm-zero.ko
> echo "Loading dm-snapshot.ko module"
> insmod /lib/dm-snapshot.ko
> echo Making device-mapper control node
> mkdmnod
> mkblkdevs
> echo Scanning logical volumes
> lvm vgscan --ignorelockingfailure
>   
So right ^^^here^^^ is where we need to test if old snapshot was 
read-only or read-write (how to do that? grep vgdisplay -v for some value?)
So maybe something like this?:

set removemissing flag false
if old snapshot is the only PV missing:
--if old snapshot was read-write:
------stop boot sequence
------show user some info about old snapshot
------prompt whether to remove the old snapshot from the VG so it can be 
activated
------if response yes:
----------set removemissing flag true
------else
----------alert user that manual action is required to correct problem 
related to old snapshot before VG can be activated
----------abort boot
--elsif old snapshot was read-only:
------set removemissing flag true
if removemissing flag true:
----vgreduce --removemissing VG PV(old snapshot)

Gerry

> echo Activating logical volumes
> lvm vgchange -ay --ignorelockingfailure  rootvg
> ...
>
> This will fail if any of the physical volumes are missing.  More logic
> in initrd is required to boot with missing PVs.
>
> So yes, a ram disk is guaranteed to be missing on reboot, and thus to 
> fail in the RH/Fedora distros.  There is no problem taking snapshots
> of the root fs otherwise.  If there is a robust way to reduce 
> missing PVs automagically in the above script from initrd, Fedora could
> use the contribution.
>
>   


[-- Attachment #2: Type: text/html, Size: 2729 bytes --]

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Marek Podmaka]

Hi Gerry,

Thursday, May 1, 2008, 22:21:00, Gerry Reno wrote:

> At least in the case where the snapshot is read-only (LVM1 default,
> LVM2 by config?), if the snapshot is lost, invalid, not removed from
> VG prior to reboot, when LVM comes back up it should see this and
> immediately know that it can just vgreduce VG --removemissing for
> the old snapshot. In the case of rw (no LVM1, LVM2 default), it
> should be a user choice and LVM should prompt the user at boot as to
> whether to remove this old snapshot so it can attempt to activate
> the VG. Unless the user knows that there were non-backup related lvm
> mods written during the snapshot (eg: pvmove) then the user will
> just answer yes and the system should boot. This is how LVM should
> operate in this scenario.

I don't think that automatically making changes to the VG is a good
idea. Imagine that you have a snapshot on disk mapped from SAN storage
array and that becomes temporary unavailable (SAN switch failure or
whatever). It would be a bad idea to remove this PV from VG just
because it is TEMPORARY unavailable. And how do you distinguish
between ram disk and normal disk and if it is totally gone or it is
only temp issue? Also if real HDD fails, you don't want to vgreduce
it, you want to replace it, vgcfgrestore it and sync it (in case you
are using mirroring). Also imagine that you can have also real data on
the same PV, not only snapshot.

What would help is the ability to activate the VG also with missing
PVs, I'm not sure if that is possible in linux LVM.



-- 
  bYE, Marki

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Gerry Reno]

Marek Podmaka wrote:
> Hi Gerry,
>
> Thursday, May 1, 2008, 22:21:00, Gerry Reno wrote:
>
>   
>> At least in the case where the snapshot is read-only (LVM1 default,
>> LVM2 by config?), if the snapshot is lost, invalid, not removed from
>> VG prior to reboot, when LVM comes back up it should see this and
>> immediately know that it can just vgreduce VG --removemissing for
>> the old snapshot. In the case of rw (no LVM1, LVM2 default), it
>> should be a user choice and LVM should prompt the user at boot as to
>> whether to remove this old snapshot so it can attempt to activate
>> the VG. Unless the user knows that there were non-backup related lvm
>> mods written during the snapshot (eg: pvmove) then the user will
>> just answer yes and the system should boot. This is how LVM should
>> operate in this scenario.
>>     
>
> I don't think that automatically making changes to the VG is a good
> idea. Imagine that you have a snapshot on disk mapped from SAN storage
> array and that becomes temporary unavailable (SAN switch failure or
> whatever). It would be a bad idea to remove this PV from VG just
> because it is TEMPORARY unavailable. 
This was part of my point about the ramdisk, in that other types of 
devices can also disappear not just ramdisk.
Once the device with the snapshot disappears, I'm not sure I would trust 
that snapshot even if the device comes back online.
The backup process would try to keep running and it would probably get 
all kinds of errors at this point so to me it
makes sense to consider that snapshot invalid from that standpoint of 
the backup. Now if it was a read-write snapshot
and you've used some lvm commands to pvmove something onto the snapshot 
then that is why I suggest that the user
be able to make decision at reboot about whether to remove the PV. If 
they answer 'no' to the removal then the boot will
stop and they will have to perform some recovery through the rescue mode 
to see if they can recover some data from
the device before allowing the PV to be removed from the VG so the 
system can boot.

> And how do you distinguish
> between ram disk and normal disk and if it is totally gone or it is
> only temp issue? Also if real HDD fails, you don't want to vgreduce
> it, you want to replace it, vgcfgrestore it and sync it (in case you
> are using mirroring). Also imagine that you can have also real data on
> the same PV, not only snapshot.
>   
You don't vgreduce the entire hdd. Only for the PV that contained the 
snapshot.
If you used an entire hdd to be the snapshot then you can restore it. 
Recover data
in the case of read-write and then vgreduce it from the VG so system can 
boot.

> What would help is the ability to activate the VG also with missing
> PVs, 
You cannot activate a VG with missing PV's because at some point then 
LVM would attempt to write
on one of the missing PV and then you get panic.


Regards,
Gerry


> I'm not sure if that is possible in linux LVM.
>
>
>
>   

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Charles Marcus]

On 5/2/2008, Gerry Reno (greno@verizon.net) wrote:
> This was part of my point about the ramdisk, in that other types of 
> devices can also disappear not just ramdisk.

Sure - but a ramdisk is *much* more volatile than a hard drive... and 
data on a hard drive will almost *always* survive a reboot, where data 
on a ramdisk will *never* survive a reboot.

Personally, for me, the dangers of using a ramdisk as a snapshot volume 
*far* outweigh any questionable benefits...

-- 

Best regards,

Charles

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Larry Dickson]

[-- Attachment #1: Type: text/plain, Size: 3745 bytes --]

> Once the device with the snapshot disappears, I'm not sure I would trust
> that snapshot even if the device comes back online.

The COW operation implies you can trust that snapshot if, and only if,
nothing was written to the origin volume between when the snapshot
disappeared and when it came back on line. Perhaps a "Has been written" flag
could be maintained for the origin volume starting at reboot.

Larry Dickson

On 5/2/08, Gerry Reno <greno@verizon.net> wrote:
>
> Marek Podmaka wrote:
>
>> Hi Gerry,
>>
>> Thursday, May 1, 2008, 22:21:00, Gerry Reno wrote:
>>
>>
>>
>>> At least in the case where the snapshot is read-only (LVM1 default,
>>> LVM2 by config?), if the snapshot is lost, invalid, not removed from
>>> VG prior to reboot, when LVM comes back up it should see this and
>>> immediately know that it can just vgreduce VG --removemissing for
>>> the old snapshot. In the case of rw (no LVM1, LVM2 default), it
>>> should be a user choice and LVM should prompt the user at boot as to
>>> whether to remove this old snapshot so it can attempt to activate
>>> the VG. Unless the user knows that there were non-backup related lvm
>>> mods written during the snapshot (eg: pvmove) then the user will
>>> just answer yes and the system should boot. This is how LVM should
>>> operate in this scenario.
>>>
>>>
>>
>> I don't think that automatically making changes to the VG is a good
>> idea. Imagine that you have a snapshot on disk mapped from SAN storage
>> array and that becomes temporary unavailable (SAN switch failure or
>> whatever). It would be a bad idea to remove this PV from VG just
>> because it is TEMPORARY unavailable.
>>
> This was part of my point about the ramdisk, in that other types of devices
> can also disappear not just ramdisk.
> Once the device with the snapshot disappears, I'm not sure I would trust
> that snapshot even if the device comes back online.
> The backup process would try to keep running and it would probably get all
> kinds of errors at this point so to me it
> makes sense to consider that snapshot invalid from that standpoint of the
> backup. Now if it was a read-write snapshot
> and you've used some lvm commands to pvmove something onto the snapshot
> then that is why I suggest that the user
> be able to make decision at reboot about whether to remove the PV. If they
> answer 'no' to the removal then the boot will
> stop and they will have to perform some recovery through the rescue mode to
> see if they can recover some data from
> the device before allowing the PV to be removed from the VG so the system
> can boot.
>
> And how do you distinguish
>> between ram disk and normal disk and if it is totally gone or it is
>> only temp issue? Also if real HDD fails, you don't want to vgreduce
>> it, you want to replace it, vgcfgrestore it and sync it (in case you
>> are using mirroring). Also imagine that you can have also real data on
>> the same PV, not only snapshot.
>>
>>
> You don't vgreduce the entire hdd. Only for the PV that contained the
> snapshot.
> If you used an entire hdd to be the snapshot then you can restore it.
> Recover data
> in the case of read-write and then vgreduce it from the VG so system can
> boot.
>
> What would help is the ability to activate the VG also with missing
>> PVs,
>>
> You cannot activate a VG with missing PV's because at some point then LVM
> would attempt to write
> on one of the missing PV and then you get panic.
>
>
> Regards,
> Gerry
>
>
> I'm not sure if that is possible in linux LVM.
>>
>>
>>
>>
>>
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>

[-- Attachment #2: Type: text/html, Size: 5341 bytes --]

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Marek Podmaka]

Hello,

Friday, May 2, 2008, 16:00:13, Gerry Reno wrote:

> You cannot activate a VG with missing PV's because at some point
> then LVM would attempt to write on one of the missing PV and then
> you get panic.

As I told - I'm not sure if it is possible on Linux, but for example
on HP-UX LVM it is normally possible to activate VG even with some
missing PVs (there is also special Low Quorum mode - which is not
enabled by default - to activate VG when you have available <=50% of
mirrored PVs).

Of course trying to read/write from missing PV should (and on HP-UX
does) produce an I/O error to application, but should NOT panic. From
my point there is almost no difference between activating VG with
missing PV and loosing that PV while VG is activated (SAN or HDD
failure).

Having snaphot on failed/missing PV is a little bit different, because
there is question what to do with original lvol. But that can/should
prevent activating the lvol and not entire VG (which can have more
lvols which don't have any data/snapshots on the missing PV).


-- 
  bYE, Marki

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Stuart D. Gathman]

On Fri, 2 May 2008, Marek Podmaka wrote:

> I don't think that automatically making changes to the VG is a good
> idea. Imagine that you have a snapshot on disk mapped from SAN storage
> array and that becomes temporary unavailable (SAN switch failure or
> whatever). It would be a bad idea to remove this PV from VG just
> because it is TEMPORARY unavailable. And how do you distinguish
> between ram disk and normal disk and if it is totally gone or it is
> only temp issue? Also if real HDD fails, you don't want to vgreduce
> it, you want to replace it, vgcfgrestore it and sync it (in case you
> are using mirroring). Also imagine that you can have also real data on
> the same PV, not only snapshot.

When activating a VG, missing PVs should be marked "missing", and show as 
such with pvs.  Missing PVs should behave exactly as if they were
not missing, but have I/O errors on every operation.  Every attempted
I/O on a missing PV should cause a synthetic "missing PV" error.  This
should make mirrors, etc, do the right thing.  LVs partially on the
missing PV could be mountable, getting errors for spots on the missing PV.

The could be an option on vgchange for --ignoremissingpv to activate
with missing PVs.  You might not want to activate with a root filesystem
partially on a missing PV (filesystem corruption).  

It could be helpful if LVs partially on missing PVs were made read-only.  If
the PV was missing only temporarily, this prevents filesystem corruption.  On
the other hand, if the PV is really gone, you want to be able to read it
to recover as much data as you can.  This determination is straightforward
for regular and snapshot LVs (snapshots should be read-only if they or their
origin is partially missing).  It would be more complicated for mirrored LVs.

Finally, there is the issue of metadata changes made while some PVs are
missing.  Suppose there are 2 PVs, one is temporarily missing, and you
make changes to LVM (add/remove LVs and snapshots).  When you reboot and
the missing PV comes back, which version of metadata do you use?
"Most recent version" has some failure scenarios.  AIX solves this
problem with "quorum".  A quorum of PVs must have the same metadata 
version for the VG to be automatically activated.  Otherwise, an admin must
pick which version to use.  A quorum is a simple majority by default.

In any case, it should be obvious that supporting activation of VGs with
missing PVs is not trivial, and using a ram disk for a PV is insane until
missing PVs are fully supported.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Bryn M. Reeves]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stuart D. Gathman wrote:
> When activating a VG, missing PVs should be marked "missing", and show as 
> such with pvs.  Missing PVs should behave exactly as if they were
> not missing, but have I/O errors on every operation.  Every attempted
> I/O on a missing PV should cause a synthetic "missing PV" error.  This
> should make mirrors, etc, do the right thing.  LVs partially on the
> missing PV could be mountable, getting errors for spots on the missing PV.
> 
> The could be an option on vgchange for --ignoremissingpv to activate
> with missing PVs.  You might not want to activate with a root filesystem
> partially on a missing PV (filesystem corruption).  

Am I missing something here? vgchange --partial seems like it will do
what you want (man 8 lvm):

- -P | --partial
 When set, the tools will do their best to provide access to volume
 groups that are only partially available.  Where part of a logical
 volume is missing, /dev/ioerror  will be substituted, and you could use
 dmsetup (8) to set this up to return I/O errors when accessed, or
 create it as a large block device of nulls.  Metadata may not be
 changed with this option. To insert a replacement physical volume of
 the same or large size use pvcreate -u to set the uuid to match the
 original followed by vgcfgrestore (8).

You can also choose the device used for mapping segments on any
incomplete logical volumes by setting the missing_stripe_filler
parameter in the activation section of lvm.conf.

Regards,
Bryn.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFIGyme6YSQoMYUY94RAuqLAJ0SNvliyJFj+oPca084RnCrjhtLPQCeMc42
mPcsR4yUr4WSJq1K+0eyZQY=
=Fu2C
-----END PGP SIGNATURE-----

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Marek Podmaka]

Hello,

Friday, May 2, 2008, 16:47:58, Bryn M. Reeves wrote:

> - -P | --partial
>  When set, the tools will do their best to provide access to volume
>  groups that are only partially available.  Where part of a logical
>  volume is missing, /dev/ioerror  will be substituted, and you could use
>  dmsetup (8) to set this up to return I/O errors when accessed, or
>  create it as a large block device of nulls.  Metadata may not be
>  changed with this option. To insert a replacement physical volume of
>  the same or large size use pvcreate -u to set the uuid to match the
>  original followed by vgcfgrestore (8).

> You can also choose the device used for mapping segments on any
> incomplete logical volumes by setting the missing_stripe_filler
> parameter in the activation section of lvm.conf.

It is not the same. I was doing this about 2 years ago when one HDD
died and I wanted to recover as much data as possible (I mean data
changes since last backup), but it was not possible to activate the VG
with missing PV (or activate only read-only, which prevented me from
mounting the filesystem, because it required fsck, which cannot be
done on readonly lvol). I had to google quite a lot on how to create
this "virtual" PV with dmsetup which returned all zeros (or I/O
errors). It had to be the same size as original (but fortunately you
can create 100 GB sparse file in RAM and use loop to turn it into
PV)... But this is only for recovery - you are making "virtual"
identical PV instead of the original missing... So it is definitely
not the same as activating VG/LV with missing PV.


-- 
  bYE, Marki

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Stuart D. Gathman]

On Fri, 2 May 2008, Bryn M. Reeves wrote:

> Am I missing something here? vgchange --partial seems like it will do
> what you want (man 8 lvm):

It does part of what I want.  It would at least allow a system to boot
from a partial VG containing the root fs.  It does not allow any
metadata changes while VG is partial - not even reducing missing PV.
If a PV is temporarily missing, and you using the pvcreate -u option,
then the original PV comes back as well, you have fireworks.
Also, it doesn't ensure partially missing LVs are readonly (including
where a snapshot is on a missing PV).

In any case, the --partial might be good for initrd.  Although a newbie
admin would be terribly confused by the "no metadatachanges" restriction.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Gerry Reno]

Stuart D. Gathman wrote:
> On Fri, 2 May 2008, Bryn M. Reeves wrote:
>
>   
>> Am I missing something here? vgchange --partial seems like it will do
>> what you want (man 8 lvm):
>>     
>
> It does part of what I want.  It would at least allow a system to boot
> from a partial VG containing the root fs.  It does not allow any
> metadata changes while VG is partial - not even reducing missing PV.
>   
Well, what good is this if you cannot fix anything? You are stuck in 
'broken' mode.

Regards,
Gerry

> If a PV is temporarily missing, and you using the pvcreate -u option,
> then the original PV comes back as well, you have fireworks.
> Also, it doesn't ensure partially missing LVs are readonly (including
> where a snapshot is on a missing PV).
>
> In any case, the --partial might be good for initrd.  Although a newbie
> admin would be terribly confused by the "no metadatachanges" restriction.
>
>   

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Bryn M. Reeves]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Gerry Reno wrote:
> Stuart D. Gathman wrote:
>> On Fri, 2 May 2008, Bryn M. Reeves wrote:
>>
>>  
>>> Am I missing something here? vgchange --partial seems like it will do
>>> what you want (man 8 lvm):
>>>     
>>
>> It does part of what I want.  It would at least allow a system to boot
>> from a partial VG containing the root fs.  It does not allow any
>> metadata changes while VG is partial - not even reducing missing PV.
>>   
> Well, what good is this if you cannot fix anything? You are stuck in
> 'broken' mode.
> 
> Regards,
> Gerry

You recover what you can and then either remove the stray PVs from the
volume group via vgreduce --removemissing or replace them with new
devices that you have prepared using pvcreate --uuid --restorefile=.

Regards,
Bryn.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFIGzTy6YSQoMYUY94RAszRAJ9d0xHvZJbktyps+iF1dhTRSM4oKgCfeWz2
WnV7jmyJLWIknLsddgDNpuM=
=vnhu
-----END PGP SIGNATURE-----

Re: [linux-lvm] F7 will not boot after running backup w/snapshot

[Mikulas Patocka]

On Thu, 1 May 2008, Gerry Reno wrote:

> Mikulas Patocka wrote:
>> ... If you mount the origin device with missing snapshot, you destroy the 
>> snapshot (even if you don't touch it). The snapshot can no longer be 
>> repaired.
>> 
> So it is safer to not activate device in this case then destroy data.
> Why? What value is the old snapshot at this point? You just had a system 
> reboot in the middle of a snapshotted backup so all you need to do is get the 
> system up, redo another snapshot and retake your backup. I'm not interested 
> in the old snapshot.

If you use snapshots for something other than backups (for example 
version-control using snapshots --- to enable admin to revert changes if 
he messes something), then the snapshot is valuable and should survive 
reboot.

Anyway, placing a ramdisk to volume group is bad idea and it must not be 
done on any production system --- note that any lvcreate, lvconvert, etc 
command can allocate anything on that ramdisk --- without the 
administrator knowing it.
--- so I don't see any reason why we should do extra hacks to lvm for 
people who placed ramdisk into vg.


To use temporary storage for snapshot, a special command for lvm would be 
more appropriate --- a command that would setup snapshot and write nothing 
about it to metadata, so that the snapshot would be forgotten on next 
reboot --- then, you can setup the snapshot on any device outside the 
volume group. You can already do this with dmsetup.

Mikulas

[linux-lvm] Temporary shapshots

[Stuart D. Gathman]

On Fri, 2 May 2008, Mikulas Patocka wrote:

> To use temporary storage for snapshot, a special command for lvm would be 
> more appropriate --- a command that would setup snapshot and write nothing 
> about it to metadata, so that the snapshot would be forgotten on next 
> reboot --- then, you can setup the snapshot on any device outside the 
> volume group. You can already do this with dmsetup.

This is an excellent idea - and not just for ramdisks.  The hardest part
of using snapshots for backups is making sure they are deleted afterward.
Not having to check at reboot would be one less 'i' to dot.

Now supposing this feature is added to lvm - what if an enterprising 
admin decides to create *two* snapshots using the same block device?
This could be a caveat emptor - or it could check that the block device
was not in use by anything else, like another snapshot.

Couldn't a LV be used for a temporary snapshot also?  It would just
look like the COW instead of the snapshot image.  I could allocate
a backup_cow LV for backup use only.

The only problem I see is newbies getting very confused wondering why
their "snapshot" won't mount.  That's why I suggest *not* calling the
examples (of the temporary snapshot feature) in the docs anything with "snap"
in the name.  I suggest "cow" as above.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Temporary shapshots

[Larry Dickson]

[-- Attachment #1: Type: text/plain, Size: 2212 bytes --]

I love this idea because it nicely covers one of the main purposes of
snapshotting, namely, making backups at a well-defined time point. Would
there ever be any use of a temporary read-write snapshot? If not, perhaps
making it read-only would simplify the design.

Larry Dickson
Cutting Edge Networked Storage

On 5/3/08, Stuart D. Gathman <stuart@bmsi.com> wrote:
>
> On Fri, 2 May 2008, Mikulas Patocka wrote:
>
> > To use temporary storage for snapshot, a special command for lvm would be
> > more appropriate --- a command that would setup snapshot and write
> nothing
> > about it to metadata, so that the snapshot would be forgotten on next
> > reboot --- then, you can setup the snapshot on any device outside the
> > volume group. You can already do this with dmsetup.
>
> This is an excellent idea - and not just for ramdisks.  The hardest part
> of using snapshots for backups is making sure they are deleted afterward.
> Not having to check at reboot would be one less 'i' to dot.
>
> Now supposing this feature is added to lvm - what if an enterprising
> admin decides to create *two* snapshots using the same block device?
> This could be a caveat emptor - or it could check that the block device
> was not in use by anything else, like another snapshot.
>
> Couldn't a LV be used for a temporary snapshot also?  It would just
> look like the COW instead of the snapshot image.  I could allocate
> a backup_cow LV for backup use only.
>
> The only problem I see is newbies getting very confused wondering why
> their "snapshot" won't mount.  That's why I suggest *not* calling the
> examples (of the temporary snapshot feature) in the docs anything with
> "snap"
> in the name.  I suggest "cow" as above.
>
> --
>              Stuart D. Gathman <stuart@bmsi.com>
>    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
> "Confutatis maledictis, flammis acribus addictis" - background song for
> a Microsoft sponsored "Where do you want to go from here?" commercial.
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>

[-- Attachment #2: Type: text/html, Size: 2984 bytes --]

Re: [linux-lvm] Temporary shapshots

[Charles Marcus]

On 5/5/2008 10:30 AM, Larry Dickson wrote:
>>> To use temporary storage for snapshot, a special command for lvm 
>>> would be more appropriate --- a command that would setup snapshot
>>> and write nothing about it to metadata, so that the snapshot would
>>> be forgotten on next reboot --- then, you can setup the snapshot on
>>> any device outside the volume group. You can already do this with
>>> dmsetup.

>> This is an excellent idea - and not just for ramdisks. The hardest
>> part of using snapshots for backups is making sure they are deleted 
>> afterward. Not having to check at reboot would be one less 'i' to
>> dot.

> I love this idea because it nicely covers one of the main purposes of
> snapshotting, namely, making backups at a well-defined time point. 
> Would there ever be any use of a temporary read-write snapshot? If
> not, perhaps making it read-only would simplify the design.

Ditto...

I feel much better now about using snapshots after realizing the OP's 
problem was due to the use of a RAMdisk for a snapshot volume (combined 
with an unclean shutdown), but anything that makes these safer and 
easier gets my vite...

-- 

Best regards,

Charles