[linux-lvm] Best practice: metadata backup

[linux-lvm] Best practice: metadata backup

[Stuart D. Gathman]

LVM keeps backups of metadata in /etc/lvm (or some distro specific place).  
This is not especially useful when the root filesystem itself is in LVM.  
I have been making rsync backups of /etc/lvm to /boot (which is non-LVM
since grub doesn't support LVM until 1.95).  Should I make /etc/lvm a
symlink to /boot/lvm?

When grub supports LVM, and the entire disk is LVM (no partition table),
how will backups be made?  AIX treats the boot LV specially, and ensures
that its PEs are contiguous at the beginning of the disk(s).  Will Linux
LVM do something similar?  Or will we just keep allocating a non-LVM boot
partition?

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Best practice: metadata backup

[Alasdair G Kergon]

On Sun, Aug 05, 2007 at 12:17:15AM -0400, Stuart D. Gathman wrote:
> LVM keeps backups of metadata in /etc/lvm (or some distro specific place).  
> This is not especially useful when the root filesystem itself is in LVM.  
> I have been making rsync backups of /etc/lvm to /boot (which is non-LVM
> since grub doesn't support LVM until 1.95).  Should I make /etc/lvm a
> symlink to /boot/lvm?
 
If you give /boot enough free space.

You can also change the location by setting the environment variable
LVM_SYSTEM_DIR.

> When grub supports LVM, and the entire disk is LVM (no partition table),
> how will backups be made?  

Just like any other file backups - tape, CD, internet: you choose!

> AIX treats the boot LV specially, and ensures
> that its PEs are contiguous at the beginning of the disk(s).  

Nothing is stopping a linux installer doing that today:
  --alloc contiguous and specify the extents.

But if grub supports LVM2 properly (add a trivial mapping layer from LV
extents to physical sectors, then populate it from the LVM2 metadata
across an agreed interface from a library maintained as part of the lvm2
package) it won't require anything like that of course - the hardest
problem I think is actually for it to identify the right disks to use.

Alasdair
-- 
agk@redhat.com

Re: [linux-lvm] Best practice: metadata backup

[Stuart D. Gathman]

On Tue, 7 Aug 2007, Alasdair G Kergon wrote:

> > When grub supports LVM, and the entire disk is LVM (no partition table),
> > how will backups be made?  
> 
> Just like any other file backups - tape, CD, internet: you choose!

I meant metadata backups.  AIX keeps a copy on each physical volume
(using quorum to ensure a majority of the copies agree).  I was wondering
how linux LVM would handle metadata backup on an all LVM system.  Seems
like you would still need a small non LVM filesystem just for that.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Best practice: metadata backup

[Alasdair G Kergon]

On Tue, Aug 07, 2007 at 04:56:01PM -0400, Stuart D. Gathman wrote:
> I meant metadata backups.  AIX keeps a copy on each physical volume
> (using quorum to ensure a majority of the copies agree).  I was wondering
> how linux LVM would handle metadata backup on an all LVM system.  Seems
> like you would still need a small non LVM filesystem just for that.

The backups in /etc/lvm are backups in case you lost or overwrote the metadata
accidentally.

There are already copies (and backups of previous copies) on each physical
volume (in the default configuration) and there's an automated process to deal
with simple problems like power failure during a metadata update leading to
disagreement between the copies.
 
Alasdair
-- 
agk@redhat.com

Re: [linux-lvm] Best practice: metadata backup

[paddy]

On Tue, Aug 07, 2007 at 08:01:53PM +0100, Alasdair G Kergon wrote:
> > AIX treats the boot LV specially, and ensures
> > that its PEs are contiguous at the beginning of the disk(s).  
> 
> Nothing is stopping a linux installer doing that today:
>   --alloc contiguous and specify the extents.

(its been a long time but) last time I looked there was no facility
to flag an LV in a manner that would accord it treatment from the
various utilities appropriate to it's status as a boot volume
(such as, for example, not moving about). Of course, when I looked
there wasn't any functionality in the utilities that did anything
to worry about :-) (but user-space code is easier written than 
on-disk representations are changed)

(unless its already been done ...) by the time anyone gets around to it,
the various bios limitation of PCs, etc will all be just a memory.

> But if grub supports LVM2 properly (add a trivial mapping layer from LV
> extents to physical sectors, then populate it from the LVM2 metadata
> across an agreed interface from a library maintained as part of the lvm2
> package) it won't require anything like that of course - the hardest
> problem I think is actually for it to identify the right disks to use.

(and yet strangely we all manage to boot ... !?!? )

... or everyone uses a boot-loader that reads filesystems, and they all
learn to read LVM too :-)

Regards,
Paddy

Re: [linux-lvm] Best practice: metadata backup

[Stuart D. Gathman]

On Tue, 7 Aug 2007, Alasdair G Kergon wrote:

> There are already copies (and backups of previous copies) on each physical
> volume (in the default configuration) and there's an automated process to deal
> with simple problems like power failure during a metadata update leading to
> disagreement between the copies.

That's good.  But I've been lurking and noticed that when things
go disastrously wrong, and some poor admin posts here hoping for a miracle,
the key to the miracle is having a copy of the LVM metadata somewhere for
use with vgcfgrestore.  Apparently, the PV copies are not enough for
current tools to allow recovery from some situations.

I wish I could take a sabbatical and spend a year setting up VGs,
breaking them in evil ways, and fixing the tools to intelligently
recover.  Given the nature of open source, OSS friendly employers should
offer sabbaticals for developers, like clergy and academics get.  A year
every 7 years to work full-time on whatever project (hardware or
software related) grabs their interest.

Certainly, automatically bringing all remaining LVs online after
PV failure is a must.

I've always been interested in data storage that can recover from
disaster.  I designed a database filesystem that stores a table id
in every block, so that usable tables and directories can be recovered
regardless of what portion of the filesystem is wiped out (missing records,
of course).  It has saved our customers butt several times ("No, we don't
know where the backup tapes have gone...we took them offsite like you said.
You didn't say anything about bringing them back.").

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Best practice: metadata backup

[Alasdair G Kergon]

On Wed, Aug 08, 2007 at 03:03:38PM -0400, Stuart D. Gathman wrote:
> That's good.  But I've been lurking and noticed that when things
> go disastrously wrong, and some poor admin posts here hoping for a miracle,
> the key to the miracle is having a copy of the LVM metadata somewhere for
> use with vgcfgrestore.  Apparently, the PV copies are not enough for
> current tools to allow recovery from some situations.
 
No distribution that I'm aware of uses the existing facilities in full
to solve problems at system boot.  If things aren't exactly right they
fail and leave the user to work out how to clean up the mess.  There's
quite a bit of logic that could be added to the initrd/initscripts to
handle - or at least report on - certain failure situations using the
existing LVM2 features.

Secondly, by the time admins seek help, they've sometimes already made
their situation many times worse by attempting to fix things the wrong
way.  Worse, they sometimes realise they did something stupid and
convince themselves they can still get sensible advice without revealing
everything they know they did.  So someone providing that help needs to
try to check everything in more than one way in case relevant
information is being withheld.  Mechanisms to help skilled support
engineers deal with such situations efficiently pervade the LVM2 design.  

The lvm_dump.sh script is being developed to capture the essential
information about an LVM2 system, and eventually we hope to have tools
to analyse and report upon the contents of the dump.  'pvck' is also
under development to aid recovery from one class of problems.

> Certainly, automatically bringing all remaining LVs online after
> PV failure is a must.
 
That's an example in point.  The LVM2 tools have offered facilities to
do that for several years.  The distributions do not yet make use of
them.

Alasdair
-- 
agk@redhat.com

Re: [linux-lvm] Best practice: metadata backup

[John Hearns]

On Wed, 2007-08-08 at 15:03 -0400, Stuart D. Gathman wrote:

> 
> I've always been interested in data storage that can recover from
> disaster.  I designed a database filesystem that stores a table id
> in every block, so that usable tables and directories can be recovered
> regardless of what portion of the filesystem is wiped out (missing records,
> of course).  It has saved our customers butt several times ("No, we don't
> know where the backup tapes have gone...we took them offsite like you said.
> You didn't say anything about bringing them back.").

Something along the lines of an object-based filesystem?
Great minds think alike!

I do a lot of work with Panasas  www.panasas.com
and Lustre   www.clusterfs.com


Not that familiar with ZFS - anyone care to comment?