[linux-lvm] LVs corrupted after pvresize

[linux-lvm] LVs corrupted after pvresize

[Christian Völker]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Yohoo!

I'm currently in big troubles :-(
What happened?

I have a single VG running on three RAID arrays.

I resized one of these RAID arrays for additional 250GB. After this was
done I needed to reboot for the kernel to pick up the new size.
Then I went into fdisk, deleted the 8e-LVM partition and re-created a
new one with the same starting sector but a later end sector.
Again, for the kernel to pick up the new partition table I rebooted.

After the reboot my VG was inaccessible because it complained about a
missing PV.
I checked
http://www.novell.com/coolsolutions/appnote/19386.html#DiskPermanentlyRemoved
Symptom2, Solution2.

vgcfgrestore showed the same error: "Couldn't find device with uuid ..."

So I re-created th PV with "pvcreate --uuid [...] /dev/sdd1.

vgscan and I had the new size and a running VG showing all my LVs. :)

Unfortunately ALL my LVs on the /dev/sdd1 PV are severely corrupted- the
complain about missing partition tables, fsck.ext2 prints thousands of
errors and so on.

I tried to reduce the PV again, but the LVs remain corrupted.

I don't expect my data to be recoverable- but I want to know what I did
wrong.

Though, if you can recover my LVs it would be fine ;-)


Any ideas?


Christian


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFI96130XNIYlAXmzsRAqMGAKDK13dabQD/LFGl460R410PQlNFCQCgt7ZT
iVCXd/QCUoSQIdfTaODAbPU=
=XMWS
-----END PGP SIGNATURE-----

Re: [linux-lvm] LVs corrupted after pvresize

[Stuart D. Gathman]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 757 bytes --]

On Thu, 16 Oct 2008, Christian V�lker wrote:

> I'm currently in big troubles :-(
> What happened?
> 
> I have a single VG running on three RAID arrays.
> 
> I resized one of these RAID arrays for additional 250GB. After this was

Resizing a RAID-5 array is often not what you think, and doing so often
scrambles the data, depending on what RAID implementation you are using.  That
is why I stick with RAID-1.  Resizing is simple.  In any case, LVM
was not your problem.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] LVs corrupted after pvresize

[Christian Völker]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Yohoo!

|> I resized one of these RAID arrays for additional 250GB. After this was
| Resizing a RAID-5 array is often not what you think, and doing so often
| scrambles the data, depending on what RAID implementation you are using.
I'm using a hardware RAID 3ware 9500S Controller. And this guy offers
migration from RAID10 (4disks) to RAID5 (4disks, but with larger capacity).
And I already did this task several times on these types of controllers.


| In any case, LVM was not your problem.
This answer is too simple to be true.
After the resize the partition and the PV were recognized correctly! So
I'm pretty sure the migration was ok.
But after the deletion of the partion, recreation and pvcreate
everything went wrong. So for me it is obvious, LVM had an issue. But
which one?

Greets
Christian
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFI+A770XNIYlAXmzsRAicJAKCpogPKLWhuOtABRI/cBthIRVwlHACZAcoi
b7+z+RaIKUKnui+b8XLqtw0=
=TSoe
-----END PGP SIGNATURE-----

Re: [linux-lvm] LVs corrupted after pvresize

[Peter Larsen]

On Fri, 2008-10-17 at 06:05 +0200, Christian Völker wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Yohoo!
> 
> |> I resized one of these RAID arrays for additional 250GB. After this was
> | Resizing a RAID-5 array is often not what you think, and doing so often
> | scrambles the data, depending on what RAID implementation you are using.
> I'm using a hardware RAID 3ware 9500S Controller. And this guy offers
> migration from RAID10 (4disks) to RAID5 (4disks, but with larger capacity).
> And I already did this task several times on these types of controllers.

That wasn't really the issue raised. Extending raids by adding new disks
doesn't do a balanced increase. In other words, you don't get a very
well functioning raid. Doing true raid migration is not straight
forward. So unless you fully restructure all data, you aren't getting
the performance you should be getting.

> | In any case, LVM was not your problem.
> This answer is too simple to be true.

Not really. Your approach was very poor.

> After the resize the partition and the PV were recognized correctly! 

Because at that time, the PVs were intact.

> So
> I'm pretty sure the migration was ok.

Again, you misunderstood Stuart's comment. He didn't say your RAID
migration didn't work. He said it wasn't optimal.

> But after the deletion of the partion, recreation and pvcreate

That's the error. There's a pvresize for a reason. You also need to use
the same UUID. Removing the PV makes your vg inconsistent and you've
basically @#$@# up LVM.

> everything went wrong. So for me it is obvious, LVM had an issue. But
> which one?

No - your use of LVM was wrong. Not LVM itself. 

--- 
Regards
    Peter Larsen

"Right now I'm having amnesia and deja vu at the same time."
		-- Steven Wright

Re: [linux-lvm] LVs corrupted after pvresize

[Stuart D. Gathman]

On Fri, 17 Oct 2008, Peter Larsen wrote:

> Again, you misunderstood Stuart's comment. He didn't say your RAID
> migration didn't work. He said it wasn't optimal.

Actually, I *was* hypothesizing that the RAID migration scrambled the data.
But apparently, that was not the problem because:

> > But after the deletion of the partion, recreation and pvcreate
> 
> That's the error. There's a pvresize for a reason. You also need to use

Yes, the pvcreate just hosed your metadata.  If you have a backup of the
metadata (somewhere other than an affected LV), you can restore it,
and then do the pvresize.  

> No - your use of LVM was wrong. Not LVM itself. 

Having used AIX LVM, I can say that Linux LVM has a way to go before
it reaches that level of ease of use.  But it keeps getting better,
and I get to participate!

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] LVs corrupted after pvresize

[Peter Larsen]

On Mon, 2008-10-20 at 11:41 -0400, Stuart D. Gathman wrote:

> Having used AIX LVM, I can say that Linux LVM has a way to go before
> it reaches that level of ease of use.  But it keeps getting better,
> and I get to participate!

I'm currious - my exposure to LVM like systems is mainly on Linux.
Where's the catch? I find the basic LVM command structure pretty
straight forward and easy. There's even a couple of GUI's available now
to make the LVM management easier.

Can you give an example or two on how the LVM differs on the
user-friendliness areas?

--- 
Regards
    Peter Larsen

"Here's something to think about:  How come you never see a headline like
`Psychic Wins Lottery'?"
		-- Jay Leno

Re: [linux-lvm] LVs corrupted after pvresize

[Morten Torstensen]

Peter Larsen wrote:
> Can you give an example or two on how the LVM differs on the
> user-friendliness areas?

Say you want to mirror a LV, then you just do a "mklvcopy mylvname 2" to 
have 2 copies of data (one extra, like raid-1). If you want to migrate 
your disks to a new system, add the new PV to your VG and do a "mklvcopy 
mylvname 3 newpv", then when it finish remove your old mirrors and 
delete the old PVs.

The AIX concept of physical extent/logical extent mapping is sorely 
missed on linux.

Generally speaking, LVM on AIX is also more mature. Exporting and 
importing VGs (to move VGs between systems) works better in my 
experience, clustering is more stable, but that is also something where 
YMMV can show.

Managing mirrors, data placement and hotspot detection are what I mainly 
miss on Linux.

-- 

//Morten Torstensen
//Email: morten@mortent.org
//IM: Cartoon@jabber.no morten.torstensen@gmail.com

And if it turns out that there is a God, I don't believe that he is evil.
The worst that can be said is that he's an underachiever.

Re: [linux-lvm] LVs corrupted after pvresize

[Larry Dickson]

[-- Attachment #1: Type: text/plain, Size: 1476 bytes --]

I thought lvdisplay --maps gave you the PV to LV mappings.

Larry Dickson
Cutting Edge Networked Storage

On 10/21/08, Morten Torstensen <morten@mortent.org> wrote:
>
> Peter Larsen wrote:
>
>> Can you give an example or two on how the LVM differs on the
>> user-friendliness areas?
>>
>
> Say you want to mirror a LV, then you just do a "mklvcopy mylvname 2" to
> have 2 copies of data (one extra, like raid-1). If you want to migrate your
> disks to a new system, add the new PV to your VG and do a "mklvcopy mylvname
> 3 newpv", then when it finish remove your old mirrors and delete the old
> PVs.
>
> The AIX concept of physical extent/logical extent mapping is sorely missed
> on linux.
>
> Generally speaking, LVM on AIX is also more mature. Exporting and importing
> VGs (to move VGs between systems) works better in my experience, clustering
> is more stable, but that is also something where YMMV can show.
>
> Managing mirrors, data placement and hotspot detection are what I mainly
> miss on Linux.
>
> --
>
> //Morten Torstensen
> //Email: morten@mortent.org
> //IM: Cartoon@jabber.no morten.torstensen@gmail.com
>
> And if it turns out that there is a God, I don't believe that he is evil.
> The worst that can be said is that he's an underachiever.
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>

[-- Attachment #2: Type: text/html, Size: 2776 bytes --]

Re: [linux-lvm] LVs corrupted after pvresize

[Stuart D. Gathman]

On Tue, 21 Oct 2008, Larry Dickson wrote:

> I thought lvdisplay --maps gave you the PV to LV mappings.

> > The AIX concept of physical extent/logical extent mapping is sorely missed
> > on linux.

He misses the AIX *flavor* of PE to LE mapping (not PV to LV).  In
AIX, each LE is mapped to 1 or more (for mirroring) PEs in arbitrary
locations.  Each LE is tracked independently in AIX - the focus is on
LEs, with LVs being simply a collection of LEs.  If a PV dies, some
LEs in a LV may have mirrors and others not.  When bringing a PV back
on line, each LE is resynchronized independently.

BTW, AIX also has "migratepv mylv srcpv dstpv" which creates a copy of
each LE on dstpv, then removes a copy of each LE on srcpv.

Why did we leave?  It had nothing to do with quality of hardware or
software from IBM.  As a small VAR, we couldn't deal with IBM directly,
and it was too expensive to deal with additional middlemen who knew less
than we did and couldn't provide support.

Additionally, I love diving into the source when I get spare moments -
something you can't do with AIX LVM.

Solaris on Sun hardware looks attractive, offering a robust LVM/FS (ZFS), 
fully supported enterprise hardware, and open source.  But it is past
the price point of our current customers, and Solaris on PCs doesn't 
justify the learning curve compared to Linux which we are already 
familiar with.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.