[linux-lvm] LVM with large arrays, one large or mutiple small PV's?

[linux-lvm] LVM with large arrays, one large or mutiple small PV's?

[David S. Broome]

Hello - This is a best practices style question.

LVM with large arrays, one large or mutiple small PV's?

What is the suggested physical partition layout with large hardware RAID arrays (is NAS or SAN) in the case where there can be one VG?

ie I have two Raid 5 Arrays that can either present one large logical volume ie 3.5T and 2.9T each or a multiple smaller logical volumes ie 500G.

Would it be better have the hardware present add multiple smaller logical volumes to as PV's or the single larger volumes?

If there is a need for multiple VG's then obviously there needs to be multiple PVs.

The smaller volumes would have the advantage of being able to pvmoved to other hardware in the future with more flexibility due to the size of PV's.

There would be some overhead space lost due to the multiple PV's, although this would be small.

Thanks in advance for any comments.

Dave,
--
David Broome  Sr. Programmer Analyst @ FineArts.UVic.CA /BSc
250.721-6307 | dbroome@uvic.ca | FIA 221

Re: [linux-lvm] LVM with large arrays, one large or mutiple small PV's?

[Marek Podmaka]

Hello,

Tuesday, August 12, 2008, 5:53:26, David S. Broome wrote:

> Hello - This is a best practices style question.
> LVM with large arrays, one large or mutiple small PV's?

> What is the suggested physical partition layout with large hardware
> RAID arrays (is NAS or SAN) in the case where there can be one VG?
> ie I have two Raid 5 Arrays that can either present one large
> logical volume ie 3.5T and 2.9T each or a multiple smaller logical volumes ie 500G.

I think from LVM side it doesn't matter if you have 10x PV of size 100
GB or 1 PV of size 1 TB. But it really depends on the SAN disk array
you have and how it handles each virtual disk. Does it have queues,
caches for each vdisk or for the entire array? For example for HP EVA
arrays it is recommended to prefer smaller vdisks if possible.

If array has for example queue for 100 outstanding requests per vdisk
then it means 100 req for 1 TB vdisk, or 10x100 req for the same 1TB
over 10 vdisks.

The same applies also for OS. Don't know how are the defaults on
linux, but for HP-UX we have to increase scsi queue depth for large
disks.

Also more PV is better when you somewhere in future want to shrink the
VG... you just pvmove some data and can remove PV. This can't be done
when you will have only one big PV.


-- 
  bYE, Marki

Re: [linux-lvm] LVM with large arrays, one large or mutiple small PV's?

[Bryn M. Reeves]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Marek Podmaka wrote:
> I think from LVM side it doesn't matter if you have 10x PV of size 100
> GB or 1 PV of size 1 TB. But it really depends on the SAN disk array
> you have and how it handles each virtual disk. Does it have queues,
> caches for each vdisk or for the entire array? For example for HP EVA
> arrays it is recommended to prefer smaller vdisks if possible.

It does affect LVM - the performance of the LVM2 tools is affected by
the number of physical volumes in a volume group. Strictly speaking,
it's the number of metadata areas which defaults to one per physical
volume. This means that for large volume groups you can use the
"--metadatacopies" option to pvcreate to control the number of metadata
areas present (set it to zero for some PVs) and avoid the sluggish tool
performance you might otherwise see.

Recent releases of LVM2 improve performance for large VGs as they
perform caching of metadata read from disk under some circumstances.
There were also a series of fixes to ensure that the tools behave
correctly when they find PVs with no metadata.

See earlier threads in the archives on this subject for more information
& examples of creating VGs with reduced numbers of metadata areas.

Regards,
Bryn.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFIorLz6YSQoMYUY94RAhMRAJ4hwLFf1dGfQRmNq5+C3F/hjCwVNgCfawJJ
hm3SO7u8tJSPz23rZcwT5XQ=
=0PG6
-----END PGP SIGNATURE-----

RE: [linux-lvm] LVM with large arrays, one large or mutiple small PV's?

[Andrea Ghelardi]

Hi all,

Could I "push" this request?

I'm in a very similar situation, and I googled for two entire days but it
seems that there is no "advanced" documentation on this topic.

Also, do we have some ideas about the overhead introduced by having more
smaller PVs instead of a few bigger ones?

Thanks!


Andrea Ghelardi

Via San Martino, 52, 56125 Pisa, ITALY
T: +39 050 220 371   
a.ghelardi@iontrading.com
http://www.iontrading.com
 
 

-----Original Message-----
From: linux-lvm-bounces@redhat.com [mailto:linux-lvm-bounces@redhat.com] On
Behalf Of David S. Broome
Sent: Tuesday, August 12, 2008 5:53 AM
To: linux-lvm@redhat.com
Subject: [linux-lvm] LVM with large arrays, one large or mutiple small PV's?

Hello - This is a best practices style question.

LVM with large arrays, one large or mutiple small PV's?

What is the suggested physical partition layout with large hardware RAID
arrays (is NAS or SAN) in the case where there can be one VG?

ie I have two Raid 5 Arrays that can either present one large logical volume
ie 3.5T and 2.9T each or a multiple smaller logical volumes ie 500G.

Would it be better have the hardware present add multiple smaller logical
volumes to as PV's or the single larger volumes?

If there is a need for multiple VG's then obviously there needs to be
multiple PVs.

The smaller volumes would have the advantage of being able to pvmoved to
other hardware in the future with more flexibility due to the size of PV's.

There would be some overhead space lost due to the multiple PV's, although
this would be small.

Thanks in advance for any comments.

Dave,
--
David Broome  Sr. Programmer Analyst @ FineArts.UVic.CA /BSc
250.721-6307 | dbroome@uvic.ca | FIA 221



_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

RE: [linux-lvm] LVM with large arrays, one large or mutiple smallPV's?

[Allen, Jack]

 

-----Original Message-----
From: linux-lvm-bounces@redhat.com [mailto:linux-lvm-bounces@redhat.com]
On Behalf Of Bryn M. Reeves
Sent: Wednesday, August 13, 2008 6:10 AM
To: Marek Podmaka; LVM general discussion and development
Subject: Re: [linux-lvm] LVM with large arrays, one large or mutiple
smallPV's?

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Marek Podmaka wrote:
> I think from LVM side it doesn't matter if you have 10x PV of size 100
> GB or 1 PV of size 1 TB. But it really depends on the SAN disk array
> you have and how it handles each virtual disk. Does it have queues,
> caches for each vdisk or for the entire array? For example for HP EVA
> arrays it is recommended to prefer smaller vdisks if possible.

It does affect LVM - the performance of the LVM2 tools is affected by
the number of physical volumes in a volume group. Strictly speaking,
it's the number of metadata areas which defaults to one per physical
volume. This means that for large volume groups you can use the
"--metadatacopies" option to pvcreate to control the number of metadata
areas present (set it to zero for some PVs) and avoid the sluggish tool
performance you might otherwise see.

Recent releases of LVM2 improve performance for large VGs as they
perform caching of metadata read from disk under some circumstances.
There were also a series of fixes to ensure that the tools behave
correctly when they find PVs with no metadata.

See earlier threads in the archives on this subject for more information
& examples of creating VGs with reduced numbers of metadata areas.

Regards,
Bryn.
============================================================
You are saying it affects the performance of some of the LVM tools, but
what about File Systems or databases that use the Logical Volume
directly?

Thanks:
Jack Allen


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFIorLz6YSQoMYUY94RAhMRAJ4hwLFf1dGfQRmNq5+C3F/hjCwVNgCfawJJ
hm3SO7u8tJSPz23rZcwT5XQ=
=0PG6
-----END PGP SIGNATURE-----

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] LVM with large arrays, one large or mutiple smallPV's?

[Bryn M. Reeves]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Allen, Jack wrote:
>> On Behalf Of Bryn M. Reeves
>> Recent releases of LVM2 improve performance for large VGs as they
>> perform caching of metadata read from disk under some circumstances.
>> There were also a series of fixes to ensure that the tools behave
>> correctly when they find PVs with no metadata.
>> 
>> See earlier threads in the archives on this subject for more information
>> & examples of creating VGs with reduced numbers of metadata areas.
>> 
>> Regards,
>> Bryn.
> ============================================================
> You are saying it affects the performance of some of the LVM tools, but
> what about File Systems or databases that use the Logical Volume
> directly?

Correct. It has no effect on I/O performance - at least not in typical
real-world cases; you could probably construct a worst case scenario
where the VG is severely fragmented across 100s and 100s of PVs so tthat
I/O performance suffers but I have never in practice encountered this
situation (and it could just as easily occur in a VG containing a
smaller number of PVs with the right sequence of LV manipulations and
allocation policy).

On the other hand, poor tool performance with VGs containing 10s or 100s
of PVs has been reported in real world situations on numerous occasions
(particularly on s390/zVM systems using IBM DASD storage since these are
limited to 2GiB per volume so tend to be used in large numbers).

These problems tend not to be noticeable with modest numbers of PVs (~10
or so) although the impact can be measured. With hundreds of metadata
containing PVs the problem can be very noticeable - especially with LVM2
versions that do not include Milan's metadata caching enhancements.

Regards,
Bryn.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFIoyFz6YSQoMYUY94RAu6DAJ0WvnIlFjAizDj2Zzyx0bqX1TNRSACbBZU5
778C4VXaUo2xHe3Wmr7ODFM=
=K0Su
-----END PGP SIGNATURE-----