[linux-lvm] LVM snapshot?

[linux-lvm] LVM snapshot?

[Tom Diehl]

Hi,

I have a machine that is running EL5 with xen. One of the DOMU's has an LV
from DOM0 mounted on it to allow access to the data. Is it safe to do 
a snapshot of the LV from DOM0 while it is still mounted rw on the DOMU?

What I would like to do is snapshot the LV, fsck the snapshot and if it comes
up clean, continue on. If it shows errors, then I will schedule downtime
for a real fsck.

Does this sound safe or am I going to get into trouble if I do that?

Regards,

-- 
Tom Diehl		tdiehl@rogueind.com		Spamtrap address mtd123@rogueind.com

Re: [linux-lvm] LVM snapshot?

[Tim Post]

On Sun, 2008-12-21 at 09:33 -0500, Tom Diehl wrote:
> Hi,
> 
> I have a machine that is running EL5 with xen. One of the DOMU's has an LV
> from DOM0 mounted on it to allow access to the data. Is it safe to do 
> a snapshot of the LV from DOM0 while it is still mounted rw on the DOMU?

Yes, provided:

1 - You send a sysrq 's' to the guest to sync anything dirty
2 - You pause the guest prior to taking the snapshot

After taking the snapshot, of course unpause the guest. I have a tool
that I wrote using libxenctrl / liblvm2cmd which does this very quickly,
if you'd like it I'll send you the source code when I get back to my
office next week. Its also rather easy to export mount stats (how much
is used on any given mount) via xenbus, which helps automatically size
the snapshots from dom-0.

A fsck -n /dev/xvda(x) within the running guest itself should tell you
(more reliably) if any problems exist, depending on the file system and
journal mode. It need not run completely, ~15 seconds in you'll be able
to tell and just ctrl-c. Moreover, this gives you a general indication
of how big the downtime window should be. Plus, the i/o of the test is
in the dom-u, which is better.

A "All Clean" message will look like the normal "clean" message, plus a
warning that journal recovery is not possible since fsck is not going to
make any changes to the FS.

HTH

Cheers,
--Tim

[linux-lvm] LVM2 robustness w/ large (>100TB) name spaces?

[Steve Costaras]

[-- Attachment #1: Type: text/plain, Size: 1279 bytes --]


I have not seen much commentary in my searches concerning LVM under linux
with very large name spaces (single file systems), and figured I would pose
the question out to the forum as I am in the planning stages here to rebuild
and consolidate systems.

Initially I have a couple questions regarding the underlying LVM structure
and what it supports:

- Are PV sizes >=16TB possible?   (I am looking to add external RAID chassis
which will present to the system as large (~13TB (16-drive, raid-6, + hot
spare)) drive.   The subsystem drive will be replaced w/ 2TB or greater
models when they are available in 2010-2011 and am trying to avoid a
forklift upgrade of subsystem components.

- What are the limits on PE/LE's per logical volume (>200,000,000? A
problem?)  (I will be attaching multiple external chassis like above to
several HBA's and will be using LVM striping to increase performance.   So a
small PE size (4MB-8MB) would be best to aid in the distribution of requests
across the physical subsystems.)

- Are there any references or examples of users who have created/run large
(100TB - 1PB) LVM installations that are functional in the real-world?   (I
don't see why this would not work on paper, but theory and practice are two
different things).


Thanks,

Steve

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 3916 bytes --]

Re: [linux-lvm] LVM2 robustness w/ large (>100TB) name spaces?

[Alasdair G Kergon]

 From the LVM2 point of view, extents (PEs/LEs) are counted using 32-bit
numbers so that's where the limit comes from.
(You can change extent size later, shuffling extent boundaries with pvmove
first as necessary.)

The stripe size is controlled by a separate parameter essentially independent
of the PE size.

When creating the PVs, use a larger-than-default size for the metadata area:
it is awkward to change later if you fill it up.

Alasdair
-- 
agk@redhat.com

RE: [linux-lvm] LVM2 robustness w/ large (>100TB) name spaces?

[Steve Costaras]

[-- Attachment #1: Type: text/plain, Size: 1472 bytes --]

Thanks, I was planning on using a larger metadata area anyway, just to play
it safe as I'm expecting a 3-4x growth for the life of the array before next
hard rebuild and all the comments I've seen on the Inet are talking about
much smaller array sizes than what I'm building.     Is there any rule of
thumb or sizing guidelines or is it a black art?

With PE/LE's being 32bit (or is it a signed 32bit so 31 bits?)  Either way,
that's more than enough even w/ 4MiB sizes for the expected end-state of the
name space.

    

-----Original Message-----
From: linux-lvm-bounces@redhat.com [mailto:linux-lvm-bounces@redhat.com] On
Behalf Of Alasdair G Kergon
Sent: Monday, December 22, 2008 19:25
To: LVM general discussion and development
Subject: Re: [linux-lvm] LVM2 robustness w/ large (>100TB) name spaces?

 From the LVM2 point of view, extents (PEs/LEs) are counted using 32-bit
numbers so that's where the limit comes from.
(You can change extent size later, shuffling extent boundaries with pvmove
first as necessary.)

The stripe size is controlled by a separate parameter essentially
independent of the PE size.

When creating the PVs, use a larger-than-default size for the metadata area:
it is awkward to change later if you fill it up.

Alasdair
--
agk@redhat.com

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 3916 bytes --]

Re: [linux-lvm] LVM2 robustness w/ large (>100TB) name spaces?

[Marek Podmaka]

Hello,

Tuesday, December 23, 2008, 1:15:28, Steve Costaras wrote:

> - What are the limits on PE/LE's per logical volume (>200,000,000? A
> problem?)  (I will be attaching multiple external chassis like above to
> several HBA's and will be using LVM striping to increase performance.   So a
> small PE size (4MB-8MB) would be best to aid in the distribution of requests
> across the physical subsystems.)

I think 4-8 MB for PE size is too small when you will be using such
big (and probably advanced arrays).
LVM stripping (strip size in hundreds of kB) would kill any array,
because when you request for example 512 kB from one array and next
512 kB from another array, they can't handle it efficiently. You won't
see the benefit of reading from all 16 spindles - everytime it will
just load 512 kB from one physical disk. Also detection of sequential
read might not work well in array in this case.

In HP-UX LVM with enterprise arrays like HP EVA or HP XP we use 32-64 MB
PE and enable distribution - that means "stripe" size = PE size.
LE1 = PV1_1
LE2 = PV2_1
LE3 = PV1_2
LE4 = PV2_2 and so on.
Using this you request for example 32 MB from one array. Given the
cache sizes of arrays and readahead, so should get much better
performance, because those 32 MB will be fetched partially from all 16
drives.
Also we don't use ditribution among 2 arrays, just using different
paths to one array (different HBA, different SAN switch and different
array FC controller). We use 2 arrays only for mirroring data to other
datacentre for clusters.
The main reason for us for that PE distribution is that HP-UX does not
have loadbalancing multipath built-in. But even when you will have it,
using more PVs is better because of the architectural limits of arrays
(no. of outstanding request for single virtual drive, scsi queue depth
on server and on array, cache memory limits per virtual drive, etc.)


-- 
  bYE, Marki

RE: [linux-lvm] LVM2 robustness w/ large (>100TB) name spaces?

[Steve Costaras]

[-- Attachment #1: Type: text/plain, Size: 3724 bytes --]


Thanks, yes the workload that this system will do is a lot of small I/O
requests.    From the system that I have logs from (I am trying to gather
logs from all systems to fine tune the sizes) the requests are in the
~64-128KiB range to the drive subsystem and very random in nature.    So a
4MiB PE or even 8MiB PE shouldn't be much of a problem (assuming the
workloads on the other boxes is similar).    

This is why I am planning on adding as many spindles as I can to grow that
way more so than streaming I/O.    I understand the limits of outstanding
commands IOPS (read/write) and scsi command queue depth, which the later
(command queue depth) is the biggest item that I see what would push me to
create more/smaller (instead of raid6 (13+2+1) for each PV something like a
raid-6 of (5+2+1) which would lower storage efficiency but increase the
number of PV's and command queue depth aggregate (as well as increasing
write iops but this is mainly a read-request array not many writes).

I am more looking for examples of builds that have real-world broached the
100+TB range under linux and what kinds of gotcha's I'm in for.   My current
arrays that are going to be merged into this are on average ~20-30TiB in
size each, since all serve similar (and somewhat overlapping functions)
merging them is in order.

Steve 

-----Original Message-----
From: linux-lvm-bounces@redhat.com [mailto:linux-lvm-bounces@redhat.com] On
Behalf Of Marek Podmaka
Sent: Tuesday, December 23, 2008 04:28
To: LVM general discussion and development
Subject: Re: [linux-lvm] LVM2 robustness w/ large (>100TB) name spaces?

Hello,

Tuesday, December 23, 2008, 1:15:28, Steve Costaras wrote:

> - What are the limits on PE/LE's per logical volume (>200,000,000? A
> problem?)  (I will be attaching multiple external chassis like above to
> several HBA's and will be using LVM striping to increase performance.   So
a
> small PE size (4MB-8MB) would be best to aid in the distribution of 
> requests across the physical subsystems.)

I think 4-8 MB for PE size is too small when you will be using such big (and
probably advanced arrays).
LVM stripping (strip size in hundreds of kB) would kill any array, because
when you request for example 512 kB from one array and next
512 kB from another array, they can't handle it efficiently. You won't see
the benefit of reading from all 16 spindles - everytime it will just load
512 kB from one physical disk. Also detection of sequential read might not
work well in array in this case.

In HP-UX LVM with enterprise arrays like HP EVA or HP XP we use 32-64 MB PE
and enable distribution - that means "stripe" size = PE size.
LE1 = PV1_1
LE2 = PV2_1
LE3 = PV1_2
LE4 = PV2_2 and so on.
Using this you request for example 32 MB from one array. Given the cache
sizes of arrays and readahead, so should get much better performance,
because those 32 MB will be fetched partially from all 16 drives.
Also we don't use ditribution among 2 arrays, just using different paths to
one array (different HBA, different SAN switch and different array FC
controller). We use 2 arrays only for mirroring data to other datacentre for
clusters.
The main reason for us for that PE distribution is that HP-UX does not have
loadbalancing multipath built-in. But even when you will have it, using more
PVs is better because of the architectural limits of arrays (no. of
outstanding request for single virtual drive, scsi queue depth on server and
on array, cache memory limits per virtual drive, etc.)


--
  bYE, Marki


_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 3916 bytes --]