[linux-lvm] Snapshots and disk re-use

[linux-lvm] Snapshots and disk re-use

[Jonathan Tripathy]

Hi Everyone,

We host many Xen VMs for customers. We use LVM as the fisk storage for 
these VMs. When a customer cancels, we generally dd their LV with 
/dev/zero, before removing the LV.

If however we want to create a snapshot of a customer's LV, is there the 
chance that the data may appear on a new future LV if we were to create 
one (for a new customer)? Is is my understanding that most filesystems 
don't actually remove data from a disk when deleting a file, but just 
set an "ignore" tag of some sort...

Thanks

Re: [linux-lvm] Snapshots and disk re-use

[Joe Thornber]

On Wed, 2011-02-23 at 12:36 +0000, Jonathan Tripathy wrote:
> Hi Everyone,
> 
> We host many Xen VMs for customers. We use LVM as the fisk storage for 
> these VMs. When a customer cancels, we generally dd their LV with 
> /dev/zero, before removing the LV.
> 
> If however we want to create a snapshot of a customer's LV, is there the 
> chance that the data may appear on a new future LV if we were to create 
> one (for a new customer)? Is is my understanding that most filesystems 
> don't actually remove data from a disk when deleting a file, but just 
> set an "ignore" tag of some sort...

I'm a bit confused about your scenario.  But I think this will help:

The snapshot device is composed of 2 sub devices, the origin and the
exception store.  Someone who only has visibility of the snapshot (ie.
your vm if your scenario is what I think it is) will have visibility of
all of the origin.  So you _do_ need to worry about data leaks on the
origin device.  Their view of the exception store is always a subset
which has been overwritten by origin data first.  So data leaks are not
an issue for the exception store.

- Joe

Re: [linux-lvm] Snapshots and disk re-use

[Sunil_Gupta2]

A file when removed from linux OS does not remove the data it just unlink the inode associated with the file which makes data unaccessible. If the disk is /dev/zeroed I don't think it will reappear.

--Sunil  

-----Original Message-----
From: linux-lvm-bounces@redhat.com [mailto:linux-lvm-bounces@redhat.com] On Behalf Of Jonathan Tripathy
Sent: Wednesday, February 23, 2011 6:06 PM
To: linux-lvm@redhat.com
Subject: [linux-lvm] Snapshots and disk re-use

Hi Everyone,

We host many Xen VMs for customers. We use LVM as the fisk storage for these VMs. When a customer cancels, we generally dd their LV with /dev/zero, before removing the LV.

If however we want to create a snapshot of a customer's LV, is there the chance that the data may appear on a new future LV if we were to create one (for a new customer)? Is is my understanding that most filesystems don't actually remove data from a disk when deleting a file, but just set an "ignore" tag of some sort...

Thanks

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Snapshots and disk re-use

[Jonathan Tripathy]

On 23/02/11 13:09, Joe Thornber wrote:
> On Wed, 2011-02-23 at 12:36 +0000, Jonathan Tripathy wrote:
>> Hi Everyone,
>>
>> We host many Xen VMs for customers. We use LVM as the fisk storage for
>> these VMs. When a customer cancels, we generally dd their LV with
>> /dev/zero, before removing the LV.
>>
>> If however we want to create a snapshot of a customer's LV, is there the
>> chance that the data may appear on a new future LV if we were to create
>> one (for a new customer)? Is is my understanding that most filesystems
>> don't actually remove data from a disk when deleting a file, but just
>> set an "ignore" tag of some sort...
> I'm a bit confused about your scenario.  But I think this will help:
>
> The snapshot device is composed of 2 sub devices, the origin and the
> exception store.  Someone who only has visibility of the snapshot (ie.
> your vm if your scenario is what I think it is) will have visibility of
> all of the origin.  So you _do_ need to worry about data leaks on the
> origin device.  Their view of the exception store is always a subset
> which has been overwritten by origin data first.  So data leaks are not
> an issue for the exception store.
>
> - Joe
>
Sorry Joe, I think I wrote my scenario in a confusing way.

The snapshot will be done in the Xen host, so the Xen host will see the 
snapshot. The VM will only see the "normal" LV (and continue to use it 
as normal). My concern is what happens when I remove the snapshot and 
later on I create a new additional LV for a new customer.

Please understand where my confusion lies. My current procedure for 
adding and removing customers is as follows:

When a customer leaves, zero their LV, then remove it
When a customer joins, create a new LV

I'm just not sure where "snapshots" fit into my above procedure

Thanks

Re: [linux-lvm] Snapshots and disk re-use

[Joe Thornber]

On Wed, 2011-02-23 at 13:57 +0000, Jonathan Tripathy wrote:

> The snapshot will be done in the Xen host, so the Xen host will see the 
> snapshot. The VM will only see the "normal" LV (and continue to use it 
> as normal). My concern is what happens when I remove the snapshot and 
> later on I create a new additional LV for a new customer.

When you remove the snapshot the exception store will have bits of the
origin device on it.  Zero it before you use the space to provision a
volume for a new customer.

- Joe

Re: [linux-lvm] Snapshots and disk re-use

[Jonathan Tripathy]

On 23/02/11 14:16, Joe Thornber wrote:
> On Wed, 2011-02-23 at 13:57 +0000, Jonathan Tripathy wrote:
>
>> The snapshot will be done in the Xen host, so the Xen host will see the
>> snapshot. The VM will only see the "normal" LV (and continue to use it
>> as normal). My concern is what happens when I remove the snapshot and
>> later on I create a new additional LV for a new customer.
> When you remove the snapshot the exception store will have bits of the
> origin device on it.  Zero it before you use the space to provision a
> volume for a new customer.
>
> - Joe
>
>
And how do I zero just exception store?

Thanks

Re: [linux-lvm] Snapshots and disk re-use

[Ray Morris]

On 23/02/11 14:16, Joe Thornber wrote:
>  And how do I zero just exception store?


In this example, the normal lv is c1_s1.
It has two snapshots, c1_s1_snap0 and c1_s1_snap1.

List the underlying volumes like so:

[root@clonebox2 ~]# ls -1  /dev/mapper/clones-c1_s1*
/dev/mapper/clones-c1_s1
/dev/mapper/clones-c1_s1-real
/dev/mapper/clones-c1_s1_snap0
/dev/mapper/clones-c1_s1_snap0-cow
/dev/mapper/clones-c1_s1_snap1
/dev/mapper/clones-c1_s1_snap1-cow

Half of those would be hidden to lvdisplay, though the man 
page will probably tell you how to show hidden LVs.  "ls"
shows them all.

c1_s1_snap0-cow and c1_s1_snap1-cow are the copy on write 
volumes, the exception stores, I believe.  -real is where 
the current data lives, and c1_s1, the "main" LV is a basically 
a proxy which reads directly from -rela, but diverts writes
so that old data is copied to -cow.  -snap0 is a proxy which 
reads from either -cow or -real as needed.  Hopefully someone 
will tell me where I'm wrong, if I am.

Zeroing c1_s1 first, then c1_s1_snap0-cow would zero everything, 
or zeroing -real and -cow would be more efficient because you 
avoid the copy on write penalty when zeroing the origin. I haven't
actually tested this, only reasoned through it.
-- 
Ray Morris
support@bettercgi.com

Strongbox - The next generation in site security:
http://www.bettercgi.com/strongbox/

Throttlebox - Intelligent Bandwidth Control
http://www.bettercgi.com/throttlebox/

Strongbox / Throttlebox affiliate program:
http://www.bettercgi.com/affiliates/user/register.php




On Wed, 23 Feb 2011 14:18:29 +0000
Jonathan Tripathy <jonnyt@abpni.co.uk> wrote:

> 

> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> 

Re: [linux-lvm] Snapshots and disk re-use

[Jonathan Tripathy]

On 23/02/11 16:12, Ray Morris wrote:
> On 23/02/11 14:16, Joe Thornber wrote:
>>   And how do I zero just exception store?
>
> In this example, the normal lv is c1_s1.
> It has two snapshots, c1_s1_snap0 and c1_s1_snap1.
>
> List the underlying volumes like so:
>
> [root@clonebox2 ~]# ls -1  /dev/mapper/clones-c1_s1*
> /dev/mapper/clones-c1_s1
> /dev/mapper/clones-c1_s1-real
> /dev/mapper/clones-c1_s1_snap0
> /dev/mapper/clones-c1_s1_snap0-cow
> /dev/mapper/clones-c1_s1_snap1
> /dev/mapper/clones-c1_s1_snap1-cow
>
> Half of those would be hidden to lvdisplay, though the man
> page will probably tell you how to show hidden LVs.  "ls"
> shows them all.
>
> c1_s1_snap0-cow and c1_s1_snap1-cow are the copy on write
> volumes, the exception stores, I believe.  -real is where
> the current data lives, and c1_s1, the "main" LV is a basically
> a proxy which reads directly from -rela, but diverts writes
> so that old data is copied to -cow.  -snap0 is a proxy which
> reads from either -cow or -real as needed.  Hopefully someone
> will tell me where I'm wrong, if I am.
>
> Zeroing c1_s1 first, then c1_s1_snap0-cow would zero everything,
Thanks for the explanation. But what if I don't want to zero everything? 
I only want to return everything back to the state it was before I 
created the snapshot.

Re: [linux-lvm] Snapshots and disk re-use

[Stuart D. Gathman]

On Wed, 23 Feb 2011, Jonathan Tripathy wrote:

> > Zeroing c1_s1 first, then c1_s1_snap0-cow would zero everything,
> Thanks for the explanation. But what if I don't want to zero everything? I
> only want to return everything back to the state it was before I created the
> snapshot.

So you're not worried about the security implication of leftovers in free
space, and just want a base image to clone for new customers?

The logical thing to do is to keep the origin volume untouched (except 
for upgrading now and then), and take a snapshot for each customer.
Each snapshot would then be a new clone of the origin.  Unfortunately,
large numbers of snapshots are inefficient for writes to new data,
so you'd likely have to "dd" to an independent LV instead.  (This is being
worked on, and there are 3rd party products like Zumastor that fix it now.)

Or maybe you want to *revert* the origin back to the state saved
in a snapshot.  A specific revert command is in the works, but
to do it manually, unmount the origin, and copy the snapshot over
the *-real hidden LV, then remove the snapshot.  This does unncessary copying
of unchanged blocks, but requires only one copy, and doesn't copy each
write back to the cow table.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Snapshots and disk re-use

[Jonathan Tripathy]

On 23/02/11 17:54, Stuart D. Gathman wrote:
> On Wed, 23 Feb 2011, Jonathan Tripathy wrote:
>
>>> Zeroing c1_s1 first, then c1_s1_snap0-cow would zero everything,
>> Thanks for the explanation. But what if I don't want to zero everything? I
>> only want to return everything back to the state it was before I created the
>> snapshot.
> So you're not worried about the security implication of leftovers in free
> space, and just want a base image to clone for new customers?
Yes I am. This is exactly what I'm worried about. I just need to know 
how to zero out any "left over" bits of data that would be left over 
after removing a snapshot. I use snapshots for backing up a volume (by 
running rsync against the snapshot to a remote server)

Re: [linux-lvm] Snapshots and disk re-use

[Stuart D. Gathman]

On Wed, 23 Feb 2011, Stuart D. Gathman wrote:

> So you're not worried about the security implication of leftovers in free
> space, and just want a base image to clone for new customers?
> 
> The logical thing to do is to keep the origin volume untouched (except 
> for upgrading now and then), and take a snapshot for each customer.
> Each snapshot would then be a new clone of the origin.  Unfortunately,
> large numbers of snapshots are inefficient for writes to new data,
> so you'd likely have to "dd" to an independent LV instead.  (This is being
> worked on, and there are 3rd party products like Zumastor that fix it now.)

Actually, if you never (or rarely) write to the origin, lots of snapshots
should be fine.  But every write to the origin will first copy the 
original origin data to every snapshot.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Snapshots and disk re-use

[Jonathan Tripathy]

On 23/02/11 18:05, Stuart D. Gathman wrote:
> On Wed, 23 Feb 2011, Stuart D. Gathman wrote:
>
>> So you're not worried about the security implication of leftovers in free
>> space, and just want a base image to clone for new customers?
>>
>> The logical thing to do is to keep the origin volume untouched (except
>> for upgrading now and then), and take a snapshot for each customer.
>> Each snapshot would then be a new clone of the origin.  Unfortunately,
>> large numbers of snapshots are inefficient for writes to new data,
>> so you'd likely have to "dd" to an independent LV instead.  (This is being
>> worked on, and there are 3rd party products like Zumastor that fix it now.)
> Actually, if you never (or rarely) write to the origin, lots of snapshots
> should be fine.

> But every write to the origin will first copy the
> original origin data to every snapshot.
>
Why would origin data be copied over to the snapshot after the snapshot 
has been created? Surely the point of a snapshot is to have "frozen" data?

Re: [linux-lvm] Snapshots and disk re-use

[Les Mikesell]

On 2/23/2011 12:19 PM, Jonathan Tripathy wrote:
>
>>> So you're not worried about the security implication of leftovers in
>>> free
>>> space, and just want a base image to clone for new customers?
>>>
>>> The logical thing to do is to keep the origin volume untouched (except
>>> for upgrading now and then), and take a snapshot for each customer.
>>> Each snapshot would then be a new clone of the origin. Unfortunately,
>>> large numbers of snapshots are inefficient for writes to new data,
>>> so you'd likely have to "dd" to an independent LV instead. (This is
>>> being
>>> worked on, and there are 3rd party products like Zumastor that fix it
>>> now.)
>> Actually, if you never (or rarely) write to the origin, lots of snapshots
>> should be fine.
>
>> But every write to the origin will first copy the
>> original origin data to every snapshot.
>>
> Why would origin data be copied over to the snapshot after the snapshot
> has been created? Surely the point of a snapshot is to have "frozen" data?

Yes, is the way this actually works explained somewhere?  I would have 
expected the 'copy-on-write' blocks to be copied only on the side where 
the write is happening and relocated instead of rewriting all the 
snapshots that might be outstanding with the old data.

-- 
   Les Mikesell
     lesmikesell@gmail.com

[linux-lvm] Problem executing lvm related commands

[Tinni]

Hi All,

 Need the suggession for the folling issues :

   Operating  System : rhel 5.5 ,  86_64 

  In our environment,  16 nodes are in cluster and shared storage is being used.
  Multipath is used, and there was no aliases.  The  admins  created the disk,
  with the /dev/dm....

  Now they changed the  multipath  config  file and incorporated  the aliases.

  The alias names are as follows:

             HOST-1TB-CLVM-VG04-DISK1
             HOST-1TB-CLVM-VG02-DISK1

  and so on. 

  They  use  the filter  in lvm.conf  as   filter = [ "r/scsi.*/","r/pci.*/","r/sd.*/","a/.*/" ]

  They did not enable  the  "preferred_names"   option in lvm.conf


  Now the problem is, lvm related  any commands are getting hung. 

  While debugging the issue I activated the "preferred_names"  option   in lvm.conf

  and found the errors  as follows:

     [root@server ~]# lvs
  File descriptor 4 (/dev/pts/0) leaked on lvs invocation. Parent PID 20817: bash
  WARNING: Ignoring duplicate config node: preferred_names (seeking preferred_names)

I mentioned literally all the vgnames  under "preferred_names"  in lvm.conf but  still the 
errors are same. 

If anybody gives some suggesstion , how to rectify the issue  that  will be really helpful.


Thanks  in advance

Re: [linux-lvm] Snapshots and disk re-use

[Stuart D. Gathman]

On Wed, 23 Feb 2011, Jonathan Tripathy wrote:

> > Actually, if you never (or rarely) write to the origin, lots of snapshots
> > should be fine.
> 
> > But every write to the origin will first copy the
> > original origin data to every snapshot.
> >
> Why would origin data be copied over to the snapshot after the snapshot has
> been created? Surely the point of a snapshot is to have "frozen" data?

That is how the data is "frozen" despite being written to on the origin.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Snapshots and disk re-use

[Phillip Susi]

On 2/23/2011 12:54 PM, Stuart D. Gathman wrote:
> Or maybe you want to *revert* the origin back to the state saved
> in a snapshot.  A specific revert command is in the works, but
> to do it manually, unmount the origin, and copy the snapshot over

That is lvconvert --merge, and it has been around for a while now.  It
is quite handy for testing a dist-upgrade and then rolling back.

Re: [linux-lvm] Snapshots and disk re-use

[Stuart D. Gathman]

On Wed, 23 Feb 2011, Jonathan Tripathy wrote:

> On 23/02/11 17:54, Stuart D. Gathman wrote:
> > On Wed, 23 Feb 2011, Jonathan Tripathy wrote:
> >
> > > > Zeroing c1_s1 first, then c1_s1_snap0-cow would zero everything,
> > > Thanks for the explanation. But what if I don't want to zero everything? I
> > > only want to return everything back to the state it was before I created
> > > the
> > > snapshot.
> > So you're not worried about the security implication of leftovers in free
> > space, and just want a base image to clone for new customers?
> Yes I am. This is exactly what I'm worried about. I just need to know how to
> zero out any "left over" bits of data that would be left over after removing a
> snapshot. I use snapshots for backing up a volume (by running rsync against
> the snapshot to a remote server)

If the origin is not mounted, you can just copy /dev/zero to the *-cow, then
delete the snapshot.  I am a little concerned whether that is safe while the
origin is mounted, however.  Perhaps an LVM expert can elucidate.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Snapshots and disk re-use

[Stuart D. Gathman]

On Wed, 23 Feb 2011, Les Mikesell wrote:

> > > Actually, if you never (or rarely) write to the origin, lots of snapshots
> > > should be fine.
> >
> > > But every write to the origin will first copy the
> > > original origin data to every snapshot.
> > >
> > Why would origin data be copied over to the snapshot after the snapshot
> > has been created? Surely the point of a snapshot is to have "frozen" data?
> 
> Yes, is the way this actually works explained somewhere?  I would have
> expected the 'copy-on-write' blocks to be copied only on the side where the
> write is happening and relocated instead of rewriting all the snapshots that
> might be outstanding with the old data.

It should be obvious when you think about it.  You go to write to the origin, 
to a block you've never written to since creating the snapshot.  Where
are you going to save that original data?  Why in the *-cow LV of course.

The origin and snapshot LVs do not have data of their own.  The
origin data is in *-real, and new writes to the origin first read *-real,
write *-cow, then finally do the new write to *-real.  This read-write-write
cycle is tolerable for a single snapshot, but quickly gets out of hand 
with multiple snapshots.

Writes to the snapshot, on the other hand, simply write to the *-cow with
some mapping, no need to update the origin.

Reads from origin simply read from *-real.  Reads from snapshot, read
from *-cow if mapped, and from *-real otherwise.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Snapshots and disk re-use

[Stuart D. Gathman]

On Wed, 23 Feb 2011, Phillip Susi wrote:

> On 2/23/2011 12:54 PM, Stuart D. Gathman wrote:
> > Or maybe you want to *revert* the origin back to the state saved
> > in a snapshot.  A specific revert command is in the works, but
> > to do it manually, unmount the origin, and copy the snapshot over
> 
> That is lvconvert --merge, and it has been around for a while now.  It
> is quite handy for testing a dist-upgrade and then rolling back.

Yes, I see that it is in Fedora, but not EL5.5.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Snapshots and disk re-use

[Nataraj]

On 02/23/2011 10:39 AM, Les Mikesell wrote:
> On 2/23/2011 12:19 PM, Jonathan Tripathy wrote:
>>
>>>> So you're not worried about the security implication of leftovers in
>>>> free
>>>> space, and just want a base image to clone for new customers?
>>>>
>>>> The logical thing to do is to keep the origin volume untouched (except
>>>> for upgrading now and then), and take a snapshot for each customer.
>>>> Each snapshot would then be a new clone of the origin. Unfortunately,
>>>> large numbers of snapshots are inefficient for writes to new data,
>>>> so you'd likely have to "dd" to an independent LV instead. (This is
>>>> being
>>>> worked on, and there are 3rd party products like Zumastor that fix it
>>>> now.)
>>> Actually, if you never (or rarely) write to the origin, lots of
>>> snapshots
>>> should be fine.
>>
>>> But every write to the origin will first copy the
>>> original origin data to every snapshot.
>>>
>> Why would origin data be copied over to the snapshot after the snapshot
>> has been created? Surely the point of a snapshot is to have "frozen"
>> data?
The content of the snapshot is not changing. Previously the data in the
snapshot was a pointer to the origin.  When you rewrite data in the
origin, the old data gets moved to the cow device which is owned by the
snapshot, not by the origin.  So the view of the snapshot does not
change, only that for the underlying data that was changed in the
origin, the previous data is moved to the snapshot's cow and now the
snapshot points to that data in the cow instead of in the origin volume.

 I would be careful of using the word frozen, because you can actually
make changes to the snapshot volume if you wanted to, so it is not
really frozen.

>
> Yes, is the way this actually works explained somewhere?  I would have
> expected the 'copy-on-write' blocks to be copied only on the side
> where the write is happening and relocated instead of rewriting all
> the snapshots that might be outstanding with the old data.
>
I don't know for sure, but it sounds like it is currently done this
way.  Ideally there could be some kind of data sharing between multiple
snapshots.  I suspect the data gets copied (instead of just changing
pointers) to avoid excessive fragmentation in the origin volume.

Nataraj

Re: [linux-lvm] Snapshots and disk re-use

[Nataraj]

On 02/23/2011 11:45 AM, Stuart D. Gathman wrote:
> On Wed, 23 Feb 2011, Phillip Susi wrote:
>
>> On 2/23/2011 12:54 PM, Stuart D. Gathman wrote:
>>> Or maybe you want to *revert* the origin back to the state saved
>>> in a snapshot.  A specific revert command is in the works, but
>>> to do it manually, unmount the origin, and copy the snapshot over
>> That is lvconvert --merge, and it has been around for a while now.  It
>> is quite handy for testing a dist-upgrade and then rolling back.
> Yes, I see that it is in Fedora, but not EL5.5.
>
It is in EL6.0 as well.  From what I can tell, and please correct me if
this is not the case, I believe there are no changes to the lvm
structure, so you could take a redhat 5 system and boot a redhat 6
recovery disk or live CD and use lvconvert --merge to revert to the
origin to a snapshot.  You probably need to reboot anyway in order to
use 'lvconvert --merge', since you can't revert an active volume.

Nataraj

Re: [linux-lvm] Snapshots and disk re-use

[Les Mikesell]

On 2/23/2011 1:39 PM, Stuart D. Gathman wrote:
> On Wed, 23 Feb 2011, Les Mikesell wrote:
>
>>>> Actually, if you never (or rarely) write to the origin, lots of snapshots
>>>> should be fine.
>>>
>>>> But every write to the origin will first copy the
>>>> original origin data to every snapshot.
>>>>
>>> Why would origin data be copied over to the snapshot after the snapshot
>>> has been created? Surely the point of a snapshot is to have "frozen" data?
>>
>> Yes, is the way this actually works explained somewhere?  I would have
>> expected the 'copy-on-write' blocks to be copied only on the side where the
>> write is happening and relocated instead of rewriting all the snapshots that
>> might be outstanding with the old data.
>
> It should be obvious when you think about it.

Well, maybe if you've never used a NetApp.  They seem to make a big deal 
out of only writing once.

> You go to write to the origin,
> to a block you've never written to since creating the snapshot.  Where
> are you going to save that original data?  Why in the *-cow LV of course.
>
> The origin and snapshot LVs do not have data of their own.  The
> origin data is in *-real, and new writes to the origin first read *-real,
> write *-cow, then finally do the new write to *-real.  This read-write-write
> cycle is tolerable for a single snapshot, but quickly gets out of hand
> with multiple snapshots.

Which is why re-writing stuff that was already correct and not being 
modified doesn't make much sense.

> Writes to the snapshot, on the other hand, simply write to the *-cow with
> some mapping, no need to update the origin.

I would have guessed that snapshots were read-only. If you can write, it 
isn't really a snapshot, is it?

-- 
   Les Mikesell
     lesmikesell@gmail.com

Re: [linux-lvm] Snapshots and disk re-use

[Stuart D. Gathman]

On Wed, 23 Feb 2011, Les Mikesell wrote:

> Which is why re-writing stuff that was already correct and not being modified
> doesn't make much sense.

So set the origin to readonly.

> > Writes to the snapshot, on the other hand, simply write to the *-cow with
> > some mapping, no need to update the origin.
> 
> I would have guessed that snapshots were read-only. If you can write, it isn't
> really a snapshot, is it?

So set the snapshot to readonly.  

But to be useful, you need to write to one or the other.  Using the origin as a
(mostly) readonly "template" for many VM customers works efficiently with VMs
running in snapshots.  In that case, writes to the snapshots are just one
write.  You *could* consider the origin a "snapshot" in that case.  But then
you are defining "snapshot" to be the readonly branch, which is a different
sense than how LVM uses the term.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Snapshots and disk re-use

[Jonathan Tripathy]

> But to be useful, you need to write to one or the other.  Using the origin as a
> (mostly) readonly "template" for many VM customers works efficiently with VMs
> running in snapshots.  In that case, writes to the snapshots are just one
> write.  You *could* consider the origin a "snapshot" in that case.  But then
> you are defining "snapshot" to be the readonly branch, which is a different
> sense than how LVM uses the term.
>
This is really eye opening to the useful of LVM snapshots. I never once 
considered to use them in this way :)

But back to my original question. Let me try and word it a bit better.

Give that I currently follow the current procedure for removing and 
adding customers:

To remove customer: zero out customer LV, then remove LV
To add customer: create a new LV

And I want to run backups of a customer's LV using snapshots I just:
create a snapshot of the customer LV, then use rsync, then remove the 
snapshot. Is there anything I should do to prevent cross-customer data 
leakage?

Thanks

Re: [linux-lvm] Snapshots and disk re-use

[Stuart D. Gathman]

On Wed, 23 Feb 2011, Jonathan Tripathy wrote:

> Give that I currently follow the current procedure for removing and adding
> customers:
> 
> To remove customer: zero out customer LV, then remove LV
> To add customer: create a new LV
> 
> And I want to run backups of a customer's LV using snapshots I just:
> create a snapshot of the customer LV, then use rsync, then remove the
> snapshot. Is there anything I should do to prevent cross-customer data
> leakage?

You are still ambiguous.  If by "create a new LV", you mean a new LV
that is not an LVM snapshot, then just zero it out when you create it
(rather than when you delete it).

If by "create a new LV", you mean create a new snapshot on a (mostly)
readonly origin, then you don't need to do anything.  The entire logical
volume is (logically) initialized to the origin.  There is no need to
zero when deleting either.

If, in addition to "cross-customer data leakage", you are worried about
FBI raids finding old data from deleted customers in LVM free space, then zero
independent LVs when deleting (as you planned), and zero *-cow for snapshots
before deleting with the origin unmounted (but check with an expert whether
that is safe while other snapshots for the same origin are mounted - I think it
should be).

IDEA - it seems that the device mapper could logically zero an LV by
simply returning blocks of zero on reads until the corresponding block
it written.  Yeah, would require overhead to track which blocks have
been written.  That overhead could be 1 bit for each of fairly large blocks,
and be fairly small, fit into ram easily, and be stored in a logically
zeroed block and discarded when the last block is written.  So effectively
it only requires storing a pointer in meta-data to the current block where
the bitmap is stored.  I can see that compared to the simplicity of
simply writing zeroes on allocation, it might not be worth it.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Snapshots and disk re-use

[Stuart D. Gathman]

On Wed, 23 Feb 2011, Stuart D. Gathman wrote:

> If, in addition to "cross-customer data leakage", you are worried about
> FBI raids finding old data from deleted customers in LVM free space, then zero
> independent LVs when deleting (as you planned), and zero *-cow for snapshots
> before deleting with the origin unmounted (but check with an expert whether
> that is safe while other snapshots for the same origin are mounted - I think
> it should be).

Also, if worried about free space leakage, you should periodically allocate a
(huge) LV that uses all remaining space, and zero it.  That will clean up
after any time an operator may not have followed the procedure.  

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Snapshots and disk re-use

[Jonathan Tripathy]

On 23/02/11 23:25, Stuart D. Gathman wrote:
> On Wed, 23 Feb 2011, Jonathan Tripathy wrote:
>
>> Give that I currently follow the current procedure for removing and adding
>> customers:
>>
>> To remove customer: zero out customer LV, then remove LV
>> To add customer: create a new LV
>>
>> And I want to run backups of a customer's LV using snapshots I just:
>> create a snapshot of the customer LV, then use rsync, then remove the
>> snapshot. Is there anything I should do to prevent cross-customer data
>> leakage?
> You are still ambiguous.  If by "create a new LV", you mean a new LV
> that is not an LVM snapshot, then just zero it out when you create it
> (rather than when you delete it).
This is exactly what I mean. However I was hoping to avoid zeroing an LV 
on creation due to the long amount of time it takes to perform such a 
task (about 40 mins). It takes so long as I have to select a small 
enough block size so that disk bandwidth is still available to the other 
customers using their VPSes.

This is why I'm trying to understand where data is stored while a 
snapshot is created for an origin. All my snapshots will be read only.

So I'm guessing then, that when a snapshot is created for an origin, 
then there are 2 physical copies of the data on disk? (Albeit only one 
is accessible at the regular filesystem level)

> IDEA - it seems that the device mapper could logically zero an LV by
> simply returning blocks of zero on reads until the corresponding block
> it written.  Yeah, would require overhead to track which blocks have
> been written.  That overhead could be 1 bit for each of fairly large blocks,
> and be fairly small, fit into ram easily, and be stored in a logically
> zeroed block and discarded when the last block is written.  So effectively
> it only requires storing a pointer in meta-data to the current block where
> the bitmap is stored.  I can see that compared to the simplicity of
> simply writing zeroes on allocation, it might not be worth it.
>
This sounds like a really good idea. Would be very useful in a 
multi-tenant environment. However it should be an option feature.

Re: [linux-lvm] Snapshots and disk re-use

[Stuart D. Gathman]

On Thu, 24 Feb 2011, Jonathan Tripathy wrote:

> So I'm guessing then, that when a snapshot is created for an origin, then
> there are 2 physical copies of the data on disk? (Albeit only one is
> accessible at the regular filesystem level)

NO, NO, NO.  There is still only *one* physical copy of the data
after creating a snapshot.  You have simply created a "branch point"
which can now diverge as each branch is written to.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Snapshots and disk re-use

[Jonathan Tripathy]

On 24/02/11 00:32, Stuart D. Gathman wrote:
> On Thu, 24 Feb 2011, Jonathan Tripathy wrote:
>
>> So I'm guessing then, that when a snapshot is created for an origin, then
>> there are 2 physical copies of the data on disk? (Albeit only one is
>> accessible at the regular filesystem level)
> NO, NO, NO.  There is still only *one* physical copy of the data
> after creating a snapshot.  You have simply created a "branch point"
> which can now diverge as each branch is written to.
>
Then why was it suggested that I should zero my new customer LVs upon 
creation? Please remember that my snapshots will not be written to

Thanks

Re: [linux-lvm] Snapshots and disk re-use

[Jonathan Tripathy]

On 24/02/11 00:37, Jonathan Tripathy wrote:
>
> On 24/02/11 00:32, Stuart D. Gathman wrote:
>> On Thu, 24 Feb 2011, Jonathan Tripathy wrote:
>>
>>> So I'm guessing then, that when a snapshot is created for an origin, 
>>> then
>>> there are 2 physical copies of the data on disk? (Albeit only one is
>>> accessible at the regular filesystem level)
>> NO, NO, NO.  There is still only *one* physical copy of the data
>> after creating a snapshot.  You have simply created a "branch point"
>> which can now diverge as each branch is written to.
>>
> Then why was it suggested that I should zero my new customer LVs upon 
> creation? Please remember that my snapshots will not be written to
>
However the origin will be written to as it will be in use...

Re: [linux-lvm] Snapshots and disk re-use

[Stuart D. Gathman]

On Thu, 24 Feb 2011, Jonathan Tripathy wrote:

> > > > So I'm guessing then, that when a snapshot is created for an origin,
> > > > then
> > > > there are 2 physical copies of the data on disk? (Albeit only one is
> > > > accessible at the regular filesystem level)
> > > NO, NO, NO.  There is still only *one* physical copy of the data
> > > after creating a snapshot.  You have simply created a "branch point"
> > > which can now diverge as each branch is written to.
> > >
> > Then why was it suggested that I should zero my new customer LVs upon
> > creation? Please remember that my snapshots will not be written to
> >
> However the origin will be written to as it will be in use...

There were 3 cases of what you might be asking.  One of the 3 cases was:
If you are taking snapshots for backup, then it was suggested to zero the *-cow
(which will have any blocks written to the origin since the snapshot was taken)
before deleting the snapshot.  However, I wasn't sure if this was safe to while
the origin is mounted, since writes to the origin consult the *-cow to see
whether origin blocks need to be copied before begin overwritten.

IDEA - this could be done safely in lvremove by disconnecting snapshot
from origin, then optionally zeroing the *-cow before finally removing it.

The option could be as simple as a --keep option to keep the *-cow around
as a normal LV (under a new name, perhaps) for the admin to run shred or copy
zeros to before finally removing it.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Snapshots and disk re-use

[Jonathan Tripathy]

On 24/02/11 02:00, Stuart D. Gathman wrote:
> On Thu, 24 Feb 2011, Jonathan Tripathy wrote:
>
>>>>> So I'm guessing then, that when a snapshot is created for an origin,
>>>>> then
>>>>> there are 2 physical copies of the data on disk? (Albeit only one is
>>>>> accessible at the regular filesystem level)
>>>> NO, NO, NO.  There is still only *one* physical copy of the data
>>>> after creating a snapshot.  You have simply created a "branch point"
>>>> which can now diverge as each branch is written to.
>>>>
>>> Then why was it suggested that I should zero my new customer LVs upon
>>> creation? Please remember that my snapshots will not be written to
>>>
>> However the origin will be written to as it will be in use...
> There were 3 cases of what you might be asking.  One of the 3 cases was:
> If you are taking snapshots for backup, then it was suggested to zero the *-cow
> (which will have any blocks written to the origin since the snapshot was taken)
> before deleting the snapshot.  However, I wasn't sure if this was safe to while
> the origin is mounted, since writes to the origin consult the *-cow to see
> whether origin blocks need to be copied before begin overwritten.
But I want to keep this data. As the data written to the origin while a 
respective snapshot is in use would be data during normal operation of a 
customer VPS.

All I want to do is make sure that my use of snapshot for backup 
purposes (to rsync the snapshot to a remote server) doesn't leave loose 
data lying anywhere on physical disk.

Re: [linux-lvm] Snapshots and disk re-use

[Stuart D. Gathman]

On Thu, 24 Feb 2011, Jonathan Tripathy wrote:

> > > However the origin will be written to as it will be in use...
> > There were 3 cases of what you might be asking.  One of the 3 cases was:
> > If you are taking snapshots for backup, then it was suggested to zero the
> > *-cow (which will have any blocks written to the origin since the snapshot
> > was taken) before deleting the snapshot.  However, I wasn't sure if this
> > was safe to while the origin is mounted, since writes to the origin consult
> > the *-cow to see whether origin blocks need to be copied before begin
> > overwritten.

> But I want to keep this data. As the data written to the origin while a
> respective snapshot is in use would be data during normal operation of a
> customer VPS.
> 
> All I want to do is make sure that my use of snapshot for backup purposes (to
> rsync the snapshot to a remote server) doesn't leave loose data lying anywhere
> on physical disk.

Yes. To be more pedantic, the COW has copies of the original contents
of blocks written to the origin since the snapshot.  That is why you
need to clear it to achieve your stated purpose.  The origin blocks
are written normally to the *-real volume (you can see these in
the /dev/mapper directory).

I am only concerned that since the COW has both the saved origin blocks, and a
table of which blocks from the origin have been saved, that clearing
the table might cause problems for an active origin.  It should be safe
with the origin unmounted (but having to shutdown the VM defeats the
purpose of a snapshot!)  I am hoping an expert will weigh in.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Snapshots and disk re-use

[Jonathan Tripathy]

On 24/02/11 14:50, Stuart D. Gathman wrote:
> On Thu, 24 Feb 2011, Jonathan Tripathy wrote:
>
>>>> However the origin will be written to as it will be in use...
>>> There were 3 cases of what you might be asking.  One of the 3 cases was:
>>> If you are taking snapshots for backup, then it was suggested to zero the
>>> *-cow (which will have any blocks written to the origin since the snapshot
>>> was taken) before deleting the snapshot.  However, I wasn't sure if this
>>> was safe to while the origin is mounted, since writes to the origin consult
>>> the *-cow to see whether origin blocks need to be copied before begin
>>> overwritten.
>> But I want to keep this data. As the data written to the origin while a
>> respective snapshot is in use would be data during normal operation of a
>> customer VPS.
>>
>> All I want to do is make sure that my use of snapshot for backup purposes (to
>> rsync the snapshot to a remote server) doesn't leave loose data lying anywhere
>> on physical disk.
> Yes. To be more pedantic, the COW has copies of the original contents
> of blocks written to the origin since the snapshot.  That is why you
> need to clear it to achieve your stated purpose.  The origin blocks
> are written normally to the *-real volume (you can see these in
> the /dev/mapper directory).
But didn't you say that there is only one copy of the files stored 
physically on disk?
> I am only concerned that since the COW has both the saved origin blocks, and a
> table of which blocks from the origin have been saved, that clearing
> the table might cause problems for an active origin.  It should be safe
> with the origin unmounted (but having to shutdown the VM defeats the
> purpose of a snapshot!)  I am hoping an expert will weigh in.
>

Re: [linux-lvm] Snapshots and disk re-use

[Stuart D. Gathman]

On Thu, 24 Feb 2011, Jonathan Tripathy wrote:

> > Yes. To be more pedantic, the COW has copies of the original contents
> > of blocks written to the origin since the snapshot.  That is why you
> > need to clear it to achieve your stated purpose.  The origin blocks
> > are written normally to the *-real volume (you can see these in
> > the /dev/mapper directory).
> But didn't you say that there is only one copy of the files stored physically
> on disk?

Yes.  When you make the snapshot, there is only one copy, and the COW table
is empty.  AS YOU WRITE to the origin, each chunk written is saved to
*-cow first before being written to *-real.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Snapshots and disk re-use

[Jonathan Tripathy]

On 24/02/11 15:13, Stuart D. Gathman wrote:
> On Thu, 24 Feb 2011, Jonathan Tripathy wrote:
>
>>> Yes. To be more pedantic, the COW has copies of the original contents
>>> of blocks written to the origin since the snapshot.  That is why you
>>> need to clear it to achieve your stated purpose.  The origin blocks
>>> are written normally to the *-real volume (you can see these in
>>> the /dev/mapper directory).
>> But didn't you say that there is only one copy of the files stored physically
>> on disk?
> Yes.  When you make the snapshot, there is only one copy, and the COW table
> is empty.  AS YOU WRITE to the origin, each chunk written is saved to
> *-cow first before being written to *-real.
Got ya. So data that is being written to the origin, while the snapshot 
exists, is the data that may leak, as it's saved to the COW first, then 
copied over to real.

Hopefully an expert will let me know weather its safe to zero the COW 
after I�ve finished with the snapshot.

Re: [linux-lvm] Snapshots and disk re-use

[Jonathan Tripathy]

On 24/02/11 15:20, Jonathan Tripathy wrote:
>
> On 24/02/11 15:13, Stuart D. Gathman wrote:
>> On Thu, 24 Feb 2011, Jonathan Tripathy wrote:
>>
>>>> Yes. To be more pedantic, the COW has copies of the original contents
>>>> of blocks written to the origin since the snapshot.  That is why you
>>>> need to clear it to achieve your stated purpose.  The origin blocks
>>>> are written normally to the *-real volume (you can see these in
>>>> the /dev/mapper directory).
>>> But didn't you say that there is only one copy of the files stored 
>>> physically
>>> on disk?
>> Yes.  When you make the snapshot, there is only one copy, and the COW 
>> table
>> is empty.  AS YOU WRITE to the origin, each chunk written is saved to
>> *-cow first before being written to *-real.
> Got ya. So data that is being written to the origin, while the 
> snapshot exists, is the data that may leak, as it's saved to the COW 
> first, then copied over to real.
>
> Hopefully an expert will let me know weather its safe to zero the COW 
> after I�ve finished with the snapshot.
>
>
Oh, I forgot to mention. My LVM sits on top of linux software RAID1. 
Does that complicate things at all?

Re: [linux-lvm] Snapshots and disk re-use

[Nataraj]

On 02/24/2011 08:41 AM, Jonathan Tripathy wrote:
>
> On 24/02/11 15:20, Jonathan Tripathy wrote:
>>
>> On 24/02/11 15:13, Stuart D. Gathman wrote:
>>> On Thu, 24 Feb 2011, Jonathan Tripathy wrote:
>>>
>>>>> Yes. To be more pedantic, the COW has copies of the original contents
>>>>> of blocks written to the origin since the snapshot.  That is why you
>>>>> need to clear it to achieve your stated purpose.  The origin blocks
>>>>> are written normally to the *-real volume (you can see these in
>>>>> the /dev/mapper directory).
>>>> But didn't you say that there is only one copy of the files stored
>>>> physically
>>>> on disk?
>>> Yes.  When you make the snapshot, there is only one copy, and the
>>> COW table
>>> is empty.  AS YOU WRITE to the origin, each chunk written is saved to
>>> *-cow first before being written to *-real.
>> Got ya. So data that is being written to the origin, while the
>> snapshot exists, is the data that may leak, as it's saved to the COW
>> first, then copied over to real.
>>
>> Hopefully an expert will let me know weather its safe to zero the COW
>> after I�ve finished with the snapshot.
>>
>>
> Oh, I forgot to mention. My LVM sits on top of linux software RAID1.
> Does that complicate things at all?
>

One other thing that wasn't mentioned here.... As far as I understand,
if these are lvms on the host and the snapshots are being taken on the
host, there is no guaranteed integrity of the filesystems unless you
shutdown the guest while the snapshot is taken.

It's too bad they don't implement a system call that could do something
like sync filesystem and sleep until I tell you to continue.  This would
be perfect for snapshot backups of virtual hosts.

Nataraj

Re: [linux-lvm] Snapshots and disk re-use

[Stuart D. Gathman]

On Thu, 24 Feb 2011, Jonathan Tripathy wrote:

> Oh, I forgot to mention. My LVM sits on top of linux software RAID1. Does that
> complicate things at all?

Nope.  If it were RAID5, or other scheme with a RWM cycle, you would need to
ensure alignment between RAID chunks and LVM chunks for good performance, but
RAID1 does not have that problem.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Snapshots and disk re-use

[Les Mikesell]

On 2/24/2011 1:15 PM, Nataraj wrote:
>
> One other thing that wasn't mentioned here.... As far as I understand,
> if these are lvms on the host and the snapshots are being taken on the
> host, there is no guaranteed integrity of the filesystems unless you
> shutdown the guest while the snapshot is taken.
>
> It's too bad they don't implement a system call that could do something
> like sync filesystem and sleep until I tell you to continue.  This would
> be perfect for snapshot backups of virtual hosts.

Syncing a filesystem's OS buffers isn't really enough to ensure that 
running apps have saved their data in a consistent state.  The best you 
are going to get without shuting things down is more or less what you'd 
have if the server crashed at some point in time.

-- 
   Les Mikesell
    lesmikesell@gmail.com

Re: [linux-lvm] Snapshots and disk re-use

[Stuart D. Gathman]

On Thu, 24 Feb 2011, Jonathan Tripathy wrote:

> > Yes.  When you make the snapshot, there is only one copy, and the COW table
> > is empty.  AS YOU WRITE to the origin, each chunk written is saved to
> > *-cow first before being written to *-real.
> Got ya. So data that is being written to the origin, while the snapshot
> exists, is the data that may leak, as it's saved to the COW first, then copied
> over to real.
> 
> Hopefully an expert will let me know weather its safe to zero the COW after
> I've finished with the snapshot.

What *is* safe is to zero the snapshot.  This will overwrite any blocks
in the COW copied from the origin.  The problem is that if the snapshot runs
out of room, it is invalidated, and you may or may not have overwritten
all blocks copied from the origin.

So if you don't hear from an expert, a safe prodecure is to allocate 
snapshots for backup that are as big as the origin + 1 PP (which should
be enough for the COW table as well unless we are talking terabytes).  Then you
can zero the snapshot (not the COW) after making a backup. That will overwrite
any data copied from the origin.  The only leftover data will just be the COW
table which is a bunch of block #s, so shouldn't be a security problem.

This procedure is less efficient than zeroing LVs on allocation, and takes
extra space for worst case snapshot allocation.  But if you want allocation
to be "instant", and can pay for it when recycling, that should solve your
problem.  You should still zero all free space (by allocating a huge LV
with all remaining space and zeroing it) periodically in case anything
got missed.

IDEA, since you are on raid1, reads are faster than writes (twice as fast),
and your snapshots will be mostly unused (the COW will only have a few blocks
copied from the origin).  So you can write a "clear" utility that scans 
a block device for non-zero chunks, and only writes over those with zeros.
This might be a good application for mmap().  

Furthermore, if you have 3 copies of the data in your raid1 system, then reads
are 3 times as fast, and the clear utility might be fast enough to use at
allocation time - which is simple and safe.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Snapshots and disk re-use

[Stuart D. Gathman]

On Thu, 24 Feb 2011, Nataraj wrote:

> One other thing that wasn't mentioned here.... As far as I understand,
> if these are lvms on the host and the snapshots are being taken on the
> host, there is no guaranteed integrity of the filesystems unless you
> shutdown the guest while the snapshot is taken.
> 
> It's too bad they don't implement a system call that could do something
> like sync filesystem and sleep until I tell you to continue.  This would
> be perfect for snapshot backups of virtual hosts.

The hosting service backup script can attempt to send a message to 
the VM on a network port before taking the snapshot.  If the VM
accepts the connection and implements the handshake API, it will respond with
a "WAIT A SEC", tell databases to sync and suspend writes, then
tell the host to "GO AHEAD".  The host then takes the snapshot and
tells the VM, "ALL DONE, CARRY ON".

This network API ought to be standardized.  In fact, I am just about
to write this API for a client to reduce their backup window from
6 minutes to a few seconds.  If I am reinventing the wheel here,
please let me know!  If not, I'll share here.  Code will likely be python.

For many applications, filesystem journalling is sufficient for snapshots
to be useful without further involvement.  But when database are involved,
you often need the handshake.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Snapshots and disk re-use

[Jonathan Tripathy]

On 24/02/11 19:45, Stuart D. Gathman wrote:
> On Thu, 24 Feb 2011, Jonathan Tripathy wrote:
>
>>> Yes.  When you make the snapshot, there is only one copy, and the COW table
>>> is empty.  AS YOU WRITE to the origin, each chunk written is saved to
>>> *-cow first before being written to *-real.
>> Got ya. So data that is being written to the origin, while the snapshot
>> exists, is the data that may leak, as it's saved to the COW first, then copied
>> over to real.
>>
>> Hopefully an expert will let me know weather its safe to zero the COW after
>> I've finished with the snapshot.
> What *is* safe is to zero the snapshot.  This will overwrite any blocks
> in the COW copied from the origin.  The problem is that if the snapshot runs
> out of room, it is invalidated, and you may or may not have overwritten
> all blocks copied from the origin.
>
> So if you don't hear from an expert, a safe prodecure is to allocate
> snapshots for backup that are as big as the origin + 1 PP (which should
> be enough for the COW table as well unless we are talking terabytes).  Then you
> can zero the snapshot (not the COW) after making a backup. That will overwrite
> any data copied from the origin.  The only leftover data will just be the COW
> table which is a bunch of block #s, so shouldn't be a security problem.
>
> This procedure is less efficient than zeroing LVs on allocation, and takes
> extra space for worst case snapshot allocation.  But if you want allocation
> to be "instant", and can pay for it when recycling, that should solve your
> problem.  You should still zero all free space (by allocating a huge LV
> with all remaining space and zeroing it) periodically in case anything
> got missed.
Hmm this sounds like it would work. However I would rather zero the LVs 
on allocation than do this, as we would run many backups, and it would 
be highly inefficient to zero out all the snapshots (unless I made the 
snapshot really small, but that would cause other problems, wouldn't it?)


> IDEA, since you are on raid1, reads are faster than writes (twice as fast),
> and your snapshots will be mostly unused (the COW will only have a few blocks
> copied from the origin).  So you can write a "clear" utility that scans
> a block device for non-zero chunks, and only writes over those with zeros.
> This might be a good application for mmap().
This would be a fantastic idea. Since LVM is very commonily used in 
shared tennant environments, if would be a fantastic feature if LVM 
could make sure that snapshots didn't cause data leakage

Hopefully an expert will help me out with my zeroing issues

Re: [linux-lvm] Snapshots and disk re-use

[Jonathan Tripathy]

On 24/02/2011 15:20, Jonathan Tripathy wrote:
>
> On 24/02/11 15:13, Stuart D. Gathman wrote:
>> On Thu, 24 Feb 2011, Jonathan Tripathy wrote:
>>
>>>> Yes. To be more pedantic, the COW has copies of the original contents
>>>> of blocks written to the origin since the snapshot.  That is why you
>>>> need to clear it to achieve your stated purpose.  The origin blocks
>>>> are written normally to the *-real volume (you can see these in
>>>> the /dev/mapper directory).
>>> But didn't you say that there is only one copy of the files stored 
>>> physically
>>> on disk?
>> Yes.  When you make the snapshot, there is only one copy, and the COW 
>> table
>> is empty.  AS YOU WRITE to the origin, each chunk written is saved to
>> *-cow first before being written to *-real.
> Got ya. So data that is being written to the origin, while the 
> snapshot exists, is the data that may leak, as it's saved to the COW 
> first, then copied over to real.
>
> Hopefully an expert will let me know weather its safe to zero the COW 
> after I�ve finished with the snapshot.
Are any "experts" available to help me answer the above question? I feel 
that this is a really important issue for those of us in the hosting 
industry.

Just to sum up my question: when a customer leaves our service, we zero 
their drive before removing the LV. This hopefully ensures that there is 
no data "leakage" when we create a new LV for a new customer. However, 
we need to take into consideration about what happens when we create 
snapshots of LVs to perform backups (using rsync).

Any help would be appreciated.

Re: [linux-lvm] Snapshots and disk re-use

[Stuart D. Gathman]

On Tue, 5 Apr 2011, Jonathan Tripathy wrote:

>> Hopefully an expert will let me know weather its safe to zero the COW after 
>> I?ve finished with the snapshot.
> Are any "experts" available to help me answer the above question? I feel that 
> this is a really important issue for those of us in the hosting industry.
>
> Just to sum up my question: when a customer leaves our service, we zero their 
> drive before removing the LV. This hopefully ensures that there is no data 
> "leakage" when we create a new LV for a new customer. However, we need to 
> take into consideration about what happens when we create snapshots of LVs to 
> perform backups (using rsync).
>
> Any help would be appreciated.

At this point, we'll just have to try it on a non-production server. 
Hopefully, worst case the kernel crashes.  I run Fedora (14 currently)
and CentOS-5.5.  My guess as an amateur is that zeroing the COW while
the origin is open is a problem.  I would suggest this for backup:

1) make snapshot
2) make backup
3) pause VM (check that this closes the origin LV, if not save the VM)
4) with both origin and snapshot not active, zero COW
5) remove snapshot
6) unpause VM (or restore if saved)

--
 	      Stuart D. Gathman <stuart@bmsi.com>
     Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Snapshots and disk re-use

[Jonathan Tripathy]

On 05/04/2011 21:41, Stuart D. Gathman wrote:
> On Tue, 5 Apr 2011, Jonathan Tripathy wrote:
>
>>> Hopefully an expert will let me know weather its safe to zero the 
>>> COW after I?ve finished with the snapshot.
>> Are any "experts" available to help me answer the above question? I 
>> feel that this is a really important issue for those of us in the 
>> hosting industry.
>>
>> Just to sum up my question: when a customer leaves our service, we 
>> zero their drive before removing the LV. This hopefully ensures that 
>> there is no data "leakage" when we create a new LV for a new 
>> customer. However, we need to take into consideration about what 
>> happens when we create snapshots of LVs to perform backups (using 
>> rsync).
>>
>> Any help would be appreciated.
>
> At this point, we'll just have to try it on a non-production server. 
> Hopefully, worst case the kernel crashes.  I run Fedora (14 currently)
> and CentOS-5.5.  My guess as an amateur is that zeroing the COW while
> the origin is open is a problem.  I would suggest this for backup:
>
> 1) make snapshot
> 2) make backup
> 3) pause VM (check that this closes the origin LV, if not save the VM)
> 4) with both origin and snapshot not active, zero COW
> 5) remove snapshot
> 6) unpause VM (or restore if saved)
>
> -- 
>           Stuart D. Gathman <stuart@bmsi.com>
>     Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 
> 591-6154
> "Confutatis maledictis, flammis acribus addictis" - background song for
> a Microsoft sponsored "Where do you want to go from here?" commercial.
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Yeah, I'll have to try this on a non-production server. Which kernel do 
you expect to crash? The Dom0 (Xen + LVM host) or the VM? Anyway, just 
thinking about it, it seems that pausing/saving/shutting down the VM is 
a must, as the VM may be writing to disk at the time of zeroing the cow 
(!!).

In the hosting industy, what does everyone else do? Do they just ignore 
the issue???

Re: [linux-lvm] Snapshots and disk re-use

[James Hawtin]

On 05/04/2011 20:48, Jonathan Tripathy wrote:
>
> Yeah, I'll have to try this on a non-production server. Which kernel 
> do you expect to crash? The Dom0 (Xen + LVM host) or the VM? Anyway, 
> just thinking about it, it seems that pausing/saving/shutting down the 
> VM is a must, as the VM may be writing to disk at the time of zeroing 
> the cow (!!).
>
> In the hosting industy, what does everyone else do? Do they just 
> ignore the issue???
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>
>
> -----
> No virus found in this message.
> Checked by AVG - www.avg.com
> Version: 10.0.1209 / Virus Database: 1500/3552 - Release Date: 04/05/11

As an alturnative could you not?

1) Create snapshot lv using specific physical extents.
lvcreate -s -l 100 ..... /dev/sdx1:0-99
2) Backup VM
3) Delete snapshot lv
4) create a normal lv using same physical extents
lvcreate -l 100 ..... /dev/sdx1:0-99
5) zero normal lv
6) delete normal lv

James

Re: [linux-lvm] Snapshots and disk re-use

[Jonathan Tripathy]

On 05/04/2011 21:59, James Hawtin wrote:
> On 05/04/2011 20:48, Jonathan Tripathy wrote:
>>
>> Yeah, I'll have to try this on a non-production server. Which kernel 
>> do you expect to crash? The Dom0 (Xen + LVM host) or the VM? Anyway, 
>> just thinking about it, it seems that pausing/saving/shutting down 
>> the VM is a must, as the VM may be writing to disk at the time of 
>> zeroing the cow (!!).
>>
>> In the hosting industy, what does everyone else do? Do they just 
>> ignore the issue???
>>
>> _______________________________________________
>> linux-lvm mailing list
>> linux-lvm@redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-lvm
>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>>
>>
>> -----
>> No virus found in this message.
>> Checked by AVG - www.avg.com
>> Version: 10.0.1209 / Virus Database: 1500/3552 - Release Date: 04/05/11
>
> As an alturnative could you not?
>
> 1) Create snapshot lv using specific physical extents.
> lvcreate -s -l 100 ..... /dev/sdx1:0-99
> 2) Backup VM
> 3) Delete snapshot lv
> 4) create a normal lv using same physical extents
> lvcreate -l 100 ..... /dev/sdx1:0-99
> 5) zero normal lv
> 6) delete normal lv
>
> James 
Hi James,

Interesting, didn't know you could do that! However, how do I know that 
the PEs aren't being used by LVs? Also, could you please explain the 
syntax? Normally to create a snapshot, I would do:

lvcreate -L20G -s -n backup /dev/vg0/customerID

Thanks

Thanks

Re: [linux-lvm] Snapshots and disk re-use

[James Hawtin]

On 05/04/2011 21:36, Jonathan Tripathy wrote:
> Hi James,
>
> Interesting, didn't know you could do that! However, how do I know 
> that the PEs aren't being used by LVs? Also, could you please explain 
> the syntax? Normally to create a snapshot, I would do:
>
> lvcreate -L20G -s -n backup /dev/vg0/customerID
>

Hmmm well you have two options, you could use pvdisplay --map or 
lvdisplay --map to work out exactly which PEs have been used to build 
you snapshot cow and then use that information to allow you to create a 
blanking PV in the same place or you could do it the easy way :-

1 hog the space to specific PEs
2 delete the hog
3 create the snapshot on same PEs
4 backup
5 delete the snapshot
6 create the hog on the same PEs
7 zero the hog

This has the advantage that the creation commands will fail if the PEs 
you want are not available the problem with it is you probably need more 
space for snapshots. As its less flexible in space use. Below i have 
illustrated all the commands, you need to do this. you don;t need all 
the display commands but they are there to prove to you this has worked, 
and the lvs are in the same place.

#pvdisplay --map /dev/cciss/c0d1p1
   --- Physical volume ---
   PV Name               /dev/cciss/c0d1p1
   VG Name               test_vg
   PV Size               683.51 GB / not usable 5.97 MB
   Allocatable           yes
   PE Size (KByte)       131072
   Total PE              5468
   Free PE               4332
   Allocated PE          1136
   PV UUID               YXjplf-EfLh-8Jkr-2utT-gmi5-gjAH-UOCcN0

   --- Physical Segments ---
   Physical extent 0 to 15:
     Logical volume      /dev/test_vg/test_lv
     Logical extents     0 to 15
   Physical extent 16 to 815:
     Logical volume      /dev/test_vg/mail_lv
     Logical extents     0 to 799
   Physical extent 816 to 975:
     Logical volume      /dev/test_vg/data_lv
     Logical extents     0 to 159
   Physical extent 976 to 2255:
     FREE
   Physical extent 2256 to 2335:
     Logical volume      /dev/test_vg/srv_lv
     Logical extents     0 to 79
   Physical extent 2336 to 2415:
     Logical volume      /dev/test_vg/data_lv
     Logical extents     160 to 239
   Physical extent 2416 to 5467:
     FREE

#lvcreate -l 20 -n hog_lv test_vg /dev/cciss/c0d1p1:5448-5467

#pvdisplay --map /dev/cciss/c0d1p1
   --- Physical volume ---
   PV Name               /dev/cciss/c0d1p1
   VG Name               test_vg
   PV Size               683.51 GB / not usable 5.97 MB
   Allocatable           yes
   PE Size (KByte)       131072
   Total PE              5468
   Free PE               4312
   Allocated PE          1156
   PV UUID               YXjplf-EfLh-8Jkr-2utT-gmi5-gjAH-UOCcN0

   --- Physical Segments ---
   Physical extent 0 to 15:
     Logical volume      /dev/test_vg/test_lv
     Logical extents     0 to 15
   Physical extent 16 to 815:
     Logical volume      /dev/test_vg/mail_lv
     Logical extents     0 to 799
   Physical extent 816 to 975:
     Logical volume      /dev/test_vg/data_lv
     Logical extents     0 to 159
   Physical extent 976 to 2255:
     FREE
   Physical extent 2256 to 2335:
     Logical volume      /dev/test_vg/srv_lv
     Logical extents     0 to 79
   Physical extent 2336 to 2415:
     Logical volume      /dev/test_vg/data_lv
     Logical extents     160 to 239
   Physical extent 2416 to 5447:
     FREE
   Physical extent 5448 to 5467:
     Logical volume      /dev/test_vg/hog_lv
     Logical extents     0 to 19

#lvremove /dev/test_vg/hog_lv
Do you really want to remove active logical volume hog_lv? [y/n]: y
   Logical volume "hog_lv" successfully removed
#lvcreate -l 20 -s -n data_snap /dev/test_vg/data_lv 
/dev/cciss/c0d1p1:5448-5467
   Logical volume "data_snap" created
#pvdisplay --map /dev/cciss/c0d1p1
                                    --- Physical volume ---
   PV Name               /dev/cciss/c0d1p1
   VG Name               test_vg
   PV Size               683.51 GB / not usable 5.97 MB
   Allocatable           yes
   PE Size (KByte)       131072
   Total PE              5468
   Free PE               4312
   Allocated PE          1156
   PV UUID               YXjplf-EfLh-8Jkr-2utT-gmi5-gjAH-UOCcN0

   --- Physical Segments ---
   Physical extent 0 to 15:
     Logical volume      /dev/test_vg/restricted_lv
     Logical extents     0 to 15
   Physical extent 16 to 815:
     Logical volume      /dev/test_vg/mail_lv
     Logical extents     0 to 799
   Physical extent 816 to 975:
     Logical volume      /dev/test_vg/data_lv
     Logical extents     0 to 159
   Physical extent 976 to 2255:
     FREE
   Physical extent 2256 to 2335:
     Logical volume      /dev/test_vg/srv_lv
     Logical extents     0 to 79
   Physical extent 2336 to 2415:
     Logical volume      /dev/test_vg/data_lv
     Logical extents     160 to 239
   Physical extent 2416 to 5447:
     FREE
   Physical extent 5448 to 5467:
     Logical volume      /dev/test_vg/data_snap
     Logical extents     0 to 19


#lvdisplay /dev/test_vg/data_snap
   --- Logical volume ---
   LV Name                /dev/test_vg/data_snap
   VG Name                test_vg
   LV UUID                bdqB77-f0vb-ZucS-Ka1l-pCr3-Ebeq-kOchmk
   LV Write Access        read/write
   LV snapshot status     active destination for /dev/test_vg/data_lv
   LV Status              available
   # open                 0
   LV Size                30.00 GB
   Current LE             240
   COW-table size         2.50 GB
   COW-table LE           20
   Allocated to snapshot  0.00%
   Snapshot chunk size    4.00 KB
   Segments               1
   Allocation             inherit
   Read ahead sectors     auto
   - currently set to     256
   Block device           253:5

#lvdisplay --map /dev/test_vg/data_snap
   --- Logical volume ---
   LV Name                /dev/test_vg/data_snap
   VG Name                test_vg
   LV UUID                IBBvOq-Bg0U-c69v-p7fQ-tR63-T8UV-gM1Ncu
   LV Write Access        read/write
   LV snapshot status     active destination for /dev/test_vg/data_lv
   LV Status              available
   # open                 0
   LV Size                30.00 GB
   Current LE             240
   COW-table size         2.50 GB
   COW-table LE           20
   Allocated to snapshot  0.00%
   Snapshot chunk size    4.00 KB
   Segments               1
   Allocation             inherit
   Read ahead sectors     auto
   - currently set to     256
   Block device           253:5

   --- Segments ---
   Logical extent 0 to 19:
     Type                linear
     Physical volume     /dev/cciss/c0d1p1
     Physical extents    5448 to 5467

<NOW BACKUP>

#lvremove /dev/test_vg/data_snap
Do you really want to remove active logical volume data_snap? [y/n]: y
   Logical volume "data_snap" successfully removed

#lvcreate -l 20 -n hog_lv test_vg /dev/cciss/c0d1p1:5448-5467
       Logical volume "hog_lv" created

#pvdisplay --map /dev/cciss/c0d1p1                                   --- 
Physical volume ---
   PV Name               /dev/cciss/c0d1p1
   VG Name               test_vg
   PV Size               683.51 GB / not usable 5.97 MB
   Allocatable           yes
   PE Size (KByte)       131072
   Total PE              5468
   Free PE               4312
   Allocated PE          1156
   PV UUID               YXjplf-EfLh-8Jkr-2utT-gmi5-gjAH-UOCcN0

   --- Physical Segments ---
   Physical extent 0 to 15:
     Logical volume      /dev/test_vg/restricted_lv
     Logical extents     0 to 15
   Physical extent 16 to 815:
     Logical volume      /dev/test_vg/mail_lv
     Logical extents     0 to 799
   Physical extent 816 to 975:
     Logical volume      /dev/test_vg/data_lv
     Logical extents     0 to 159
   Physical extent 976 to 2255:
     FREE
   Physical extent 2256 to 2335:
     Logical volume      /dev/test_vg/srv_lv
     Logical extents     0 to 79
   Physical extent 2336 to 2415:
     Logical volume      /dev/test_vg/data_lv
     Logical extents     160 to 239
   Physical extent 2416 to 5447:
     FREE
   Physical extent 5448 to 5467:
     Logical volume      /dev/test_vg/hog_lv
     Logical extents     0 to 19

#dd if=/dev/zero of=/dev/hog_lv

#lvremove /dev/test_vg/hog_lv
Do you really want to remove active logical volume hog_lv? [y/n]: y
   Logical volume "hog_lv" successfully removed

Enjoy

James

Re: [linux-lvm] Snapshots and disk re-use

[Jonathan Tripathy]

On 05/04/2011 23:42, James Hawtin wrote:
> On 05/04/2011 21:36, Jonathan Tripathy wrote:
>> Hi James,
>>
>> Interesting, didn't know you could do that! However, how do I know 
>> that the PEs aren't being used by LVs? Also, could you please explain 
>> the syntax? Normally to create a snapshot, I would do:
>>
>> lvcreate -L20G -s -n backup /dev/vg0/customerID
>>
>
> Hmmm well you have two options, you could use pvdisplay --map or 
> lvdisplay --map to work out exactly which PEs have been used to build 
> you snapshot cow and then use that information to allow you to create 
> a blanking PV in the same place or you could do it the easy way :-
>
> 1 hog the space to specific PEs
> 2 delete the hog
> 3 create the snapshot on same PEs
> 4 backup
> 5 delete the snapshot
> 6 create the hog on the same PEs
> 7 zero the hog
>
> This has the advantage that the creation commands will fail if the PEs 
> you want are not available the problem with it is you probably need 
> more space for snapshots. As its less flexible in space use. Below i 
> have illustrated all the commands, you need to do this. you don;t need 
> all the display commands but they are there to prove to you this has 
> worked, and the lvs are in the same place.
>
> #pvdisplay --map /dev/cciss/c0d1p1
>   --- Physical volume ---
>   PV Name               /dev/cciss/c0d1p1
>   VG Name               test_vg
>   PV Size               683.51 GB / not usable 5.97 MB
>   Allocatable           yes
>   PE Size (KByte)       131072
>   Total PE              5468
>   Free PE               4332
>   Allocated PE          1136
>   PV UUID               YXjplf-EfLh-8Jkr-2utT-gmi5-gjAH-UOCcN0
>
>   --- Physical Segments ---
>   Physical extent 0 to 15:
>     Logical volume      /dev/test_vg/test_lv
>     Logical extents     0 to 15
>   Physical extent 16 to 815:
>     Logical volume      /dev/test_vg/mail_lv
>     Logical extents     0 to 799
>   Physical extent 816 to 975:
>     Logical volume      /dev/test_vg/data_lv
>     Logical extents     0 to 159
>   Physical extent 976 to 2255:
>     FREE
>   Physical extent 2256 to 2335:
>     Logical volume      /dev/test_vg/srv_lv
>     Logical extents     0 to 79
>   Physical extent 2336 to 2415:
>     Logical volume      /dev/test_vg/data_lv
>     Logical extents     160 to 239
>   Physical extent 2416 to 5467:
>     FREE
>
> #lvcreate -l 20 -n hog_lv test_vg /dev/cciss/c0d1p1:5448-5467
>
> #pvdisplay --map /dev/cciss/c0d1p1
>   --- Physical volume ---
>   PV Name               /dev/cciss/c0d1p1
>   VG Name               test_vg
>   PV Size               683.51 GB / not usable 5.97 MB
>   Allocatable           yes
>   PE Size (KByte)       131072
>   Total PE              5468
>   Free PE               4312
>   Allocated PE          1156
>   PV UUID               YXjplf-EfLh-8Jkr-2utT-gmi5-gjAH-UOCcN0
>
>   --- Physical Segments ---
>   Physical extent 0 to 15:
>     Logical volume      /dev/test_vg/test_lv
>     Logical extents     0 to 15
>   Physical extent 16 to 815:
>     Logical volume      /dev/test_vg/mail_lv
>     Logical extents     0 to 799
>   Physical extent 816 to 975:
>     Logical volume      /dev/test_vg/data_lv
>     Logical extents     0 to 159
>   Physical extent 976 to 2255:
>     FREE
>   Physical extent 2256 to 2335:
>     Logical volume      /dev/test_vg/srv_lv
>     Logical extents     0 to 79
>   Physical extent 2336 to 2415:
>     Logical volume      /dev/test_vg/data_lv
>     Logical extents     160 to 239
>   Physical extent 2416 to 5447:
>     FREE
>   Physical extent 5448 to 5467:
>     Logical volume      /dev/test_vg/hog_lv
>     Logical extents     0 to 19
>
> #lvremove /dev/test_vg/hog_lv
> Do you really want to remove active logical volume hog_lv? [y/n]: y
>   Logical volume "hog_lv" successfully removed
> #lvcreate -l 20 -s -n data_snap /dev/test_vg/data_lv 
> /dev/cciss/c0d1p1:5448-5467
>   Logical volume "data_snap" created
> #pvdisplay --map /dev/cciss/c0d1p1
>                                    --- Physical volume ---
>   PV Name               /dev/cciss/c0d1p1
>   VG Name               test_vg
>   PV Size               683.51 GB / not usable 5.97 MB
>   Allocatable           yes
>   PE Size (KByte)       131072
>   Total PE              5468
>   Free PE               4312
>   Allocated PE          1156
>   PV UUID               YXjplf-EfLh-8Jkr-2utT-gmi5-gjAH-UOCcN0
>
>   --- Physical Segments ---
>   Physical extent 0 to 15:
>     Logical volume      /dev/test_vg/restricted_lv
>     Logical extents     0 to 15
>   Physical extent 16 to 815:
>     Logical volume      /dev/test_vg/mail_lv
>     Logical extents     0 to 799
>   Physical extent 816 to 975:
>     Logical volume      /dev/test_vg/data_lv
>     Logical extents     0 to 159
>   Physical extent 976 to 2255:
>     FREE
>   Physical extent 2256 to 2335:
>     Logical volume      /dev/test_vg/srv_lv
>     Logical extents     0 to 79
>   Physical extent 2336 to 2415:
>     Logical volume      /dev/test_vg/data_lv
>     Logical extents     160 to 239
>   Physical extent 2416 to 5447:
>     FREE
>   Physical extent 5448 to 5467:
>     Logical volume      /dev/test_vg/data_snap
>     Logical extents     0 to 19
>
>
> #lvdisplay /dev/test_vg/data_snap
>   --- Logical volume ---
>   LV Name                /dev/test_vg/data_snap
>   VG Name                test_vg
>   LV UUID                bdqB77-f0vb-ZucS-Ka1l-pCr3-Ebeq-kOchmk
>   LV Write Access        read/write
>   LV snapshot status     active destination for /dev/test_vg/data_lv
>   LV Status              available
>   # open                 0
>   LV Size                30.00 GB
>   Current LE             240
>   COW-table size         2.50 GB
>   COW-table LE           20
>   Allocated to snapshot  0.00%
>   Snapshot chunk size    4.00 KB
>   Segments               1
>   Allocation             inherit
>   Read ahead sectors     auto
>   - currently set to     256
>   Block device           253:5
>
> #lvdisplay --map /dev/test_vg/data_snap
>   --- Logical volume ---
>   LV Name                /dev/test_vg/data_snap
>   VG Name                test_vg
>   LV UUID                IBBvOq-Bg0U-c69v-p7fQ-tR63-T8UV-gM1Ncu
>   LV Write Access        read/write
>   LV snapshot status     active destination for /dev/test_vg/data_lv
>   LV Status              available
>   # open                 0
>   LV Size                30.00 GB
>   Current LE             240
>   COW-table size         2.50 GB
>   COW-table LE           20
>   Allocated to snapshot  0.00%
>   Snapshot chunk size    4.00 KB
>   Segments               1
>   Allocation             inherit
>   Read ahead sectors     auto
>   - currently set to     256
>   Block device           253:5
>
>   --- Segments ---
>   Logical extent 0 to 19:
>     Type                linear
>     Physical volume     /dev/cciss/c0d1p1
>     Physical extents    5448 to 5467
>
> <NOW BACKUP>
>
> #lvremove /dev/test_vg/data_snap
> Do you really want to remove active logical volume data_snap? [y/n]: y
>   Logical volume "data_snap" successfully removed
>
> #lvcreate -l 20 -n hog_lv test_vg /dev/cciss/c0d1p1:5448-5467
>       Logical volume "hog_lv" created
>
> #pvdisplay --map /dev/cciss/c0d1p1                                   
> --- Physical volume ---
>   PV Name               /dev/cciss/c0d1p1
>   VG Name               test_vg
>   PV Size               683.51 GB / not usable 5.97 MB
>   Allocatable           yes
>   PE Size (KByte)       131072
>   Total PE              5468
>   Free PE               4312
>   Allocated PE          1156
>   PV UUID               YXjplf-EfLh-8Jkr-2utT-gmi5-gjAH-UOCcN0
>
>   --- Physical Segments ---
>   Physical extent 0 to 15:
>     Logical volume      /dev/test_vg/restricted_lv
>     Logical extents     0 to 15
>   Physical extent 16 to 815:
>     Logical volume      /dev/test_vg/mail_lv
>     Logical extents     0 to 799
>   Physical extent 816 to 975:
>     Logical volume      /dev/test_vg/data_lv
>     Logical extents     0 to 159
>   Physical extent 976 to 2255:
>     FREE
>   Physical extent 2256 to 2335:
>     Logical volume      /dev/test_vg/srv_lv
>     Logical extents     0 to 79
>   Physical extent 2336 to 2415:
>     Logical volume      /dev/test_vg/data_lv
>     Logical extents     160 to 239
>   Physical extent 2416 to 5447:
>     FREE
>   Physical extent 5448 to 5467:
>     Logical volume      /dev/test_vg/hog_lv
>     Logical extents     0 to 19
>
> #dd if=/dev/zero of=/dev/hog_lv
>
> #lvremove /dev/test_vg/hog_lv
> Do you really want to remove active logical volume hog_lv? [y/n]: y
>   Logical volume "hog_lv" successfully removed
>
> Enjoy
>
> James
>
>
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

James,

That's fantastic! Thanks very much! I have a couple of questions:

1) If I wanted to create a script that backed up lots of customer-data 
LVs, could I just do one zero at the end (and still have no data leakage)?

2) On average, each of my data LVs are 20GB each, and if I were to 
create a snapshot of 20GB, this would take about 20 mins to erase. If I 
made the snapshot only 1GB, that means it would be quick to erase at the 
end (however only 1GB of data could be created on the respect origin, 
correct?)

Thanks

Re: [linux-lvm] Snapshots and disk re-use

[James Hawtin]

> James,
>
> That's fantastic! Thanks very much! I have a couple of questions:
>
> 1) If I wanted to create a script that backed up lots of customer-data 
> LVs, could I just do one zero at the end (and still have no data 
> leakage)?

Yes you could, because COW means COPY ON WRITE, so the original block is 
copied onto the COW with the data from the original disk overwritting 
any data currently on it. Before that point any data on it was not 
addressable from the snapshot lv (* see my final point)
> 2) On average, each of my data LVs are 20GB each, and if I were to 
> create a snapshot of 20GB, this would take about 20 mins to erase. If 
> I made the snapshot only 1GB, that means it would be quick to erase at 
> the end (however only 1GB of data could be created on the respect 
> origin, correct?)

You are right you only have to erase the snapshot cow space, which is 
normally only 10-15% of the whole original disk. 2GB is pretty fast to 
over right on any system I have used these days. To be sure though you 
do need to overwrite the whole cow even if only a few percent was used 
as you cannot tell which few percent that was.

* I do wonder why you are so worried, leakage is only a problem if the 
COW is assigned to a future customer LV. If you always used the same 
space for backups perhaps had a PE just for backups it would never be 
used in a customer lv therefore you could argue that you never have to 
erase it. If its on a pe you only use for snapshotting you also don't 
need to hog the space as any bit of that disk is ok.

James

Re: [linux-lvm] Snapshots and disk re-use

[Jonathan Tripathy]

On 06/04/2011 00:11, James Hawtin wrote:
>
>> James,
>>
>> That's fantastic! Thanks very much! I have a couple of questions:
>>
>> 1) If I wanted to create a script that backed up lots of 
>> customer-data LVs, could I just do one zero at the end (and still 
>> have no data leakage)?
>
> Yes you could, because COW means COPY ON WRITE, so the original block 
> is copied onto the COW with the data from the original disk 
> overwritting any data currently on it. Before that point any data on 
> it was not addressable from the snapshot lv (* see my final point)
>> 2) On average, each of my data LVs are 20GB each, and if I were to 
>> create a snapshot of 20GB, this would take about 20 mins to erase. If 
>> I made the snapshot only 1GB, that means it would be quick to erase 
>> at the end (however only 1GB of data could be created on the respect 
>> origin, correct?)
>
> You are right you only have to erase the snapshot cow space, which is 
> normally only 10-15% of the whole original disk. 2GB is pretty fast to 
> over right on any system I have used these days. To be sure though you 
> do need to overwrite the whole cow even if only a few percent was used 
> as you cannot tell which few percent that was.
Actually, I meant just making a snapshot of 1GB, not just erase the 
first 1GB of a 20GB snapshot. Bu tthis may be moot (See below)
>
> * I do wonder why you are so worried, leakage is only a problem if the 
> COW is assigned to a future customer LV. If you always used the same 
> space for backups perhaps had a PE just for backups it would never be 
> used in a customer lv therefore you could argue that you never have to 
> erase it. If its on a pe you only use for snapshotting you also don't 
> need to hog the space as any bit of that disk is ok.
Excellent point! As long as I use the same PEs for making the snapshot 
everytime, I don't need to ever erase it (And it can be a nice big size 
like 50GB, so even my largest customers won't outgrow the snapshot). 
However though, wouldn't I need to keep the "hog" around just to make 
sure that the snapshot PEs don't get assigned to a new customer LV in 
the future (Currently, we don't specify PEs to use when creating normal 
LVs)?

An even better question: Does the snapshot have to be on the same 
physical disk as the LV its mirroring?

Re: [linux-lvm] Snapshots and disk re-use

[James Hawtin]

On 05/04/2011 23:19, Jonathan Tripathy wrote:
> Excellent point! As long as I use the same PEs for making the snapshot 
> everytime, I don't need to ever erase it (And it can be a nice big 
> size like 50GB, so even my largest customers won't outgrow the 
> snapshot). However though, wouldn't I need to keep the "hog" around 
> just to make sure that the snapshot PEs don't get assigned to a new 
> customer LV in the future (Currently, we don't specify PEs to use when 
> creating normal LVs)?
>

I think you missed the point of why I suggested using a separate PV, the 
space could be divided using fdisk it does not have to separate physical 
disk (This is ok as you will never use this space for mirroring). If 
snapshots are created on a separate PV you can use pvchange - x n and 
pvchange -x y to change if it is allocatable and only when you are 
creating snaps do you make in allocatable that will prevent accidental 
reuse in customer lvs without lots of hassle.  If you don't use pvchange 
you will need to specify the PVs when ever you create a customer LV.

# an lvcreate with specifing the PEs to use.
lvcreate -l 20 -s -n data_snap /dev/test_vg/data_lv /dev/cciss/c0d1p2


Also this allows you to use 50 GB area or 4 x 10 GB ones, depending on 
what you are backing up without having to worry about creating a hog 
over the top of anything you haven't used. This allows you more 
flexibility with simultaneous backups.

> An even better question: Does the snapshot have to be on the same 
> physical disk as the LV its mirroring?
>
No it does not.

Do I get cake now?

James

Re: [linux-lvm] Snapshots and disk re-use

[Jonathan Tripathy]

On 06/04/2011 00:39, James Hawtin wrote:
> On 05/04/2011 23:19, Jonathan Tripathy wrote:
>> Excellent point! As long as I use the same PEs for making the 
>> snapshot everytime, I don't need to ever erase it (And it can be a 
>> nice big size like 50GB, so even my largest customers won't outgrow 
>> the snapshot). However though, wouldn't I need to keep the "hog" 
>> around just to make sure that the snapshot PEs don't get assigned to 
>> a new customer LV in the future (Currently, we don't specify PEs to 
>> use when creating normal LVs)?
>>
>
> I think you missed the point of why I suggested using a separate PV, 
> the space could be divided using fdisk it does not have to separate 
> physical disk (This is ok as you will never use this space for 
> mirroring). If snapshots are created on a separate PV you can use 
> pvchange - x n and pvchange -x y to change if it is allocatable and 
> only when you are creating snaps do you make in allocatable that will 
> prevent accidental reuse in customer lvs without lots of hassle.  If 
> you don't use pvchange you will need to specify the PVs when ever you 
> create a customer LV.
Ok now things are really getting interesting!

Actually, when I create new customer LVs, I always specify which volume 
group I want to add it to. E.g:

lvcreate -nNewCustomerLV -L20G vg0

where vg0 is /dev/vg0
and vg0 is a volume group which uses an entire physical partition (Which 
I guess is called a PV).

Now, if I were to create my snapshots on a seperate vg, eg:

lvcreate -L 20G -s -n data_snap /dev/vg0/NewCustomerLV /dev/vg1

does that mean I never need to use pvchange to "switch off" vg1? And I 
never need to zero or create a "hog"? And no leakage will ever occur?

>
>
> Do I get cake now? 
Only if it's not a lie.... :)
(Just incase you didn't get the referance 
http://en.wikipedia.org/wiki/The_cake_is_a_lie)

Re: [linux-lvm] Snapshots and disk re-use

[Stuart D. Gathman]

On Wed, 6 Apr 2011, Jonathan Tripathy wrote:

> lvcreate -nNewCustomerLV -L20G vg0
>
> where vg0 is /dev/vg0
> and vg0 is a volume group which uses an entire physical partition (Which I 
> guess is called a PV).
>
> Now, if I were to create my snapshots on a seperate vg, eg:

You can't do that.  Snapshots must be on the same VG.  James is suggesting you
create them on the same VG, but a different PV reserved for backup snapshots.

--
 	      Stuart D. Gathman <stuart@bmsi.com>
     Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Snapshots and disk re-use

[Jonathan Tripathy]

On 06/04/2011 01:08, Stuart D. Gathman wrote:
> On Wed, 6 Apr 2011, Jonathan Tripathy wrote:
>
>> lvcreate -nNewCustomerLV -L20G vg0
>>
>> where vg0 is /dev/vg0
>> and vg0 is a volume group which uses an entire physical partition 
>> (Which I guess is called a PV).
>>
>> Now, if I were to create my snapshots on a seperate vg, eg:
>
> You can't do that.  Snapshots must be on the same VG.  James is 
> suggesting you
> create them on the same VG, but a different PV reserved for backup 
> snapshots. 
Guess I need to go back to LVM basics. I had a "hierarchical" structure 
in my head (which must be wrong):

An LV sites on top of a VG which sits on top of a PV

But going by your comment above, you can add multiple PVs (Physical 
partitions) to a single VG and create some form of a JBOD configuration?

Re: [linux-lvm] Snapshots and disk re-use

[James Hawtin]

On 06/04/2011 00:00, Jonathan Tripathy wrote:
> Ok now things are really getting interesting!
>
> Actually, when I create new customer LVs, I always specify which 
> volume group I want to add it to. E.g:
>
> lvcreate -nNewCustomerLV -L20G vg0

Sadly that is just normal... However a volume group could be made up of 
more than one disk, you can with an option specify which of the disks 
(pv) to use. Rather than the system select for you.

> where vg0 is /dev/vg0
> and vg0 is a volume group which uses an entire physical partition 
> (Which I guess is called a PV).

That is a vg (volume group) not a pv (physical volume).

vgdisplay -v
pvs
pvdisplay --map

and you should get the idea.

>
> Now, if I were to create my snapshots on a seperate vg, eg:
>
> lvcreate -L 20G -s -n data_snap /dev/vg0/NewCustomerLV /dev/vg1
>
> does that mean I never need to use pvchange to "switch off" vg1? And I 
> never need to zero or create a "hog"? And no leakage will ever occur?

No you are confused pvs are created on disk partitions, one or more pvs 
then make you vg.

LVs are created on top of VGs, VGs are created on top of PVs, PVs are 
created on partitions on block devices or whole block devices.

(Ok we shall stop here however blocks devices could be loop back, meta 
devices like raid 0/1/5 etc hardware/software or real disks however that 
is not LVM any more.)

>
>> Do I get cake now? 
> Only if it's not a lie.... :)

I saw the cake but I got none (Portal)

James

Re: [linux-lvm] Snapshots and disk re-use

[Jonathan Tripathy]

> No you are confused pvs are created on disk partitions, one or more 
> pvs then make you vg.
>
> LVs are created on top of VGs, VGs are created on top of PVs, PVs are 
> created on partitions on block devices or whole block devices.
>
> (Ok we shall stop here however blocks devices could be loop back, meta 
> devices like raid 0/1/5 etc hardware/software or real disks however 
> that is not LVM any more.)

Ok, i think I get it now. At the minute, my vg (vg0) only has on PV in 
it (/dev/md3 which you can tell is a mdadm RAID device). I wasn't aware 
you could add more PVs (that's pretty cool!). So, let's say I had a 
spare partition (/dev/hdb7 as an example). To my vg0 volume group, I 
would firstly:

pvcreate /dev/hdb7
vgextend /dev/hdb7

Then, every time I create a new customer LV, I would do:

lvcreate -nNewCustomerLV -L20G vg0 /dev/md3

Then, every time I wanted to create a snapshot:

lvcreate -L20G -s -n data_snap /dev/vg0/NewCustomerLV /dev/hdb7

Is that correct? No Leakage? And no zeroing needed?

Side note: Since I didn't partition my servers with this in mind, my new 
PV will probably have to be an iSCSI device located on a remote target 
:( Either that or use a loopback device with an image, but I'd be scared 
that the system would not boot properly. Can you give me any tips on how 
to use an image file as a PV just for snapshots?

Thanks

Re: [linux-lvm] Snapshots and disk re-use

[Stuart D. Gathman]

On Wed, 6 Apr 2011, Jonathan Tripathy wrote:

> pvcreate /dev/hdb7
> vgextend /dev/hdb7
>
> Then, every time I create a new customer LV, I would do:
>
> lvcreate -nNewCustomerLV -L20G vg0 /dev/md3
>
> Then, every time I wanted to create a snapshot:
>
> lvcreate -L20G -s -n data_snap /dev/vg0/NewCustomerLV /dev/hdb7
>
> Is that correct? No Leakage? And no zeroing needed?

Correct.  Except vgextend needs to specify a VG.  And you still need to zero a
customer LV before removing it.  The PV reserved for snapshots approach only
avoids the need to zero snapshots.

I wonder if there is an elegant way to automate things like this via
"allocation policies" or similar in the metadata.

--
 	      Stuart D. Gathman <stuart@bmsi.com>
     Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Snapshots and disk re-use

[James Hawtin]

On 06/04/2011 00:28, Jonathan Tripathy wrote:
> Ok, i think I get it now. At the minute, my vg (vg0) only has on PV in 
> it (/dev/md3 which you can tell is a mdadm RAID device). I wasn't 
> aware you could add more PVs (that's pretty cool!). So, let's say I 
> had a spare partition (/dev/hdb7 as an example). To my vg0 volume 
> group, I would firstly:
>
> pvcreate /dev/hdb7
> vgextend /dev/hdb7

Right however danger warnings are going off in my mind now!

> Then, every time I create a new customer LV, I would do:
>
> lvcreate -nNewCustomerLV -L20G vg0 /dev/md3

yes that would work

>
> Then, every time I wanted to create a snapshot:
>
> lvcreate -L20G -s -n data_snap /dev/vg0/NewCustomerLV /dev/hdb7
>

Yes

> Is that correct? No Leakage? And no zeroing needed?
Indeed

>
> Side note: Since I didn't partition my servers with this in mind, my 
> new PV will probably have to be an iSCSI device located on a remote 
> target :( Either that or use a loopback device with an image, but I'd 
> be scared that the system would not boot properly. Can you give me any 
> tips on how to use an image file as a PV just for snapshots?

Ok, there has been alot of dangerous talk here, i assume you are using 
an md device so you can mirror things. If you added a single disk to 
that, and that disk failed you would have a major problem. Likewise if 
you rebooted with open snaps and iscsi you would need that iscsi device 
available during computer boot. I REALLY hope you do not have your local 
FS on the same vg as your data. As this would result in a non booting 
machine.

BTW new use pvmove on /var as it stores data there and will freeze the 
whole system.

All hope is not lost, if you can add disk temperately you can use :-
1) add a new pv to the exisiting disk (vgextend)
2) move the data lvs to the new pv (pvmove)
3) remove the old disk (vgreduce)
   a) check with pvscan that the old disk really is not in use
4) resize the partiton (fdisk)
5) create a new pv (pvcreate --force) you need that to overwrite... take 
care now.
6) add the old disk back in (vgextend)
7) move the data lvs back to the old disk (pvmove)
8) remove the temp disk (vgreduce)

Now that is worth cake!

James

Re: [linux-lvm] Snapshots and disk re-use

[Stuart D. Gathman]

On Tue, 5 Apr 2011, Stuart D. Gathman wrote:

>> lvcreate -L20G -s -n data_snap /dev/vg0/NewCustomerLV /dev/hdb7
>> 
>> Is that correct? No Leakage? And no zeroing needed?
>
> Correct.  Except vgextend needs to specify a VG.  And you still need to zero
> a customer LV before removing it.  The PV reserved for snapshots approach
> only avoids the need to zero snapshots.

In fact, since you still have to zero an LV before removing, you might as well
zero an LV when you allocate.  Same overhead.  Then you don't need to mess
with any of these schemes to deal with snapshots.  Did we already cover
that obvious solution?  We did discuss a utility to optimize zeroing an
already mostly zero LV.

--
 	      Stuart D. Gathman <stuart@bmsi.com>
     Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Snapshots and disk re-use

[Jonathan Tripathy]

On 06/04/2011 01:38, Stuart D. Gathman wrote:
> On Wed, 6 Apr 2011, Jonathan Tripathy wrote:
>
>> pvcreate /dev/hdb7
>> vgextend /dev/hdb7
>>
>> Then, every time I create a new customer LV, I would do:
>>
>> lvcreate -nNewCustomerLV -L20G vg0 /dev/md3
>>
>> Then, every time I wanted to create a snapshot:
>>
>> lvcreate -L20G -s -n data_snap /dev/vg0/NewCustomerLV /dev/hdb7
>>
>> Is that correct? No Leakage? And no zeroing needed?
>
> Correct.  Except vgextend needs to specify a VG.  And you still need 
> to zero a
> customer LV before removing it.  The PV reserved for snapshots 
> approach only
> avoids the need to zero snapshots.
Excellent! Cake for all! :)
>
> I wonder if there is an elegant way to automate things like this via
> "allocation policies" or similar in the metadata. 
Indeed, there does seem to be some scope here for a code feature request 
as this does seem like an aweful lot of work for something that could 
very easily be overlooked. Every book or article I've read about hosting 
seems to never mention anything about zeroing....

Re: [linux-lvm] Snapshots and disk re-use

[Jonathan Tripathy]

On 06/04/2011 01:42, James Hawtin wrote:
> On 06/04/2011 00:28, Jonathan Tripathy wrote:
>> Ok, i think I get it now. At the minute, my vg (vg0) only has on PV 
>> in it (/dev/md3 which you can tell is a mdadm RAID device). I wasn't 
>> aware you could add more PVs (that's pretty cool!). So, let's say I 
>> had a spare partition (/dev/hdb7 as an example). To my vg0 volume 
>> group, I would firstly:
>>
>> pvcreate /dev/hdb7
>> vgextend /dev/hdb7
>
> Right however danger warnings are going off in my mind now!
>
>> Then, every time I create a new customer LV, I would do:
>>
>> lvcreate -nNewCustomerLV -L20G vg0 /dev/md3
>
> yes that would work
>
>>
>> Then, every time I wanted to create a snapshot:
>>
>> lvcreate -L20G -s -n data_snap /dev/vg0/NewCustomerLV /dev/hdb7
>>
>
> Yes
>
>> Is that correct? No Leakage? And no zeroing needed?
> Indeed
>
>>
>> Side note: Since I didn't partition my servers with this in mind, my 
>> new PV will probably have to be an iSCSI device located on a remote 
>> target :( Either that or use a loopback device with an image, but I'd 
>> be scared that the system would not boot properly. Can you give me 
>> any tips on how to use an image file as a PV just for snapshots?
>
> Ok, there has been alot of dangerous talk here, i assume you are using 
> an md device so you can mirror things. 
Correct
> If you added a single disk to that, and that disk failed you would 
> have a major problem. Likewise if you rebooted with open snaps and 
> iscsi you would need that iscsi device available during computer boot. 
> I REALLY hope you do not have your local FS on the same vg as your 
> data. As this would result in a non booting machine.
Nope. My root partition is not on LVM at all, but just on a regular md 
partition. The LVM LVs are used for virtual machine hosting for 
customers. Also, I used just /dev/hdb7 as an example, but it does bring 
up some interesting questions:

If the PV used for snapshots were to fail while the snapshot was open, 
or the server rebooted and the PV wasn't available at boot, what would 
happen? I ask these questions because a loopback device or iSCSI is 
really my only feasible option right now for the snapshot PV...

Re: [linux-lvm] Snapshots and disk re-use

[James Hawtin]

On 06/04/2011 00:50, Jonathan Tripathy wrote:
>
> If the PV used for snapshots were to fail while the snapshot was open, 
> or the server rebooted and the PV wasn't available at boot, what would 
> happen? I ask these questions because a loopback device or iSCSI is 
> really my only feasible option right now for the snapshot PV...

What would happen is... if the file systems are mounted at boot time 
(in  the fstab) it will fail the fsck because the device is not there. 
and drop to single user mode, you could then edit the fstab to to remove 
those file systems that would bring the system online, at which point 
you could fix what stopped the iscsi from working, and mount the file 
systems.

At one place I worked they never mounted the data file systems at "boot" 
but in a rc.local so the system always got to be interactive before any 
problems so it was easy to go in and fix.

DO NOT... create a loop back device on the same on a file system that 
the loopback that will then form a pv of, if you do your system is 
DOOOMMMED! to get it to boot again your have to mount a part volume 
group and block copy the devices to a new one, even worse if you 
extended the file system with the loop back on it onto the pv of the 
loop back it will NEVER work again. So the only place you can create a 
loopback device is outside of a vg it is to be a part of and frankley 
better that its in NO vg as you may get recursion problems.

The problem with a loop back is that you need to do a the loopback setup 
to enable the device before the vgscan and vgchange can bring it online 
in the volume, very hard to get right at boot time.  If you have 
partitioned it you will also need to do kpartx.

If you use loopbacks i  would extend the volume group onto the disk only 
during backups then nreduce it out afterwards to reduce risks.

Steal space from somewhere you say you have the OS on a physical 
paritions, so LVM everything but / and /boot redo the make a pv on the 
space freed. to rescue a system is easy if you can mount / everything 
else does not matter.

If you have everything in / ... you are insane as you should set /var, 
/tmp and perhaps even /home to noexec as if you get an automated break 
in that is normally where the write stage two, to get executed, this 
normally stops them in there tracks, No public writiable filesystems 
should be executable.... !

James

Re: [linux-lvm] Snapshots and disk re-use

[James Hawtin]

On 06/04/2011 00:43, Stuart D. Gathman wrote:
> In fact, since you still have to zero an LV before removing, you might 
> as well
> zero an LV when you allocate.  Same overhead.  Then you don't need to 
> mess
> with any of these schemes to deal with snapshots.  Did we already cover
> that obvious solution?  We did discuss a utility to optimize zeroing an
> already mostly zero LV.
>

I think you mean and lv of a virtual machine rather than snap shot, not 
sure how you could zero the cow. Stuart does have a good point here, 
there is another source of leak outside of snapshots (and probably 
worse), if you delete an lv the create a new one over the same PEs the 
data from then old lv will be there, save a few k that is zeroed at the 
front (to help with fs/label creation). This is worse as it is in order 
data that the cow from a snap is not. Geting a while fs back is very 
possible.

Try this test

lvcreate -L 256m -n test_lv osiris_l1_vg
yes | dd of=/dev/osiris_l1_vg/test_lv
pvdisplay --map
lvremove /dev/osiris_l1_vg/test_lv
lvcreate -L 256m -n test_lv osiris_l1_vg
pvdisplay --map
od -ha /dev/osiris_l1_vg/test_lv

James

Re: [linux-lvm] Snapshots and disk re-use

[Jonathan Tripathy]

On 06/04/2011 02:20, James Hawtin wrote:
> On 06/04/2011 00:50, Jonathan Tripathy wrote:
>>
>> If the PV used for snapshots were to fail while the snapshot was 
>> open, or the server rebooted and the PV wasn't available at boot, 
>> what would happen? I ask these questions because a loopback device or 
>> iSCSI is really my only feasible option right now for the snapshot PV...
>
> What would happen is... if the file systems are mounted at boot time 
> (in  the fstab) it will fail the fsck because the device is not there. 
> and drop to single user mode, you could then edit the fstab to to 
> remove those file systems that would bring the system online, at which 
> point you could fix what stopped the iscsi from working, and mount the 
> file systems.
>
> At one place I worked they never mounted the data file systems at 
> "boot" but in a rc.local so the system always got to be interactive 
> before any problems so it was easy to go in and fix.
Since my vg is used purly for virtualisation, I don't mount any LVs in 
fstab in my host OS. LVs only get mounted once VMs start up.
>
> DO NOT... create a loop back device on the same on a file system that 
> the loopback that will then form a pv of, if you do your system is 
> DOOOMMMED! to get it to boot again your have to mount a part volume 
> group and block copy the devices to a new one, even worse if you 
> extended the file system with the loop back on it onto the pv of the 
> loop back it will NEVER work again. So the only place you can create a 
> loopback device is outside of a vg it is to be a part of and frankley 
> better that its in NO vg as you may get recursion problems.
Oh no, wouldn't dare to do that :) I was thinking of creating the image 
file on a non-LVM area of the drive (somewhere in the host OS partition)
>
> The problem with a loop back is that you need to do a the loopback 
> setup to enable the device before the vgscan and vgchange can bring it 
> online in the volume, very hard to get right at boot time.  If you 
> have partitioned it you will also need to do kpartx.
Well I wouldn't partition the image file, seems more trouble than it's 
worth. I did find an article from Novell that seems to guide me through 
what to do if my machine were to reboot with a PV missing:

http://www.novell.com/coolsolutions/appnote/19386.html

>
> If you use loopbacks i  would extend the volume group onto the disk 
> only during backups then nreduce it out afterwards to reduce risks.
Good Idea.

Re: [linux-lvm] Snapshots and disk re-use

[Jonathan Tripathy]

On 06/04/2011 02:36, James Hawtin wrote:
> On 06/04/2011 00:43, Stuart D. Gathman wrote:
>> In fact, since you still have to zero an LV before removing, you 
>> might as well
>> zero an LV when you allocate.  Same overhead.  Then you don't need to 
>> mess
>> with any of these schemes to deal with snapshots.  Did we already cover
>> that obvious solution?  We did discuss a utility to optimize zeroing an
>> already mostly zero LV.
>>
>
> I think you mean and lv of a virtual machine rather than snap shot, 
> not sure how you could zero the cow. Stuart does have a good point 
> here, there is another source of leak outside of snapshots (and 
> probably worse), if you delete an lv the create a new one over the 
> same PEs the data from then old lv will be there, save a few k that is 
> zeroed at the front (to help with fs/label creation). This is worse as 
> it is in order data that the cow from a snap is not. Geting a while fs 
> back is very possible. 

Currently, as part of our procedure, we zero LVs once a customer has 
left our service. The reason why we don't zero on allocation is that 
customer usually expect quick setup times (under 10 minutes) and zeroing 
gigabytes worth of space can take too long. Getting new zeroed LVs ready 
before sign ups also isn't an option but for other reasons.

Re: [linux-lvm] Snapshots and disk re-use

[James Hawtin]

On 06/04/2011 01:47, Jonathan Tripathy wrote:
>
> Currently, as part of our procedure, we zero LVs once a customer has 
> left our service. The reason why we don't zero on allocation is that 
> customer usually expect quick setup times (under 10 minutes) and 
> zeroing gigabytes worth of space can take too long. Getting new zeroed 
> LVs ready before sign ups also isn't an option but for other reasons.

Very reasonable

It however is a shame than lvm cannot zero PE (physical extents) on 
first access or do a proper background initialisation. As would happen 
on hardware raids.

James