Re: [linux-lvm] access or interface to list of blocks that have, changed via C.O.W.?

Re: [linux-lvm] access or interface to list of blocks that have, changed via C.O.W.?

[Mark Woodward]

There are a couple projects that do this. They are pretty much based on 
ddsnap. You can google it.
In LVM2 world, it is fairly trivial to do what you want to do.

(1) create a virtual disk.
(2) take the "old" snapshot.
(3) write to lvdisk
(4) take the "new" snapshot.


At this stage the COW device of the "old" snapshot has all the data that 
has changed up to and including the "new" snapshot. You can back that 
up. As a differential. Then delete the "old" snapshot. The "new" 
snapshot is now renamed to the old snapshot.

Take the next "new" snapshot. The renamed "old" snapshot has the changes 
since the previous snapshot up to and including the latest "new" 
snapshot. Just repeat this process, and you can do incremental backups 
of your LVM disks.

The biggest issue with performance is the COW aspect of snapshots. I 
have found using 64K chunk sizes greatly increase performance by 
reducing COW to snapshots. The default size if 4K.



On 10/03/2012 10:36 AM, linux-lvm-request@redhat.com wrote:
> From: Linda Walsh<lvm@tlinx.org>
> To: LVM general discussion and development<linux-lvm@redhat.com>
> Subject: [linux-lvm] access or interface to list of blocks that have
> 	changed	via C.O.W.?
> Message-ID:<50651097.2060200@tlinx.org>
> Content-Type: text/plain; charset=UTF-8; format=flowed
>
> I'm trying to use snapshots that are active for a day or so to grab
> differences between successive days for allowing me to display previous
> versions of files over a month or so...
>
> They aren't a replacement for my daily towero'hanoi-backups, but
> they are a great deal easier to use.
>
> The chink in my plan is the time it is taking to do a diff -- not really
> a fatal chink but annoying.  It takes anywhere from 80-100 minutes for
> rsync to sum up the changes and copy them to a third volume each day.
>
> I figured well sheesh, the lvm should already know what sectors are
> different.  If I could combine that info against a inode+blockmap, I might
> get that diff computation down to under 10 minutes maybe under 5.
>
> The contents of the files that are computed as differences by rsync usually
> take a minute or less to actually copy into final volume -- anywhere between
> 800Mb - 2.5GB out of a 1TB volume.   So I I could shortcut that diff
> process,
> might make it practical to do more than one snapshot a day;...
>
> Is there some interface for getting at that information or would it have
> to be written from scratch?

Re: [linux-lvm] access or interface to list of blocks that have, changed via C.O.W.?

[Linda Walsh]

Mark Woodward wrote:
> There are a couple projects that do this. They are pretty much based 
> on ddsnap. You can google it.
> In LVM2 world, it is fairly trivial to do what you want to do.
---
    I figured it was likely -- I as LVM2 has to to know what blocks
change to make realtime snapshots.  I just am trying to figure out how
to get a list of those blocks -- can I query some util and get the blocks
that are different at that point?   I was figuring on using that with
a blockmap of the fs, to get files that have changed, as I'm wanting to 
export
the files for smb(win client ) usage.
>
> (1) create a virtual disk.
> (2) take the "old" snapshot.
> (3) write to lvdisk
> (4) take the "new" snapshot.
>
>
> At this stage the COW device of the "old" snapshot has all the data 
> that has changed up to and including the "new" snapshot. You can back 
> that up. As a differential. Then delete the "old" snapshot. The "new" 
> snapshot is now renamed to the old snapshot.
----
    Now here's a confusion -- back it up as a differential?  Do you
mean from a backup utility or going from some list of blocks that have 
changed?


>
> Take the next "new" snapshot. The renamed "old" snapshot has the 
> changes since the previous snapshot up to and including the latest 
> "new" snapshot. Just repeat this process, and you can do incremental 
> backups of your LVM disks.
----
    I'm sorta already doing the above -- it's just that I'm doing my 'diff'
with 'rsync' and it's dog-slow.  100-120 minutes for ~800GB resulting in
about 2.5G of diff.  Then I shuffle that off to another static vol sized for
the content -- and the 'cp' usually takes about 60-70 seconds.

    What's hurting me is that "incremental backup" by having to scan the 
file
system.
>
> The biggest issue with performance is the COW aspect of snapshots. I 
> have found using 64K chunk sizes greatly increase performance by 
> reducing COW to snapshots. The default size if 4K.
----
    I didn't know it was that low as a default -- but am using 64K 
already --
as that's my RAID's 'chunksize' (I thought about experimenting with 
larger sizes, but would like it to run in a reasonable time first.

    Also I a relevant question 0-- when I do a dmsetup list, I see a 
bunch of
cow volumes for drives that I **had** snaps going from at one point.  
Seems like
the COW volumes didn't go away when halted...though it looks like, from 
the dates, that maybe they get cleaned up at a boot(?)

I only have 1 snapshot going but I see 14 cow partitions....looking like

VG-Home (254, 3)
VG-Home--2012.09.30--00.52.54   (254, 50)
VG-Home--2012.09.30--00.52.54-cow       (254, 51)
VG-Home--2012.10.01--04.58.11   (254, 52)
VG-Home--2012.10.01--04.58.11-cow       (254, 53)
VG-Home--2012.10.02--07.22.14   (254, 54)
VG-Home--2012.10.02--07.22.14-cow       (254, 55)
VG-Home--2012.10.03--09.08.27   (254, 56)
VG-Home--2012.10.03--09.08.27-cow       (254, 57)
VG-Home-real    (254, 2)

So would those be the list of blocks that changed upto the point they
were halted?

Do I need to worry about those "cow" vols taking up space?

Re: [linux-lvm] access or interface to list of blocks that have, changed via C.O.W.?

[Mark Woodward]

On 10/03/2012 10:52 PM, Linda Walsh wrote:
> Mark Woodward wrote:
>> There are a couple projects that do this. They are pretty much based 
>> on ddsnap. You can google it.
>> In LVM2 world, it is fairly trivial to do what you want to do.
> ---
>    I figured it was likely -- I as LVM2 has to to know what blocks
> change to make realtime snapshots.  I just am trying to figure out how
> to get a list of those blocks -- can I query some util and get the blocks
> that are different at that point?   I was figuring on using that with
> a blockmap of the fs, to get files that have changed, as I'm wanting 
> to export
> the files for smb(win client ) usage.
Well, I can honestly say that you are doing it the hard way. If you are 
connecting to a Linux box through samba, you can log the file changes.

>>
>> (1) create a virtual disk.
>> (2) take the "old" snapshot.
>> (3) write to lvdisk
>> (4) take the "new" snapshot.
>>
>>
>> At this stage the COW device of the "old" snapshot has all the data 
>> that has changed up to and including the "new" snapshot. You can back 
>> that up. As a differential. Then delete the "old" snapshot. The "new" 
>> snapshot is now renamed to the old snapshot.
> ----
>    Now here's a confusion -- back it up as a differential?  Do you
> mean from a backup utility or going from some list of blocks that have 
> changed?
I was talking about backing up the raw block level device.
>
>
>>
>> Take the next "new" snapshot. The renamed "old" snapshot has the 
>> changes since the previous snapshot up to and including the latest 
>> "new" snapshot. Just repeat this process, and you can do incremental 
>> backups of your LVM disks.
> ----
>    I'm sorta already doing the above -- it's just that I'm doing my 
> 'diff'
> with 'rsync' and it's dog-slow.  100-120 minutes for ~800GB resulting in
> about 2.5G of diff.  Then I shuffle that off to another static vol 
> sized for
> the content -- and the 'cp' usually takes about 60-70 seconds.
>
>    What's hurting me is that "incremental backup" by having to scan 
> the file
> system.
The file system is the hard way.
>>
>> The biggest issue with performance is the COW aspect of snapshots. I 
>> have found using 64K chunk sizes greatly increase performance by 
>> reducing COW to snapshots. The default size if 4K.
> ----
>    I didn't know it was that low as a default -- but am using 64K 
> already --
> as that's my RAID's 'chunksize' (I thought about experimenting with 
> larger sizes, but would like it to run in a reasonable time first.
>
>    Also I a relevant question 0-- when I do a dmsetup list, I see a 
> bunch of
> cow volumes for drives that I **had** snaps going from at one point.  
> Seems like
> the COW volumes didn't go away when halted...though it looks like, 
> from the dates, that maybe they get cleaned up at a boot(?)
>
> I only have 1 snapshot going but I see 14 cow partitions....looking like
>
> VG-Home (254, 3)
> VG-Home--2012.09.30--00.52.54   (254, 50)
> VG-Home--2012.09.30--00.52.54-cow       (254, 51)
> VG-Home--2012.10.01--04.58.11   (254, 52)
> VG-Home--2012.10.01--04.58.11-cow       (254, 53)
> VG-Home--2012.10.02--07.22.14   (254, 54)
> VG-Home--2012.10.02--07.22.14-cow       (254, 55)
> VG-Home--2012.10.03--09.08.27   (254, 56)
> VG-Home--2012.10.03--09.08.27-cow       (254, 57)
> VG-Home-real    (254, 2)
>
> So would those be the list of blocks that changed upto the point they
> were halted?
>
> Do I need to worry about those "cow" vols taking up space?
>
If they are active, not only are they taking up space, but they are also 
being updated with every write.
>
>

Re: [linux-lvm] access or interface to list of blocks that have, changed via C.O.W.?

[Linda Walsh]

Mark Woodward wrote:
> On 10/03/2012 10:52 PM, Linda Walsh wrote:
>> Mark Woodward wrote:
>>> There are a couple projects that do this. They are pretty much based 
>>> on ddsnap. You can google it.
>>> In LVM2 world, it is fairly trivial to do what you want to do.
>> ---
>>    I figured it was likely -- I as LVM2 has to to know what blocks
>> change to make realtime snapshots.  I just am trying to figure out how
>> to get a list of those blocks -- can I query some util and get the 
>> blocks
>> that are different at that point?   I was figuring on using that with
>> a blockmap of the fs, to get files that have changed, as I'm wanting 
>> to export
>> the files for smb(win client ) usage.
> Well, I can honestly say that you are doing it the hard way. If you 
> are connecting to a Linux box through samba, you can log the file 
> changes.
----
    Changes can come in via samba or locally so logging through samba 
wouldn't
cut it.
>
>>>
>>> (1) create a virtual disk.
>>> (2) take the "old" snapshot.
>>> (3) write to lvdisk
>>> (4) take the "new" snapshot.
>>>
>>>
>>> At this stage the COW device of the "old" snapshot has all the data 
>>> that has changed up to and including the "new" snapshot. You can 
>>> back that up. As a differential. Then delete the "old" snapshot. The 
>>> "new" snapshot is now renamed to the old snapshot.
>> ----
>>    Now here's a confusion -- back it up as a differential?  Do you
>> mean from a backup utility or going from some list of blocks that 
>> have changed?
> I was talking about backing up the raw block level device.
----
    I'm not sure that would work for me -- as I'm planning on just storing
the files that have changed for ~ a month and rotating them out...

That's why I'm hoping to get the block numbers that have changed -- if I can
map those to 1 or more inodes -- I could just back them up rather than 
walking
a 5Million+ file tree.


>>
>>
>>>
>>> Take the next "new" snapshot. The renamed "old" snapshot has the 
>>> changes since the previous snapshot up to and including the latest 
>>> "new" snapshot. Just repeat this process, and you can do incremental 
>>> backups of your LVM disks.
>> ----
>>    I'm sorta already doing the above -- it's just that I'm doing my 
>> 'diff'
>> with 'rsync' and it's dog-slow.  100-120 minutes for ~800GB resulting in
>> about 2.5G of diff.  Then I shuffle that off to another static vol 
>> sized for
>> the content -- and the 'cp' usually takes about 60-70 seconds.
>>
>>    What's hurting me is that "incremental backup" by having to scan 
>> the file
>> system.
> The file system is the hard way.
----
    Yep... tell me about it...

>>>
>>> The biggest issue with performance is the COW aspect of snapshots. I 
>>> have found using 64K chunk sizes greatly increase performance by 
>>> reducing COW to snapshots. The default size if 4K.
>> ----
>>    I didn't know it was that low as a default -- but am using 64K 
>> already --
>> as that's my RAID's 'chunksize' (I thought about experimenting with 
>> larger sizes, but would like it to run in a reasonable time first.
>>
>>    Also I a relevant question 0-- when I do a dmsetup list, I see a 
>> bunch of
>> cow volumes for drives that I **had** snaps going from at one point.  
>> Seems like
>> the COW volumes didn't go away when halted...though it looks like, 
>> from the dates, that maybe they get cleaned up at a boot(?)
>>
>> I only have 1 snapshot going but I see 14 cow partitions....looking like
>>
>> VG-Home (254, 3)
>> VG-Home--2012.09.30--00.52.54   (254, 50)
>> VG-Home--2012.09.30--00.52.54-cow       (254, 51)
>> VG-Home--2012.10.01--04.58.11   (254, 52)
>> VG-Home--2012.10.01--04.58.11-cow       (254, 53)
>> VG-Home--2012.10.02--07.22.14   (254, 54)
>> VG-Home--2012.10.02--07.22.14-cow       (254, 55)
>> VG-Home--2012.10.03--09.08.27   (254, 56)
>> VG-Home--2012.10.03--09.08.27-cow       (254, 57)
>> VG-Home-real    (254, 2)
>>
>> So would those be the list of blocks that changed upto the point they
>> were halted?
>>
>> Do I need to worry about those "cow" vols taking up space?
>>
> If they are active, not only are they taking up space, but they are 
> also being updated with every write.
----
    I doubt that -- their corresponding snapshot volumes are gone
Odd that they cow volumesdon't go as well.


    When I delete an active snap, I first remove the snapshot volume
with dmsetup remove -- then I lvremove it.  Seems to work without me getting
warnings about removing an active volume -- I'd assume when I lvremove'd
a snap, that'd be all I need to do...

    FWIW -- only cow files are the ones since the last reboot.  That's 
why I
wondered if a reboot cleaned out the spurious entries..

    I try to only keep one active snapshot going at a time due to the write
penalty...

Re: [linux-lvm] access or interface to list of blocks that have, changed via C.O.W.?

[Mark Woodward]

I was going to try to answer in line, but decided that it would be too 
much work. There are utilities to extract the exception table out of the 
LVM2 snapshot, and if you can code in almost any language, you can write 
your own. It is dead simple. You can google for ddsnap and zumastore to 
get the code. It old and not supported currently, but still works. I 
have my own snapshotting differential backup running on my servers on a 
cron. (sorry, block level)

Essentially the COW device is formatted in blocks the size of the chunk 
size. The first chunk only contains a header. The second chunk contains 
an array of maps. This chunk is followed by all the data for the array. 
If the snapshot grows beyond this, a new chunk with an array is written.

The format of the array is simple: old_address (The offset in the 
volume) followed by the new_address (the offset in the COW device). An 
array of all the "old_address" values is the changed block list. You 
don't even need to worry about the data if you can really get a file 
list by blocks.

BTW: I would be curious how you do that and what file system supports it?

On 10/04/2012 01:05 AM, Linda Walsh wrote:
> Mark Woodward wrote:
>> On 10/03/2012 10:52 PM, Linda Walsh wrote:
>>> Mark Woodward wrote:
>>>> There are a couple projects that do this. They are pretty much 
>>>> based on ddsnap. You can google it.
>>>> In LVM2 world, it is fairly trivial to do what you want to do.
>>> ---
>>>    I figured it was likely -- I as LVM2 has to to know what blocks
>>> change to make realtime snapshots.  I just am trying to figure out how
>>> to get a list of those blocks -- can I query some util and get the 
>>> blocks
>>> that are different at that point?   I was figuring on using that with
>>> a blockmap of the fs, to get files that have changed, as I'm wanting 
>>> to export
>>> the files for smb(win client ) usage.
>> Well, I can honestly say that you are doing it the hard way. If you 
>> are connecting to a Linux box through samba, you can log the file 
>> changes.
> ----
>    Changes can come in via samba or locally so logging through samba 
> wouldn't
> cut it.
>>
>>>>
>>>> (1) create a virtual disk.
>>>> (2) take the "old" snapshot.
>>>> (3) write to lvdisk
>>>> (4) take the "new" snapshot.
>>>>
>>>>
>>>> At this stage the COW device of the "old" snapshot has all the data 
>>>> that has changed up to and including the "new" snapshot. You can 
>>>> back that up. As a differential. Then delete the "old" snapshot. 
>>>> The "new" snapshot is now renamed to the old snapshot.
>>> ----
>>>    Now here's a confusion -- back it up as a differential?  Do you
>>> mean from a backup utility or going from some list of blocks that 
>>> have changed?
>> I was talking about backing up the raw block level device.
> ----
>    I'm not sure that would work for me -- as I'm planning on just storing
> the files that have changed for ~ a month and rotating them out...
>
> That's why I'm hoping to get the block numbers that have changed -- if 
> I can
> map those to 1 or more inodes -- I could just back them up rather than 
> walking
> a 5Million+ file tree.
>
>
>>>
>>>
>>>>
>>>> Take the next "new" snapshot. The renamed "old" snapshot has the 
>>>> changes since the previous snapshot up to and including the latest 
>>>> "new" snapshot. Just repeat this process, and you can do 
>>>> incremental backups of your LVM disks.
>>> ----
>>>    I'm sorta already doing the above -- it's just that I'm doing my 
>>> 'diff'
>>> with 'rsync' and it's dog-slow.  100-120 minutes for ~800GB 
>>> resulting in
>>> about 2.5G of diff.  Then I shuffle that off to another static vol 
>>> sized for
>>> the content -- and the 'cp' usually takes about 60-70 seconds.
>>>
>>>    What's hurting me is that "incremental backup" by having to scan 
>>> the file
>>> system.
>> The file system is the hard way.
> ----
>    Yep... tell me about it...
>
>>>>
>>>> The biggest issue with performance is the COW aspect of snapshots. 
>>>> I have found using 64K chunk sizes greatly increase performance by 
>>>> reducing COW to snapshots. The default size if 4K.
>>> ----
>>>    I didn't know it was that low as a default -- but am using 64K 
>>> already --
>>> as that's my RAID's 'chunksize' (I thought about experimenting with 
>>> larger sizes, but would like it to run in a reasonable time first.
>>>
>>>    Also I a relevant question 0-- when I do a dmsetup list, I see a 
>>> bunch of
>>> cow volumes for drives that I **had** snaps going from at one 
>>> point.  Seems like
>>> the COW volumes didn't go away when halted...though it looks like, 
>>> from the dates, that maybe they get cleaned up at a boot(?)
>>>
>>> I only have 1 snapshot going but I see 14 cow partitions....looking 
>>> like
>>>
>>> VG-Home (254, 3)
>>> VG-Home--2012.09.30--00.52.54   (254, 50)
>>> VG-Home--2012.09.30--00.52.54-cow       (254, 51)
>>> VG-Home--2012.10.01--04.58.11   (254, 52)
>>> VG-Home--2012.10.01--04.58.11-cow       (254, 53)
>>> VG-Home--2012.10.02--07.22.14   (254, 54)
>>> VG-Home--2012.10.02--07.22.14-cow       (254, 55)
>>> VG-Home--2012.10.03--09.08.27   (254, 56)
>>> VG-Home--2012.10.03--09.08.27-cow       (254, 57)
>>> VG-Home-real    (254, 2)
>>>
>>> So would those be the list of blocks that changed upto the point they
>>> were halted?
>>>
>>> Do I need to worry about those "cow" vols taking up space?
>>>
>> If they are active, not only are they taking up space, but they are 
>> also being updated with every write.
> ----
>    I doubt that -- their corresponding snapshot volumes are gone
> Odd that they cow volumesdon't go as well.
>
>
>    When I delete an active snap, I first remove the snapshot volume
> with dmsetup remove -- then I lvremove it.  Seems to work without me 
> getting
> warnings about removing an active volume -- I'd assume when I lvremove'd
> a snap, that'd be all I need to do...
>
>    FWIW -- only cow files are the ones since the last reboot. That's 
> why I
> wondered if a reboot cleaned out the spurious entries..
>
>    I try to only keep one active snapshot going at a time due to the 
> write
> penalty...
>
>
>

Re: [linux-lvm] access or interface to list of blocks that have, changed via C.O.W.?

[Bryn M. Reeves]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 10/04/2012 11:17 AM, Mark Woodward wrote:
> I was going to try to answer in line, but decided that it would be
> too much work. There are utilities to extract the exception table
> out of the LVM2 snapshot, and if you can code in almost any
> language, you can write your own. It is dead simple. You can google
> for ddsnap and zumastore to

ddsnap was the Zumastor snapshot tool which uses a different in-kernel
snapshot target and metadata format. It does not share code with
current (or historic afaik) LVM2/device-mapper and has been dead for a
number of years (last commits around 2008).

> get the code. It old and not supported currently, but still works.
> I

Really? I'd be surprised if it even builds against modern kernels or
device-mapper.

> The format of the array is simple: old_address (The offset in the 
> volume) followed by the new_address (the offset in the COW device).
> An array of all the "old_address" values is the changed block list.
> You don't even need to worry about the data if you can really get a
> file list by blocks.

If you really want to poke into the CoW store format I'd start by
reading dm-snap-persistent.c which is the traditional device-mapper
snapshot format.

Snapshots using the thinp target use the metadata format described in
dm-thin-metadata.c.

Regards,
Bryn.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAlBtoLQACgkQ6YSQoMYUY95MgACdFnOXbJC03HjQA8/p8kKxXeLD
ALMAmQFhD+OfKiA2YV9RS3kaUGFzvzE0
=wsdZ
-----END PGP SIGNATURE-----

Re: [linux-lvm] access or interface to list of blocks that have, changed via C.O.W.?

[Mark Woodward]

On 10/04/2012 10:44 AM, Bryn M. Reeves wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 10/04/2012 11:17 AM, Mark Woodward wrote:
>> I was going to try to answer in line, but decided that it would be
>> too much work. There are utilities to extract the exception table
>> out of the LVM2 snapshot, and if you can code in almost any
>> language, you can write your own. It is dead simple. You can google
>> for ddsnap and zumastore to
> ddsnap was the Zumastor snapshot tool which uses a different in-kernel
> snapshot target and metadata format. It does not share code with
> current (or historic afaik) LVM2/device-mapper and has been dead for a
> number of years (last commits around 2008).
>
>> get the code. It old and not supported currently, but still works.
>> I
> Really? I'd be surprised if it even builds against modern kernels or
> device-mapper.
Well.....
The LVM2 code that ships with most distros still uses the old interfaces 
and still (Ubuntu 12.04) seemst to create a COW file with SnAp

>
>> The format of the array is simple: old_address (The offset in the
>> volume) followed by the new_address (the offset in the COW device).
>> An array of all the "old_address" values is the changed block list.
>> You don't even need to worry about the data if you can really get a
>> file list by blocks.
> If you really want to poke into the CoW store format I'd start by
> reading dm-snap-persistent.c which is the traditional device-mapper
> snapshot format.
>
> Snapshots using the thinp target use the metadata format described in
> dm-thin-metadata.c.

Where are these files to be found?
>
> Regards,
> Bryn.
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.12 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAlBtoLQACgkQ6YSQoMYUY95MgACdFnOXbJC03HjQA8/p8kKxXeLD
> ALMAmQFhD+OfKiA2YV9RS3kaUGFzvzE0
> =wsdZ
> -----END PGP SIGNATURE-----

Re: [linux-lvm] access or interface to list of blocks that have, changed via C.O.W.?

[Stuart D Gathman]

On 10/04/2012 11:02 AM, Mark Woodward expounded in part:
>
>> snapshot target and metadata format. It does not share code with
>> current (or historic afaik) LVM2/device-mapper and has been dead for a
>> number of years (last commits around 2008).
>>
>>> get the code. It old and not supported currently, but still works.
>>> I
>> Really? I'd be surprised if it even builds against modern kernels or
>> device-mapper.
> ddsnap was the Zumastor snapshot tool which uses a different in-kernel
> Well.....
> The LVM2 code that ships with most distros still uses the old 
> interfaces and still (Ubuntu 12.04) seemst to create a COW file with SnAp

This utility (ddsnap work alike) requires no kernel changes, and only 
depends on the layout of the COW (and only on the changed block table):

http://www.opensvc.com/init/default/doc_storage_dds

Re: [linux-lvm] access or interface to list of blocks that have, changed via C.O.W.?

[Bryn M. Reeves]

On 10/04/2012 11:00 PM, Stuart D Gathman wrote:
> On 10/04/2012 11:02 AM, Mark Woodward expounded in part:
>>
>>> snapshot target and metadata format. It does not share code with
>>> current (or historic afaik) LVM2/device-mapper and has been dead for a
>>> number of years (last commits around 2008).
>>>
>>>> get the code. It old and not supported currently, but still works.
>>>> I
>>> Really? I'd be surprised if it even builds against modern kernels or
>>> device-mapper.
>> ddsnap was the Zumastor snapshot tool which uses a different in-kernel
>> Well.....
>> The LVM2 code that ships with most distros still uses the old 
>> interfaces and still (Ubuntu 12.04) seemst to create a COW file with SnAp
> 
> This utility (ddsnap work alike) requires no kernel changes, and only 
> depends on the layout of the COW (and only on the changed block table):
> 
> http://www.opensvc.com/init/default/doc_storage_dds

This is closer to the kind of thing that Linda was originally asking
about. It seems to let you extract deltas and move them around and
re-apply them for replication purposes so it's similar to ddsnap in that
it's distributed although as you say it's entirely in userspace and
doesn't requite the ddsnap device-mapper target.

Regards,
Bryn.

Re: [linux-lvm] access or interface to list of blocks that have, changed via C.O.W.?

[Bryn M. Reeves]

On 10/04/2012 04:02 PM, Mark Woodward wrote:
> On 10/04/2012 10:44 AM, Bryn M. Reeves wrote:
>> Really? I'd be surprised if it even builds against modern kernels or
>> device-mapper.
> Well.....
> The LVM2 code that ships with most distros still uses the old interfaces 
> and still (Ubuntu 12.04) seemst to create a COW file with SnAp

Yes; that's the traditional dm-snapshot target that has been in the
kernel for years. The Zumastor/ddsnap stuff was a separate project to
replace that using a new metadata format and target - they are not the
same and the Zumastor snapshot code was never merged in the kernel.

>> Snapshots using the thinp target use the metadata format described in
>> dm-thin-metadata.c.
> 
> Where are these files to be found?

In drivers/md in the kernel sources - the dm-thin stuff was merged after
3.1 (October last year); see commits starting with 991d9fa.

Regards,
Bryn.

Re: [linux-lvm] access or interface to list of blocks that have, changed via C.O.W.?

[Mark Woodward]

On 10/05/2012 05:26 AM, Bryn M. Reeves wrote:
> On 10/04/2012 04:02 PM, Mark Woodward wrote:
>> On 10/04/2012 10:44 AM, Bryn M. Reeves wrote:
>>> Really? I'd be surprised if it even builds against modern kernels or
>>> device-mapper.
>> Well.....
>> The LVM2 code that ships with most distros still uses the old interfaces
>> and still (Ubuntu 12.04) seemst to create a COW file with SnAp
> Yes; that's the traditional dm-snapshot target that has been in the
> kernel for years. The Zumastor/ddsnap stuff was a separate project to
> replace that using a new metadata format and target - they are not the
> same and the Zumastor snapshot code was never merged in the kernel.
So, after this discussion, I went to my Ubuntu 12.04 system (3.2.0) and 
tried some of my LVM2 utilities, and they read the header but found no 
changes in a snapshot when clearly there had been some.

It seems I am wrong, but I'm a little confused, did the developers 
change the format of the snapshot data COW file without changing the 
file header signature? If so, that is very bad thing to do. If the COW 
format has changed and the device header signature has not, how does one 
correctly determine how the data is stored?
>
>>> Snapshots using the thinp target use the metadata format described in
>>> dm-thin-metadata.c.
>> Where are these files to be found?
> In drivers/md in the kernel sources - the dm-thin stuff was merged after
> 3.1 (October last year); see commits starting with 991d9fa.
>
> Regards,
> Bryn.

Re: [linux-lvm] access or interface to list of blocks that have, changed via C.O.W.?

[Mark Woodward]

On 10/04/2012 10:44 AM, Bryn M. Reeves wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 10/04/2012 11:17 AM, Mark Woodward wrote:
>> I was going to try to answer in line, but decided that it would be
>> too much work. There are utilities to extract the exception table
>> out of the LVM2 snapshot, and if you can code in almost any
>> language, you can write your own. It is dead simple. You can google
>> for ddsnap and zumastore to
> ddsnap was the Zumastor snapshot tool which uses a different in-kernel
> snapshot target and metadata format. It does not share code with
> current (or historic afaik) LVM2/device-mapper and has been dead for a
> number of years (last commits around 2008).
>
>> get the code. It old and not supported currently, but still works.
>> I
> Really? I'd be surprised if it even builds against modern kernels or
> device-mapper.
>
>> The format of the array is simple: old_address (The offset in the
>> volume) followed by the new_address (the offset in the COW device).
>> An array of all the "old_address" values is the changed block list.
>> You don't even need to worry about the data if you can really get a
>> file list by blocks.
> If you really want to poke into the CoW store format I'd start by
> reading dm-snap-persistent.c which is the traditional device-mapper
> snapshot format.
>
> Snapshots using the thinp target use the metadata format described in
> dm-thin-metadata.c.

Sorry, I got pulled away a bit.

Let me understand. The CoW format changed, but the header signature 
stayed the same? I'm having a bit of an issue understanding this. The 
"old" LVM format CoW device had the signature "SnAp" and that had a 
fairly well understood layout.  Has this or has this not changed? I had 
some code that worked in the 2.6 series of kernels using the LVM tools. 
However, when I run it on a later kernel version (3.2.0) it seems not to 
work. I am more than willing to say its a bug on my end, but can you 
clarify whether or not it is expected that this format would change? I 
am planning a server upgrade and I rely on the snapshot format for my 
differential backup tools.

As for the thin provisioned snapshot I have been long waiting for this. 
Is it "production ready?" if so, in what kernel version moving forward?
>
> Regards,
> Bryn.
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.12 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAlBtoLQACgkQ6YSQoMYUY95MgACdFnOXbJC03HjQA8/p8kKxXeLD
> ALMAmQFhD+OfKiA2YV9RS3kaUGFzvzE0
> =wsdZ
> -----END PGP SIGNATURE-----

Re: [linux-lvm] access or interface to list of blocks that have, changed via C.O.W.?

[Lars Ellenberg]

On Tue, Oct 16, 2012 at 08:15:34AM -0400, Mark Woodward wrote:
> On 10/04/2012 10:44 AM, Bryn M. Reeves wrote:
> >-----BEGIN PGP SIGNED MESSAGE-----
> >Hash: SHA1
> >
> >On 10/04/2012 11:17 AM, Mark Woodward wrote:
> >>I was going to try to answer in line, but decided that it would be
> >>too much work. There are utilities to extract the exception table
> >>out of the LVM2 snapshot, and if you can code in almost any
> >>language, you can write your own. It is dead simple. You can google
> >>for ddsnap and zumastore to
> >ddsnap was the Zumastor snapshot tool which uses a different in-kernel
> >snapshot target and metadata format. It does not share code with
> >current (or historic afaik) LVM2/device-mapper and has been dead for a
> >number of years (last commits around 2008).
> >
> >>get the code. It old and not supported currently, but still works.
> >>I
> >Really? I'd be surprised if it even builds against modern kernels or
> >device-mapper.
> >
> >>The format of the array is simple: old_address (The offset in the
> >>volume) followed by the new_address (the offset in the COW device).
> >>An array of all the "old_address" values is the changed block list.
> >>You don't even need to worry about the data if you can really get a
> >>file list by blocks.
> >If you really want to poke into the CoW store format I'd start by
> >reading dm-snap-persistent.c which is the traditional device-mapper
> >snapshot format.
> >
> >Snapshots using the thinp target use the metadata format described in
> >dm-thin-metadata.c.
> 
> Sorry, I got pulled away a bit.
> 
> Let me understand. The CoW format changed, but the header signature
> stayed the same?

Not at all.

> I'm having a bit of an issue understanding this.

That's probably because it is nonsense ;-)


So we have the "traditional" COW snapshots.
And we have the "new" "thin provisioned pool" based snapshots.

The thinp target went in in upstream 3.2.
It is also present in RHEL 6.3 ("2.6.32"+rhel patches).

This thing has some "pool" from which it "allocates" chunks,
and some "meta" device where it persists its logical to physical mapping
and chunk refcounting in some btree.

The relevant source files are in the kernel source tree,
drivers/md/dm-thin*

It comes with userland tools to dump its meta data: thin_dump,
from package device-mapper-persistent-data, respectively
https://github.com/jthornber/thin-provisioning-tools


The "traditional" COW based snapshot lives in kernel source as
drivers/md/dm-snap*, and has a very simple meta data layout,
which you probably know about already.

If not, here are some links:
http://thread.gmane.org/gmane.linux.lvm.general/9898/focus=9899
http://thread.gmane.org/gmane.linux.lvm.general/10886
http://docs.opensvc.com/storage.dds.html
http://git.opensvc.com/?p=dds/.git;a=summary

No, that meta data layout did not change (afaik).

> I had some code that worked in the 2.6 series of kernels
> using the LVM tools. However, when I run it on a later kernel
> version (3.2.0) it seems not to work. I am more than willing to say
> its a bug on my end, but can you clarify whether or not it is
> expected that this format would change?

Nope.

Maybe you can be more specific as to what your code does,
and how/when it fails, currently?

> I am planning a server upgrade and I rely on the snapshot format for
> my differential backup tools.
> 
> As for the thin provisioned snapshot I have been long waiting for
> this. Is it "production ready?" if so, in what kernel version moving
> forward?

It is "tech preview" in RHEL 6.3.
It is in upstream kernel since 3.2.
It "learned" a few things since (e.g. discard support with 3.6).

"Production ready"...
Well. I'm sure some use it in production alright.
Wether that means it is fit for *your* use case, I don't know.

Hth,

-- 
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBD� and LINBIT� are registered trademarks of LINBIT, Austria.

[linux-lvm] access or interface to list of blocks that have changed via C.O.W.?

[Linda Walsh]

I'm trying to use snapshots that are active for a day or so to grab
differences between successive days for allowing me to display previous
versions of files over a month or so...

They aren't a replacement for my daily towero'hanoi-backups, but
they are a great deal easier to use.

The chink in my plan is the time it is taking to do a diff -- not really
a fatal chink but annoying.  It takes anywhere from 80-100 minutes for
rsync to sum up the changes and copy them to a third volume each day.

I figured well sheesh, the lvm should already know what sectors are
different.  If I could combine that info against a inode+blockmap, I might
get that diff computation down to under 10 minutes maybe under 5.

The contents of the files that are computed as differences by rsync usually
take a minute or less to actually copy into final volume -- anywhere between
800Mb - 2.5GB out of a 1TB volume.   So I I could shortcut that diff 
process,
might make it practical to do more than one snapshot a day;...

Is there some interface for getting at that information or would it have
to be written from scratch?


Thnks...

Linda