[linux-lvm] [RFC] Out of order snapshot deletion

[linux-lvm] [RFC] Out of order snapshot deletion

[Vijai Babu Madhavan]

Hi,

I thought I would bring this topic in a separate thread so that 
it can be discussed independently.

As users start taking a bunch of snapshots and keep them as 
backups, we see the following use pattern when the snapshots 
get recycled.

Different snapshots have different life cycle. When some one 
decides to keep 'x' snapshots, at the recycle phase, the least 
recent does not necessarily get deleted.

As users want to keep the daily snapshots for a much longer period
than the bi-hourly snapshots. In the same way, weekly snapshots 
live longer than daily snapshots.

I think this is pretty much similar to the way the current backup tapes
are managed.

I want to know if this is the kind of use case that users are having or 
the developers are thinking.

The reason why I am posting this is that this out of snapshot deletion
scenario, poses some interesting challenges in the design on the solution 
of multiple snapshots that share blocks with each other, as it is a lot easier 
to design a solution where the least recent gets deleted always.

Vijai

[linux-lvm] Re: [dm-devel] [RFC] Out of order snapshot deletion

[Ming Zhang]

On Thu, 2007-01-11 at 22:19 -0700, Vijai Babu Madhavan wrote:
> Hi,
> 
> I thought I would bring this topic in a separate thread so that 
> it can be discussed independently.
> 
> As users start taking a bunch of snapshots and keep them as 
> backups, we see the following use pattern when the snapshots 
> get recycled.
> 
> Different snapshots have different life cycle. When some one 
> decides to keep 'x' snapshots, at the recycle phase, the least 
> recent does not necessarily get deleted.
> 
> As users want to keep the daily snapshots for a much longer period
> than the bi-hourly snapshots. In the same way, weekly snapshots 
> live longer than daily snapshots.
> 
> I think this is pretty much similar to the way the current backup tapes
> are managed.

make sense.

> 
> I want to know if this is the kind of use case that users are having or 
> the developers are thinking.
> 
> The reason why I am posting this is that this out of snapshot deletion
> scenario, poses some interesting challenges in the design on the solution 
> of multiple snapshots that share blocks with each other, as it is a lot easier 
> to design a solution where the least recent gets deleted always.

this does not hold true. for snapshot used as backup purpose, you 100%
can not assume the oldest or newest will be delete first. there is no
such rule.

and possibly, you create one snapshot base on policy and later mount it
and check data consistency. if it turns out to be a bad one, you want to
delete it. then you have no order to follow at all.

ps, any comment on http://sourceware.org/cluster/csnap/? this also try
to solve the current multiple snapshot issues.

thanks

Ming

> 
> Vijai
> 
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
-- 
http://blackmagic02881.wordpress.com/

[linux-lvm] RE: [dm-devel] [RFC] Out of order snapshot deletion

[Wilson, Christopher J]

My opinion is that "snapshots" should be less persistent than full clone
copies.  Longer term retention should move to full clone then you don't
have the snapshot overhead you've raised on your other thread.  I can't
imagine that there are so many administrators out there wanting to
maintain week old snapshots.  Those older ones would most likely have
been mounted somewhere and spooled to tape.  I could be way off base but
I'd not feel so confident in a week old cow as I would with a week old
clone.

We here have been tossing around the idea of treating snapshot (or full
clone) as backup media.  Media that would contain a barcode and which
could be migrated to physical tape.  Currently this is the realm of VTL.
VTL is a copy of your data written out in a sequential manner, if you
could somehow create disk media via snapshot that the disk caches
(Falconstor, Sepaton) use then you could get the long sequential
read/write overhead out of the backup application and the host.

Our issue with snapshot, as I mentioned on the other thread, is that
there are no close ties to the upper level applications, this includes
backup software like NetBackup and Legato.  NetBackup has the "virtual
backup" piece but I've not really seen it implemented.  Ideally an
interface for a database application to take a snapshot of its data and
call it barcode xxxx03 could then be imported into your backup
application as a virtual tape xxxx03.  That "tape" could be cloned via
NetBackup or disk cache software to real tape with barcode xxxx03 for
off site storage (through an encryption device of course).  I know there
are data alignment issues that exist here with data sizes and physical
tape sizes but we believe it's an avenue worth exploring.

There are several opportunities to better admin quality of life here.
One is that backup administrators would be able to continually manage
their media in what they know, tapes with barcodes they can touch and
feel.  The other is DBA's could maintain a list of snapshots through
something other than "glue code".  It would integrate into the processes
that exist around data protection.

Christopher Wilson
Storage Architect
Verizon Business
IT Solutions - IP Application Hosting
240 264 4136
vnet: 364 4136

 

-----Original Message-----
From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com]
On Behalf Of Vijai Babu Madhavan
Sent: Friday, January 12, 2007 12:19 AM
To: evms-devel@lists.sourceforge.net; dm-devel@redhat.com;
linux-lvm@redhat.com
Subject: [dm-devel] [RFC] Out of order snapshot deletion

Hi,

I thought I would bring this topic in a separate thread so that it can
be discussed independently.

As users start taking a bunch of snapshots and keep them as backups, we
see the following use pattern when the snapshots get recycled.

Different snapshots have different life cycle. When some one decides to
keep 'x' snapshots, at the recycle phase, the least recent does not
necessarily get deleted.

As users want to keep the daily snapshots for a much longer period than
the bi-hourly snapshots. In the same way, weekly snapshots live longer
than daily snapshots.

I think this is pretty much similar to the way the current backup tapes
are managed.

I want to know if this is the kind of use case that users are having or
the developers are thinking.

The reason why I am posting this is that this out of snapshot deletion
scenario, poses some interesting challenges in the design on the
solution of multiple snapshots that share blocks with each other, as it
is a lot easier to design a solution where the least recent gets deleted
always.

Vijai


--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

______________________________________________________________________
This e-mail has been scanned by Verizon Managed Email Content Service,
using Skeptic(tm) technology powered by MessageLabs. For more
information on Verizon Managed Email Content Service, visit
http://www.verizonbusiness.com.
______________________________________________________________________


______________________________________________________________________
This e-mail has been scanned by Verizon Managed Email Content Service, using Skeptic� technology powered by MessageLabs. For more information on Verizon Managed Email Content Service, visit http://www.verizonbusiness.com.
______________________________________________________________________