[linux-lvm] What is holding back clustered snapshotting?

[linux-lvm] What is holding back clustered snapshotting?

[Madison Kelly]

Hi all,

   I know that, currently, this isn't supported. Would someone be able 
to explain or point me at a place to read up on what is holding this 
feature back? What are the difficulties? Is it just a question of time, 
or are there certain technical hurdles in the way?

   Thanks for any input!

Madi

Re: [linux-lvm] What is holding back clustered snapshotting?

[Stuart D. Gathman]

On Tue, 5 Jan 2010, Madison Kelly wrote:

>   I know that, currently, this isn't supported. Would someone be able to
> explain or point me at a place to read up on what is holding this feature
> back? What are the difficulties? Is it just a question of time, or are there
> certain technical hurdles in the way?

Setting up the shapshot is just a matter of locking and coordination.  
However, writes to the origin or snapshot (may) require allocating a cluster,
copying the origin data, then writing the origin.  All of this coordinated
with all the machines using the VG.  Apart from some cleven invention, this
requires global locking on many writes.  This is just too inefficient.

However, you can obtain the same effect using a SAN.  Have one machine
run LVM (and raid, etc), and export LVs via AoE or iSCSI.  Of course,
that LVM machine now becomes a single point of failure...

Here's an idea (someone probably already thought of this, but..), 
have one machine in a cluster elected "master" for a VG, and have all
reads/writes from other machines go through the master via AoE or iSCSI.
When failure of the "master" is detected, elect another machine to take over
as master.  Sort of a rotating SAN server.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] What is holding back clustered snapshotting?

[Ray Van Dolson]

On Tue, Jan 05, 2010 at 11:11:08AM -0800, Madison Kelly wrote:
> Hi all,
> 
>    I know that, currently, this isn't supported. Would someone be able 
> to explain or point me at a place to read up on what is holding this 
> feature back? What are the difficulties? Is it just a question of time, 
> or are there certain technical hurdles in the way?
> 
>    Thanks for any input!
> 

I'm sure there are some technical hurdles... but perhaps customers need
to put more of a priority on it.

I keep forgetting to open an RFE SR for this feature.  That sorta stuff
makes a difference in prioritization for RH's engineering though for
sure!

Ray

Re: [linux-lvm] What is holding back clustered snapshotting?

[Ray Van Dolson]

On Tue, Jan 05, 2010 at 11:30:01AM -0800, Stuart D. Gathman wrote:
> On Tue, 5 Jan 2010, Madison Kelly wrote:
> 
> >   I know that, currently, this isn't supported. Would someone be able to
> > explain or point me at a place to read up on what is holding this feature
> > back? What are the difficulties? Is it just a question of time, or are there
> > certain technical hurdles in the way?
> 
> Setting up the shapshot is just a matter of locking and coordination.
> However, writes to the origin or snapshot (may) require allocating a
> cluster, copying the origin data, then writing the origin.  All of
> this coordinated with all the machines using the VG.  Apart from some
> cleven invention, this requires global locking on many writes.  This
> is just too inefficient.

Writeable snapshots I guess would be a challenge.  But even read only
snapshots would be great as it would theoretically make backing up
large, clustered filesystems simpler.

> 
> However, you can obtain the same effect using a SAN.  Have one
> machine run LVM (and raid, etc), and export LVs via AoE or iSCSI.  Of
> course, that LVM machine now becomes a single point of failure...
> 
> Here's an idea (someone probably already thought of this, but..),
> have one machine in a cluster elected "master" for a VG, and have all
> reads/writes from other machines go through the master via AoE or
> iSCSI.  When failure of the "master" is detected, elect another
> machine to take over as master.  Sort of a rotating SAN server.
> 

Interesting idea. :)

Ray

Re: [linux-lvm] What is holding back clustered snapshotting?

[Stuart D. Gathman]

On Tue, 5 Jan 2010, Ray Van Dolson wrote:

> > Setting up the shapshot is just a matter of locking and coordination.
> > However, writes to the origin or snapshot (may) require allocating a
> > cluster, copying the origin data, then writing the origin.  All of
> > this coordinated with all the machines using the VG.  Apart from some
> > cleven invention, this requires global locking on many writes.  This
> > is just too inefficient.
> 
> Writeable snapshots I guess would be a challenge.  But even read only
> snapshots would be great as it would theoretically make backing up
> large, clustered filesystems simpler.

Even read-only snapshots have to handle writes to the origin.  And
anytime any machine writes a new cluster to the origin, all machines need their
cluster maps updated so that reads don't pick up the wrong cluster.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] What is holding back clustered snapshotting?

[Jonathan Brassow]

On Jan 5, 2010, at 1:11 PM, Madison Kelly wrote:

> Hi all,
>
>  I know that, currently, this isn't supported. Would someone be able  
> to explain or point me at a place to read up on what is holding this  
> feature back? What are the difficulties? Is it just a question of  
> time, or are there certain technical hurdles in the way?
>
>  Thanks for any input!

The solution isn't ridiculously hard.  It has actually been  
implemented a couple times by me and at least once by Mikulas  
leveraging a locking interface I had proposed.  (You can search the  
various dm-devel archives for them.)

Our current priority is to improve the snapshot implementation (shared  
snapshots), but cluster snapshots is on the todo list.  It does tend  
to be a matter of time.  As I said, a couple implementations have come  
forward, but it requires a team effort to push it upstream - the  
priority hasn't been high enough for it to surpass other things on the  
list.

  brassow