Adding snapshot capability to Linux

Adding snapshot capability to Linux

[Libor Vaněk]

Hi,
I'm going to start my dissertation work which is "Adding snapshop 
capability to Linux kernel with copy-on-write support". My idea is add 
it as another VFS - I know that there is some snapshot support in LVM 
but it's working on "device-level" and I'd like/have to do it on fs level.

My idea is to use it this way:
- I have running system with some "/foo" dir
- I want to make snapshot of "/foo/bar" to "/foo/snap1"
- I run "mount -t snapshot /foo/bar /foo/snap1"
- This creates virtual image of "/foo/bar" to the "/foo/snap1" with 
hidden file (something like journal) in "/foo/snap1" - all files are 
linked to "/foo/bar"
- Whenever is some file/dir changed in "/foo/bar" there is created 
physical copy of it to the snapshot(s) before writing changes (for 
making records about this will be used the hidden file)
- Of course that one directory can be snapshoted more times
- Probably the hidden file with records about all snapshots and details 
should be stored in "/foo/bar"
- Question is how to handle ACLs and EA for XFS/JFS/... and if it won't 
collide with journal

I'd like to do it not only because I have to but I want people to use it 
(I want to make it GPL) and maybe it will be one nice day part of Linux 
kernel ;-)

So I'd like if you can send me any suggestions/tips/warnings/links etc. 
before I start coding so I know what should I avoid/use.

Thanks,
Libor

Re: Adding snapshot capability to Linux

[Peter Wächtler]

Libor Vanìk wrote:
> Hi,
> I'm going to start my dissertation work which is "Adding snapshop 
> capability to Linux kernel with copy-on-write support". My idea is add 
> it as another VFS - I know that there is some snapshot support in LVM 
> but it's working on "device-level" and I'd like/have to do it on fs level.
> 

This functionality already exists in LVM (logical volume manager).

Re: Adding snapshot capability to Linux

[Libor Vanek]

Peter Wächtler wrote:

> Libor Vaněk wrote:
>
>> Hi,
>> I'm going to start my dissertation work which is "Adding snapshop 
>> capability to Linux kernel with copy-on-write support". My idea is 
>> add it as another VFS - I know that there is some snapshot support in 
>> LVM but it's working on "device-level" and I'd like/have to do it on 
>> fs level.
>>
> This functionality already exists in LVM (logical volume manager). 


But AFAIK it doesn't work on fs level but (i.e. you cannot make snapshot 
of some directory which contains NFS/Samba mapped dirs).

Libor

Re: Adding snapshot capability to Linux

[Jasper Spaans]

On Mon, Apr 22, 2002 at 02:19:20PM +0200, Libor Vanek wrote:

> > > I'm going to start my dissertation work which is "Adding snapshop 
> > > capability to Linux kernel with copy-on-write support". My idea is 
> > > add it as another VFS - I know that there is some snapshot support in 
> > > LVM but it's working on "device-level" and I'd like/have to do it on 
> > > fs level.
> 
> But AFAIK it doesn't work on fs level but (i.e. you cannot make snapshot 
> of some directory which contains NFS/Samba mapped dirs).

I'm sorry to have to say this, but making snapshots of remote filesystems is
kinda silly and impossible - if you implement it at the FS level.

To make a snapshot, you will have to be able to transfer the whole state of
the (remote) filesystem atomically (atomically on the remote server) which
will not happen when you don't have support for such functionality on that
server (think of other clients writing to that FS). Besides that, think of
the possible amounts of memory needed to store this snapshot locally, and
you should come to the conclusion that implementing snapshots at the FS
level is hairy, if not impossible.

You could however try to set up a snapshotting-system outside of LVM, though
that could become tricky too since LVM provides a nice way of reserving
space to store changes on the snapshotted fs. (To get back to your
original idea, you could implement a system like this and add some server
scripts to it, so a client can create and see a snapshotted version)

Regards,
-- 
Jasper Spaans
http://jsp.ds9a.nl/contact/
Tel/Fax: +31-84-8749842

Re: Adding snapshot capability to Linux

[Alvaro Figueroa]

> I'm sorry to have to say this, but making snapshots of remote filesystems is
> kinda silly and impossible - if you implement it at the FS level.

FWIW, this kinda sounds like mirroring an EVMS object that is on a
remote machine[1], or the cache in CODA.

[1] Plugins for working on objetcs in a SAN or network attatched disks.

-- 
Alvaro Figueroa

Re: Adding snapshot capability to Linux

[rpm]

On Monday 22 April 2002 05:36 pm, Peter Wächtler wrote:
> Libor Vanìk wrote:
> > Hi,
> > I'm going to start my dissertation work which is "Adding snapshop
> > capability to Linux kernel with copy-on-write support". My idea is add
> > it as another VFS - I know that there is some snapshot support in LVM
> > but it's working on "device-level" and I'd like/have to do it on fs
> > level.
> 


Instead of changing VFS you can probably make a generic stackable FS module 
.....that can stack on top of the physical filesystems  and happily take 
snapshots at "FS" level :) ! and you can use the FIST to create a basic 
stackable FS and then modify it to take care of snapshoting ! ( the good part 
is it can work on scores of other OSs like solaris , freeBSD etc ) 
i created one for ext2 but then it was not genric enough to work for other FS 
! 

cheers,
rpm

Re: Adding snapshot capability to Linux

[Alvaro Figueroa]

> Instead of changing VFS you can probably make a generic stackable FS module 
> .....that can stack on top of the physical filesystems  and happily take 
> snapshots at "FS" level :) ! and you can use the FIST to create a basic 
> stackable FS and then modify it to take care of snapshoting !

Since this solution doens't solve Lisbor's request of using it on smb
filesystems, well, you could as well save up all of the programmer
cycles and use EVMS.

It has a pluggin for treating normal partitions as EVMS objets, so you
don't need to translate them or so; and with EVMS you can even use RW
snapshots.

-- 
Alvaro Figueroa

Re: Adding snapshot capability to Linux

[Alexander Viro]

On 23 Apr 2002, Alvaro Figueroa wrote:

> > Instead of changing VFS you can probably make a generic stackable FS module 
> > .....that can stack on top of the physical filesystems  and happily take 
> > snapshots at "FS" level :) ! and you can use the FIST to create a basic 
> > stackable FS and then modify it to take care of snapshoting !
> 
> Since this solution doens't solve Lisbor's request of using it on smb
> filesystems, well, you could as well save up all of the programmer
> cycles and use EVMS.
> 
> It has a pluggin for treating normal partitions as EVMS objets, so you
> don't need to translate them or so; and with EVMS you can even use RW
> snapshots.

You _can't_ get consistent snapshots without cooperation from fs.  LVM,
EVMS, whatever.  Only filesystem knows what IO needs to be pushed to
make what we have on device consistent and what IO needs to be held
back.  Neither VFS nor device driver do not and can not have such
knowledge - it depends both on fs layout and on implementation details.

Re: Adding snapshot capability to Linux

[Xavier Bestel]

le mar 23-04-2002 à 16:45, Alexander Viro a écrit :
> You _can't_ get consistent snapshots without cooperation from fs.  LVM,
> EVMS, whatever.  Only filesystem knows what IO needs to be pushed to
> make what we have on device consistent and what IO needs to be held
> back.  Neither VFS nor device driver do not and can not have such
> knowledge - it depends both on fs layout and on implementation details.

I always thought that with a journalled fs, data was always consistent
on disk - i.e. always in a state where remounting the image (and
replaying the journal) makes it consistent.

Re: Adding snapshot capability to Linux

[Libor Vanek]

Alexander Viro wrote:

>You _can't_ get consistent snapshots without cooperation from fs.  LVM,
>EVMS, whatever.  Only filesystem knows what IO needs to be pushed to
>make what we have on device consistent and what IO needs to be held
>back.  Neither VFS nor device driver do not and can not have such
>knowledge - it depends both on fs layout and on implementation details.
>

My idea was to "catch" functions for writing to fs (file/metadata) and 
hold them till I copy file/metadata to snapshot. I thought that when 
done in the correct place this could work with virtualy any "normal" fs...

Libor

Re: Adding snapshot capability to Linux

[Jeremy Jackson]

This type of snapshot is very desirable.  It can be done by remounting
the fs ro, then taking EVMS or LVM snapshot, but you can't do that with open
files.

For file level COW (not block level),
 I wonder if DMAPI available with SGI's XFS (IRIX & Linux)
provides the VFS like hooks that could be used to build such a system.

Perhaps it could be implemented on other FS or generically in VFS.  The nice
thing is it's a standard and already has a working implementation (XFS)

from dmapi man page (oss.sgi.com XFS project):

DESCRIPTION
dmi is a system interface used to implement the interface defined in the
X/Open document: Systems Management: Data Storage Management (XDSM) API
dated February 1997. This interface is made available on Silicon Graphics
systems by means of the libdm library.

That spec is available from opengroup.org to view online for free.

For a journaling fs cooperation with snapshot while rw,
the fs must accept a snapshot request,
pause in flight IO, sync all pending buffers, flush it's log, mark fs clean
(almost like umount)
continue the block dev snapshot, mark fs in use, resume io.

Not so bad for journaling fs?  Non journaling would be easier, I think.

How about having all FS export methods for this, and VFS export to
userspace.
The holy grail is to generically support consistent online snapshot backup,
no?

Jeremy

----- Original Message -----
From: "Alexander Viro" <viro@math.psu.edu>
To: "Alvaro Figueroa" <fede2@fuerzag.ulatina.ac.cr>
Cc: "LKML" <linux-kernel@vger.kernel.org>
Sent: Tuesday, April 23, 2002 7:45 AM
Subject: Re: Adding snapshot capability to Linux


>
>
> On 23 Apr 2002, Alvaro Figueroa wrote:
>
> > > Instead of changing VFS you can probably make a generic stackable FS
module
> > > .....that can stack on top of the physical filesystems  and happily
take
> > > snapshots at "FS" level :) ! and you can use the FIST to create a
basic
> > > stackable FS and then modify it to take care of snapshoting !
> >
> > Since this solution doens't solve Lisbor's request of using it on smb
> > filesystems, well, you could as well save up all of the programmer
> > cycles and use EVMS.
> >
> > It has a pluggin for treating normal partitions as EVMS objets, so you
> > don't need to translate them or so; and with EVMS you can even use RW
> > snapshots.
>
> You _can't_ get consistent snapshots without cooperation from fs.  LVM,
> EVMS, whatever.  Only filesystem knows what IO needs to be pushed to
> make what we have on device consistent and what IO needs to be held
> back.  Neither VFS nor device driver do not and can not have such
> knowledge - it depends both on fs layout and on implementation details.
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: Adding snapshot capability to Linux

[Andreas Dilger]

On Apr 23, 2002  12:04 -0700, Jeremy Jackson wrote:
> This type of snapshot is very desirable.  It can be done by remounting
> the fs ro, then taking EVMS or LVM snapshot, but you can't do that with open
> files.
> 
> For a journaling fs cooperation with snapshot while rw, the fs must accept
> a snapshot request, pause in flight IO, sync all pending buffers, flush
> it's log, mark fs clean (almost like umount) continue the block dev
> snapshot, mark fs in use, resume io.
> 
> How about having all FS export methods for this, and VFS export to
> userspace.

Please do your homework.  What you describe already exists.  The VFS
methods (write_super_lockfs and unlockfs) are already there, and all
of the journaling filesystems support this.  The LVM and EVMS code has
patches to call these VFS methods.  For some reason the VFS-lock patch
has not yet been included in the kernel, but it should be.

As for filesystem-level snapshots (i.e. blocking write requests at the
VFS layer and doing COW) that is what snapfs (previously mentioned) does.

Cheers, Andreas
--
Andreas Dilger
http://www-mddsp.enel.ucalgary.ca/People/adilger/
http://sourceforge.net/projects/ext2resize/

Re: Adding snapshot capability to Linux

[Andreas Dilger]

On Apr 22, 2002  12:58 +0200, Libor Vanlk wrote:
> I'm going to start my dissertation work which is "Adding snapshop 
> capability to Linux kernel with copy-on-write support". My idea is add 
> it as another VFS - I know that there is some snapshot support in LVM 
> but it's working on "device-level" and I'd like/have to do it on fs level.
> 
> My idea is to use it this way:
> - I have running system with some "/foo" dir
> - I want to make snapshot of "/foo/bar" to "/foo/snap1"
> - I run "mount -t snapshot /foo/bar /foo/snap1"
> - This creates virtual image of "/foo/bar" to the "/foo/snap1" with 
> hidden file (something like journal) in "/foo/snap1" - all files are 
> linked to "/foo/bar"
> - Whenever is some file/dir changed in "/foo/bar" there is created 
> physical copy of it to the snapshot(s) before writing changes (for 
> making records about this will be used the hidden file)
> - Of course that one directory can be snapshoted more times
> - Probably the hidden file with records about all snapshots and details 
> should be stored in "/foo/bar"
> - Question is how to handle ACLs and EA for XFS/JFS/... and if it won't 
> collide with journal
> 
> I'd like to do it not only because I have to but I want people to use it 
> (I want to make it GPL) and maybe it will be one nice day part of Linux 
> kernel ;-)
> 
> So I'd like if you can send me any suggestions/tips/warnings/links etc. 
> before I start coding so I know what should I avoid/use.

Please see:
http://www-mddsp.enel.ucalgary.ca/People/adilger/snapfs/

What you describe is exactly what snapfs does.  The Sourceforge project
is currently inactive, but the code itself is GPL and only needs some
polishing up and maintenance to be useful (probably also some work to
get it all OK under 2.4 again).  There is already ext2 and ext3 support
for snapfs, and it would probably still be a worthwhile project to add
snapfs support for reiserfs.

Cheers, Andreas
--
Andreas Dilger
http://www-mddsp.enel.ucalgary.ca/People/adilger/
http://sourceforge.net/projects/ext2resize/

Re: Adding snapshot capability to Linux

[Kent Borg]

On Mon, Apr 22, 2002 at 11:07:46AM -0600, Andreas Dilger wrote:
> Please see:
> http://www-mddsp.enel.ucalgary.ca/People/adilger/snapfs/
> 
> What you describe is exactly what snapfs does.  The Sourceforge project
> is currently inactive, but the code itself is GPL and only needs some
> polishing up and maintenance to be useful (probably also some work to
> get it all OK under 2.4 again).

Very interesting.  Firing up google.com/linux I found a LWN story on
snapfs from a year ago March: <http://lwn.net/2001/0308/kernel.php3>.
Other than looking to me like it might not be bootable, this looks
very much like something someone named Kent Borg was asking about a
few days ago.


-kb

Re: Adding snapshot capability to Linux

[Andreas Dilger]

On Apr 22, 2002  13:41 -0400, Kent Borg wrote:
> On Mon, Apr 22, 2002 at 11:07:46AM -0600, Andreas Dilger wrote:
> > Please see:
> > http://www-mddsp.enel.ucalgary.ca/People/adilger/snapfs/
> > 
> > What you describe is exactly what snapfs does.  The Sourceforge project
> > is currently inactive, but the code itself is GPL and only needs some
> > polishing up and maintenance to be useful (probably also some work to
> > get it all OK under 2.4 again).
> 
> Very interesting.  Firing up google.com/linux I found a LWN story on
> snapfs from a year ago March: <http://lwn.net/2001/0308/kernel.php3>.
> Other than looking to me like it might not be bootable, this looks
> very much like something someone named Kent Borg was asking about a
> few days ago.

Actually, because ext2/ext3 is used as the underlying on-disk format,
you could probably boot from a snapfs filesystem if needed.  The on-disk
layout is designed such that the "current" version of the snapshot is
what you would always get (the snapshot data is hung off EAs on each
inode).  However, since you would then have the root filesystem mounted
as ext2/ext3 and not snapfs, you could not use the snapshot features.

Cheers, Andreas
--
Andreas Dilger
http://www-mddsp.enel.ucalgary.ca/People/adilger/
http://sourceforge.net/projects/ext2resize/

Re: Adding snapshot capability to Linux

[Sandy Harris]

Andreas Dilger wrote:
> 
> On Apr 22, 2002  12:58 +0200, Libor Vanlk wrote:
> > I'm going to start my dissertation work which is "Adding snapshop
> > capability to Linux kernel with copy-on-write support". ...

> > So I'd like if you can send me any suggestions/tips/warnings/links etc.
> > before I start coding so I know what should I avoid/use.
> 
> Please see:
> http://www-mddsp.enel.ucalgary.ca/People/adilger/snapfs/
> 
> What you describe is exactly what snapfs does. ...

A related thing to look at would be the Plan 9 file system.
http://www.cs.bell-labs.com/plan9dist/

It is an entirely different design using copy-on-write and a
form of snapshotting such that something like:

cd //2001/12/25/foo

puts the user in directory /foo as it was at daily backup time
on Christmas day last year.

The original Plan 9 system used WORM drives with hard drives
acting as cache. I suspect the design could work without WORM.