CheckFS: Checkpoints and Block Level Incremental Backup (BLIB)

CheckFS: Checkpoints and Block Level Incremental Backup (BLIB)

[Milind Dumbare]

Hi,

	LinSysSoft Technologies  has taken up the challenge to incorporate
Checkpoint and Block Level Incremental Backup (BLIB) support in the open
source's Ext3 File System, which is very well known for its stability,
to create a new file system called CheckFS. Block Level Incremental
Backup enables truly efficient backup and restore mechanisms.
Checkpoints provide administrators to create point-in-time copies of a
live file system by keeping track of the data blocks modified in real
time. 

For further information and a downloadable working prototype of this
technology go to : http://checkfs.linsyssoft.com

....Milind Dumbare
(www.linsyssoft.com)

Re: CheckFS: Checkpoints and Block Level Incremental Backup (BLIB)

[Theodore Ts'o]

On Fri, Jul 22, 2005 at 07:34:38PM +0530, Milind Dumbare wrote:
> Hi,
> 
> 	LinSysSoft Technologies  has taken up the challenge to incorporate
> Checkpoint and Block Level Incremental Backup (BLIB) support in the open
> source's Ext3 File System, which is very well known for its stability,
> to create a new file system called CheckFS. Block Level Incremental
> Backup enables truly efficient backup and restore mechanisms.
> Checkpoints provide administrators to create point-in-time copies of a
> live file system by keeping track of the data blocks modified in real
> time. 
> 
> For further information and a downloadable working prototype of this
> technology go to : http://checkfs.linsyssoft.com

This looks like very interesting technology, but out of curiosity, why
did you develop this as separate filesystem with a new filesystem
name, and doing a global search-and-replace of "ext3" with "checkfs"
in the source files, instead of simply just modifying ext3 and posting
diffs?  Especially since that the apparent intention is to keep ext3
compatibility using the same ext3 magic numbers, data formats, and
user-mode utilities.

I'll reserve the superblock fields and compatibility bitmap fields
used by your code in e2fsprogs to help avoid the possibility of other
folks working on ext3 extensions from colliding with the codepoints
which you "borrowed", but in the future, it would be good if people
contact me in advance so I ensure that there are no collisions with
other development groups.

What version of the source base did you originally fork checkfs from?
That way we can do a "s/checkfs/ext3/g" search and replace and then
generate some diffs to see what you changed, or alternatively, it
would be even better if you minimized differences between your version
and mainline and generated the diffs yourself.

Thanks, regards, 

						- Ted

Re: CheckFS: Checkpoints and Block Level Incremental Backup (BLIB)

[Amit S. Kale]

Ted,

Thanks for your suggestions and help.

We started it from 2.6.7 last year and then it was sitting idle for several 
months for lack of resources. We'll go back to that version and generate a 
diff that's easier to read.

Yes, changing the name has made the task of rebasing wrt. changing kernels lot 
difficult. Our original intention was to make testing easier by keeping ext3 
and checkfs filesystems in the same kernel. Had we continued it at that 
point, we would have posted differences wrt. ext3 sources themselves. There 
was compelling reason to change the name.

Regards.
-Amit


> This looks like very interesting technology, but out of curiosity, why
> did you develop this as separate filesystem with a new filesystem
> name, and doing a global search-and-replace of "ext3" with "checkfs"
> in the source files, instead of simply just modifying ext3 and posting
> diffs? Especially since that the apparent intention is to keep ext3
> compatibility using the same ext3 magic numbers, data formats, and
> user-mode utilities.
>
> I'll reserve the superblock fields and compatibility bitmap fields
> used by your code in e2fsprogs to help avoid the possibility of other
> folks working on ext3 extensions from colliding with the codepoints
> which you "borrowed", but in the future, it would be good if people
> contact me in advance so I ensure that there are no collisions with
> other development groups.
>
> What version of the source base did you originally fork checkfs from?
> That way we can do a "s/checkfs/ext3/g" search and replace and then
> generate some diffs to see what you changed, or alternatively, it
> would be even better if you minimized differences between your version
> and mainline and generated the diffs yourself.
> 
> Thanks, regards,
> 
> - Ted

Re: CheckFS: Checkpoints and Block Level Incremental Backup (BLIB)

[Theodore Ts'o]

On Sat, Jul 23, 2005 at 11:30:07AM +0530, Amit S. Kale wrote:
> 
> We started it from 2.6.7 last year and then it was sitting idle for several 
> months for lack of resources. We'll go back to that version and generate a 
> diff that's easier to read.
> 
> Yes, changing the name has made the task of rebasing wrt. changing kernels lot 
> difficult. Our original intention was to make testing easier by keeping ext3 
> and checkfs filesystems in the same kernel. Had we continued it at that 
> point, we would have posted differences wrt. ext3 sources themselves. There 
> was compelling reason to change the name.

One easier way of doing development, particularly for people doing
filesystem work, is to compile the kernel with the test filesystem
code using user-mode linux (UML) architecture.  This significantly
shortens your edit-compile-debug cycle time, and it makes it easier to
run your filesystem code under a debugger.  That way you also don't
have to worry about your filesystem changes toasting your system,
either.  This technique doesn't work all that well for people doing
architecture-specific code or for device drivers, but for filesystems,
it's ideal.

Regards,

						- Ted

Re: CheckFS: Checkpoints and Block Level Incremental Backup (BLIB)

[Pavel Machek]

Hi!

> Thanks for your suggestions and help.
> 
> We started it from 2.6.7 last year and then it was sitting idle for several 
> months for lack of resources. We'll go back to that version and generate a 
> diff that's easier to read.
> 
> Yes, changing the name has made the task of rebasing wrt. changing kernels lot 
> difficult. Our original intention was to make testing easier by keeping ext3 
> and checkfs filesystems in the same kernel. Had we continued it at that 
> point, we would have posted differences wrt. ext3 sources themselves. There 
> was compelling reason to change the name.

Maybe you want to put your development machines on ext*2* while doing
this ;-). Or perhaps reiserfs/xfs/something.
								Pavel

-- 
teflon -- maybe it is a trademark, but it should not be.

Re: CheckFS: Checkpoints and Block Level Incremental Backup (BLIB)

[Jan Engelhardt]

>Maybe you want to put your development machines on ext*2* while doing
>this ;-). Or perhaps reiserfs/xfs/something.

Or perhaps into at the VFS level, so any fs can benefit from it.



Jan Engelhardt
-- 

Re: CheckFS: Checkpoints and Block Level Incremental Backup (BLIB)

[Amit S. Kale]

On Sunday 24 Jul 2005 8:44 pm, Jan Engelhardt wrote:
> >Maybe you want to put your development machines on ext*2* while doing
> >this ;-). Or perhaps reiserfs/xfs/something.
>
> Or perhaps into at the VFS level, so any fs can benefit from it.

We thought about that. While it's possible to do that, it would need hooks 
into all filesystems etc. Definitely worth trying once we get some more basic 
stuff working for ext3

After all the things that need to be saved at the time of taking a checkpoint 
for any filesystem would be the superblock and inode table (or their 
equivalents). Everything else is automatically taken care of.

-Amit

Re: CheckFS: Checkpoints and Block Level Incremental Backup (BLIB)

[Theodore Ts'o]

On Mon, Jul 25, 2005 at 11:24:43AM +0530, Amit S. Kale wrote:
> On Sunday 24 Jul 2005 8:44 pm, Jan Engelhardt wrote:
> > >Maybe you want to put your development machines on ext*2* while doing
> > >this ;-). Or perhaps reiserfs/xfs/something.
> >
> > Or perhaps into at the VFS level, so any fs can benefit from it.
> 
> We thought about that. While it's possible to do that, it would need
> hooks into all filesystems etc. Definitely worth trying once we get
> some more basic stuff working for ext3
> 
> After all the things that need to be saved at the time of taking a
> checkpoint for any filesystem would be the superblock and inode
> table (or their equivalents). Everything else is automatically taken
> care of.

You are aware of the block-device-level snapshot solutions, right?
They can be more painful to use, although that's mostly a UI issue,
and they have the added advantage that you can safely run e2fsck on
the snapshot image if you want to check the consistency of the
filesystem while it is mounted.  (If it is clean, you can then use
tune2fs to reset the max-mount-count and last-checked-time on the
original filesystem image; if it is not clean, you can send e-mail to
the system administrator suggesting that she schedule downtime ASAP
and do a e2fsck on the filesystem image.  This is a good thing that a
paranoid sysadmin might schedule via cron every Saturday morning at
3am, for example.)

							- Ted

Re: CheckFS: Checkpoints and Block Level Incremental Backup (BLIB)

[Amit S. Kale]

On Monday 25 Jul 2005 6:02 pm, Theodore Ts'o wrote:
> On Mon, Jul 25, 2005 at 11:24:43AM +0530, Amit S. Kale wrote:
> > On Sunday 24 Jul 2005 8:44 pm, Jan Engelhardt wrote:
> > > >Maybe you want to put your development machines on ext*2* while doing
> > > >this ;-). Or perhaps reiserfs/xfs/something.
> > >
> > > Or perhaps into at the VFS level, so any fs can benefit from it.
> >
> > We thought about that. While it's possible to do that, it would need
> > hooks into all filesystems etc. Definitely worth trying once we get
> > some more basic stuff working for ext3
> >
> > After all the things that need to be saved at the time of taking a
> > checkpoint for any filesystem would be the superblock and inode
> > table (or their equivalents). Everything else is automatically taken
> > care of.
>
> You are aware of the block-device-level snapshot solutions, right?
> They can be more painful to use, although that's mostly a UI issue,
> and they have the added advantage that you can safely run e2fsck on
> the snapshot image if you want to check the consistency of the
> filesystem while it is mounted.

That would be a lesser advantage IMHO compared to the advantage of being able 
to manage disks in a better way.

Block-device level snapshots are simple to implement, though making them 
intelligent is quite difficult. Block device level snapshots can't detect 
which block are allocated and which are not (unless they look into a 
filesystem's block allocation map, which requires a device->fs interface). A 
snapshot may hence require an exhorbitant amount of space when it's not 
really needed.

The BLIB procedure to be used with checkfs is to create a checkpoint. Transfer 
it to disk as the first full backup. This operation uses the free space 
available within the filesystem till the time the data is transfered to a 
tape. A similar procedure used with block-level backup will require an 
auxilliary device.

Generally filesystem level snapshots/checkpoints can be more intelligent hence 
easier to use and requires fewer resources.

-Amit

> (If it is clean, you can then use 
> tune2fs to reset the max-mount-count and last-checked-time on the
> original filesystem image; if it is not clean, you can send e-mail to
> the system administrator suggesting that she schedule downtime ASAP
> and do a e2fsck on the filesystem image.  This is a good thing that a
> paranoid sysadmin might schedule via cron every Saturday morning at
> 3am, for example.)

Re: CheckFS: Checkpoints and Block Level Incremental Backup (BLIB)

[Milind Dumbare]

Hi,

        The diff file generated by comparing ext3/ and
kernel/fs/checkfs/ can also be accessed from the link
http://checkfs.linsyssoft.com/temp/
        I cleaned it, but still it has some unnecessary differences. I
am working on that to make it more clean and will send it to u as soon
as I finish with that.

Milind Dumbare
(www.linsyssoft.com)

Re: CheckFS: Checkpoints and Block Level Incremental Backup (BLIB)

[Theodore Ts'o]

On Mon, Aug 01, 2005 at 01:20:39PM +0530, Milind Dumbare wrote:
> Hi,
> 
>         The diff file generated by comparing ext3/ and
> kernel/fs/checkfs/ can also be accessed from the link
> http://checkfs.linsyssoft.com/temp/
>         I cleaned it, but still it has some unnecessary differences. I
> am working on that to make it more clean and will send it to u as soon
> as I finish with that.

Thanks for working on it; it's much appreciated.  One very quick
comment; it's generally considered poor form to remove other people's
copyright notices; it's a do unto others as you would do unto them
(lest their lawyers do unto you what your lawyers might do unto them
if the positions were reversed :-) sort of thing.  

						- Ted

Re: CheckFS: Checkpoints and Block Level Incremental Backup (BLIB)

[Milind Dumbare]

Hi,

This is just an intermediate diff between the original source and our
source with checkfs replaced by ext3. We will be careful about
maintaining the copyright notices when finishing this merge.

Thank you,
Milind Dumbare
(www.linsyssoft.com)

On Mon, 2005-08-01 at 06:08 -0400, Theodore Ts'o wrote:
> On Mon, Aug 01, 2005 at 01:20:39PM +0530, Milind Dumbare wrote:
> > Hi,
> > 
> >         The diff file generated by comparing ext3/ and
> > kernel/fs/checkfs/ can also be accessed from the link
> > http://checkfs.linsyssoft.com/temp/
> >         I cleaned it, but still it has some unnecessary differences. I
> > am working on that to make it more clean and will send it to u as soon
> > as I finish with that.
> 
> Thanks for working on it; it's much appreciated.  One very quick
> comment; it's generally considered poor form to remove other people's
> copyright notices; it's a do unto others as you would do unto them
> (lest their lawyers do unto you what your lawyers might do unto them
> if the positions were reversed :-) sort of thing.  
> 
> 						- Ted
> 
>