Garbage Collection Schemes

Garbage Collection Schemes

[John Andersen]

I am new to NILFS, so bear with me.

I would like to ask, (without reading the entire archive of messages) if
any thought to other garbage collection schemes has been contemplated.

For instance: This file system looks like it might be very useful for 
certain types of environments where N sequential revisions of documents
must be maintained, regardless of the  age or how any given document relates
to other documents.

Checkpoints do not seem to do this.

I would like to see garbage collection options that allow (for example)
10 copies retained (of EACH file), each bearing different dates.  

Garbage collection would erase the 11th (oldest copy) if, (and only if) the 
other 10 all had different dates.  This prevents the situation where one user 
(or rogue process) re-saving a document 15 times a day wipes out all prior 
copies.

An option might be garbage collection not based on a NUMBER of iterations
but rather a period of time (3 years, 7 years, etc, while still purging same 
date (or same hour, minute, etc) copies.

This type of garbage collection is found almost nowhere, and this
file system seems to be the closest possible candidate to do this.

There are applications, where a consistent representation of all 
files at a specific point in time (snapshot or checkpoint) is LESS
important than the ability to roll back individual documents to 
a number of prior iterations or a number of years.

Legal documents, financial records, code version archiving, etc, all
have these requirements.


-- 
__________________________
John Andersen	
Screenio.com

Re: Garbage Collection Schemes

[Gergely Gábor]

[-- Attachment #1.1: Type: text/plain, Size: 2289 bytes --]

On Mon, 26 Nov 2007 15:12:10 -0800
John Andersen <jsa-MJZuZBXGww0Jmq/UkTHWNaxOck334EZe@public.gmane.org> wrote:

> I am new to NILFS, so bear with me.
> 
> I would like to ask, (without reading the entire archive of messages) if
> any thought to other garbage collection schemes has been contemplated.
> 
> For instance: This file system looks like it might be very useful for 
> certain types of environments where N sequential revisions of documents
> must be maintained, regardless of the  age or how any given document relates
> to other documents.
> 
> Checkpoints do not seem to do this.
> 
> I would like to see garbage collection options that allow (for example)
> 10 copies retained (of EACH file), each bearing different dates.  
> 
> Garbage collection would erase the 11th (oldest copy) if, (and only if) the 
> other 10 all had different dates.  This prevents the situation where one user 
> (or rogue process) re-saving a document 15 times a day wipes out all prior 
> copies.
> 
> An option might be garbage collection not based on a NUMBER of iterations
> but rather a period of time (3 years, 7 years, etc, while still purging same 
> date (or same hour, minute, etc) copies.
> 

AFAIK there is a protection interval in which GC wont clean up a checkpoint. (maybe i remember badly... and some other LFS fs was that...)

> This type of garbage collection is found almost nowhere, and this
> file system seems to be the closest possible candidate to do this.
> 
> There are applications, where a consistent representation of all 
> files at a specific point in time (snapshot or checkpoint) is LESS
> important than the ability to roll back individual documents to 
> a number of prior iterations or a number of years.
> 
> Legal documents, financial records, code version archiving, etc, all
> have these requirements.
> 
> 
> -- 
> __________________________
> John Andersen	
> Screenio.com
> _______________________________________________
> users mailing list
> users-JrjvKiOkagjYtjvyW6yDsg@public.gmane.org
> https://www.nilfs.org/mailman/listinfo/users

BTW: does/will nilfs support named snapshots?

Best Regards: Gergely Gábor

-- 
Gergely Gábor <elentirmo.gilgalad-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 158 bytes --]

_______________________________________________
users mailing list
users-JrjvKiOkagjYtjvyW6yDsg@public.gmane.org
https://www.nilfs.org/mailman/listinfo/users

Re: Garbage Collection Schemes

[Koji Sato]

Hello John.
Sorry for my late reply.

> I am new to NILFS, so bear with me.
> 
> I would like to ask, (without reading the entire archive of messages) if
> any thought to other garbage collection schemes has been contemplated.
> 
> For instance: This file system looks like it might be very useful for 
> certain types of environments where N sequential revisions of documents
> must be maintained, regardless of the  age or how any given document relates
> to other documents.
> 
> Checkpoints do not seem to do this.
> 
> I would like to see garbage collection options that allow (for example)
> 10 copies retained (of EACH file), each bearing different dates.  
> 
> Garbage collection would erase the 11th (oldest copy) if, (and only if) the 
> other 10 all had different dates.  This prevents the situation where one user 
> (or rogue process) re-saving a document 15 times a day wipes out all prior 
> copies.
> 
> An option might be garbage collection not based on a NUMBER of iterations
> but rather a period of time (3 years, 7 years, etc, while still purging same 
> date (or same hour, minute, etc) copies.
> 
> This type of garbage collection is found almost nowhere, and this
> file system seems to be the closest possible candidate to do this.
> 
> There are applications, where a consistent representation of all 
> files at a specific point in time (snapshot or checkpoint) is LESS
> important than the ability to roll back individual documents to 
> a number of prior iterations or a number of years.
> 
> Legal documents, financial records, code version archiving, etc, all
> have these requirements.

In the current implementation, NILFS does not automatically create
snapshots. If daily or weekly snapshots are required to be made, the
chcp command should be periodically invoked (using cron or other
mechanisms). The implementation of the automatic snapshot creation based 
on user-defined retention policy is one of our future works. Please let 
us know if you have any suggestion or comment.

Thanks you.

-- 
Koji Sato
Open Source Software Computing Project
NTT Cyber Space Laboratories
Nippon Telegraph and Telephone Corporation