srfs - a new file system.

srfs - a new file system.

[Nir Tzachar]

hello all.

We're proud to announce the availability of a _proof of concept_ file
system, called srfs. ( http://www.cs.bgu.ac.il/~srfs/ ).
a quick overview: [from the home page]
srfs is a global file system designed to be distributed geographicly over
multiple locations and provide a consistent, high available and durable
infrastructure for information.

Started as a research project into file systems and self-stabilization in
Ben Gurion University of the Negev Department of Computer Science, the
project aims to integrate self-stabilization methods and algorithms into
the file (and operation) systems to provide a system with a desired
behavior in the presence of transient faults.

Based on layered self-stabilizing algorithms, provide a tree replication
structure based on auto-discovery of servers using local and global IP
multicasting. The tree structure is providing the command and timing
infrastructure required for a distributed file system.

The project is basically divided into two components:
1) a kernel module, which provides the low level functionality, and
   disk management.
2) a user space caching daemon, which provide the stabilization and
   replication properties of the file system.
these two components communicate via a character device.

more info on the system architecture can be find on the web page, and
here: http://www.cs.bgu.ac.il/~tzachar/srfs.pdf

We hope some will find this interesting enough to take for a test drive,
and wont mind the latencies ( currently, the caching daemon is a bit slow.
hopefully, we will improve it in the future. )
anyway, please keep in mind this is a very early version that only works,
and keeps the stabilization properties. no posix compliance whatsoever...

the code contains several hacks and design flaws that we're aware of,
and probably many that we're not... so please be gentle ;)

if someone found this interesting, please contact us with ur insights.
cheers,
the srfs team.

p.s I would like to thank all members of this mailing list (fsdevel), for
ur continual help with problems we encountered during the development.
thanks guys (and girls???).

========================================================================
nir.

Re: srfs - a new file system.

[Eric Sandall]

Quoting Nir Tzachar <tzachar@cs.bgu.ac.il>:
> hello all.
> 
> We're proud to announce the availability of a _proof of concept_ file
> system, called srfs. ( http://www.cs.bgu.ac.il/~srfs/ ).
> a quick overview: [from the home page]
> srfs is a global file system designed to be distributed geographicly over
> multiple locations and provide a consistent, high available and durable
> infrastructure for information.
> 
> Started as a research project into file systems and self-stabilization in
> Ben Gurion University of the Negev Department of Computer Science, the
> project aims to integrate self-stabilization methods and algorithms into
> the file (and operation) systems to provide a system with a desired
> behavior in the presence of transient faults.
> 
> Based on layered self-stabilizing algorithms, provide a tree replication
> structure based on auto-discovery of servers using local and global IP
> multicasting. The tree structure is providing the command and timing
> infrastructure required for a distributed file system.
> 
> The project is basically divided into two components:
> 1) a kernel module, which provides the low level functionality, and
>    disk management.
> 2) a user space caching daemon, which provide the stabilization and
>    replication properties of the file system.
> these two components communicate via a character device.
> 
> more info on the system architecture can be find on the web page, and
> here: http://www.cs.bgu.ac.il/~tzachar/srfs.pdf
> 
> We hope some will find this interesting enough to take for a test drive,
> and wont mind the latencies ( currently, the caching daemon is a bit slow.
> hopefully, we will improve it in the future. )
> anyway, please keep in mind this is a very early version that only works,
> and keeps the stabilization properties. no posix compliance whatsoever...
> 
> the code contains several hacks and design flaws that we're aware of,
> and probably many that we're not... so please be gentle ;)
> 
> if someone found this interesting, please contact us with ur insights.
> cheers,
> the srfs team.
> 
> p.s I would like to thank all members of this mailing list (fsdevel), for
> ur continual help with problems we encountered during the development.
> thanks guys (and girls???).
> 
> ========================================================================
> nir.

This sounds fairly similar to Coda[0], which is already in development and use.

-sandalle

[0] http://www.coda.cs.cmu.edu/

-- 
PGP Key Fingerprint:  FCFF 26A1 BE21 08F4 BB91  FAED 1D7B 7D74 A8EF DD61
http://search.keyserver.net:11371/pks/lookup?op=get&search=0xA8EFDD61

-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCS/E/IT$ d-- s++:+>: a-- C++(+++) BL++++VIS>$ P+(++) L+++ E-(---) W++ N+@ o?
K? w++++>-- O M-@ V-- PS+(+++) PE(-) Y++(+) PGP++(+) t+() 5++ X(+) R+(++)
tv(--)b++(+++) DI+@ D++(+++) G>+++ e>+++ h---(++) r++ y+
------END GEEK CODE BLOCK------

Eric Sandall                     |  Source Mage GNU/Linux Developer
eric@sandall.us                  |  http://www.sourcemage.org/
http://eric.sandall.us/          |  SysAdmin @ Inst. Shock Physics @ WSU
http://counter.li.org/  #196285  |  http://www.shock.wsu.edu/

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

Re: srfs - a new file system.

[Nir Tzachar]

>
> This sounds fairly similar to Coda[0], which is already in development and use.
>

not at all.

coda is not self stabilizing at all.
srfs is also a totally distributed file system -> see the doc.
bye
========================================================================
nir.

Re: srfs - a new file system.

[Brian Beattie]

On Tue, 2003-10-21 at 08:07, Nir Tzachar wrote:
> >
> > This sounds fairly similar to Coda[0], which is already in development and use.
> >
> 
> not at all.
> 
> coda is not self stabilizing at all.
> srfs is also a totally distributed file system -> see the doc.

what does "self stabilizing" mean in this context?

> bye
bye bye
-- 
Brian Beattie            | Experienced kernel hacker/embedded systems
beattie@beattie-home.net | programmer, direct or contract, short or
www.beattie-home.net     | long term, available immediately.

"Honor isn't about making the right choices.
It's about dealing with the consequences." -- Midori Koto

Re: srfs - a new file system.

[Erik Andersen]

On Mon Oct 20, 2003 at 11:12:07AM +0200, Nir Tzachar wrote:
> more info on the system architecture can be find on the web page, and
> here: http://www.cs.bgu.ac.il/~tzachar/srfs.pdf

Suppose I install srfs on both my laptop and my server.  I then
move the CVS repository for my pet project onto the new srfs
filesystem and I take off for the weekend with my laptop.   Over
the weekend I commit several changes to file X.  Over the weekend
my friend also commits several changes to file X.

When I get home and plug in my laptop, presumably the caching
daemon will try to stabalize the system by deciding which version
of file X was changed last and replicating that latest version.  

Who's work will the caching daemon overwrite?  My work, or my
friends work?

Of course, this need not involve anything so extreme as days of
disconnected independent operation.  A rebooting router between
two previously syncd srfs peers seems sufficient to trigger this
kind of data loss, unless you make the logging daemon fail all
writes when disconnected.

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: srfs - a new file system.

[Nir Tzachar]

first, id like to thank u guys for playing around with the idea.

now, i want to apologize if my explanation was not clear enough:
self stabilization (original idea by Dijkstra) - A self stabilizing system
is a system that can automatically recover following the occurrence of
( transient ) faults. The idea is to design a system which can be started
in an arbitrary state and still converge to a desired behavior.

Our file system behaves like this:
lets say you have several servers, with different file system trees on
them. If (and when ...) you connect these file systems with an srfs
framework, all servers will display the same file system tree, which is
somewhat of a union between them all.
if you wish to talk in coda terms, you can say all servers operated
disconnectedly, and then were connected at the same time. the conflict
resolving mechanism we use, is by majority.

We differ from coda in the sense we don't have a main server, which pushes
Volumes to sub-servers (im not sure what the coda terminology is... ), and
data is served in a load-balanced way. In Srfs, all the data resides on
all servers (hosts) and is replicated between them.
replication takes place at two levels: tree view (plus meta data) and the
actual data.
tree view - the tree view on all hosts is the same. an `ls` on a dir
            on any host will produce the same output.
data - data will be replicated to all hosts upon a successful write,
       and upon each access to a dirty file on each host.

all replication is lazy, and happens only on access to dirs / files
(and on successful writes - when the file is being closed.)

Thus, the following behavior can be achieved:
lets say you have 2N+1 hosts, all with coherent file system trees.
now, take N of them offline, change the tree, put those N back online,
and their tree will be the same as the other N+1 other hosts.

The main goal of the file system is self stabilization, over long periods
of time and long distances. you can use it as a SAN, or as a data farm,
using system like LinuxVirtualServer to balance the load between nodes.

cheers.

========================================================================
nir.

Re: srfs - a new file system.

[Nir Tzachar]

> Who's work will the caching daemon overwrite?My work, or my
> friends work?

well, in our system, unless u break the symmetry, the daemon will
pick a random file. Since no majority can be found, this is the default.
but, lets say your friend was connected to a third server, and his work
was saved there also. when you'll connect ur laptop, all of ur work will
be lost, and what ull see in only his work ;)


========================================================================
nir.

Re: srfs - a new file system.

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 625 bytes --]

On Tue, 21 Oct 2003 22:57:09 MDT, Erik Andersen said:

> Suppose I install srfs on both my laptop and my server.  I then
> move the CVS repository for my pet project onto the new srfs
> filesystem and I take off for the weekend with my laptop.   Over
> the weekend I commit several changes to file X.  Over the weekend
> my friend also commits several changes to file X.
> 
> When I get home and plug in my laptop, presumably the caching
> daemon will try to stabalize the system by deciding which version
> of file X was changed last and replicating that latest version.  

Hey Larry - potential BitKeeper customer here. :)

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: srfs - a new file system.

[Erik Andersen]

[-- Attachment #1: Type: text/plain, Size: 1069 bytes --]

On Wed Oct 22, 2003 at 12:05:46PM -0400, Valdis.Kletnieks@vt.edu wrote:
> On Tue, 21 Oct 2003 22:57:09 MDT, Erik Andersen said:
> 
> > Suppose I install srfs on both my laptop and my server.  I then
> > move the CVS repository for my pet project onto the new srfs
> > filesystem and I take off for the weekend with my laptop.   Over
> > the weekend I commit several changes to file X.  Over the weekend
> > my friend also commits several changes to file X.
> > 
> > When I get home and plug in my laptop, presumably the caching
> > daemon will try to stabalize the system by deciding which version
> > of file X was changed last and replicating that latest version.  
> 
> Hey Larry - potential BitKeeper customer here. :)

Not so much a potential BitKeeper customer, as pointing out that
the distributed filesystems prople are attacking the same
fundamental problem as the distributed version control folks.

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: srfs - a new file system.

[Miles Bader]

Erik Andersen <andersen@codepoet.org> writes:
> Not so much a potential BitKeeper customer, as pointing out that
> the distributed filesystems prople are attacking the same
> fundamental problem as the distributed version control folks.

It may be the same at some level, but there's an important difference:
distributed filesystems are usually (AFAIK) attempting to maintain the
illusion of a single global filesystem that looks more or less to the
users like a local filesystem, and usually just an average unixy
filesystem.  This is very, very, hard...

Distributed version control systems, OTOH, because they're at a somewhat
higher level, have the huge advantage of distinct operational boundaries
which are exposed the user and can be used to manage the distribution.
Since users are used to these boundaries, and they usually occur at
fairly obvious and reasonable places, this isn't such a burden on the
users.

-Miles
-- 
97% of everything is grunge

Re: srfs - a new file system.

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 752 bytes --]

On Thu, 23 Oct 2003 14:20:21 +0900, Miles Bader said:

> Distributed version control systems, OTOH, because they're at a somewhat
> higher level, have the huge advantage of distinct operational boundaries
> which are exposed the user and can be used to manage the distribution.
> Since users are used to these boundaries, and they usually occur at
> fairly obvious and reasonable places, this isn't such a burden on the
> users.

On the flip side, a filesystem only has to worry about who wrote which blocks
in what order.  I suspect if you tried to push the idea of a filesystem that did
the sort of intuiting of intent that BitKeeper has to do on a merge, it would
quickly get shouted down.

Unless of course somebody does BK as a Reiser4 module. :)

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: srfs - a new file system.

[Pavel Machek]

Hi!

> > This sounds fairly similar to Coda[0], which is already in development and use.
> >
> 
> not at all.
> 
> coda is not self stabilizing at all.
> srfs is also a totally distributed file system -> see the doc.
> bye

Yes, but perhaps differences can be localized to userspace daemon,
having same kernel part for coda and srfs?
That would be *good*.

				Pavel
-- 
				Pavel
Written on sharp zaurus, because my Velo1 broke. If you have Velo you don't need...

Re: srfs - a new file system.

[Daniel Egger]

[-- Attachment #1: Type: text/plain, Size: 554 bytes --]

Am Mon, den 20.10.2003 schrieb Eric Sandall um 23:00:

> This sounds fairly similar to Coda[0], which is already in development and use.

The last time I looked Coda was a horrible mess of a code, closely
impossible to get it compile let alone configure and it seems to have
the same interoperability problems like intermezzo i.e. it didn't work
between i386<->powerpc. I haven't looked at Lustre light or srfs yet but
I certainly welcome any fresh projects in the area of distributed or
replicating filesystems.

-- 
Servus,
       Daniel

[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: srfs - a new file system.

[Eric Sandall]

Quoting Daniel Egger <degger@fhm.edu>:
> The last time I looked Coda was a horrible mess of a code, closely
> impossible to get it compile let alone configure and it seems to have
> the same interoperability problems like intermezzo i.e. it didn't work
> between i386<->powerpc. I haven't looked at Lustre light or srfs yet but
> I certainly welcome any fresh projects in the area of distributed or
> replicating filesystems.
> 
> -- 
> Servus,
>        Daniel

Agreed, more DFS' are always good.  As for Coda, it has compiled fine for me for
the last year (with some bison patches), but I have not actually tried it yet. 
NFS may be slow, but at least it works and I haven't lost any files due to
using it.

-sandalle

-- 
PGP Key Fingerprint:  FCFF 26A1 BE21 08F4 BB91  FAED 1D7B 7D74 A8EF DD61
http://search.keyserver.net:11371/pks/lookup?op=get&search=0xA8EFDD61

-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCS/E/IT$ d-- s++:+>: a-- C++(+++) BL++++VIS>$ P+(++) L+++ E-(---) W++ N+@ o?
K? w++++>-- O M-@ V-- PS+(+++) PE(-) Y++(+) PGP++(+) t+() 5++ X(+) R+(++)
tv(--)b++(+++) DI+@ D++(+++) G>+++ e>+++ h---(++) r++ y+
------END GEEK CODE BLOCK------

Eric Sandall                     |  Source Mage GNU/Linux Developer
eric@sandall.us                  |  http://www.sourcemage.org/
http://eric.sandall.us/          |  SysAdmin @ Inst. Shock Physics @ WSU
http://counter.li.org/  #196285  |  http://www.shock.wsu.edu/

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

Re: srfs - a new file system.

[Daniel Egger]

[-- Attachment #1: Type: text/plain, Size: 603 bytes --]

Am Don, den 23.10.2003 schrieb Eric Sandall um 20:47:

> Agreed, more DFS' are always good.  As for Coda, it has compiled fine for me for
> the last year (with some bison patches), but I have not actually tried it yet. 
> NFS may be slow, but at least it works and I haven't lost any files due to
> using it.

The slowless of NFS is not an issue for me (actually I'm getting quite
good performance over a switched 100Mbit network). However it doesn't
replicate which is annoying when being much on the road but normally
also using lots of computers in the lab.

-- 
Servus,
       Daniel

[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: srfs - a new file system.

[Nir Tzachar]

> The last time I looked Coda was a horrible mess of a code, closely
> impossible to get it compile let alone configure and it seems to have
> the same interoperability problems like intermezzo i.e. it didn't work
> between i386<->powerpc. I haven't looked at Lustre light or srfs yet but
> I certainly welcome any fresh projects in the area of distributed or
> replicating filesystems.

well, i hope u'll take a look at our file system, but keep in mind:
a) the code is in an early beta state.
b) srfs _cheats_ a bit, by enslaving a local file system.
c) the code is not as neat as i'd like it to be.
d) we havnt checked interoperability, since we dont have other platform.
   although, srfs should work on any platform running java.

cheers.
-- 
========================================================================
nir.

Re: srfs - a new file system.

[Nir Tzachar]

hi

> Yes, but perhaps differences can be localized to userspace daemon,
> having same kernel part for coda and srfs?
> That would be *good*.
> 

in essence, ur correct. we would have taken that approach, if we were not 
aiming at building a file system on top of an object storage. this 
approach simplifies things a bit, and the kernel part is reduced.

-- 
========================================================================
nir.

Re: srfs - a new file system.--OT

[Gadgeteer]

On Thursday 23 October 2003 12:47, Eric Sandall wrote:
> > I certainly welcome any fresh projects in the area of distributed or
> > replicating filesystems.

Just to interject here the November issue of Linux Magazine has an article on 
DRBD - Distributed Replicated Block Device (www.drbd.org).  I would be very 
interested in thoughts/comments regarding this project.

thanks in advance,
ken