Developing a Distributed File System

Developing a Distributed File System

[UZAIR LAKHANI]

Hello All,

Our team want to develop a distributed file system, a
file system like NFS or in other words a remote file
access system.

In our design we have multiple clients and a single
server. I want to identify main issues involved. What
I found after doing some study is this:

(1) Handling of multiple clients (concurrent
read/write problem)

(2) Continuation of (1) i.e. the distributed lock
management

(3) For performance reasons caching on the client side
and the cache coherency problem
 (this can be avoided for our prototype release)

(4) Data integrity problem.

To solve these issues, I think I have to study NFS,
AFS and coda. Also I am doing general googling for
finding out distributed file system issues.

Since both linux and fs (especially a distributed one)
are new to me, I want help in this regard that how I
start this development of a prototype and which
documents/books to study for this purpose.

Thanks,
Uzair Lakhani,
Karachi, Pakistan.

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: Developing a Distributed File System

[Andreas Dilger]

On Jul 09, 2006  22:23 -0700, UZAIR LAKHANI wrote:
> Our team want to develop a distributed file system, a
> file system like NFS or in other words a remote file
> access system.

First you need to explain why you want to develop a
new distributed file system?  What is wrong with any
of the existing filesystems?

> Since both linux and fs (especially a distributed one)
> are new to me, I want help in this regard that how I
> start this development of a prototype and which
> documents/books to study for this purpose.

This is by no means an easy task to start with.  How about
fixing some bug or simple feature to implement on an
existing filesystem (even a local one) before tackling
the hardest problem first?

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

Re: Developing a Distributed File System

[UZAIR LAKHANI]

--- Andreas Dilger <adilger@clusterfs.com> wrote:

> On Jul 09, 2006  22:23 -0700, UZAIR LAKHANI wrote:
> > Our team want to develop a distributed file
> system, a
> > file system like NFS or in other words a remote
> file
> > access system.
> 
> First you need to explain why you want to develop a
> new distributed file system?  What is wrong with any
> of the existing filesystems?

Hello All,
Thanks for the reply. Basically we want to move
towards developing a fail-over and high performance
cluster file system. Although in the simplest design
in which there will be a master server and a backup
server. 

Therefore we first have to figure out the issues
involved in a distributed file system (single server
and multiple clients). 

You may have noticed earlier that I had asked
questions regarding the definition of a cluster file
system. This was for the same reason.
> 
> > Since both linux and fs (especially a distributed
> one)
> > are new to me, I want help in this regard that how
> I
> > start this development of a prototype and which
> > documents/books to study for this purpose.
> 
> This is by no means an easy task to start with.  How
> about
> fixing some bug or simple feature to implement on an
> existing filesystem (even a local one) before
> tackling
> the hardest problem first?
> 
> Cheers, Andreas
> --
> Andreas Dilger
> Principal Software Engineer
> Cluster File Systems, Inc.

Thanks,
Uzair Lakhani,
Karachi, Pakistan.
> 
> -
> To unsubscribe from this list: send the line
> "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at 
> http://vger.kernel.org/majordomo-info.html
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: Developing a Distributed File System

[Andreas Dilger]

On Jul 10, 2006  21:55 -0700, UZAIR LAKHANI wrote:
> Thanks for the reply. Basically we want to move
> towards developing a fail-over and high performance
> cluster file system. Although in the simplest design
> in which there will be a master server and a backup
> server. 
>
> Therefore we first have to figure out the issues
> involved in a distributed file system (single server
> and multiple clients). 

I hate to state the obvious, but this is a VERY HARD PROBLEM.

I may be biased in sugessing this, but why not spend your time
and efforts in improving some existing cluster filesystem instead
of spending 20 work years of effort just to get a decently usable
system?  That way, instead of the ever-popular process of having
20 mediocre implementations of something Linux instead gains a
small number of very good implementations?

Possible targets of your efforts might include OCFS2, Lustre, GFS2.

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.