plugin for cached network file system

plugin for cached network file system

[Michael James]

How hard would it be to write a Reiser4 plug-in
 to implement a network file system with a cache on the local disk?

By caching I mean an incomplete local mirror
 of the master filesystem on the network server
 with on-demand loading and a simple flushing algorithm.
So open calls to files in the cached area are intercepted
 and if the file isn't already in place a script is called to get it there.
THEN the open call returns the file-handle normally.

I only need the simplest case of a distributed file system:
  - nodes are only reading the files,
  - files only change slowly (~weekly) 
	so cached files could be assumed fresh for a day at a time
	or even flushed with:		rsh node 'rm -r /cache/*'
  - any network code could be avoided by NFS mounting the master copy,
	so fetching a new file is still just a local cp off the NFS mount.


The situation is that CSIRO'S Bioinformatics Facility
 has two 400Gig filesystems on a server,
 that need to be available to 66 cluster nodes.
Just NFS exporting one server to so many nodes leaves the cluster NFS bound.
(Should I say they are Dell Perc 3 controllers and slow-as...)

Jobs come in big batches each reading the same 2 - 6 Gig file.
Each node has 27 Gig of free space so it's heaps for the working set
 I just don't want to have to groom what is in this working set for each job.
Nor do I want to have to pre-pend an rsync to a batch.

Has anyone written something that could be bent to this purpose?
Is there template code for a Reiser4 plug-in?
Or what would it cost to get it written?

michaelj


-- 
Michael James                         michael.james@csiro.au
System Administrator                    voice:  02 6246 5040
CSIRO Bioinformatics Facility             fax:  02 6246 5166

Re: plugin for cached network file system

[Hans Reiser]

Michael James wrote:

>How hard would it be to write a Reiser4 plug-in
> to implement a network file system with a cache on the local disk?
>
>By caching I mean an incomplete local mirror
> of the master filesystem on the network server
> with on-demand loading and a simple flushing algorithm.
>So open calls to files in the cached area are intercepted
> and if the file isn't already in place a script is called to get it there.
>THEN the open call returns the file-handle normally.
>
>I only need the simplest case of a distributed file system:
>  - nodes are only reading the files,
>  - files only change slowly (~weekly) 
>	so cached files could be assumed fresh for a day at a time
>	or even flushed with:		rsh node 'rm -r /cache/*'
>  - any network code could be avoided by NFS mounting the master copy,
>	so fetching a new file is still just a local cp off the NFS mount.
>
>
>The situation is that CSIRO'S Bioinformatics Facility
> has two 400Gig filesystems on a server,
> that need to be available to 66 cluster nodes.
>Just NFS exporting one server to so many nodes leaves the cluster NFS bound.
>(Should I say they are Dell Perc 3 controllers and slow-as...)
>
>Jobs come in big batches each reading the same 2 - 6 Gig file.
>Each node has 27 Gig of free space so it's heaps for the working set
> I just don't want to have to groom what is in this working set for each job.
>Nor do I want to have to pre-pend an rsync to a batch.
>
>Has anyone written something that could be bent to this purpose?
>Is there template code for a Reiser4 plug-in?
>Or what would it cost to get it written?
>
>michaelj
>
>
>  
>
It is easy to do a hack, and hard to do it right so that the world would 
want to drop nfs and move to it.  We are trying to find a sponsor to do 
it right, and have a proposal called Reiser5 that costs $1.8 million, 
and provides global scalability (which nfs/afs/clusterfs/intermezzo do 
not, in my view).

Hans

Re: plugin for cached network file system

[Michael James]

> > Michael James asked:
> >
> >How hard would it be to write a Reiser4 plug-in
> > to implement a network file system with a cache on the local disk?

Then, On Thursday 13 May 2004 15:49, Hans Reiser wrote:

> It is easy to do a hack, and hard to do it right so that the world would 
> want to drop nfs and move to it.  We are trying to find a sponsor to do 
> it right, and have a proposal called Reiser5 that costs $1.8 million, 
> and provides global scalability (which nfs/afs/clusterfs/intermezzo do 
> not, in my view).

At this stage, I'm looking for the, "Don't worry, be scrappy" solution.

1) Just kludge the basic functionality.
	(read-only, file-fetching/flushing in sh or perl,
	 NFS mounted master, manual deletion of outdated copies)

2) Let time, necessity and the GPL refine it and add features.
	(read-write-locking, automatic updating,
	  built-in file fetching/flushing and network code)

3) Always keeping an eye on 2) during 1).

There doesn't seem to be anything like this in Linux, and having it
 would be a boon (and give reiser4 a leg-up) in the cluster community.

michaelj


-- 
Michael James                         michael.james@csiro.au
System Administrator                    voice:  02 6246 5040
CSIRO Bioinformatics Facility             fax:  02 6246 5166

Re: plugin for cached network file system

[mjt]

On Thu, May 13, 2004 at 04:53:52PM +1000, Michael James wrote:
>
>At this stage, I'm looking for the, "Don't worry, be scrappy" solution.

I'm no expert, but I wouldn't go around kludging something like this.

>There doesn't seem to be anything like this in Linux, and having it
> would be a boon (and give reiser4 a leg-up) in the cluster community.

I'd rather start an online money-collection campaign, pass the hat around,
to give Namesys money to implement Reiser5. It would probably be faster
finished, the basic kludge even, and the quality would be better.

-- 
mjt

Re: plugin for cached network file system

[Miguel]

On Thu, 13 May 2004 10:03:55 +0300
mjt@nysv.org (Markus Törnqvist) wrote:

> On Thu, May 13, 2004 at 04:53:52PM +1000, Michael James wrote:
> >
> >At this stage, I'm looking for the, "Don't worry, be scrappy"
> >solution.
> 
> I'm no expert, but I wouldn't go around kludging something like this.
> 
> >There doesn't seem to be anything like this in Linux, and having it
> > would be a boon (and give reiser4 a leg-up) in the cluster
> > community.
> 
> I'd rather start an online money-collection campaign, pass the hat
> around, to give Namesys money to implement Reiser5. It would probably
> be faster finished, the basic kludge even, and the quality would be
> better.

wow the problem here maybe to find a hat with 1.8million dollar capacity

:D

> 
> -- 
> mjt
> 


-- 
La resistencia es fútil todos seréis asimilados

Re: plugin for cached network file system

[mjt]

On Thu, May 13, 2004 at 09:21:06AM +0200, Miguel wrote:

>wow the problem here maybe to find a hat with 1.8million dollar capacity

Getting the bugger bootstrapped may require less money.

-- 
mjt

Re: plugin for cached network file system

[Hans Reiser]

Markus Törnqvist wrote:

>On Thu, May 13, 2004 at 09:21:06AM +0200, Miguel wrote:
>
>  
>
>>wow the problem here maybe to find a hat with 1.8million dollar capacity
>>    
>>
>
>Getting the bugger bootstrapped may require less money.
>
>  
>
There are pleny of people working on cluster file systems.  Their stuff 
does not scale globally, it all scales to single machine room only.  I 
would like to address this problem, but only if well funded enough to 
win the competition.

Re: plugin for cached network file system

[Enrique Perez-Terron]

On Thu, 2004-05-13 at 18:09, Hans Reiser wrote:

> There are pleny of people working on cluster file systems.  Their stuff 
> does not scale globally, it all scales to single machine room only.  I 
> would like to address this problem, but only if well funded enough to 
> win the competition.

In what ways do the scalability problems manifest themselves? What
happens?

Enrique

Re: plugin for cached network file system

[Hans Reiser]

Michael James wrote:

>>>Michael James asked:
>>>
>>>How hard would it be to write a Reiser4 plug-in
>>>to implement a network file system with a cache on the local disk?
>>>      
>>>
>
>Then, On Thursday 13 May 2004 15:49, Hans Reiser wrote:
>
>  
>
>>It is easy to do a hack, and hard to do it right so that the world would 
>>want to drop nfs and move to it.  We are trying to find a sponsor to do 
>>it right, and have a proposal called Reiser5 that costs $1.8 million, 
>>and provides global scalability (which nfs/afs/clusterfs/intermezzo do 
>>not, in my view).
>>    
>>
>
>At this stage, I'm looking for the, "Don't worry, be scrappy" solution.
>
>1) Just kludge the basic functionality.
>	(read-only, file-fetching/flushing in sh or perl,
>	 NFS mounted master, manual deletion of outdated copies)
>
>2) Let time, necessity and the GPL refine it and add features.
>	(read-write-locking, automatic updating,
>	  built-in file fetching/flushing and network code)
>
>3) Always keeping an eye on 2) during 1).
>
>There doesn't seem to be anything like this in Linux, and having it
> would be a boon (and give reiser4 a leg-up) in the cluster community.
>
>michaelj
>
>
>  
>
I am very skeptical of the value of it (that is how NFS came about, just 
doing things quickly....), but to answer the question asked, it would be 
easy.

Re: plugin for cached network file system

[Michael James]

> >>>Michael James asked:
> >>>
> >>>How hard would it be to write a Reiser4 plug-in
> >>>to implement a network file system with a cache on the local disk?


> >On Thursday 13 May 2004 15:49, Hans Reiser wrote:
> >
> >>It is easy to do a hack, and hard to do it right so that the world would 
> >>want to drop nfs and move to it.  We are trying to find a sponsor to do 
> >>it right, and have a proposal called Reiser5 that costs $1.8 million, 
> >>and provides global scalability (which nfs/afs/clusterfs/intermezzo do 
> >>not, in my view).

NFS has its strength: Easy, overlays onto an existing (trusted) filesystem
 and its many weaknesses: bad locking, single server, etc.
AFS and its derivatives Coda and Intermezzo
 use ACLs which map badly in a Unix environment.
Also to support disconnected operation so they need to be complete copies,
 (no good for my case) and because of the penalty of re-syncing on re-connect
 they aren't designed for big partitions (again no good).
Lustre I need to study more.

It's Hans' determination to to go right down to basics and do it properly
 that makes me think reiserfs could be the one to carry Linux to new levels.

That said, I don't have $1.8 M or even the time to spend it
 before my last kludge (rsync complete local copies onto the nodes)
 turns around and bites me. (the files grew, not enough space anymore)

So I'm still looking for the, "Don't worry, be scrappy" solution.


> >On Friday 14 May 2004 01:58, Hans Reiser wrote:
>
> I am very skeptical of the value of it (that is how NFS came about, just 
> doing things quickly....), but to answer the question asked, it would be 
> easy.

Where do I start?

-- 
Michael James                         michael.james@csiro.au
System Administrator                    voice:  02 6246 5040
CSIRO Bioinformatics Facility             fax:  02 6246 5166

Re: plugin for cached network file system

[Hubert Chan]

>>>>> "Hans" == Hans Reiser <reiser@namesys.com> writes:

Hans> It is easy to do a hack, and hard to do it right so that the world
Hans> would want to drop nfs and move to it.  We are trying to find a
Hans> sponsor to do it right, and have a proposal called Reiser5 that
Hans> costs $1.8 million, and provides global scalability (which
Hans> nfs/afs/clusterfs/intermezzo do not, in my view).

Purely out of curiosity (I have no access to funding of that scale, and
I don't influence anyone who does), do you have an estimate for how much
it would cost to get Reiser6 functionality?

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: plugin for cached network file system

[Hans Reiser]

Same amount.

Hans

On Friday 14 May 2004 21:40, Hubert Chan wrote:
> >>>>> "Hans" == Hans Reiser <reiser@namesys.com> writes:
>
> Hans> It is easy to do a hack, and hard to do it right so that the world
> Hans> would want to drop nfs and move to it.  We are trying to find a
> Hans> sponsor to do it right, and have a proposal called Reiser5 that
> Hans> costs $1.8 million, and provides global scalability (which
> Hans> nfs/afs/clusterfs/intermezzo do not, in my view).
>
> Purely out of curiosity (I have no access to funding of that scale, and
> I don't influence anyone who does), do you have an estimate for how much
> it would cost to get Reiser6 functionality?

Re: plugin for cached network file system

[Christian Mayrhuber]

You could try coda. http://www.coda.cs.cmu.edu/
It supports disconnected operation and network caching.
I don't know if their "venus" caching daemon blows up if you tell it 
to cache 20Gig's. Their admin doc says max. 300Meg's for caching.

From their homepage:
Coda is a distributed filesystem with its origin in AFS2. It has many features 
that are very desirable for network filesystems. Currently, Coda has several 
features not found elsewhere.

   1. disconnected operation for mobile computing
   2. is freely available under a liberal license
   3. high performance through client side persistent caching
   4. server replication
   5. security model for authentication, encryption and access control
   6. continued operation during partial network failures in server network
   7. network bandwith adaptation
   8. good scalability
   9. well defined semantics of sharing, even in the presence of network 
failures 



On Thursday 13 May 2004 02:59, Michael James wrote:
> How hard would it be to write a Reiser4 plug-in
>  to implement a network file system with a cache on the local disk?
>
> By caching I mean an incomplete local mirror
>  of the master filesystem on the network server
>  with on-demand loading and a simple flushing algorithm.
> So open calls to files in the cached area are intercepted
>  and if the file isn't already in place a script is called to get it there.
> THEN the open call returns the file-handle normally.
>
> I only need the simplest case of a distributed file system:
>   - nodes are only reading the files,
>   - files only change slowly (~weekly)
> 	so cached files could be assumed fresh for a day at a time
> 	or even flushed with:		rsh node 'rm -r /cache/*'
>   - any network code could be avoided by NFS mounting the master copy,
> 	so fetching a new file is still just a local cp off the NFS mount.
>
>
> The situation is that CSIRO'S Bioinformatics Facility
>  has two 400Gig filesystems on a server,
>  that need to be available to 66 cluster nodes.
> Just NFS exporting one server to so many nodes leaves the cluster NFS
> bound. (Should I say they are Dell Perc 3 controllers and slow-as...)
>
> Jobs come in big batches each reading the same 2 - 6 Gig file.
> Each node has 27 Gig of free space so it's heaps for the working set
>  I just don't want to have to groom what is in this working set for each
> job. Nor do I want to have to pre-pend an rsync to a batch.
>
> Has anyone written something that could be bent to this purpose?
> Is there template code for a Reiser4 plug-in?
> Or what would it cost to get it written?
>
> michaelj

-- 
lg, Chris