fistgen-0.1.1 released (linux-2.6 support)

fistgen-0.1.1 released (linux-2.6 support)

[Erez Zadok]

We're pleased to announce the release of fistgen-0.1.1, the package of
stackable templates.  This release includes numerous bug fixes, but biggest
new thing is a port to Linux 2.6!

We've tested the templates under various conditions for more than two
months: over ext2/3, over nfs, with low-memory conditions, small/large
files, large compile benchmarks, small and large postmark configurations,
fsx-linux, and other tests -- and permutations of these.  This exhaustive
testing was done to help ensure that the templates are as stable as possible
(many people are now using fist in production environments).

Get the new tarball from

	ftp://ftp.filesystems.org/pub/fist/fistgen-0.1.1.tar.gz

Here's the relevant portion of the NEWS file:

*** Notes specific to fistgen 0.1.1:

Bugs fixed in the Linux 2.6 templates:

- handle Fedora Core 2 kernels, which have changed some kernel APIs, as well
  as vanilla 2.6 kernels (esp. since 2.6.7 changed an internal kernel API)

- compile wrapfs.h cleanly with gcc-3.3, which doesn't like
  (const struct foo*)(something) = something_else.

Bug fixes in the Linux 2.4 and 2.6 templates:

- doit.sh script was exiting before running setky (cryptfs)

*** Notes specific to fistgen 0.1:

Linux 2.6 templates!  Tested with base0fs, base1fs, base2fs, base3fs (aka
wrapfs), and cryptfs.  (SCA/Gzipfs and Attach-mode mounts not yet supported
in 2.6.)

Renamed all aux.c files to fist_aux.c to avoid conflicts with MS-DOS file
systems.

Malloc_DEBUG support for tracking memory leaks.  See explanation and use in
script match-malloc.pl.

Bugs fixed in Linux 2.4 templates:

- copy i_blocks correctly when mounted on top of ext2.

- wrapfs_commit_write() now behaves correctly for HIGHMEM systems
  (kmap/kunmap needed)

- for attach mode mounts the default maximum file size is taken from the
  root file system (and can be overridden using the maxbytes= mount time
  option).  This allows attach-mode file systems to support files larger
  than 2GB.

- fix refcnt leak on vfsmnt

- wrapfs_lock() needs struct flock's ->file set.  Work if called by
  posix_test_lock().

- readdir bug fix when exporting over NFS

- show dir= and debug= options in /proc/mounts

- move kmalloc outside spinlock

- bzero wrapfs_sb_info struct to avoid oops when lookup(hidden_root) fails

- don't generate attach code unless needed.

- ASSERT that hidden_file isn't NULL in various places where it shouldn't be

- up() the semaphore only if needed in wrapfs_commit_write

- kmalloc whole struct wrapfs_file_info, not just pointer, in wrapfs_open

- in wrapfs_lookup and wrapfs_symlink, don't use unsigned int for
  encoded_namelen, or we could never return an -ERRNO.

- in truncate, replace the hacky fake_file code with one that actually
  dentry_open's the real hidden_file.

- fail to sync files when mounted over NFS: correctly convert append-only mode
  writes to read-write in wrapfs_open.


Happy Stacking,
Erez and the fistgen team.

Re: fistgen-0.1.1 released (linux-2.6 support)

[Jeff Garzik]

I ask again...

where is the README or URL that describes what fistgen _is_?

Your announcement is tailored such that only people who already know 
what fistgen (version 0.1.1) is would want to download it.

	Jeff

Re: fistgen-0.1.1 released (linux-2.6 support)

[Erez Zadok]

In message <410ABE18.1080600@pobox.com>, Jeff Garzik writes:
> I ask again...
> 
> where is the README or URL that describes what fistgen _is_?
> 
> Your announcement is tailored such that only people who already know 
> what fistgen (version 0.1.1) is would want to download it.
> 
> 	Jeff

I'll say again...

   http://www.filesystems.org/

Erez.

Re: fistgen-0.1.1 released (linux-2.6 support)

[Jamie Lokier]

Erez Zadok wrote:
> > where is the README or URL that describes what fistgen _is_?
> > 
> > Your announcement is tailored such that only people who already know 
> > what fistgen (version 0.1.1) is would want to download it.
> > 
> > 	Jeff
> 
> I'll say again...
> 
>    http://www.filesystems.org/

I'm with Jeff.

A one line description of what fistgen is in the email would still be helpful.

I'm not going to waste 30 seconds faffing about starting a web
browser, if you can't be bothered to include a simple brief
description.

Like many people, I read lots of mails, quickly.  Yesterday it was
nearly 2000.  I spent an average time of 2 seconds on each.  It's not
possible to follow links to web sites on that time scale.

If a quick scan says its interesting, I'll read more, maybe even break
out the browser.  Something as trivial as an informative subject line
or a brief description makes all the difference.

This is constructive criticism, the point being please include a brief
description of what the thing is when making public announcements.
It's not hard, and very useful.

In the interests of generosity, I've just had a look at
www.filesystems.org.  From your email announcement, I had guessed that
FiST and fistgen were some sort of userspace filesystem tool, like
podfuk and a few others (including an NFS-based one I worked on).  But
now I see that it's not.

To be honest, from reading the main page of the web site I still don't
understand what FiST is useful for at the moment.  As research it
looks very promising.  Btw, does it offer the facility for lazy
copy-on-write trees that was discussed on linux-kernel a few months
ago -- useful for jails and cloned compilation trees?

-- Jamie

Re: fistgen-0.1.1 released (linux-2.6 support)

[David Chow]

FiST is a stackable file system project, or you can say it is a file 
filtering project. fistgen has is a template-based file system 
development tool, and of couse runs at user space. But fs'es (C sources) 
which generated by fistgen run entirely in the kernel. You can write 
FiST file systems in C language without the need to bother kernel VFS 
integration or internal understanding , simple use FiST language macros 
to write file systems. For example, wrapfs (wrapper file system) is a 
simply fs on fs file system which is used for testing and demonstration 
the most basic stackable file system. Instead of mounting block devices, 
you mount directories. You can even develop encryption and other useful 
storage tools which requires fs-level integration but no need to hack or 
write fs from scratch .

Personally, I suggests newbies of fs development to work with FiST to 
get start on understanding VFS, as you can generate as many of fs'es as 
you like, it is useful for fs-devel beginners to play around with.

regards,
David Chow

Jamie Lokier wrote:

>Erez Zadok wrote:
>  
>
>>>where is the README or URL that describes what fistgen _is_?
>>>
>>>Your announcement is tailored such that only people who already know 
>>>what fistgen (version 0.1.1) is would want to download it.
>>>
>>>	Jeff
>>>      
>>>
>>I'll say again...
>>
>>   http://www.filesystems.org/
>>    
>>
>
>I'm with Jeff.
>
>A one line description of what fistgen is in the email would still be helpful.
>
>I'm not going to waste 30 seconds faffing about starting a web
>browser, if you can't be bothered to include a simple brief
>description.
>
>Like many people, I read lots of mails, quickly.  Yesterday it was
>nearly 2000.  I spent an average time of 2 seconds on each.  It's not
>possible to follow links to web sites on that time scale.
>
>If a quick scan says its interesting, I'll read more, maybe even break
>out the browser.  Something as trivial as an informative subject line
>or a brief description makes all the difference.
>
>This is constructive criticism, the point being please include a brief
>description of what the thing is when making public announcements.
>It's not hard, and very useful.
>
>In the interests of generosity, I've just had a look at
>www.filesystems.org.  From your email announcement, I had guessed that
>FiST and fistgen were some sort of userspace filesystem tool, like
>podfuk and a few others (including an NFS-based one I worked on).  But
>now I see that it's not.
>
>To be honest, from reading the main page of the web site I still don't
>understand what FiST is useful for at the moment.  As research it
>looks very promising.  Btw, does it offer the facility for lazy
>copy-on-write trees that was discussed on linux-kernel a few months
>ago -- useful for jails and cloned compilation trees?
>  
>

Re: fistgen-0.1.1 released (linux-2.6 support)

[Erez Zadok]

In message <410B8ACA.6000902@shaolinmicro.com>, David Chow writes:
> FiST is a stackable file system project, or you can say it is a file 
> filtering project. fistgen has is a template-based file system 
> development tool, and of couse runs at user space. But fs'es (C sources) 
> which generated by fistgen run entirely in the kernel. You can write 
> FiST file systems in C language without the need to bother kernel VFS 
> integration or internal understanding , simple use FiST language macros 
> to write file systems. For example, wrapfs (wrapper file system) is a 
> simply fs on fs file system which is used for testing and demonstration 
> the most basic stackable file system. Instead of mounting block devices, 
> you mount directories. You can even develop encryption and other useful 
> storage tools which requires fs-level integration but no need to hack or 
> write fs from scratch .
> 
> Personally, I suggests newbies of fs development to work with FiST to 
> get start on understanding VFS, as you can generate as many of fs'es as 
> you like, it is useful for fs-devel beginners to play around with.
> 
> regards,
> David Chow

Thanks David, I couldn't have said it better. :-)

A good one-liner for fist is "it's a yacc for file systems."

I'll add that very often, someone uses fistgen to produce a working
stackable file system module (with kernel sources of course) that is as
close to what they the FiST language permits; and then they start modifying
the sources to their needs.

The Wrapfs templates themselves provide a fully working "null-layer"
stackable file system.  In the seven years since I've started developing
this project, I've seen two different uses:

- Educational: it's much easier looking at the wrapfs sources to understand
  how a file systems works, than to go though the VFS sources or a native
  disk-based f/s's sources.  There are also built-in debugging/tracing that
  allows you to trace what sequence of f/s ops are actually happening when
  you run userland tools like "ls" and "cat".

  I get contacted by students from all over the globe who use fist/wrapfs as
  part of some class.  People who want to learn how the Linux VFS works tell
  me that they founds Wrapfs useful as a learning tool.

- Research/Commercial: people will start from a blank template and modify it
  themselves.  It's easy to add functionality incrementally.  And since it's
  easier to learn how Wrapfs works than write a file system from scratch,
  they find that they can develop their f/s faster with wrapfs.  At least a
  dozen papers have been published based on work derived from fist; and
  several commercial products use it too (esp. in the security area).

Caveat: file system development, even with fist, will never be as easy as
writing some userland "hello world" code.  With fist, you're still
developing kernel code.  It may be easier with fist, but it's still
dangerous: you can crash your system, corrupt your data, etc.

Cheers,
Erez.

Re: fistgen-0.1.1 released (linux-2.6 support)

[Erez Zadok]

In message <20040731105423.GB23725@mail.shareable.org>, Jamie Lokier writes:

> Btw, does it offer the facility for lazy
> copy-on-write trees that was discussed on linux-kernel a few months
> ago -- useful for jails and cloned compilation trees?

Jamie, I'll have to check that thread; I don't recall it on fsdevel (sorry,
I don't have time to read lkml often, or 2000 messages per day :-).

But from what you say, it sounds like you want some sort of combo of
caching, snapshotting, and sandboxing, right?

We've developed a stackable unioning file system for linux
called... Unionfs.  It's designed as a fan-out file system, which is a way
for a stackable file system to stack on top of more than one other f/s or
directory.  You can see a paper on it here

	http://www.fsl.cs.sunysb.edu/project-unionfs.html

Since we wrote the paper, we also enhanced Unionfs to support dynamic
addition and removal of "branches" in the union, including the ability mark
certain branches as readonly or read-write, and including the ability to
divert f/s ops to specific branches.  All this gives us a f/s that can
produce snapshots and sandboxes.

Maybe this is a good coincidence because we're now preparing a release of
unionfs (testing it throughly and fixing some bugs).  If all goes well, we
hope to release it in a couple of weeks.

> -- Jamie

Erez.

Re: fistgen-0.1.1 released (linux-2.6 support)

[Jamie Lokier]

Erez Zadok wrote:
> > Btw, does it offer the facility for lazy
> > copy-on-write trees that was discussed on linux-kernel a few months
> > ago -- useful for jails and cloned compilation trees?
> 
> Jamie, I'll have to check that thread; I don't recall it on fsdevel (sorry,
> I don't have time to read lkml often, or 2000 messages per day :-).

It wasn't on fsdevel.  About linux-kernel I concur: I've been reading
it for 10 years and stopped cold a couple of months ago due to a
technical glitch.  I've been putting off reading it again, it's a burden.

> But from what you say, it sounds like you want some sort of combo of
> caching, snapshotting, and sandboxing, right?

No.

We want an equivalent to "cp -av" that is fast and doesn't use much
disk space.  Other than timing and disk capacity, it must be
semantically equivalent to copying a tree.

Snapshotting (as in recording a current state for restoring later)
wouldn't work, because cloned trees need to evolve independently.

Think about when you have 30 versions of a common source tree, each
configured differently and built with different compilation options,
and different patches applied.  For this application, "cp -rl" is
usable but it is risky and occasionally mistakes are made.

Sandboxing is appropriate for virtual machines and daemon jails: when
you create new VMs for users or new jails, you clone a distro tree.
Then it's ok to use some kind of sandboxing, although if it's
implemented as a "unionfs" where the common underlying tree must never
be changed as all VMs and jails depend on it, that's a huge limitation.

Sandboxing isn't convenient for the multiple source trees application,
because you sometimes want to operate on several of the trees at once,
e.g. to diff between them.

> We've developed a stackable unioning file system for linux
> called... Unionfs.  It's designed as a fan-out file system, which is a way
> for a stackable file system to stack on top of more than one other f/s or
> directory.  You can see a paper on it here
> 
> 	http://www.fsl.cs.sunysb.edu/project-unionfs.html
> 
> Since we wrote the paper, we also enhanced Unionfs to support dynamic
> addition and removal of "branches" in the union, including the ability mark
> certain branches as readonly or read-write, and including the ability to
> divert f/s ops to specific branches.  All this gives us a f/s that can
> produce snapshots and sandboxes.
> 
> Maybe this is a good coincidence because we're now preparing a release of
> unionfs (testing it throughly and fixing some bugs).  If all goes well, we
> hope to release it in a couple of weeks.

It sounds like a step in a positive direction, although it also sounds
rather inconvenient to use for some purposes, in contrast to a copy
optimisation which is easy to use.

-- Jamie

Re: fistgen-0.1.1 released (linux-2.6 support)

[Erez Zadok]

In message <20040731195717.GA24760@mail.shareable.org>, Jamie Lokier writes:
> Erez Zadok wrote:
[...]
> > We've developed a stackable unioning file system for linux
> > called... Unionfs.  It's designed as a fan-out file system, which is a way
> > for a stackable file system to stack on top of more than one other f/s or
> > directory.  You can see a paper on it here
> > 
> > 	http://www.fsl.cs.sunysb.edu/project-unionfs.html
> > 
> > Since we wrote the paper, we also enhanced Unionfs to support dynamic
> > addition and removal of "branches" in the union, including the ability mark
> > certain branches as readonly or read-write, and including the ability to
> > divert f/s ops to specific branches.  All this gives us a f/s that can
> > produce snapshots and sandboxes.
> > 
> > Maybe this is a good coincidence because we're now preparing a release of
> > unionfs (testing it throughly and fixing some bugs).  If all goes well, we
> > hope to release it in a couple of weeks.
> 
> It sounds like a step in a positive direction, although it also sounds
> rather inconvenient to use for some purposes, in contrast to a copy
> optimisation which is easy to use.
> 
> -- Jamie

Right, it looks like you want to support version trees (branches off of
branches, etc. ala CVS), as well as some ability to create composable views
that mix and match version "forks."

Our unionfs offers a linear set of snapshotting.  We also developed a
stackable versioning f/s (which uses traditional CoW as well as
copy-on-change), which also offers a linear versioning model.

We considered supporting version trees, but frankly, couldn't come up with a
compelling application for those that would justify the added complexity.
Such version trees are complex and hard to manage, and the most users get
really confused with version trees (CVS branch management is enough of a
reason to mess up :-)

Jamie, I'd love to get a detailed example of how one envisions using such a
versioning system, so I can see how much our current work may support it,
and what it'd take to make it do what you want.

Nevertheless, I agree with you that Unionfs is probably a step in the right
direction.  It should be fairly efficient b/c it only copies-up files when
needed (and it can be modified to copy pages instead, as our Versioning f/s
does in "Sparse" mode; or to always copy-on-read).

Cheers,
Erez.

Re: fistgen-0.1.1 released (linux-2.6 support)

[Jamie Lokier]

Erez Zadok wrote:
> Right, it looks like you want to support version trees (branches off of
> branches, etc. ala CVS), as well as some ability to create composable views
> that mix and match version "forks."

That is a complicated way of looking at it!

> We considered supporting version trees, but frankly, couldn't come up with a
> compelling application for those that would justify the added complexity.
> Such version trees are complex and hard to manage, and the most users get
> really confused with version trees (CVS branch management is enough of a
> reason to mess up :-)
> 
> Jamie, I'd love to get a detailed example of how one envisions using such a
> versioning system, so I can see how much our current work may support it,
> and what it'd take to make it do what you want.

One uses it by typing "cp -a dir1 dir2" or "cp -r dir1 dir2".

That's all.  There are no user-visible versions, no branch
management, and no other commands.

That is what the work so far (as discussed on linux-kernel a few
months ago) began.

> Nevertheless, I agree with you that Unionfs is probably a step in the right
> direction.  It should be fairly efficient b/c it only copies-up files when
> needed (and it can be modified to copy pages instead, as our Versioning f/s
> does in "Sparse" mode; or to always copy-on-read).

Copy-on-write pages is nice if you can do it.
Do you handle copy-on-write of mmap'd pages properly?
That is one requirement.

I am not convinced that a unionfs is the right way to implement this
if the user has to do lots of "mount" commands or equivalent.  It
really is just meant to be as simple as a copy optimisation -- if you
have to remember mounts, then it's too complicated to use.

However as an implementation method, if it can be made invisible to
users, then it might work.  So simplicity is another requirement.

I suspect that you see things in terms of versioning and branching,
with all the complexities that implies for users (just like
complicated VC tools).  But the point of the COW filesystem methods is
precisely to hide that from users and offer a much simpler data model.

(Perhaps there's a nice data model in between which we haven't
explored, which is both simple to understand yet more versatile than copy.)

Another requirement is that it doesn't add much performance overhead.
It should be about as fast as the underlying filesystem, except inQA
cases where implied copying is triggered, and it must not use much
more memory e.g. by doing something silly like copying pages in RAM.
This is another reason why I'm not sure a unionfs is appropriate,
although it could be if implemented in the right way.

A filesystem filter which overlays part of an underlying filesystem
might be the right way, especially if the overlay can be made
persistent (i.e. doesn't have to be manually mounted, in effect it
becomes stuck on).  I'm not sure if this is starting to go the way of
Hurd translators but it seems there may be some similarity.

-- Jamie

Re: fistgen-0.1.1 released (linux-2.6 support)

[Erez Zadok]

In message <20040802190817.GA11754@mail.shareable.org>, Jamie Lokier writes:
> Erez Zadok wrote:

> One uses it by typing "cp -a dir1 dir2" or "cp -r dir1 dir2".
> 
> That's all.  There are no user-visible versions, no branch
> management, and no other commands.
> 
> That is what the work so far (as discussed on linux-kernel a few
> months ago) began.

Aha.  This certainly clarifies it.  I think we're looking more into some
sort of clever copy-on-something versioning than unioning.  If users can do
"cp dir1 dir2" arbitrarily, then each "dir2" could automatically become a
branch point (on top of what used to be "dir2").  I can see how this is
flexible for users, but it can create some very complex-to-manage trees
below.  Just imagine what happens whens someone will do

$ cp -a dir1 dir2
$ cp -a dir3 dir2/dir4
$ cp -a dir2/dir5 dir2/dir4

Fun.

(We're working on a release of unionfs now and once that's done, we'll start
preparing versionfs to be released -- primarily testing, code cleanup, and
documentation.  Perhaps after versionfs is out people can give it a try,
look at the sources, and decide what's useful in there to use as a base of
sorts.  In the mean time, one can read a paper on Versionfs here:
http://www.fsl.cs.sunysb.edu/project-versionfs.html)

> Do you handle copy-on-write of mmap'd pages properly?
> That is one requirement.

Yes of course.  One of the main reason why we use f/s stacking to intercept
stuff and not syscall interception, is that at the f/s level you can easily
get every page read/write request.

> However as an implementation method, if it can be made invisible to
> users, then it might work.  So simplicity is another requirement.

Right: for simplicity, such a file system has to automatically create a
branch tree for "dir2" as needed, when the user does "cp -a dir1 dir2".

> I suspect that you see things in terms of versioning and branching,
> with all the complexities that implies for users (just like
> complicated VC tools).  But the point of the COW filesystem methods is
> precisely to hide that from users and offer a much simpler data model.

BTW, has anyone looked at ext3cow?  As a native f/s, it will generally
deliver better performance than stacked file systems (stacking's advantage
would be to allow this auto-versioning to take place on top of any other
f/s).

> (Perhaps there's a nice data model in between which we haven't
> explored, which is both simple to understand yet more versatile than copy.)

If you expect a substantial portion of your copied data to remain the same,
then a reasonable model is to have a backend database of sorts of per-page
checksums, so you can avoid copying pages that haven't changed.  (Several
distributed storage projects use such ideas, including I believe Venti.)

Another possibility is to assume an object-store model, where you can tag
larger chunks/extents of data with one checksum.  Now that I think of it,
some of rsync's algorithms will be very useful here.  rsyncfs anyone?

> Another requirement is that it doesn't add much performance overhead.
> It should be about as fast as the underlying filesystem, except inQA
> cases where implied copying is triggered, and it must not use much
> more memory e.g. by doing something silly like copying pages in RAM.
> This is another reason why I'm not sure a unionfs is appropriate,
> although it could be if implemented in the right way.

If low overhead is absolutely essential, then an ext3cow-like idea would be
best.  Personally I think versioning/snapshotting should be a part of every
disk-based f/s.  Stacking, by its very nature, is more costly b/c of the
function call indirection and our need to stick with whatever the VFS gives
us and not modify it.  We could solve several problems nicely if we could
change the VFS: OK, so we *can* change it, but it's a major PITA to ask
users to patch their kernel to use a stackable f/s, so we try and avoid it
very much.

Case in point: for a while, we've had a situation where if you wanted to
intercept page read/write ops in a stackable f/s, you had to create two
pages -- one at the the upper stackable f/s layer, and one at the f/s layer
below.  While this may be ok for file systems that do encryption/compression
(b/c the page data is different), it's not good if all you want is to know
when page ops happen but you don't care to change the data.  Just this past
week we found a (rather hacky IMHO) way to avoid keeping double pages, by
modifying the page->mapping field and then restoring it at just the right
places.  It's not a clean solution, and we're still testing it, but so far
it seems to work.  I'd love to solve this more cleanly but that would
require VFS and mm/*.c changes.

> -- Jamie

Erez.

Re: fistgen-0.1.1 released (linux-2.6 support)

[Shaya Potter]

On Mon, 2004-08-02 at 20:08 +0100, Jamie Lokier wrote:
> Erez Zadok wrote:
> > Right, it looks like you want to support version trees (branches off of
> > branches, etc. ala CVS), as well as some ability to create composable views
> > that mix and match version "forks."
> 
> That is a complicated way of looking at it!
> 
> > We considered supporting version trees, but frankly, couldn't come up with a
> > compelling application for those that would justify the added complexity.
> > Such version trees are complex and hard to manage, and the most users get
> > really confused with version trees (CVS branch management is enough of a
> > reason to mess up :-)
> > 
> > Jamie, I'd love to get a detailed example of how one envisions using such a
> > versioning system, so I can see how much our current work may support it,
> > and what it'd take to make it do what you want.
> 
> One uses it by typing "cp -a dir1 dir2" or "cp -r dir1 dir2".
> 
> That's all.  There are no user-visible versions, no branch
> management, and no other commands.

I have a "working" (relative term as performance sucks) fs that can do
that.  It combined a file system based out of fist's templates and a bit
of hacking on ext3cow.

basic idea is that ext3cow provided the individual file versioning
semantic (and directories are just treated as containers) while the
stackable fs managed the branching (and deciding when to version and the
like).

was able to build 4 kernels side by side w/o a problem.

i.e. it was something along the lines of (from memory)

# mkfs.ext3cow /dev/loop0
# mount -t ext3cow /dev/loop0 /mnt
# mount -o bind /mnt /mnt1
# mount -o bind /mnt /mnt2
# mount -o bind /mnt /mnt3
# (cd /mnt ; tar xzf ~/linux.tar.gz)
# mount -t branchfs -o new_branch=new,old_branch=,(some fist
options) /mnt /mnt
# cd /mnt
# branch_ioctl new1
# mount -t branchfs -o new_branch=new2,old_branch=new,(some fist
options) /mnt1 /mnt1
# mount -t branchfs -o new_branch=new2,old_branch=new,(some fist
options) /mnt2 /mnt2
# mount -t branchfs -o new_branch=new2,old_branch=new,(some fist
options) /mnt2 /mnt2

lots of issues w/ it, but it was able to do the above as (well as then
build the kernels).

the stackable fs stored a lot of meta data on the underlying fs (i.e.
managing branches and the like) and it was my first attempt at doing it,
so probably made a lot of mistakes and hence the bad performance.

unfortunately been too busy working for msft this summer, so haven't had
a chance to try some of my ideas on fixing it up.

shaya

Re: fistgen-0.1.1 released (linux-2.6 support)

[Charles P. Wright]

On Mon, 2004-08-02 at 21:54, Shaya Potter wrote:
> > One uses it by typing "cp -a dir1 dir2" or "cp -r dir1 dir2".
> > 
> > That's all.  There are no user-visible versions, no branch
> > management, and no other commands.
> 
> I have a "working" (relative term as performance sucks) fs that can do
> that.  It combined a file system based out of fist's templates and a bit
> of hacking on ext3cow.
> 
> basic idea is that ext3cow provided the individual file versioning
> semantic (and directories are just treated as containers) while the
> stackable fs managed the branching (and deciding when to version and the
> like).
> 
> was able to build 4 kernels side by side w/o a problem.
> 
> i.e. it was something along the lines of (from memory)
> 
> # mkfs.ext3cow /dev/loop0
> # mount -t ext3cow /dev/loop0 /mnt
> # mount -o bind /mnt /mnt1
> # mount -o bind /mnt /mnt2
> # mount -o bind /mnt /mnt3
> # (cd /mnt ; tar xzf ~/linux.tar.gz)
> # mount -t branchfs -o new_branch=new,old_branch=,(some fist
> options) /mnt /mnt
> # cd /mnt
> # branch_ioctl new1
> # mount -t branchfs -o new_branch=new2,old_branch=new,(some fist
> options) /mnt1 /mnt1
> # mount -t branchfs -o new_branch=new2,old_branch=new,(some fist
> options) /mnt2 /mnt2
> # mount -t branchfs -o new_branch=new2,old_branch=new,(some fist
> options) /mnt2 /mnt2
> 
> lots of issues w/ it, but it was able to do the above as (well as then
> build the kernels).
> 
Shaya,

What does that sequence of commands actually mean?

Charles

Re: fistgen-0.1.1 released (linux-2.6 support)

[Shaya Potter]

On Tue, 2004-08-03 at 08:42 -0400, Charles P. Wright wrote:
> On Mon, 2004-08-02 at 21:54, Shaya Potter wrote:
> > > One uses it by typing "cp -a dir1 dir2" or "cp -r dir1 dir2".
> > > 
> > > That's all.  There are no user-visible versions, no branch
> > > management, and no other commands.
> > 
> > I have a "working" (relative term as performance sucks) fs that can do
> > that.  It combined a file system based out of fist's templates and a bit
> > of hacking on ext3cow.
> > 
> > basic idea is that ext3cow provided the individual file versioning
> > semantic (and directories are just treated as containers) while the
> > stackable fs managed the branching (and deciding when to version and the
> > like).
> > 
> > was able to build 4 kernels side by side w/o a problem.
> > 
> > i.e. it was something along the lines of (from memory)
> > 
> > # mkfs.ext3cow /dev/loop0
> > # mount -t ext3cow /dev/loop0 /mnt
> > # mount -o bind /mnt /mnt1
> > # mount -o bind /mnt /mnt2
> > # mount -o bind /mnt /mnt3
> > # (cd /mnt ; tar xzf ~/linux.tar.gz)
> > # mount -t branchfs -o new_branch=new,old_branch=,(some fist
> > options) /mnt /mnt
> > # cd /mnt
> > # branch_ioctl new1
> > # mount -t branchfs -o new_branch=new2,old_branch=new,(some fist
> > options) /mnt1 /mnt1
> > # mount -t branchfs -o new_branch=new2,old_branch=new,(some fist
> > options) /mnt2 /mnt2
> > # mount -t branchfs -o new_branch=new2,old_branch=new,(some fist
> > options) /mnt2 /mnt2
> > 
> > lots of issues w/ it, but it was able to do the above as (well as then
> > build the kernels).
> > 
> Shaya,
> 
> What does that sequence of commands actually mean?

ahh good point.  let comments it. (and reorganize to make clearer, and
fix a stupid bug at the end)

# create the underlying buffer store
$ mkfs.ext3cow /dev/loop0

# mount the underling buffer store to initial place
$ mount -t ext3cow /dev/loop0 /mnt

# mount the underlying buffer store to places you want to branch
$ mount -o bind /mnt /mnt1
$ mount -o bind /mnt /mnt2
$ mount -o bind /mnt /mnt3

# create initial branch on the empty fs.  call it "new"
$ mount -t branchfs -o new_branch=new,old_branch=,(some fist options) /mnt /mnt

#untar the linux kernel (it's now available
$ (cd /mnt ; tar xzf ~/linux.tar.gz)

# branch the fs from branch "new" to "new1 (i.e. branch_ioctl takes a
string that says what to create the new branch.
$ cd /mnt
$ branch_ioctl new1

// create 3 new branches based on branch "new"
$ mount -t branchfs -o new_branch=new2,old_branch=new,(some fist options) /mnt1 /mnt1
$ mount -t branchfs -o new_branch=new3,old_branch=new,(some fist options) /mnt2 /mnt2
$ mount -t branchfs -o new_branch=new4,old_branch=new,(some fist options) /mnt2 /mnt2

one can now go into /mnt (branch "new1") /mnt1 (branch "new2") /mnt2
(branch "new3") and /mnt3 (branch "new4") which each share a common
linux source tree, but can be configured (files added) and compiled
(filed added and deleted) and edited (files bits are modified)
completely independently of each other.  Whenever we need to version a
file, I hacked ext3cow's "COW" functionality to provide a "cowpy"
function that let you essentially mimic link(), but create a "cow" link
instead.